Apache Airflow Archives - Big Data & ETL

Photo by Drew Beamer on Unsplash

Apache Airflow & CeleryExecutor, PostgreSQL & Redis: Start the environment using Docker-Compose in 5 minutes!

Post author:pawel.ciesla
Post published:12 October 2019
Post category:Apache Airflow Big Data Databases Docker Docker-Compose Ubuntu
Post comments:0 Comments

In this post I will show you how to create a fully operational environment in 5 minutes, which will include: Apache Airflow WebServerApache Airflow WorkerApache Airflow SchedulerFlower - is a…

Apache Airflow: Create dynamic DAG

Post author:pawel.ciesla
Post published:27 January 2019
Post category:Apache Airflow
Post comments:2 Comments

Apache Airflow gives us possibility to create dynamic DAG. This feature is very useful when we would like to achieve flexibility in Airflow, to do not create many DAGs for…

Apache Airflow: How to setup Airflow to run multiple DAGs and tasks in parallel mode?

Post author:pawel.ciesla
Post published:27 January 2019
Post category:Apache Airflow MySQL Ubuntu
Post comments:2 Comments

At the beginning of your journey with Airflow I suppose that you encountered situation when you created multiple DAGs with some tasks inside and when you run all workflows in…

Airflow: create Frenet key

Post author:pawel.ciesla
Post published:27 January 2019
Post category:Apache Airflow
Post comments:0 Comments

In this short tutorial I will show how you can easily create Fernet key. Install cryptography if is missing pip install cryptography Create new fernet key: python -c "from cryptography.fernet…

Apache Airflow: Short introduction and simple DAG

Post author:pawel.ciesla
Post published:20 January 2019
Post category:Apache Airflow Articles
Post comments:0 Comments

In this tutorial I will show you how you can create simple DAG in Apache Airflow.