In this post I will show you how to create a fully operational environment in 5 minutes, which will include:
- Apache Airflow WebServer
- Apache Airflow Worker
- Apache Airflow Scheduler
- Flower – is a web based tool for monitoring and administrating Celery clusters
- Redis – is an open source (BSD licensed), in-memory data structure store, used as a database, cache and message broker.
docker-compose.yml script
Create the docker-compose.yml file and paste the script below. Then run the docker-compos up -d command.
(The script below was taken from the site Puckel)
version: '2.1' services: redis: image: 'redis:5.0.5' # command: redis-server --requirepass redispass postgres: image: postgres:9.6 environment: - POSTGRES_USER=airflow - POSTGRES_PASSWORD=airflow - POSTGRES_DB=airflow # Uncomment these lines to persist data on the local filesystem. # - PGDATA=/var/lib/postgresql/data/pgdata # volumes: # - ./pgdata:/var/lib/postgresql/data/pgdata webserver: image: puckel/docker-airflow:1.10.4 restart: always depends_on: - postgres - redis environment: - LOAD_EX=n - FERNET_KEY=46BKJoQYlPPOexq0OhDZnIlNepKFf87WFwLbfzqDDho= - EXECUTOR=Celery # - POSTGRES_USER=airflow # - POSTGRES_PASSWORD=airflow # - POSTGRES_DB=airflow # - REDIS_PASSWORD=redispass volumes: - ./dags:/usr/local/airflow/dags # Uncomment to include custom plugins # - ./plugins:/usr/local/airflow/plugins ports: - "8080:8080" command: webserver healthcheck: test: ["CMD-SHELL", "[ -f /usr/local/airflow/airflow-webserver.pid ]"] interval: 30s timeout: 30s retries: 3 flower: image: puckel/docker-airflow:1.10.4 restart: always depends_on: - redis environment: - EXECUTOR=Celery # - REDIS_PASSWORD=redispass ports: - "5555:5555" command: flower scheduler: image: puckel/docker-airflow:1.10.4 restart: always depends_on: - webserver volumes: - ./dags:/usr/local/airflow/dags # Uncomment to include custom plugins # - ./plugins:/usr/local/airflow/plugins environment: - LOAD_EX=n - FERNET_KEY=46BKJoQYlPPOexq0OhDZnIlNepKFf87WFwLbfzqDDho= - EXECUTOR=Celery # - POSTGRES_USER=airflow # - POSTGRES_PASSWORD=airflow # - POSTGRES_DB=airflow # - REDIS_PASSWORD=redispass command: scheduler worker: image: puckel/docker-airflow:1.10.4 restart: always depends_on: - scheduler volumes: - ./dags:/usr/local/airflow/dags # Uncomment to include custom plugins # - ./plugins:/usr/local/airflow/plugins environment: - FERNET_KEY=46BKJoQYlPPOexq0OhDZnIlNepKFf87WFwLbfzqDDho= - EXECUTOR=Celery # - POSTGRES_USER=airflow # - POSTGRES_PASSWORD=airflow # - POSTGRES_DB=airflow # - REDIS_PASSWORD=redispass command: worker
Check the container status
Before navigating to pages with the user interface, check that all containers are in “UP” status. To do this, use the command:
docker-compose ps Name Command State Ports ----------------------------------------------------------------------------------------------------------------------- airflow-docker_flower_1 /entrypoint.sh flower Up 0.0.0.0:5555->5555/tcp, 8080/tcp, 8793/tcp airflow-docker_postgres_1 docker-entrypoint.sh postgres Up 5432/tcp airflow-docker_redis_1 docker-entrypoint.sh redis ... Up 6379/tcp airflow-docker_scheduler_1 /entrypoint.sh scheduler Up 5555/tcp, 8080/tcp, 8793/tcp airflow-docker_webserver_1 /entrypoint.sh webserver Up (healthy) 5555/tcp, 0.0.0.0:8080->8080/tcp, 8793/tcp airflow-docker_worker_1 /entrypoint.sh worker Up 5555/tcp, 8080/tcp, 8793/tcp
User Interface
When all containers are running, we can open in turn:
- localhost:8080 – Apache Airflow UI
- localhost:5555 – Flower UI
Test DAG
The “dags” directory has been created in the directory where we ran the dokcer-compose.yml file. Let’s create our test DAG in it. For this purpose. I will direct you to my other post, where I described exactly how to do it.
In short: create a test dag (python file) in the “dags” directory. It will automatically appear in Airflow UI. Then just run it. In addition, check monitoring from the Flower UI level.


If you enjoyed this post please add the comment below or share this post on your Facebook, Twitter, LinkedIn or another social media webpage.
Thanks in advanced!