Running Apache Airflow in a Docker Compose setup provides a consistent, isolated environment for development and testing. This approach encapsulates the database, scheduler, web server, and workers within discrete containers, eliminating version conflicts between dependencies. With a defined docker-compose.yml file, teams can spin up a complete workflow engine with a single command, ensuring that every collaborator or CI pipeline operates on the exact same infrastructure.
Why Combine Airflow and Docker Compose
The synergy between Airflow and Docker Compose addresses common deployment friction points. Docker Compose abstracts the complexity of managing multiple linked services, allowing users to focus on DAG logic rather than infrastructure configuration. This stack is ideal for local development, where spinning up a production-like environment without virtual machines saves time and system resources.
Core Components in a Standard docker-compose.yml
A typical configuration includes several essential services that communicate over a shared network. The PostgreSQL image serves as the metadata database, storing task instances and connection logs. The Redis container handles broker messaging for task queuing, while the Airflow image provides the scheduler, web UI, and optional worker containers. Volume mounts ensure that DAG files and configuration persist outside the ephemeral container lifecycle.
Service Definitions and Networking
Each service is defined with specific environment variables that control initialization. For example, AIRFLOW__CORE__SQL_ALCHEMY_CONN points to the database URI, and AIRFLOW__CORE__LOAD_EXAMPLES disables sample datasets to conserve resources. The docker-compose network allows containers to reference each other by service name, simplifying connection strings and reducing hard-coded IP addresses.
Setting Up the Environment Variables
Environment variables are the primary mechanism for configuring Airflow inside containers. Setting AIRFLOW__CORE__EXECUTOR to LocalExecutor or CeleryExecutor determines how tasks are dispatched. Additional variables configure the secret key for the web server, time zone, and email settings if alerts are required. Using an .env file keeps sensitive data separate from the compose file and simplifies switching between development and production profiles.
Initializing the Database and Admin User
On first launch, the Airflow container must initialize the database and create an admin account. This is commonly handled with command overrides in the compose file, where airflow db migrate and airflow users create are executed in sequence. Proper dependency ordering ensures the database is ready before migration scripts run, preventing startup failures due to connection timeouts.
Scaling Workers and Handling Task Concurrency
For more realistic testing, you can scale worker containers to process multiple tasks in parallel. The docker-compose scale command or deploy profiles in newer Compose files can spin up additional Celery workers. Concurrency limits in airflow.cfg must align with the number of workers to avoid overloading the machine or overwhelming external APIs during test runs.
Monitoring Logs and Troubleshooting
Inspecting logs is straightforward when services run under Docker Compose. Using docker compose logs -f allows real-time observation of scheduler heartbeats, worker task execution, and web server requests. If a DAG fails, checking container exit codes, volume mount paths, and network DNS resolution often reveals configuration issues faster than traditional debugging methods.
