1. Pilae
2. Apps
3. Apache Airflow

Running Apache Airflow in production: what it takes

1. Build one image for five components

   The upstream Apache Airflow image at a pinned release, with your providers installed against its constraints file and apache-airflow pinned in the same command. The same image runs the API server, the scheduler, the DAG processor, the triggerer and the Celery workers, from one configuration, on PostgreSQL and Redis.

2. Read DAGs from Git

   The GitDagBundle follows a branch of your repository through a read-only deploy key, and every run keeps the commit it started on until it ends. Triggers cannot load from a bundle, so they ship in the image with the plugins, and a change to them is an image release.

3. Map directory roles to Airflow roles

   The FAB auth manager signs people in with OpenID Connect to Keycloak or Entra ID, and a short security manager class maps their directory roles to Viewer, User, Op or Admin at every login. The API server answers only on your private network.

4. Back up the database and the Fernet key together

   The metadata database goes offsite daily, encrypted, to your chosen country, and the Fernet key is escrowed beside it. Once a month we restore both into a scratch environment and test a stored connection, which proves the key matches. airflow db clean prunes old run records on a retention you set.

5. Upgrade on a copy first

   Each upgrade runs airflow db migrate after a snapshot. Providers are released separately from Airflow and are pinned and upgraded the same way. The Pilae Agent tests the upgrade and parses your DAGs on a copy, waits for your approval, applies it in your window and records it in the console.

What Apache Airflow is, and who runs it

Apache Airflow system requirements

Before anything is deployed, this is what has to exist. We size it with you in the first session, and we say so when your own hardware is already enough.

CPU and memory

4 vCPU · 8 GB, plus workersOur starting point for the API server, scheduler, DAG processor and triggerer on one machine. Workers are sized to the tasks they run: a task that loads a file into memory needs more than one that calls an API and waits.

Metadata database

PostgreSQLHolds DAG runs, task state, serialised DAGs and the encrypted connections and variables. Upstream also supports MySQL. SQLite is for its tests only and MariaDB is not tested, so we use neither.

Executor and queue

CeleryExecutor · RedisCelery workers take tasks from Redis and record results in the database, as upstream recommends. Where you already run Kubernetes, the KubernetesExecutor starts a pod per task instead.

Keys

Fernet key · JWT secretThe Fernet key encrypts connection passwords and variables in the database, and a different key cannot decrypt them, so it is escrowed with the backups. The JWT secret signs the short-lived tokens tasks use to report back, and must be the same on the scheduler and the API server.

DAG repository

Git · read-only deploy keyThe DAG processor and the workers fetch DAGs through the git provider's GitDagBundle, over SSH or HTTPS. Each run records the commit it started on.

Task logs

S3-compatible bucketRemote logging writes the log of every task attempt to a bucket, through an Airflow connection pointed at its endpoint. Without it, logs stay on the worker that ran the task and go when that worker does.

Migrating from Astronomer Astro to Apache Airflow

An Astro project is already an Airflow repository: dags, plugins, include and tests folders, with Python packages in requirements.txt and system packages in packages.txt. The DAG code comes across as it is. The Dockerfile changes: it starts from Astronomer's Astro Runtime image, which we replace with the Apache Airflow image at the same Airflow version, after comparing what each image installs so that nothing a DAG imports leaves with the base image. What stays behind is what Astro keeps outside the repository. Connections, variables and environment variables set in the Astro UI or its Environment Manager are re-created on the new deployment. Astro alerts live in the Astro UI rather than in DAG code, so they become failure callbacks, Deadline Alerts and checks in our monitoring. Run history and task logs stay with the old Deployment. If it still runs the previous major version of Airflow, the move to the current one is most of the work.

1. Inventory DAGs, packages and settings

   Every DAG with its owner, schedule and last successful run, the Airflow and provider versions the old service runs, and every connection, variable, pool and alert. DAGs nobody has run in months are archived rather than moved.

2. Move to the current major version in the repository

   ruff with its AIR rules flags most of what the current major version removes or moves, such as imports that moved to airflow.sdk, operators now in the standard provider and removed context variables like execution_date. Tasks that query the metadata database directly are rewritten against the REST API, and SLAs become Deadline Alerts. Each fix is a commit your team reviews.

3. Replace the cloud's implicit credentials

   On a managed service, tasks often reach that cloud through an identity attached to the environment: the execution role on MWAA, the environment's service account on Google's service. Off those platforms, each such call needs a connection with its own credentials, scoped to what the DAG touches. Cloud operators keep working for as long as the services they call stay in use.

4. Switch DAG by DAG, never in both places

   Each DAG is deployed paused, run once by hand against test targets, then unpaused on the new Airflow and paused on the old service the same day, so no run executes twice. Catchup is off by default in current Airflow, so nothing backfills unless you ask for it.

One repository for the DAGs, one image for everything else

What Pilae is responsible for

Where it runs

zur1, fra1, fal1, gra1, ams1, hel1, lon1, ash1, hil1, sin1, tok1, syd1, on-premZurich, Frankfurt, Falkenstein, Gravelines, Amsterdam, Helsinki, London, Ashburn, Hillsboro, Singapore, Tokyo, Sydney, Your own hardware

Who holds the credentials

You do. Ours are separate, named, logged and revocable with one command. We ask before anything changes outside an agreed window.

If you leave

The machine, the data, the compose files and the runbook are already yours. Nothing stops when our access does.

What running Apache Airflow costs

On Starter, Apache Airflow runs in Pilae Cloud for CHF 449EUR 479USD 559 a month, hosting and onboarding included. On Business, it is one of up to three apps for CHF 1,590EUR 1,690USD 1,990 a month, on your own servers or in Pilae Cloud. Enterprise is quoted. The plans set what each includes; these are the inputs that move the price around them.

Instance size

The CPU, memory and, where a model runs, the GPUs the app needs for your users and your data.

High availability

One machine with tested restores, or a replicated setup that keeps serving when a node fails.

Storage and backups

How much data it holds, how long backups are kept, and point-in-time recovery for its database.

Plan and support

Starter, Business or Enterprise: support hours, response times in the contract and how often we review the service with you.

Region

Your own hardware, where the infrastructure is already yours, or a Pilae Cloud region, where machines are priced per month. On Starter, Pilae chooses the site.

Sign-on and integrations

Single sign-on, directory sync, mail relays and the other systems the app has to reach.

Apache Airflow: common questions

Also in data and streaming