1. Pilae
2. Apps
3. Apache Flink

Running Apache Flink in production: what it takes

1. Install the operator, then one FlinkDeployment per job

   The Flink Kubernetes Operator from its Helm chart, with the admission webhook on. Each job is a FlinkDeployment in application mode, a cluster of its own built from a pinned image with the job JAR inside, so jobs never share a JobManager or TaskManagers.

2. Checkpoint to Garage every minute

   RocksDB state with incremental checkpoints every 60 seconds, in a Garage bucket spread over three zones. Upstream retains one completed checkpoint by default; we keep three, so a restart has somewhere to go if the newest one is damaged.

3. Upgrade through a savepoint

   Every job runs with upgradeMode savepoint: on each change the operator stops the job with a savepoint and starts the new version from it, or from the last checkpoint if the job is unhealthy. Every stateful operator carries a fixed uid, so its state finds it again after the job graph changes.

4. Two JobManagers, metadata in the bucket

   Kubernetes high availability with a standby JobManager. Only a pointer lives in Kubernetes; the metadata a new leader needs is in the bucket, which is also what lets the operator rebuild a JobManager deployment that was deleted by mistake.

5. Restore a savepoint every month

   Savepoints are copied offsite daily, encrypted, to your chosen country. Checkpoints are not, because upstream does not guarantee they can be moved. Once a month we start each job from its latest savepoint on a scratch cluster, with its own output topic and Kafka transactional ID prefix so it cannot interfere with production's transactions, and compare what it writes.

What Apache Flink is, and who runs it

Apache Flink system requirements

Before anything is deployed, this is what has to exist. We size it with you in the first session, and we say so when your own hardware is already enough.

Kubernetes

Kubernetes · cert-managerThe Flink Kubernetes Operator runs on a cluster you already have or on a small one we set up for Flink. Its admission webhook, which rejects an invalid FlinkDeployment before anything is deployed, gets its certificate from cert-manager.

Flink and Java

Flink · JavaThe official images default to the Java release upstream recommends. Newer Java releases are marked experimental until upstream says otherwise, and old ones are dropped as Flink moves on. Jobs written for the previous major version of Flink are rebuilt against the current one, and changed where they use the DataSet API, the Scala APIs, SourceFunction or SinkFunction, which the current major version removed.

JobManagers

2 × 1 vCPU · 2 GBOur starting point: a leader and a standby. The standby takes over leadership without waiting for a new pod, but the job still restarts from its last checkpoint when the leader fails.

TaskManagers

1 vCPU · 4 GB · 1 slot eachOur starting point, with as many as the job's parallelism needs. Upstream finds that more, smaller TaskManagers with one slot each isolate tasks best. RocksDB keeps working state on local disk, so each one also gets a local SSD volume.

Object storage

S3 bucket · GarageCheckpoints, savepoints and high-availability metadata, written through Flink's Presto S3 plugin, which upstream describes as production-proven for checkpointing. Garage accepts only requests signed for its own region name, so its region is set to the one the plugin signs with.

Connectors

Released per Flink versionConnectors ship separately from Flink, and each connector release names the Flink versions it supports, so a job moves to a new Flink release only once its connectors list it.

Migrating from Confluent Cloud for Apache Flink to Apache Flink

Flink SQL is what travels, with edits, and state is what does not. Confluent Cloud infers a table from every topic, while open-source Flink needs a CREATE TABLE for each one, with the kafka connector, a format and a watermark strategy written out. The $rowtime column becomes the connector's timestamp metadata column, and SET options go back from Confluent's names to Flink's, so sql.state-ttl becomes table.exec.state.ttl. Topics that use Schema Registry decode in Avro through the avro-confluent format; Protobuf and JSON Schema topics need a format of their own first. Confluent Cloud takes savepoints for its own use and documents no way to export one, so each stateful statement rebuilds its state by reading its source topics again, from the earliest retained offset or from a timestamp you choose. A window or a join can only be rebuilt from records the topics still hold, which is why retention is checked before anything moves.

1. List the statements and what each one keeps

   Every running statement, the tables it reads and writes, and whether it holds state. Joins, aggregations, windows and pattern matching do; filters and projections do not. The stateful statements, and how far back their state reaches, set the timetable.

2. Declare the tables Confluent inferred

   Each topic gets a CREATE TABLE with the kafka connector, its format and registry address, a timestamp metadata column in place of $rowtime, and its watermark strategy. The DDL lives in your repository, beside the statements that use it.

3. Replay the state from the topics

   Stateful statements start on Flink from the earliest retained offset, or from a timestamp, and write to new topics while the Confluent statements keep running. Outputs are compared until they agree, and a statement whose window reaches past a topic's retention is named before it is moved.

4. Repoint consumers, then stop the old statements

   Consumers move to the new output topics one at a time. The Confluent statements are stopped once their owners sign off, and their tables are left alone: on Confluent Cloud, DROP TABLE deletes the topic and every record in it.

One job, its state on Garage, every upgrade through a savepoint

# orders-enrichment.yaml (excerpt), applied by the Pilae Agent after review
apiVersion: flink.apache.org/v1beta1
kind: FlinkDeployment
metadata:
  name: orders-enrichment
  namespace: acme-flink
spec:
  image: registry.acme.internal/flink/orders-enrichment:a41f9c2  # job, Flink and S3 plugin
  flinkVersion: v2_2
  serviceAccount: flink
  flinkConfiguration:
    state.backend.type: rocksdb
    execution.checkpointing.interval: 60 s
    execution.checkpointing.incremental: true
    execution.checkpointing.num-retained: 3
    execution.checkpointing.dir: s3://acme-flink/checkpoints
    execution.checkpointing.savepoint-dir: s3://acme-flink/savepoints
    high-availability.type: kubernetes
    high-availability.storageDir: s3://acme-flink/ha
    s3.endpoint: http://garage.acme.internal:3900  # plain HTTP, private network only
    s3.path-style-access: true
    kubernetes.operator.periodic.savepoint.interval: 6h
    kubernetes.operator.savepoint.history.max.count: 10
    kubernetes.operator.savepoint.history.max.age: 72h
    kubernetes.operator.cluster.health-check.enabled: true
  jobManager:
    replicas: 2  # a leader and a standby
    resources:
      requests: { cpu: "1", memory: "2048Mi" }
  taskManager:
    resources:
      requests: { cpu: "1", memory: "4096Mi" }
  podTemplate:
    spec:
      containers:
        - name: flink-main-container
          env:
            # read by the S3 plugin's AWS SDK; never written into the spec
            - name: AWS_ACCESS_KEY_ID
              valueFrom: { secretKeyRef: { name: flink-garage-key, key: access-key-id } }
            - name: AWS_SECRET_ACCESS_KEY
              valueFrom: { secretKeyRef: { name: flink-garage-key, key: secret-access-key } }
  job:
    jarURI: local:///opt/flink/usrlib/orders-enrichment.jar
    entryClass: acme.orders.EnrichmentJob
    parallelism: 4
    upgradeMode: savepoint
    state: running

What Pilae is responsible for

Where it runs

zur1, fra1, fal1, gra1, ams1, hel1, lon1, ash1, hil1, sin1, tok1, syd1, on-premZurich, Frankfurt, Falkenstein, Gravelines, Amsterdam, Helsinki, London, Ashburn, Hillsboro, Singapore, Tokyo, Sydney, Your own hardware

Who holds the credentials

You do. Ours are separate, named, logged and revocable with one command. We ask before anything changes outside an agreed window.

If you leave

The machine, the data, the compose files and the runbook are already yours. Nothing stops when our access does.

What running Apache Flink costs

Apache Flink starts on Business, because it needs dedicated hardware. On Business, it is one of up to three apps for CHF 1,590EUR 1,690USD 1,990 a month, on your own servers or in Pilae Cloud. Enterprise is quoted. The plans set what each includes; these are the inputs that move the price around them.

Instance size

The CPU, memory and, where a model runs, the GPUs the app needs for your users and your data.

High availability

One machine with tested restores, or a replicated setup that keeps serving when a node fails.

Storage and backups

How much data it holds, how long backups are kept, and point-in-time recovery for its database.

Plan and support

Starter, Business or Enterprise: support hours, response times in the contract and how often we review the service with you.

Region

Your own hardware, where the infrastructure is already yours, or a Pilae Cloud region, where machines are priced per month. On Starter, Pilae chooses the site.

Sign-on and integrations

Single sign-on, directory sync, mail relays and the other systems the app has to reach.

Apache Flink: common questions

Also in data and streaming