1. Pilae
2. Apps
3. Apache Kafka

Running Apache Kafka in production: what it takes

1. Format the quorum, then the brokers

   Three KRaft controllers on machines of their own, and at least three brokers from a pinned Apache release, each with broker.rack set to the rack or room it sits in. Topics default to three replicas with min.insync.replicas at 2, and automatic topic creation is off, so every topic exists because someone defined it.

2. Encrypt every listener, one identity per application

   TLS between clients, brokers and controllers. Applications authenticate with SASL/SCRAM or with OAuth tokens issued by Keycloak, and the built-in StandardAuthorizer refuses whatever no ACL allows. Each application gets ACLs on its own topics and consumer groups, and nothing else.

3. Run Connect and MirrorMaker 2 as clusters of their own

   Kafka Connect runs in distributed mode, with connector configurations and offsets kept in compacted topics on the cluster, and only the connector plugins you approved in its image. MirrorMaker 2 runs where a second site needs a copy of the data.

4. Alert on replication, not only on the port

   Probes reach every listener every 60 seconds. Beside them we track under-replicated partitions, partitions under min.insync.replicas and the lag of each consumer group. An engineer is alerted when a partition drops under its minimum, because from then on producers using acks=all, the default, get errors instead of acknowledgements.

5. Roll upgrades, then raise the metadata version

   Each broker stops with a controlled shutdown, which moves its partition leadership elsewhere first, and the next one waits until every partition is fully replicated again. The Pilae Agent tests the upgrade on a copy, and raises metadata.version with kafka-features.sh only once the whole cluster runs the new release and you approve, because a metadata version that brings metadata changes cannot be downgraded.

What Apache Kafka is, and who runs it

Apache Kafka system requirements

Before anything is deployed, this is what has to exist. We size it with you in the first session, and we say so when your own hardware is already enough.

KRaft controllers

3 dedicated nodesThey hold the cluster metadata in a replicated log, and three tolerate one failure. Upstream does not recommend combining the controller and broker roles in critical deployments, so controllers get small machines of their own.

Brokers

3+ nodes · broker.rack setOne rack or room per replica, so a partition with three replicas spans three of them. Upstream advises a cluster per datacentre rather than one cluster stretched over a slow link, so a second site gets a cluster of its own.

CPU and memory

4 vCPU · 16 GB per brokerOur starting point. The JVM heap stays near the 6 GB of upstream's example settings, and the rest is left to the operating system's page cache, which Kafka relies on instead of a cache of its own.

Data disks

Dedicated volumes · XFSDisk throughput is usually the limit, so Kafka data does not share disks with the system or its logs. Upstream reports better results on XFS than on ext4, asks for at least 100,000 file descriptors per broker, and vm.max_map_count is raised to suit the partition count.

Java runtime

Java, a current LTS releaseBrokers, Connect and the tools need a recent Java LTS release, and upstream recommends the most recent one. Java clients run on older releases than the brokers accept.

Schema registry

Apicurio Registry · PostgreSQLOnly where applications use Avro, Protobuf or JSON Schema. Apicurio is Apache-2.0 and recommends PostgreSQL storage in production; its default in-memory database loses everything on restart.

Migrating from Confluent Cloud to Apache Kafka

Applications keep the Kafka protocol and their client libraries, so for most of them the move is a new bootstrap address, a new credential and a new schema registry URL. Topics and their data cross with MirrorMaker 2, set to keep topic names as they are, and consumer group offsets are translated on the way. A translated offset lands at or before where the group stopped, never after it, so a consumer may read some records twice and has to tolerate it. Service accounts and API keys stay behind, and MirrorMaker copies topic ACLs without their write access, so each application gets a new identity and ACLs written for it. Schemas move to Apicurio Registry with the IDs that existing records already carry. Fully managed connectors, Flink statements and ksqlDB queries do not move at all: they are rebuilt on Kafka Connect, Kafka Streams or Apache Flink, and that list, more than the data, sets the timetable.

1. List what Confluent Cloud does besides storing topics

   Topics, consumer groups, service accounts and their ACLs, schemas, fully managed connectors, Flink statements and ksqlDB queries. The last three do not travel with the data, and the list decides how long the rest takes.

2. Move the schemas first, with their IDs

   A record written with Confluent's serializers carries its schema ID in its first bytes. Apicurio's export utility for Confluent Schema Registry produces an archive that is imported into an empty registry, and mirrored records are decoded through it before any consumer moves. Clients keep Confluent's serializers and point schema.registry.url at Apicurio's Confluent-compatible API.

3. Mirror the topics with their names

   A dedicated MirrorMaker 2 cluster beside the new one replicates from the earliest retained offset with the identity replication policy, so topic names stay the same and every group can have its offsets translated. Topic configurations follow; ACLs are written fresh.

4. Move consumers, then producers

   A consumer group is stopped on Confluent Cloud, MirrorMaker writes its translated offsets to the new cluster once its next checkpoint and offset sync have run, each every minute by default, and the group starts there. Producers move last, once the mirror has caught up, and the Confluent Cloud cluster is kept until its owners sign off.

5. Rebuild connectors and stream jobs

   Each fully managed connector becomes a self-managed one on Kafka Connect, checked for its licence first: several of Confluent's own connectors are under the Confluent Community License. Flink statements move to Apache Flink, and ksqlDB queries to Flink or Kafka Streams.

A rolling broker upgrade, one rack at a time

What Pilae is responsible for

Where it runs

zur1, fra1, fal1, gra1, ams1, hel1, lon1, ash1, hil1, sin1, tok1, syd1, on-premZurich, Frankfurt, Falkenstein, Gravelines, Amsterdam, Helsinki, London, Ashburn, Hillsboro, Singapore, Tokyo, Sydney, Your own hardware

Who holds the credentials

You do. Ours are separate, named, logged and revocable with one command. We ask before anything changes outside an agreed window.

If you leave

The machine, the data, the compose files and the runbook are already yours. Nothing stops when our access does.

What running Apache Kafka costs

Apache Kafka starts on Business, because it needs dedicated hardware. On Business, it is one of up to three apps for CHF 1,590EUR 1,690USD 1,990 a month, on your own servers or in Pilae Cloud. Enterprise is quoted. The plans set what each includes; these are the inputs that move the price around them.

Instance size

The CPU, memory and, where a model runs, the GPUs the app needs for your users and your data.

High availability

One machine with tested restores, or a replicated setup that keeps serving when a node fails.

Storage and backups

How much data it holds, how long backups are kept, and point-in-time recovery for its database.

Plan and support

Starter, Business or Enterprise: support hours, response times in the contract and how often we review the service with you.

Region

Your own hardware, where the infrastructure is already yours, or a Pilae Cloud region, where machines are priced per month. On Starter, Pilae chooses the site.

Sign-on and integrations

Single sign-on, directory sync, mail relays and the other systems the app has to reach.

Apache Kafka: common questions

Also in data and streaming