Managed Apache NiFi hosting

Data and streamingOn-prem or your region

Data flows from the Apache Software Foundation that move files and records between systems and log every step, run beside those systems in Switzerland, the EU or your own datacentre. Pilae runs it on your own servers, or in Zurich, Switzerland, and eleven other Pilae Cloud regions.

Talk to us about Apache NiFi

Licence: Apache-2.0
Runs on: Your own hardware, or any of 10+ Pilae regions — six of them in Switzerland and the EU
Upgrades: Pinned, tested against your configuration, applied in your window
Upstream: nifi.apache.org

Running Apache NiFi in production: what it takes

Give each repository its own volume
Content, FlowFile and provenance repositories each get a volume outside the NiFi installation directory, so an upgrade leaves them where they are. Provenance is sized to the retention you ask for, where NiFi ships with 30 days or 10 GB.
Set the key before the first flow
The sensitive properties key is generated once, set identically on every node and escrowed with the backups. Changing it later means rewriting the flow with nifi.sh set-sensitive-properties-key, so it is in place before anyone enters a password.
Sign on through Keycloak, one cluster per boundary
OpenID Connect to Keycloak or Entra ID, with directory groups granted policies such as view the UI, access the controller and query provenance. NiFi has deprecated per-process-group policies for removal, so teams whose data must stay apart get separate clusters, not separate process groups.
Version flows in Git
Process groups are committed through the GitLab, GitHub, Bitbucket or Azure DevOps flow registry client to a repository you own, with the NiFi user recorded as the commit author. Sensitive parameter values are never written to the repository. NiFi Registry is deprecated, so we do not deploy it.
Back up the flow, export the provenance
The conf directory, the flow history and local state go offsite daily, encrypted, to your chosen country, with the key escrowed beside them. Provenance events are exported as JSON to object storage for as long as your auditors require. Once a month we restore a node into a scratch environment and enable a controller service that holds a password, which proves the key matches.
Upgrade the cluster together
Nodes on different NiFi versions cannot share a cluster, so an upgrade stops the sources, lets the queues drain and moves every node at once. Before each one we read nifi-deprecation.log for components and features marked for removal. The Pilae Agent tests the upgrade on a copy, waits for your approval, applies it in your window and records it in the console.

What Apache NiFi is, and who runs it

Self-hosted Apache NiFi for data flows with provenance

Apache NiFi moves data between systems and changes it on the way. A flow is drawn in the browser as processors joined by queues: fetch files over SFTP, parse them as records, look up a value, then write to PostgreSQL or publish to Kafka. Each piece of data is a FlowFile, attributes plus content, and a queue that reaches its threshold holds back the processor feeding it instead of dropping data. NiFi began at the US National Security Agency and has been an Apache top-level project since 2015.

Provenance is what sets NiFi apart. It records an event each time a FlowFile is received, changed, split, joined, sent or dropped, and can draw the lineage of any file from source to destination. It is built for continuous ingestion and file movement, not for orchestrating batch jobs that run elsewhere, which is what Airflow does, or for windows and joins over a stream, which is the job of Flink.

Apache NiFi in production: repositories, the key and the cluster

NiFi keeps its working state in three repositories on local disk: the FlowFile repository logs each FlowFile’s attributes and place in the flow, the content repository holds the bytes, and the provenance repository holds the events. Each gets its own volume, because a content repository that fills a shared disk can corrupt the FlowFile repository beside it. Nodes run on your hardware or in the Pilae Cloud region that holds the data, one of 10+, six of them in Switzerland and the EU, and answer only on your private network.

Every password in a flow is encrypted in flow.json.gz with the sensitive properties key, which is escrowed beside the backups. Flows are versioned in Git, for example in a GitLab we run for you, and provenance you must keep longer than the disk allows goes to a Garage bucket. Sign-on goes through Keycloak. Our monitoring reads repository free space and queue depth from the NiFi Prometheus endpoint and alerts an engineer before back pressure halts a flow. The Pilae Agent rehearses each upgrade on a copy first.

Apache NiFi licence and editions

NiFi has no paid edition, so there is no seat count to track and no licence to hold in your name. Two deprecations shape how we run it: NiFi Registry, and policies on individual process groups and components, are both marked for removal. We build on what stays, so the next major release is an upgrade, not a redesign. Our operation follows the published plans. Talk to us about the PowerCenter workflows or Talend jobs you want to move.

Apache NiFi is Apache-2.0 and has one edition: clustering, OpenID Connect and SAML sign-on, data provenance and the Git flow registry clients are all in the release we deploy. The binary on the Apache mirrors leaves out some optional extensions because of its size, so we add the ones your flows need. Cloudera sells Cloudera Data Flow, and Snowflake, which agreed in 2024 to acquire Datavolo, a company built on NiFi, sells Openflow. Both are built on Apache NiFi and sold under their own contracts. You do not need them for anything on this page. Apache NiFi, NiFi and the NiFi logo are trademarks of the Apache Software Foundation, which does not endorse this service.

Apache NiFi system requirements

Before anything is deployed, this is what has to exist. We size it with you in the first session, and we say so when your own hardware is already enough.

CPU and memory: 8 vCPU · 32 GB per nodeOur starting point for each of three nodes. The shipped bootstrap.conf gives the Java heap 1 GB, which we raise to 16 GB. A queue past its swap threshold, 20,000 FlowFiles by default, swaps the rest to disk instead of holding them on the heap.
Runtime: JavaCurrent NiFi requires a recent Java LTS release. Python processors also need Python on every node, and upstream still marks them as a beta feature, not for production use.
Repository disks: content · FlowFile · provenance, one volume eachUpstream warns that a content repository filling a shared disk can corrupt the FlowFile repository beside it. Provenance gets its own volume because it is sized by how long you keep it. All three are mounted noatime, as upstream advises.
Coordination: ZooKeeper, 3 nodesElects the cluster coordinator and the primary node, and holds cluster-wide processor state. On a Kubernetes cluster you already run, NiFi can use Kubernetes Leases and ConfigMaps instead.
Sensitive properties key: nifi.sensitive.props.key, 12+ charactersEncrypts every password and token stored in flow.json.gz. It must be the same on every node, and a flow restored without it cannot decrypt the credentials it holds.
Sign-on: OIDC or SAML 2 · HTTPS on 8443Group names in the token count only when a group of the same name exists in the NiFi user group provider, so groups are also declared in NiFi or synced from your directory over LDAP.

Migrating from Informatica PowerCenter to Apache NiFi

Informatica set the end of support for PowerCenter 10.5 at 31 March 2026, and the path it offers from there leads to its cloud platform, IDMC. NiFi has no importer for PowerCenter mappings or workflows, so pipelines are rebuilt, not converted. The XML that pmrep exports is still useful: it lists every source, transformation, target and schedule, and it becomes the specification for the rebuild. What changes is the model. PowerCenter runs each mapping as a session, a scheduled batch over its sources. NiFi moves data through a flow as it arrives, a batch of records at a time, so set-based work such as large joins and aggregations moves into the target database as SQL. Connections and their passwords become NiFi controller services and sensitive parameters. Run history stays in the PowerCenter repository, which is kept read-only for as long as your auditors need it.

Start from the run history
The REP_WFLOW_RUN view in the PowerCenter repository records every workflow run. Workflows that have not run in a year are exported and archived rather than rebuilt, and the rest are ranked by how often they run and what depends on them.
Export each mapping as a specification
Mappings and workflows are exported to XML with pmrep ObjectExport, dependencies included. Each export is read into a short specification of sources, transformations, targets and schedule, which the owner signs off before anything is built.
Rebuild row by row, push joins down
Expressions, filters, routers and lookups become record processors such as UpdateRecord, QueryRecord and LookupRecord. Joins and aggregations over whole tables move into the target database as SQL. Schedules become CRON-driven processors.
Run both, then unschedule PowerCenter
Each rebuilt flow writes to a staging schema beside the PowerCenter target for a full business cycle, with row counts and totals compared every day. The PowerCenter workflow is unscheduled only when its owner has signed off the comparison.

One node of three: what NiFi keeps on disk

confconfiguration · backed up daily
- flow.json.gzthe flow, passwords encrypted
- archiveprevious flows, 30 days or 500 MB
- nifi.propertiessensitive props key, escrowed
- authorizations.xmlpolicies for directory groups
/srv/nifi/flowfile_repositoryNVMe, own volume · write-ahead log, checkpoint every 20 s
/srv/nifi/content_repositoryNVMe, own volume · 2 TB
- 0–1023sections holding the content of queued FlowFiles
- 0–1023/archivecontent no longer queued, kept 24 hours for replay and download
/srv/nifi/provenance_repositoryown volume · 1 TB
- max.storage.time90 days, where NiFi ships with 30
- max.storage.size800 GB, where NiFi ships with 10 GB
- index.shard.size4 GB, as upstream advises for production
/srv/nifi/database_repositoryflow history: who changed which component, and when
/srv/nifi/state/localprocessor state for this node
s3://acme-backups/nifioffsite, daily, encrypted: conf, flow history, state
s3://acme-provenanceexported events as JSON, kept as long as you require

An example node from a three-node cluster for acme. The flow and the key that decrypts its passwords sit in conf, and go offsite every night with the flow history and the local state. Each repository has its own volume. Provenance is kept 90 days on the node and exported for longer, while the content archive behind it, which replay and download need, is kept for a day.

What Pilae is responsible for

A pinned version

A version we have run, not whatever latest resolves to that day.

A runbook

What it depends on, how it fails, what to do about it. In your repository.

A restore drill

Backups restored on a schedule. A backup nobody has restored is a file.

A patch window

Security updates in a window you agreed, with a rollback ready.

Someone watching

Every endpoint probed on the minute. An alert reaches a person, not a dashboard nobody opens.

Where it runs: zur1, fra1, fal1, gra1, ams1, hel1, lon1, ash1, hil1, sin1, tok1, syd1, on-premZurich, Frankfurt, Falkenstein, Gravelines, Amsterdam, Helsinki, London, Ashburn, Hillsboro, Singapore, Tokyo, Sydney, Your own hardware
Who holds the credentials: You do. Ours are separate, named, logged and revocable with one command. We ask before anything changes outside an agreed window.
If you leave: The machine, the data, the compose files and the runbook are already yours. Nothing stops when our access does.

What running Apache NiFi costs

On Starter, Apache NiFi runs in Pilae Cloud for CHF 449EUR 479USD 559 a month, hosting and onboarding included. On Business, it is one of up to three apps for CHF 1,590EUR 1,690USD 1,990 a month, on your own servers or in Pilae Cloud. Enterprise is quoted. The plans set what each includes; these are the inputs that move the price around them.

Instance size: The CPU, memory and, where a model runs, the GPUs the app needs for your users and your data.
High availability: One machine with tested restores, or a replicated setup that keeps serving when a node fails.
Storage and backups: How much data it holds, how long backups are kept, and point-in-time recovery for its database.
Plan and support: Starter, Business or Enterprise: support hours, response times in the contract and how often we review the service with you.
Region: Your own hardware, where the infrastructure is already yours, or a Pilae Cloud region, where machines are priced per month. On Starter, Pilae chooses the site.
Sign-on and integrations: Single sign-on, directory sync, mail relays and the other systems the app has to reach.

Apache NiFi: common questions

Is Apache NiFi open source?

Yes. It is Apache-2.0 and a top-level project of the Apache Software Foundation, with one edition: clustering, sign-on, provenance and Git versioning are all in the release we deploy. Cloudera Data Flow and Openflow from Snowflake are commercial products built on NiFi and sold under their own contracts. You do not need either of them to run NiFi.

Can NiFi import our PowerCenter, Talend or Azure Data Factory jobs?

No. NiFi has no importer for any of them, so pipelines are rebuilt. The exports still help: PowerCenter mappings export to XML and Azure Data Factory pipelines are JSON definitions, and both list the sources, steps and targets to rebuild. Talend Open Studio, the open-source edition of Talend Studio, was retired by Qlik on 31 January 2024 and no longer receives updates.

Can several teams share one NiFi and keep their data apart?

Not safely, and the project now says so. NiFi has deprecated policies on individual process groups and components for removal, because every processor runs in the same Java process with the same access to the machine, so the boundary those policies drew could not be enforced. Teams whose data must stay apart get separate NiFi clusters, each with its own sign-on groups, disks and key.

What does provenance record, and for how long?

For each FlowFile, every component that received, changed, routed, sent or dropped it, when, its attributes at that point, and for transfers the address it came from or went to. A user downloading content from the interface is an event too, and changes to the flow itself are kept in the flow history. NiFi ships with 30 days or 10 GB of provenance, which we size to your retention, and events can be exported to object storage for longer. The content behind events stays only as long as the content archive keeps it, so replay works for a shorter window than search.

What happens to data in flight when a node fails?

It waits on that node. NiFi does not copy queued data between nodes: each node works on its own share. The other nodes carry on with new data, and what was queued on the failed node is processed when it comes back, or moved to the others with an offload if it can still be reached. That is why each repository sits on its own reliable volume, and why we prefer sources that can replay, such as a Kafka topic, over ones that push once and forget.

Back to apps

Bring us your Apache NiFi. We will tell you what it takes.

Thirty minutes on the deployment you already have, or the one you are about to start.

Start with one workload Book a 30-minute briefing