Design an n8n queue-mode deployment

Design n8n queue mode around PostgreSQL, Redis and workers, with binary-data choices, licence boundaries, replay safety and failure acceptance checks.

Last updated

In short

  • Queue mode separates execution capacity from the main process.
  • Worker retries require idempotent business operations.
  • Binary storage and multi-main availability depend on version and edition.
On this page (6)

Architectures · Part of n8n

Scale the execution path you actually have

n8n queue mode separates workflow execution into workers while the main process handles coordination and application functions. PostgreSQL stores application and execution state, and Redis coordinates queued execution work. This can help scale a workload, but adding workers does not automatically make the entire service highly available or make business operations safe to repeat.

This is a reference architecture and failure exercise. It has not been load-tested for a particular workload. Use the current queue-mode documentation for the exact n8n release selected, and record database, Redis and worker versions together.

Assign responsibility to each component

Component Role Failure question
Main process Application UI, coordination and configured triggers What happens to new events when it is unavailable?
Webhook handling Accept external requests through the chosen topology When is a request acknowledged as accepted?
Redis Queue coordination What work is delayed or needs reconciliation after interruption?
PostgreSQL Workflow, credential and execution state Can accepted work and credentials be restored?
Workers Run workflow executions Can an interrupted external operation be repeated safely?
Binary storage Files used by executions Can every relevant process access the same object?

Redis is an additional infrastructure dependency in this reference design; confirm its operating scope separately from the Pilae app catalogue.

All instances that decrypt stored credentials need the same intended encryption key, protected through secret management and recoverable under the backup process. A database restore without that key can leave credentials unusable. Restrict Redis and PostgreSQL to the runtime network and authorized operators; neither should be an open public service.

Choose the storage and edition deliberately

The current queue documentation describes database binary storage by default, licensed external-storage options and shared-volume requirements when filesystem storage is used. These details are version-sensitive. Do not copy an old statement that all filesystem arrangements are unsupported, or assume a worker’s private local disk is shared with another process.

Decision Record in the design Acceptance case
Binary storage Database, supported external store or shared filesystem File accepted on one process is usable on another
Worker concurrency Initial value and measured resource envelope Burst completes without exhausting database connections
Execution retention Retained metadata and payload policy Old execution cleanup preserves required evidence
Encryption key Secret reference and recovery owner Restored worker can decrypt a synthetic credential
Main-process availability Single main or licensed multi-main design Trigger behavior during main-process loss

Multi-main is documented as a self-hosted Enterprise capability. Queue mode and multiple workers should not be described as an automatic substitute for that feature. Review n8n’s source licence and the intended commercial arrangement; n8n should not be treated as unrestricted OSI open-source software merely because its source is visible.

Build replay safety into the workflow

Suppose a workflow receives a supplier invoice and creates a draft in an accounting system. A worker can lose connectivity after the remote system accepted the invoice but before n8n recorded the response. Retrying that step can create a duplicate unless the destination or integration uses a stable idempotency key.

Use a business key such as source-system identifier plus invoice identifier, and persist the mapping to the destination record. Before retrying an ambiguous request, query the destination or the integration ledger. Do not use an execution ID as the only business key if a manual replay produces a new execution ID.

Define which errors can be retried automatically, which need human review and how long a retry remains valid. A queue provides execution coordination; it cannot decide whether paying an invoice twice is acceptable.

Measure capacity with representative flows

Create a synthetic workload containing a short API request, a slow remote operation, a binary attachment and a database operation. Measure arrival rate, queue age, execution duration, worker memory, database connections and destination rate-limit responses. Queue age is often more useful than worker CPU alone because it describes how long accepted work waits.

Increase concurrency in controlled steps while observing the actual bottleneck. A remote service’s rate limit or PostgreSQL connection budget can become the limit before worker CPU does. Record the workload and configuration with any result; a single throughput number without that context is not a sizing recommendation.

Run a failure and recovery exercise

Interrupt a worker before an external write, during the write and after the destination accepts it. Check the integration ledger and verify exactly one intended business result. Restart Redis in the pilot and reconcile accepted requests against completed and unresolved executions. Stop the main process and observe webhook and timer behavior under the chosen topology.

Back up PostgreSQL, encryption-key references, workflow definitions, configuration and binary data consistently. Redis persistence and recovery need an explicit operational decision, but a Redis snapshot is not the complete application backup. Restore into an isolated environment with outbound actions disabled so recovery does not resend real notifications or recreate transactions.

For upgrades, use the same intended n8n version across the participating processes and follow the supported migration sequence. Retain the previous configuration and a compatible database backup. Acceptance means the system handles the expected burst, reports failures clearly, preserves credentials and reconciles interrupted work without duplicate business effects.

Sources and further reading

Use the linked documentation to confirm the version, edition and contract that apply to your project.

Design our workflow runtime

Bring workflow volumes, attachment sizes, destination rate limits and the business operations that must never be duplicated.