Node Roles and High Availability
Run ByteChef as separately scalable coordinator, worker and webhook nodes, and which parts of high availability need Enterprise Edition
A ByteChef node can play three roles. Each role has its own switch, every switch defaults to on, and a node can combine roles. To scale a cluster, run several nodes and give each one the roles it needs.
The three roles
| Role | Variable | What it serves |
|---|---|---|
| Coordinator | BYTECHEF_COORDINATOR_ENABLED | The REST and GraphQL API, the UI, job coordination, and the Quartz scheduler that fires triggers. |
| Worker | BYTECHEF_WORKER_ENABLED | Executes tasks. A worker serves no API. |
| Webhook node | BYTECHEF_WEBHOOK_ENABLED | The public webhook endpoints: /webhooks/{id} (including /webhooks/{id}/sse, /webhooks/*/wss and /webhooks/twilio/**), /callback, /approvals/{id}, /job/resume/{id} and /file-entries/{id}/content. |
A coordinator with BYTECHEF_WEBHOOK_ENABLED=true serves the webhook endpoints too, so a separate webhook node is optional. Add webhook nodes when webhook traffic needs its own capacity.
A webhook node runs with:
BYTECHEF_COORDINATOR_ENABLED=false
BYTECHEF_WORKER_ENABLED=false
BYTECHEF_WEBHOOK_ENABLED=trueHow each webhook mode runs on a webhook node:
- Asynchronous webhooks publish the trigger event to the message broker, and a coordinator consumes it.
- Synchronous webhooks run the trigger and the job on the webhook node itself, because the caller is waiting on that node. The webhook node therefore executes component code for them.
- SSE streams create the job on the webhook node and receive the stream events over the message broker.
What each edition includes
| Capability | Community (CE) | Enterprise (EE) |
|---|---|---|
| 1 coordinator | Yes | Yes |
| N workers | Yes | Yes |
| N separate webhook nodes (optional) | Yes | Yes |
| N coordinators (high availability) | No | Yes |
| Clustered Quartz | Optional, enables the warning about a second coordinator | Yes, required for N coordinators |
| Cluster-wide cache invalidation over the message broker | Not applicable | Yes |
| Shared HTTP sessions (no sticky sessions) | No, memory only | Yes, Redis or JDBC |
Select the edition with BYTECHEF_EDITION. Multi-tenancy (BYTECHEF_TENANT_MODE=MULTI) also requires EE.
Between the nodes, a self-hosted cluster uses RabbitMQ (AMQP) or Redis (REDIS) as the message broker. On AWS, use AWS (SQS, EE) and set BYTECHEF_MESSAGE_BROKER_AWS_CONTROL_PROVIDER=REDIS (for example Amazon ElastiCache), which is required for more than one node - see AWS SQS control messages.
Load balancer routing
Put a load balancer in front of the cluster and route by path:
| Traffic | Goes to |
|---|---|
The webhook paths from the table above, except /webhooks/{id}/test/** | The webhook nodes, with the coordinators as backup. |
/webhooks/{id}/test/** (the editor's webhook test) | The coordinators. Only coordinators serve it. |
| Everything else | The coordinators, round robin. |
Workers serve no HTTP traffic and stay out of the load balancer pool.
Every node must also share file storage: a volume all nodes mount (NFS, EFS) or the AWS provider, so any node can read what another node wrote. The encryption key and the remember-me key are the same on every node.
The coordinator backup means webhooks keep working when every webhook node is down. Set BYTECHEF_WEBHOOK_ENABLED=false on the coordinators to remove that fallback and keep all webhook traffic on webhook nodes. With every webhook node down, webhooks then fail, and the coordinator answers 404.
In the reference nginx configuration, a request that cannot reach a node within 2 seconds is retried on the next node, and so on until one answers, so a failed node is visible for about 2 seconds per failed node. nginx re-resolves the node addresses every 10 seconds, and when that happens while a request is waiting for a stopped node, nginx cannot move the request to another node; the configuration then retries it once from the start, which also uses the new addresses. Only requests that never reached a node are retried this way. A request that was already sent is never replayed if it is not idempotent.
One coordinator in CE
CE supports one coordinator. More than one starts and runs, but CE has no cache invalidation and no shared sessions, so the coordinators serve stale data and users are logged out when the load balancer switches coordinator. Without clustered Quartz, each coordinator also fires every schedule, so scheduled workflows run more than once.
With BYTECHEF_SCHEDULER_QUARTZ_CLUSTERED=true, every running coordinator checks in to the shared Quartz job store. A CE coordinator that sees another coordinator checking in logs, about a minute after startup and every 10 minutes after that:
Another coordinator is running against this database. The Community Edition supports one coordinator: with more, caches go stale and sessions are not shared. Running more than one coordinator requires ByteChef Enterprise Edition (bytechef.edition=EE, BYTECHEF_EDITION=EE).The coordinator keeps running. Running more than one coordinator requires EE.
Scheduler clustering
Only coordinators start the Quartz scheduler and fire triggers. Other nodes keep the scheduler in standby: they do not fire triggers, but they can still add and remove scheduled jobs in the shared job store.
| Variable | Default | Purpose |
|---|---|---|
BYTECHEF_SCHEDULER_QUARTZ_CLUSTERED | false | Run Quartz in clustered mode on the shared job store, so each trigger fires on exactly one coordinator. |
BYTECHEF_SCHEDULER_QUARTZ_CLUSTER_CHECKIN_INTERVAL | 15000 | How often, in milliseconds, a clustered scheduler checks in with the job store. |
More than one coordinator requires BYTECHEF_SCHEDULER_QUARTZ_CLUSTERED=true. Without it, each coordinator fires every trigger. Setting it with a single coordinator is harmless, and the reference Compose file always sets it.
Quartz serialises its job store writes through database row locks, whether or not clustering is on.
Shared sessions
By default, login state lives in the memory of the coordinator that served the login, so a load balancer would have to keep each browser on one coordinator. BYTECHEF_SESSION_STORE moves the sessions to a shared store:
| Value | Store | Notes |
|---|---|---|
MEMORY | The coordinator's memory | The default, and the only store in CE. |
REDIS (EE) | Redis, using the BYTECHEF_REDIS_* connection | Sessions are kept under the key namespace bytechef:session and expire by Redis TTL. Startup fails if Redis is unreachable. |
JDBC (EE) | The spring_session and spring_session_attributes tables in the public schema of the main database | For deployments without Redis. In multi-tenant mode the tables stay in public, not in the tenant schema. |
With REDIS or JDBC, any coordinator serves any request, so the load balancer needs no sticky sessions (no ip_hash), and a coordinator failure does not log users out. Remember-me tokens and CSRF tokens do not depend on the session store, but every node still needs the same BYTECHEF_SECURITY_REMEMBER_ME_KEY.
A CE node started with REDIS or JDBC does not fail and keeps memory sessions, so one configuration works in both editions.
Workers and webhook nodes do not use sessions, so they keep the session store disabled even when BYTECHEF_SESSION_STORE is set. Only nodes with the coordinator role open the Redis connection or use the JDBC tables.
Cache invalidation
Every node keeps its caches in its own memory (Caffeine). The only exception is the shared state caches below, which EE keeps in Redis with BYTECHEF_CACHE_PROVIDER=REDIS. In EE, evictions are broadcast to every other node over the message broker as a control message, and no setting is needed; webhook nodes take part like coordinators. CE runs one coordinator and does not broadcast evictions. A CE webhook node does not need them: production webhooks run published workflow versions, which never change, and the editor's webhook test runs on the coordinator.
The broadcast needs a broker whose control routes reach every node: AMQP, REDIS, KAFKA (a consumer group per node), JMS (a topic), or AWS with BYTECHEF_MESSAGE_BROKER_AWS_CONTROL_PROVIDER=REDIS. With AWS and the default SQS control provider, a control message reaches one node only, so cache invalidation stays off and every node keeps its caches for up to 10 minutes. The in-memory broker has a single node and needs no invalidation.
Shared state caches
A few caches hold state instead of copies of stored data: the editor's webhook test (whether a test webhook is enabled and the output of its registration), the values that Data Storage keeps during editor test runs, the jobs that generate API connectors, and the public keys of embedded JWTs. In EE, BYTECHEF_CACHE_PROVIDER=REDIS keeps these caches in Redis under the key prefix bytechef:cache: with a 10-minute TTL, so every node sees the same entries, whatever the node role, message broker or session store. All other caches stay in each node's memory. The setting turns the Redis connection on by itself; configure it with BYTECHEF_REDIS_HOST and BYTECHEF_REDIS_PORT. A node fails at startup when Redis cannot be reached, and a CE node with REDIS logs a warning at startup and keeps these caches in memory.
With the default CAFFEINE, these caches stay in each node's memory, even when Redis is configured for the message broker or the session store; EE then logs a warning at startup. With more than one EE coordinator, the editor's webhook test and the embedded JWT checks can then fail, and an editor test run can miss Data Storage values, when a request lands on a different coordinator than the one that stored the state, so set BYTECHEF_CACHE_PROVIDER=REDIS when you run more than one coordinator.
Upgrading from a version before this setting: an EE cluster that kept these caches in Redis because it used the REDIS broker, BYTECHEF_SESSION_STORE=REDIS or BYTECHEF_MESSAGE_BROKER_AWS_CONTROL_PROVIDER=REDIS must now set BYTECHEF_CACHE_PROVIDER=REDIS.
Database migrations
Run Liquibase once, before the nodes start, and turn it off on every node:
- Run a one-shot container from the same image with the
liquibasecommand andBYTECHEF_UPGRADE_ENABLED=true. - Start every node with
BYTECHEF_UPGRADE_ENABLED=false, after the one-shot run has completed successfully.
This avoids concurrent migration attempts when many nodes start together.
Reference Compose file
docker-compose.cluster.yml covers every topology on this page through variables: coordinator, worker and webhook replicas behind nginx, RabbitMQ or Redis, PostgreSQL, and a migrate service that runs Liquibase once. The migrate service waits for PostgreSQL, RabbitMQ and Redis, and every node starts after it completes.
# CE: 1 coordinator (also serving webhooks), 3 workers
docker compose -f docker-compose.cluster.yml up -d
# CE with 2 separate webhook nodes, the coordinator stays the fallback for webhook traffic
BYTECHEF_WEBHOOKS=2 docker compose -f docker-compose.cluster.yml up -d
# EE high availability: 2 coordinators
BYTECHEF_EDITION=EE BYTECHEF_CACHE_PROVIDER=REDIS BYTECHEF_COORDINATORS=2 docker compose -f docker-compose.cluster.yml up -d
# Webhooks only on 2 webhook nodes, no coordinator fallback
BYTECHEF_WEBHOOKS=2 BYTECHEF_COORDINATOR_WEBHOOKS=false docker compose -f docker-compose.cluster.yml up -dLike docker-compose.yml, the file uses fixed credentials (PostgreSQL postgres, RabbitMQ bytechef) and a fixed remember-me key, and every node mounts ~/.bytechef, where the first node to start generates the encryption key that all of them share. That makes it a one-command evaluation cluster; for anything internet-reachable, change the credentials and the remember-me key, and back up ~/.bytechef/key with the database. The variables and their defaults:
| Variable | Default | Values |
|---|---|---|
BYTECHEF_EDITION | CE | CE, EE |
BYTECHEF_COORDINATORS | 1 | more than 1 requires EE |
BYTECHEF_WORKERS | 3 | |
BYTECHEF_WEBHOOKS | 0 | with 0, the coordinators serve webhooks |
BYTECHEF_COORDINATOR_WEBHOOKS | true | with false and every webhook node down, webhooks fail |
BYTECHEF_MESSAGE_BROKER_PROVIDER | AMQP | AMQP (RabbitMQ), REDIS |
BYTECHEF_CACHE_PROVIDER | CAFFEINE | CAFFEINE, REDIS (EE only; a CE node with REDIS logs a warning and keeps CAFFEINE) |
BYTECHEF_SESSION_STORE | REDIS | REDIS, JDBC (both EE only; a CE node keeps memory sessions) |
BYTECHEF_TENANT_MODE | SINGLE | SINGLE, MULTI (EE only) |
BYTECHEF_IMAGE | docker.bytechef.io/bytechef/bytechef:latest | a locally built image |
BYTECHEF_PUBLIC_URL | http://localhost:8080 | the address users and external services reach |
BYTECHEF_FEATURE_FLAGS | empty | comma-separated feature flags |
BYTECHEF_PORT | 8080 | published port of the load balancer |
RABBITMQ_MANAGEMENT_PORT | 15672 | port of the RabbitMQ management UI on 127.0.0.1 (user bytechef, password bytechef) |
With BYTECHEF_EDITION=CE and more than one coordinator, every coordinator starts and logs the warning above. The file storage is a volume all nodes mount; in production, mount shared storage (NFS, EFS) or use the AWS provider.
The file sets BYTECHEF_SESSION_STORE=REDIS on every node. In EE with the REDIS session store, the Redis connection opens at startup, so the nodes fail to start when Redis is unreachable. PostgreSQL, the message broker and the load balancer are single containers in this file; for production, use managed or clustered services.
How is this guide?
Last updated on
Message Brokers
Choose the broker that carries task dispatches between the coordinator and the workers - memory, Redis, RabbitMQ, Kafka, JMS, or AWS SQS.
File Storage
Where ByteChef puts the opaque bytes a workflow produces - S3, the local filesystem, or the database - and the three separate storage settings that select it.