Crash Recovery

A process can die mid-run, leaving a job marked STARTED that nothing is working on any more. ByteChef ships three monitors that detect and recover that state automatically. Detection lives outside the Atlas engine — only the heartbeat primitives are engine-owned.

Orphaned-job recovery

Workers publish a heartbeat for every in-flight task every 30 seconds. The coordinator re-saves the STARTED task row on each heartbeat, bumping its last-modified timestamp.

OrphanedJobRecoveryMonitor runs every minute and treats a job as orphaned only when the job row and all of its non-terminal task executions have gone stale past the staleness threshold — a child task's heartbeat keeps its control-flow parent tasks alive transitively, so a long-running loop or fork-join is not mistaken for a crash.

Recovery marks the tasks and the job FAILED. The normal job-status fan-out fires, and the run becomes resumable through the existing resume path.

Auto-resume is at-least-once: the interrupted task re-runs from the last completed node, so the components in that step must tolerate being executed twice.

Per-run timeouts

JobTimeoutMonitor runs every minute and fails STARTED jobs whose runtime exceeds the plan's asyncRunTimeout (see Plan limits) or the operator fallback below. With neither set it is a no-op.

There is deliberately no auto-resume for a timed-out run — it would immediately exceed again.

Retention purging

JobRetentionMonitor sweeps every 6 hours, per tenant, and permanently deletes finished jobs older than the plan's logRetentionDays or the operator fallback below. The delete cascades to the job's task executions, its job and task output blobs, its context rows, and its CURRENT_EXECUTION-scoped data-storage rows. Subflow child jobs are skipped — the parent's delete covers them.

Blob and context cleanup is best-effort: a storage failure never blocks the row delete, so a run can be purged from the database while a stored output blob briefly outlives it.

Broker redelivery

Recovering the job is only half the story; a task that was dispatched but never acknowledged has to be delivered again. On AMQP and Kafka that is the broker's own redelivery, and it works today. On the Redis broker it will be a pending-entry sweep: each consumer reads its stream's pending entries (XPENDING), claims those idle past the threshold (XCLAIM), and re-runs them through the normal invoke-then-ack path, giving the same at-least-once semantics as the other brokers. See Message brokers.

See also

• Environment variables — the full recovery, timeout, and retention variable reference.
• Plan limits — where asyncRunTimeout and logRetentionDays come from.