Durable Timer architecture
Durable Timer is a dedicated scheduling service in the Triggers platform. It owns durable waiting and wake delivery. Notify Triggers remains the authority for authoring, released flow versions, and continuation.
This boundary keeps waiting state out of Notify processes and keeps scheduling logic out of business services.
Architecture at a glance
The Timer service is built from the Notify Triggers repository but runs as a separate workload. It uses the same internal NATS cluster as Notify and the SDK-enabled services.
Ownership boundaries
| Component | Owns | Does not own |
|---|---|---|
| Studio | Timer node editing, ports, validation feedback, and layout | Runtime state or scheduling |
| Notify Triggers | Released flow versions, runtime compilation, initial routing, resume validation, resume outbox, and downstream continuation | The durable wait |
| Triggers SDK | Private Timer lineage, command publication, Condition evaluation, action isolation, and Timer feedback | Scheduling policy or Timer storage |
| Business action | Its normal inputs, result, and domain side effects | Timer identity, snapshots, schedules, or control subjects |
| Durable Timer | Occurrence state, immutable snapshot, retry generation, fences, native schedule, reconciliation, terminal archive, and resume publication | Flow authoring or domain behavior |
| NATS JetStream | Durable commands, state, snapshots, schedules, wakes, terminals, and resumes | Flow interpretation |
| MongoDB | Released flows and Notify's crash-safe resume outbox | Timer wake scheduling |
Author view and compiled runtime
An author places one Durable Timer node in Studio. The runtime requires an asynchronous handoff because the original execution stops while the event is waiting. Notify adds this internal boundary during release.
The internal resume service is generated deterministically and is not displayed as another authorable component. Studio metadata is excluded from the runtime graph. Missing edge IDs are assigned stable runtime identities.
The occurrence lineage includes:
- released flow ID and version;
- root execution ID;
- Durable Timer node ID;
- stable start, elapsed, and optional retry edge IDs;
- occurrence generation and fence;
- immutable snapshot hash and current payload hash.
Resume resolution uses the pinned released runtime. It never uses the mutable Studio working copy.
Start and wake path
The saved event is exactly the payload that entered this Timer on Start. The Timer does not know how earlier nodes produced it and does not merge later action results into it.
Schema lineage across the Timer boundary
The Timer passes through the schema of its Start input. When it resumes, its Time reached edge has that same schema. A service-in node also passes through the event it receives. An Action is different: its outgoing data schema is the action manifest's result_event.fields, because downstream nodes receive the action result rather than the action request.
Consequently, a Condition immediately after an Action evaluates the Action result schema. A Mapping immediately after that Condition also starts from the Action result schema. A Mapping produces its own projected schema for the next node. Retry is control-only: it is not a data edge and cannot replace the Timer's immutable snapshot with an Action result.
At release time, Notify resolves each Condition, Mapping, and Action's direct upstream data schema and stores it as immutable config.input_fields in the released runtime. Go and Python SDKs use those fields for local evaluation; only older releases without the field list use the compatibility fallback.
Owner is whichever runtime is currently walking the graph. Notify publishes commands reached in coordinator-owned source/control traversal. An SDK publishes commands reached after a service handoff. Notify is not a relay for SDK-originated Timer commands.
Retry and terminal feedback
max_retries: 0 creates a one-shot Timer. Each accepted Retry increments the generation. A retry beyond the configured limit produces a failed terminal record with failure_code: "max_retries_exceeded".
The runtime executing the elapsed branch owns feedback. After a service handoff, the SDK removes private continuation context before invoking the action, evaluates the Condition from the action's ordinary result, and publishes Retry only when that one explicit Condition output matches. When it does not match, the Timer lineage completes. Notify performs the equivalent control operation when the branch terminates inside coordinator-owned traversal. The business action never owns Timer feedback.
NATS topology
The platform provisions one KV bucket backed by JetStream and five named streams:
| Logical resource | NATS resource | Subjects | Owner |
|---|---|---|---|
| Active state | Bucket DURABLE_TIMER_STATE, stream KV_DURABLE_TIMER_STATE | NATS KV subjects | Durable Timer |
| Commands | DURABLE_TIMER_COMMANDS | durable_timer.commands.start.v1, .retry.v1, .complete.v1, .fail.v1 | SDK and Notify publishers, Timer consumer |
| Immutable snapshots | DURABLE_TIMER_SNAPSHOTS | durable_timer.snapshot.v1.> | Durable Timer |
| Terminal archive | DURABLE_TIMER_TERMINALS | durable_timer.terminal.v1.> | Durable Timer |
| Schedules and wakes | DURABLE_TIMER_SCHEDULES | durable_timer.schedule.v1.>, durable_timer.wake.v1.> | Durable Timer and NATS scheduler |
| Resumes | DURABLE_TIMER_RESUMES | triggers.v2.durable_timer.resume.v1 | Timer publisher, Notify consumer |
Fixed durable consumers are:
DURABLE_TIMER_COMMAND_SERVICE;DURABLE_TIMER_WAKE_SERVICE;NOTIFY_TRIGGERS_DURABLE_TIMER_RESUME.
The command, state, snapshot, schedule, and resume resources are provisioned before the Timer and Notify resume consumers become ready. Active waiting data has no time-based expiry. Snapshot and terminal cleanup is not implemented in the current version, so capacity must be monitored.
Consistency and recovery
The system crosses two durable transaction domains:
- Durable Timer commits its state and event records to NATS JetStream.
- Notify records resume delivery progress in a MongoDB outbox.
There is no atomic transaction across NATS, MongoDB, a business service, and its external side effects. The internal contract is therefore at least once.
The implementation reduces duplicate and stale work through:
- deterministic Timer references, occurrence IDs, schedule IDs, and message IDs;
- compare-and-set updates to active state;
- a monotonically changing fence and retry generation;
- validation of the pinned runtime and payload hashes;
- durable command receipts and Notify resume receipts;
- replay of unfinished outbox handoffs.
On startup and at the configured reconciliation interval, Durable Timer scans waiting states and recreates each deterministic schedule. Recreating an existing schedule is harmless. An overdue schedule becomes eligible immediately.
Terminal transitions first persist the terminal receipt in active state, then archive the exact terminal state, and only then remove the active entry. A failed archive leaves the active record available for retry.
Active state deletion is revision-fenced so an archived occurrence cannot reveal an older active value.
During shutdown, the service first withdraws readiness and then drains command and wake work. Work that cannot settle within the shutdown grace is negatively acknowledged for redelivery.
Failure behavior
| Failure | Expected behavior |
|---|---|
Duplicate Start or wake | Deterministic identity and state checks suppress duplicate advancement. |
| Stale retry generation or fence | Command is rejected without changing the active occurrence. |
| Wake arrives early | Occurrence remains waiting and is not resumed early. |
| Timer restarts before due time | Reconciliation restores the deterministic native schedule. |
| Timer restarts after due time | Reconciliation makes the overdue occurrence eligible immediately. |
| Notify stops after claiming a resume | The MongoDB outbox lease makes unfinished handoffs recoverable. |
| Terminal archive fails | Active state is retained and the terminal transition is retried. |
| Configured storage capacity is unavailable | Admission fails closed rather than evicting live state. |
| Business action repeats after redelivery | The action must use idempotency at its domain boundary. |
Trust and data boundaries
Private Timer context travels only on internal NATS deliveries. The SDK and Notify validate its flow lineage, execution identity, edge IDs, generation, fence, and hashes. Unknown, malformed, oversized, or inconsistent context is rejected.
The context is unsigned. The current version trusts access to the shared internal NATS bus and adds no Timer-specific TLS, credentials, ACLs, certificates, or signing keys. Private context is stripped before action handlers and public Kafka, HTTP, and output-topic boundaries.
Deployment and enablement
Durable Timer is enabled in both Notify Triggers and the Timer workload for the environment. All released flows use the same canonical NATS resources. Business services advertise the durable_timer.v1 SDK capability when they participate in a Timer path.
Release is permitted when every service on the Timer path advertises the capability and its service-in target matches the active manifest. Runtime commands are checked against the pinned released flow version and Timer lineage.
The required broker version is NATS 2.14 because wake delivery uses native scheduling. A production deployment uses persistent multi-replica NATS storage.
Supported contract
| Supported | Outside the contract |
|---|---|
| Independent released flows with Timer nodes | Cross-flow coordination over the same business entity |
| Multiple Timer nodes inside each flow | Arbitrary graph cycles |
| One-shot, retry, sequential Timer, and restart recovery | Global exactly-once business effects |
| Native NATS 2.14 scheduling | Polling database scheduler fallback |
| Fail-closed admission and terminal archival | Reference-safe snapshot and terminal garbage collection |
| Shared internal NATS trust boundary | Per-publisher authentication or Timer-specific transport security |
Cross-flow business coordination and per-flow fairness under noisy-neighbor load are backlog work. The current service isolates Timer occurrence state by flow, version, root execution, and Timer node, but it does not decide how separate flows should coordinate a shared business entity or reserve capacity from one another.
See Durable Timer for the author and product view, and the hosted configuration reference for deployment variables.