Operating Durable Stream Sessions

Streaming agent invocations use the same durability model as other durable agent work. Stream registrations, items, terminals, attachment epochs, consumer journals, and deletion barriers are committed to agent oplogs. Losing a CLI connection, worker process, executor, or service connection does not by itself cancel the invocation.

For the public URL family and wire contract, start with Exposing Durable Streams over HTTP. For agent code that reads or writes a Durable Streams service outside the invocation-session protocol, see External Durable Streams.

Executor configuration

Worker executors expose the protocol-v1 timing controls under [durable_stream]:

[durable_stream]
external_batch_max_size = "8388608 B"
lease_ttl = "1m"
reconciliation_interval = "30s"
reconciliation_batch_size = 256
abandoned_prepare_after = "5m"
 
[limits]
live_stream_event_broadcast_capacity = 32

external_batch_max_size is a configurable, non-zero limit for one complete external Durable Streams read or append payload; it defaults to 8 MiB. The prepared-attachment lease TTL and abandoned-prepare threshold are fixed in protocol v1. Active attachments do not expire or require renewal. reconciliation_interval is the non-zero delay before retrying unfinished or failed recovery, not a periodic check of healthy attachments. The reconciliation batch size must be non-zero. The per-reader live buffer must be between 1 and 1024 events.

Other protocol-v1 limits are fixed: 16 MiB per logical value item, 1 MiB per packed-byte batch, 1024 streams per session, 256 newly discovered streams per value, recursive traversal depth 128, and 16 live readers per stream. A limit failure terminates or rejects the affected stream operation rather than silently truncating data.

Gateway configuration

The worker service owns public HTTP request sizing, polling, admission, and fork limits:

[durable_streams]
long_poll_timeout = "30s"
max_append_body_bytes = 1048576
 
[durable_streams.load]
max_concurrent_readers_per_stream = 16
max_concurrent_readers_per_node = 4096
max_catch_up_requests_per_second_per_stream = 100
max_append_requests_per_second_per_stream = 100
 
[durable_streams.forks]
max_forks_per_session = 128
max_forks_per_second = 4
max_copied_bytes = 67108864

Route overrides and gateway counters are local admission controls, not cluster-wide quotas. Two deployed routes that resolve to the same canonical stream have separate per-route reader and append accounting, but share the canonical stream’s catch-up budget; every route also shares the node-wide reader ceiling. A request rejected by a route or node budget returns 429 with Retry-After: 1. Executor routing or shard unavailability returns 503 with Retry-After.

Fork count, rate, and copy-byte enforcement is durable and executor-side even though the values are supplied by worker-service configuration. See Forking Durable Streams for the protocol behavior.

Metrics

Worker-service gateway metrics have no golem_ prefix:

• durable_stream_load_rejection_total{reason} counts per-stream reader, per-node reader, catch-up-rate, and append-rate rejections.
• durable_stream_append_total{outcome} counts bounded append outcomes: accepted, duplicate, fenced, and gap.

Worker-executor metrics describe durable production, consumption, and cross-agent attachments:

See Metrics for scrape endpoints and exporter setup. Durable Stream spans include identifiers and bounded outcomes, but not stream values or credentials.

Retained history and capacity

Manual stream deletion, session cancellation, and expiry are logical lifecycle transitions; they do not rewrite oplogs. Golem has no configurable stream-retention horizon. Items, terminals, consumer journals, fork cuts, producer state, and manual-delete tombstones remain in durable history until the owning agent data is deleted under the normal agent lifecycle.

Plan storage for long-lived, high-volume streams and their forks. Forks share the copied logical prefix but still add target oplog records and independent continuation history. A manual tombstone reserves its stream URL for the life of that session. Session expiry hides the old URL family as 404 and allows a durable agent to bind the same public session ID to a fresh internal invocation; the old immutable history remains retained.

Resume and takeover

Use a saved session checkpoint when an operator or client must reconnect explicitly:

golem agent invoke 'MyAgent()' stream_method --save-session session.json
golem agent invoke 'MyAgent()' stream_method --resume-session session.json

--resume-session is valid after the previous transport detached. If the previous transport is still attached, use --takeover-session session.json. Every accepted resume or takeover advances the attachment epoch and fences the old transport. Frames from a stale epoch are rejected; they are never applied to the new attachment.

The CLI does not retry automatically. Keep the checkpoint file private because it contains invocation identity and cursor state. It does not contain agent values or credentials.

Attachments and leases

Cross-agent streams create durable producer-to-consumer attachments. Their states are Prepared, Active, or Finalized. Prepared attachments have a deadline; active attachments remain valid without renewal. An expired prepare deadline is not proof that the consumer disappeared: reconciliation checks the durable consumer journal and consumer incarnation before finalizing an attachment.

Recovery runs when a producer loads, after committed session changes or stream terminals, and before deletion. A read authorized by the exact active consumer topology also repairs a producer left prepared by a lost activation RPC. Healthy reads do not mutate attachment state. Once recovery completes, it waits for a lifecycle event; only unfinished or failed work gets a timed retry.

Stale attachment bookkeeping can remain on a resident producer until its next load or deletion. Deletion reconciles dependencies before deciding whether it is safe. Finishing an invocation or forwarding a stream does not itself release history needed by another reader’s replay or resume.

Use the worker-executor metrics and traces to inspect attachment activity:

• golem_durable_stream_attachment_operation_total shows prepare, activate, and finalize commits or replay no-ops.
• golem_durable_stream_lease_remaining_seconds shows observed prepared-attachment lease headroom.
• golem_durable_stream_reconciliation_total shows reconciler outcomes.
• golem_durable_stream_cascade_total shows durable deletion-cascade outcomes.
• durable_stream.attachment.* and durable_stream.resume_attempt OTLP spans carry attachment, stream, attempt, and epoch identifiers, but no stream values or credentials.

See Logs for tracing setup.

The built-in OTLP exporter also emits golem.durable_stream.event_count from durable stream summaries. Its bounded event label distinguishes registration, aggregated items, stream end/cancellation, and session result/finish/cancellation/expiry; terminal summaries may use the bounded outcome label (success or error). Item counts are aggregated into the metric value. The exporter does not create one span or label value per streamed item and never exports item payloads through this summary.

session-cancellation counts cancellation-related journal records (request, consumer intent, and application), not unique logical cancellations. One cancellation can produce several such records.

A session result and the eventual stream or session terminal are separate durable events, so an early invocation result is not stream completion. Collector delivery is best effort even though source lifecycle state advances when a batch is accepted.

Writing input streams over HTTP

Input slots accept Durable Streams POST requests by default. Output slots and $result are read-only (405 with Allow). For a stream<string> input named input, append two messages with:

POST {base}/invocations/{session}/streams/input
Content-Type: application/json
Producer-Id: uploader
Producer-Epoch: 0
Producer-Seq: 0
 
["first", "second"]

Each JSON array element is one message; a single non-array JSON value is one message. Arrays are flattened only one level, and an empty array is invalid. Every item is validated against the invocation’s pinned element schema before any item in the batch is appended. Invalid input returns 400 with an application/problem+json body containing path and detail. A stream<u8> input instead accepts one binary batch with Content-Type: application/octet-stream.

A nonempty append requires the slot’s content type (400 if missing, 409 if mismatched). Send Stream-Closed: true to append and close atomically, or send an empty body with that header to close without appending. An empty close-only request may omit Content-Type; any supplied Content-Type is ignored. Only case-insensitive true sets Stream-Closed; other textual values are treated as absent. Successful ordinary writes return 204; a newly accepted write with producer headers returns 200, and its duplicate retry returns 204. All successful writes include Stream-Next-Offset. Responses include Stream-Closed: true only when closed and otherwise omit the header. Reads report closure only at EOF; session GET/HEAD report it only when all slots are closed or deleted. The session manifest’s closed fields remain JSON booleans. Further writes after closure return 409 with the final offset and Stream-Closed: true; repeating an ordinary empty close returns 204.

The three Producer-* headers are optional but must be supplied together. Epoch and sequence are decimal integers from 0 through 9007199254740991. Sequence starts at zero and increases per request, not per item. A larger epoch fences older epochs and starts again at sequence zero. Successful producer responses echo Producer-Epoch and report the highest accepted Producer-Seq. While the stream is open, repeating a sequence returns its original offset without writing again. After closure, only the original closing producer tuple gets this successful retry; other producer tuples return 409. A stale epoch returns 403 with the current epoch; a sequence gap returns 409 with Producer-Expected-Seq and Producer-Received-Seq.

Producer state and stream data are committed in the same oplog batch before acknowledgment. After losing a response, retry with the same producer tuple to avoid duplicate items. Plain appends without producer headers do not have this guarantee, and the gateway does not transparently retry append calls. HTTP writes and an attached WebSocket can interleave on the same input: the WebSocket keeps its own continuous sequence counter, while the shared durable offsets order all messages. Reconnecting the WebSocket restores its counter from durable history, independently of HTTP producer counters.

Closing the input over HTTP also ends input from an attached WebSocket. WebSocket frames already in flight after that close are discarded without failing the invocation or adding items to durable history. A WebSocket sender must still stop after sending its own End frame; sending more items after its own End is a protocol error.

POST can create a missing session when every non-stream argument is supplied by the route’s path or query. Otherwise, first PUT {base}/invocations/{session} with the non-stream argument object; POST returns 404 with that instruction until the session exists. A deleted slot returns 410 rather than recreating it.

Worker-service limits are independent of the executor’s stream limits:

[durable_streams]
max_append_body_bytes = 1048576
 
[durable_streams.load]
max_append_requests_per_second_per_stream = 100

Oversized bodies and JSON batches with more than 4096 messages return 413. The append cap is enforced per stream on each worker-service node in one-second windows and returns 429 with Retry-After: 1; it does not consume the catch-up read budget. Append admission happens before storage access, including for closed or deleted slots, so an exhausted budget returns 429 before checking those states. Monitor durable_stream_append_total with outcomes accepted, duplicate, fenced, and gap, and durable_stream_load_rejection_total with reason append-rate-limit.

Per-route HTTP policy

Agent endpoint metadata can assign public slot names, choose an opaque media type for direct byte streams, disable external writes or deletion operations, and set stricter live-reader and append limits. See Customizing Durable Stream Routes for all five SDK forms.

These controls are gateway policy and remain compatible with the official Durable Streams protocol:

• Public names change only the route URL and OpenAPI surface. The worker and oplog continue to use the canonical method slot identity. A canonical name is not accepted at the public URL after it is renamed.
• A custom media type changes HTTP negotiation for a direct stream<u8> but leaves offsets byte-based and content opaque. It must be a concrete non-text, non-JSON type with no parameters or wildcards. JSON-shaped streams remain application/json; native text/plain for stream<string> is rejected at deployment. SSE remains text/event-stream and carries byte data as base64.
• Disabling POST or either DELETE operation narrows the protocol operations offered by that route. A disabled operation returns 405 and an exact Allow header without mutating stream or invocation state.
• max_concurrent_readers_per_stream must be from 1 through 16 and applies to long-poll and SSE readers. max_append_requests_per_second_per_stream must be positive and requires external writes. Both override the worker-service defaults only for that deployed route; ordinary catch-up reads use their separate worker-service rate budget.
• Fork creation is always present. There is no route-level fork opt-out.

Route live-reader and append accounting is separate for each route and each worker-service node, including when two routes resolve to the same underlying canonical stream. Catch-up accounting is instead shared by routes that resolve to the same canonical stream. The node-wide live-reader cap remains shared across routes. These controls are therefore admission protection, not cluster-wide quotas. Their exhaustion returns 429 with Retry-After: 1; routing, connectivity, or shard unavailability remains 503 Service Unavailable.

Cancelling a session

Delete an invocation session to cooperatively cancel all of its open streams:

DELETE {base}/invocations/{session}

The first request returns 204 No Content. Repeating the request also returns 204; an unknown session returns 404 Not Found. Cancellation closes open streams and fences output slots that have not materialized yet, so later output cannot reopen them. Streams that were already terminal keep their existing outcome.

Session cancellation retains stream history. Readers can consume recorded and buffered items before reaching the cancellation terminal. At EOF, a cancelled stream reports both Stream-Closed: true and Stream-Cancelled: true. Recorded guest observations also replay unchanged.

The upstream Durable Streams protocol’s stream deletion semantics  delete a stream URL; they do not terminate the producer’s computation. The invocation-session URL and its cooperative cancellation behavior are Golem extensions.

Deleting a stream URL

Deleting an individual stream URL atomically cancels that stream slot and writes a durable tombstone:

DELETE {base}/invocations/{session}/streams/{slot}

For an input slot, cancellation closes the external producer side; the Rust agent’s input reader observes EOF after any already-buffered items. For an output slot or $result, cancellation records that the consumer has gone away. A write already accepted by the guest transport can still report success without being added to durable history; subsequent writes fail when the producer observes the closed consumer. Transport acceptance is not an acknowledgment of durable publication.

Every stream deletion is a soft delete. After the successful deletion, authenticated requests admitted by the load limits use these statuses:

The tombstone and oplog history survive recovery; deleting a stream never removes its durable history. The URL remains reserved for the lifetime of the existing session. See Expiry for when the same public session ID can be reused for a new invocation.

Deletion failures

Normal producer deletion is fail-closed while a prepared or active attachment may still need producer history. A blocked deletion reports the dependent attachment, stream, epoch, consumer identity, and consumer incarnation fingerprint. Do not force-delete the oplog or use fork/revert to bypass this check.

Retry deletion after the consumer has durably journaled the stream terminal or after reconciliation proves that the consumer was deleted or recreated. Cascade deletion first commits a producer barrier, then records one durable outcome per dependent. If unread history remains, it writes a source-unavailable overlay to the consumer journal before completing the cascade. Retrying the operation resumes from those committed facts.

Recovery procedure

1. Confirm that the producer and consumer use durable agent mode and that their current incarnation fingerprints match the attachment diagnostics.
2. Check golem_durable_stream_producer_operation_total for replay no-ops. Replays after a lost response are expected and must not create additional logical items.
3. Check catch-up distance, journal lag, live-reader backpressure, and attachment lease metrics. A growing catch-up distance indicates a recovering reader; sustained backpressure indicates a slow attached reader.
4. Resume a detached client with its saved checkpoint. Use takeover only when intentionally fencing a still-attached client.
5. For blocked deletion, resolve the reported dependency and retry; deletion itself runs reconciliation. Investigate probe_error, journal_error, or write_error reconciliation outcomes before taking further action. Waiting for a periodic attachment poll will not resolve a dependency.

Never edit stream-related oplog entries. Registration identity and every offset are derived from committed oplog positions; rewriting that history can alias stream identity or lose journal evidence and is rejected by supported fork/revert operations.