Durable execution
Durable execution means that a running program survives failures: if the process or server running it dies, the program continues where it stopped, with the same state, and does not repeat work it already finished. In Golem, every agent is durable by default. You write ordinary code; Golem provides the durability.
Consider an agent loop that makes twelve model calls and runs a tool after each. If the server crashes during step 7, an ordinary process loses everything and starts over, paying for six model calls again and re-running six tools. A Golem agent resumes at step 7.
The oplog
Golem keeps an append-only log for each agent, called the oplog. It records every interaction between the agent’s code and the outside world, together with its result: HTTP requests (including model calls), calls to other agents, tool calls, clock reads, random numbers, and the order in which concurrent operations completed. It also records each method invocation the agent receives.
Golem records these at the boundary between the agent’s WebAssembly code and the Golem host, so your code needs no annotations, step functions, or activity wrappers. This agent loop calls a model, runs the tool the model asks for, and repeats until the model answers:
TypeScript
// Excerpt: an agent loop in a method of ReportAgent
async ask({ question }) {
this.messages.push({ role: "user", content: question });
for (let step = 0; step < 12; step++) {
// Your own function: an HTTP request to a model API
const reply = await callModel(this.messages, [echoToolSchema]);
this.messages.push(reply.message);
if (!reply.toolCall) return reply.message.content;
// A Golem tool call, through the tool's typed client
const output = await echoClient.echo({ value: reply.toolCall.arguments.value });
this.messages.push({ role: "tool", content: output });
}
return "Step limit reached";
},Every model response and every tool result goes into the oplog. If the server fails during the third model call, recovery re-runs the loop: the first two model calls and the tool calls between them return their recorded results, this.messages is rebuilt along the way, the third request is sent again, and the loop continues.
Open full-size image in a new tab.
Open full-size image in a new tab.Recovery
When an agent’s process is lost, Golem starts a fresh instance of the agent’s code, on the same server or another one, and re-runs it against the oplog:
- Each interaction already recorded in the oplog returns its recorded result instead of being performed again. The model is not called a second time, and the reviewer agent is not asked again.
- Ordinary computation and local file operations run again, which rebuilds the agent’s variables, call stack, and files exactly as they were.
- When the replay reaches the end of the log, the agent continues live from where it stopped.
The same mechanism handles a crash, a server restart, an agent moving to another server, and an idle agent being unloaded from memory and loaded again later.
The following example shows recovery in the middle of a method. OrderAgent("order-42") charges a payment over HTTP and then asks InventoryAgent to reserve stock. The server crashes while the reservation call is in progress.
Open full-size image in a new tab.
Open full-size image in a new tab.During recovery, the payment response comes from the oplog, so the customer is not charged again. The reservation call is sent again with the same idempotency key, and InventoryAgent, which had already processed it, returns its recorded result instead of reserving twice.
Execution guarantees
| Operation | Guarantee |
|---|---|
| Method invocation | Executes exactly once per idempotency key. Repeating a call with the same key returns the recorded result. |
| Call to another agent | Agent-to-agent calls execute exactly once. Retries and recovery reuse the same key, and the target runs the call once. |
| Golem tool call | Every tool call executes exactly once. A call interrupted by a failure is completed as the same call, not started again. |
| Model call | Completed model calls are recovered from the oplog rather than requested again. A model call still in flight during a failure is sent again, like any HTTP request. |
| Outgoing HTTP request, from an agent or a tool | A request that completed is answered from the oplog. A request in flight during a failure is sent again with the same idempotency-key header, which Golem adds automatically. |
| Your own code | Re-runs during recovery to rebuild memory and files, using recorded results for clocks, random numbers, and all external interactions. |
These guarantees have two scopes. A call between agents, or from an agent to a Golem tool, executes exactly once, including everything it does inside Golem: its computation, its files, and its own calls to agents and tools. An HTTP request that an agent or a tool sends to a service outside Golem is a separate operation with its own row in the table above.
Golem’s control ends at the network boundary. When a request was in flight during a failure, Golem cannot know whether the remote service received it, so it sends the request again with the same idempotency key. Services that honor the idempotency-key header process it once.
Retries
Golem retries transient failures automatically, such as network errors and interrupted executions. Retry policies can be configured per environment in golem.yaml, changed at runtime with the CLI, or applied to a block of code. Permanent errors and permission failures are not retried. Recovery does not fix bugs: code that fails the same way every time fails again.
Snapshots, updates, and transactions
- Snapshots are optional. Without them, recovery replays the agent’s history from the beginning: recorded results are reused rather than performed again, but the agent’s own code runs again. For agents with long histories, an agent type can save snapshots of its state, and recovery then loads the latest snapshot and replays only the entries after it. The oplog remains the source of truth.
- Updates move existing agents to new code. Golem replays the agent’s history against the new code and keeps the agent on the old version if they disagree. This is the one place where code must agree with recorded history; within one version, Golem records every nondeterministic input, so you follow no determinism rules. See Agent lifecycle.
- Transactions group several operations. An atomic block re-runs as a whole if interrupted. For business-level rollback, sagas run compensating actions in reverse order.
Next
Tools: how agents run shell commands, compilers, and other tools inside Golem.
Related guides: