Persistence

Introduction

The Golem Worker Executor uses a set of storage layers for persisting agent state and everything related to the execution of agents. This page gives an overview of what is stored where, as well as some pointers for extending Golem OSS with new storage implementations.

Storage types

The worker executor requires four types of storage backends:

A blob storage for storing and retrieving arbitrary sized binary blobs
A key-value storage with some extra requirements for set/sorted-set-like operations
An indexed storage for an append-only, indexable store for agent’s operation log (oplog)
A scheduler storage for durable scheduled actions

Currently Golem provides the following implementations, configurable through the worker executor’s config file:

Storage type	Implementations
Blob storage	S3, file system, SQLite, in-memory
Key-value storage	Redis, PostgreSQL, SQLite, multi-SQLite, namespace-routed, in-memory
Indexed storage	Redis, PostgreSQL, SQLite, multi-SQLite, in-memory
Scheduler storage	PostgreSQL, SQLite, in-memory

Compilation cache

The Golem registry service stores original component Wasm files in its configured blob storage, which supports S3, local filesystem, SQLite, and in-memory backends. It exposes them through its download API for the worker executor. The Wasm files must be compiled to native code before execution, which is time-consuming. To reduce agent startup time, these compiled binaries are cached in the worker executor’s blob storage.

The component compilation service is a special, horizontally scalable component of the Golem system which shares this storage with worker executors and is capable of precompiling components before they are being used by any of the executors.

Oplog

Running agents continuously persist operations into the agent’s oplog. The oplog is append-only (persisted entries are never modified) and it is indexable (during the recovery of an agent it is read from the earliest entry to the latest one). The oplog is stored in multiple configurable layers. The default configuration is the following:

The primary oplog layer uses the indexed storage. This is the one where running agent’s entries are appended to
There is a secondary layer that is also persisted to the indexed storage, but on this level each entry holds a compressed set of oplog entries.
A tertiary layer holds even larger chunks of compressed oplog entries and stores them in the blob storage.

Oplogs are moved toward lower layers to prevent the primary indexed storage from filling up. With the default layers, inactive agents’ oplog entries eventually reach blob storage. A small creation-fence key remains in primary storage.

In addition to the above, there are oplog entries holding user-defined data (such as invocation parameters). As these user-defined payloads can have an arbitrary size, the worker executor has to protect its storage layer against putting too large entries into the primary oplog. For this reason large payloads are written to the blob storage and the corresponding oplog entry only stores a reference to them. Payloads below a configurable limit are stored inline in the oplog entry.

Configuration

The [oplog] section of the worker executor configuration allows customizing the above described behavior.

The default configuration is the following:


[oplog]
archive_interval = "1day"
blob_storage_layers = 1
entry_count_limit = 1024
indexed_storage_layers = 2
max_operations_before_commit = 128
max_payload_size = 65536

These options have the following meaning:

Option	Description
`archive_interval`	The delay between scheduled archival steps that move oplog sections down one layer
`blob_storage_layers`	The number of archive layers using the blob storage
`indexed_storage_layers`	The total number of layers (one primary + archives) using the indexed storage
`entry_count_limit`	The number of oplog entries triggering the archivation of a chunk to the lower oplog layer
`max_operations_before_commit`	The number of non-critical oplog entries that can be buffered before committed to the oplog
`max_payload_size`	The maximum size of user-defined payloads that can be stored inline in an oplog entry

Agent metadata

The oplog is the primary source of truth for agent state, but the key-value store also caches aggregated agent metadata for performance. This metadata records the oplog index from which it was calculated, and Golem can refresh it by processing newer entries. Once archival is complete, Golem clears the cached status and its checkpoint. If the agent is recovered later, Golem reconstructs the metadata from the archived oplog.

Filesystem snapshots

When filesystem snapshots are enabled, the worker executor keeps copies of the files of durable agents that take snapshots in the blob storage. With S3, it uses the bucket that filesystem_snapshots_bucket names. Each incarnation of an agent has its own location. See Where the snapshots are.

Promises and schedules

The worker executor stores promise information in its key-value storage, with one entry for each promise’s completion state. Scheduled actions, including promise completion used to wake agents after long sleeps, use the separate scheduler storage.

WASI Blob store and Key Value store

Golem implements (partially) the WASI Blob Store and WASI Key-Value Store proposals. These interfaces allow agents to store blobs and key-value data in the worker executor’s configured blob storage and key-value storage layers.

Custom blob storage quotas

An account’s blob storage quota applies only to data that agents write through the WASI Blob Store API. Internal blobs such as oplog payloads, compilation caches, initial agent files, components, and snapshots do not consume this quota.

Worker executors enforce the quota from their last known account balance and send net byte changes to the registry service. Concurrent executors can each spend the same stale remaining balance before their next usage update, so a distributed deployment can temporarily exceed the configured limit. S3 deployments correct accounting drift by comparing the recorded total with the physical objects. Reconciliation runs in the background when a resource-limit read or usage update observes that the reconciliation interval has elapsed; it is not a periodic timer. Consequently, an idle account is corrected only after its next such activity, and a delta that overlaps a sweep can require the next successful sweep to converge.

For reconciliation to be accurate, registry services and worker executors must use the same S3 endpoint, custom_data_bucket, and object_prefix. The custom-data bucket and prefix must not be shared with another blob-storage namespace because namespace keys can otherwise overlap. Both services refuse to start if filesystem_snapshots_bucket equals custom_data_bucket, even when filesystem snapshots are disabled. This check keeps filesystem snapshots out of the account’s blob storage quota.

Blob mutations and usage reporting are not atomic. An executor crash can lose usage changes that have not reached the registry. Automatic physical reconciliation is enabled only for S3. Filesystem and SQLite deployments enforce quotas using incrementally reported usage, but crash-induced accounting drift can persist indefinitely, including undercounting that permits further writes beyond the configured limit.

Adding new storage implementations

The open-source version of Golem can be extended with additional storage backends beyond the built-in implementations. See these traits: