Deployment

Golem Services are distributed as Docker images. They are available in Golem Services Docker Hub . Each Docker image is built for the following architectures:

linux/amd64
linux/arm64

The services are using the following storage backends:

Relational Database (RDB) - PostgreSQL or SQLite (for running locally)
Redis
Blob storage - Shared File System or S3
etcd - optional, only for the shard manager’s distributed mode

Golem Services

The following sections provide a description of each service.

Registry Service

golem-registry-service manages accounts, applications, environments, components, plugins, tools, policies, and related registry data in the relational database and blob storage.

Worker Service

golem-worker-service provides APIs and HTTP gateway functionality for agents and acts as a routing service for worker executors. It retrieves gateway definitions from the registry service, uses blob storage for initial agent files, and stores gateway sessions in Redis or SQLite.

Worker Executor

golem-worker-executor is responsible for running the agents that belong to assigned shards. The service keeps each agent’s oplog in indexed storage - PostgreSQL, SQLite or Redis - and uses key-value and blob storage alongside it.

Shard Manager

golem-shard-manager maintains the worker executors’ shard assignments. It runs in one of two modes, selected by the persistence configuration:

Local mode (persistence.type = "Postgres" or "Sqlite") stores the shard state and the quota state in one SQL database. There is no leader election, so exactly one instance may run.
Distributed mode (persistence.type = "Etcd") stores the shard state in etcd. Any number of replicas may run: one wins an etcd lease campaign and drives all topology decisions while the rest stand by. The quota state lives in etcd too, under /golem/quota/, and a newly elected leader restores it, so executors keep the quota leases they hold across a failover. Switching an existing deployment from local to distributed mode does not carry the quota state over: executors acquire their leases again, and capacity budgets start from their full limit.

Every worker executor holds a lease on its shard assignment and renews it with the shard manager. A lease lasts shard_lease_duration (1m by default) and an executor renews at a third of the time it has left, so roughly every 20s. The shard manager refuses to start with a lease shorter than 30s and warns about one shorter than 1m: the deadline it allows each renewal call never drops below 10s, so a shorter lease cannot fit the three renewal attempts a lease is meant to get. Each renewal carries the epoch of every shard the executor holds; a shard’s epoch is its ownership generation, advanced only when the shard moves to another executor and never by a renewal. A renewal whose claim does not match the shard manager’s view is renewed all the same, and the response carries the shard manager’s set, which the executor adopts exactly as it would an assignment push: pushes deliver a change at once, and renewals guarantee it arrives within a third of the lease even if the push was lost. The shard manager records a shard’s new owner before it tells anybody about it, so a rebalance whose write is refused changes nothing at all, and every delivery carries the revision of the state it was read from. An executor ignores a delivery older than one it has already applied, so a push and a renewal that cross on the network cannot leave the older set in place. Revisions are only comparable within one shard manager process, so every delivery also names the process that sent it: an executor follows the process that answered its latest registration or renewal and starts its revisions over when that changes - which is how a shard manager that failed over, or came back on a wiped or restored store, gets its sets applied although its revisions start lower - and it ignores a push from any other process, renewing at once to hear from the one in charge. An executor that stops renewing loses its lease, and the shard manager reclaims and redistributes those shards on its next pass, which runs at least every third of shard_lease_duration even in a cluster where nothing else is happening; a crashed or unreachable executor’s shards are therefore re-homed within about one lease duration plus one pass. The lease reaches the executor as the time remaining on it rather than as an absolute time, so the two machines’ clocks are never compared and skew between them can neither lengthen nor shorten it. The executor anchors that time to the moment it sent the request the lease answers, so time the reply spends in flight comes off its copy of the lease and is never added to it, and its copy lapses no later than the shard manager’s. An assignment push carries the shard set and its revision only: it never extends a lease. The lease is the one place the two machines’ clocks must agree on a rate rather than on a value: an executor whose clock runs slow relative to the shard manager’s keeps admitting for that fraction of a lease past the shard manager reclaiming it, which is milliseconds under ordinary ntp or chrony discipline and seconds while such a daemon is slewing off a large offset, so an executor should not be started until its clock is synchronised. An executor whose own copy of the lease has expired stops admitting new work for its shards and reports that sharding is not ready, which a worker service retries after refreshing its routing table - invocations already running are not interrupted. A graceful stop - including the SIGTERM an orchestrator sends to stop a pod - deregisters the executor instead of waiting for its lease to expire, so its shards move on the next pass. A shard manager restart or failover re-grants the lease of every executor that still answers the startup health check, so a restart never evicts a healthy cluster: only executors that fail that check are removed. An executor whose own copy of the lease lapsed during the outage resumes admitting work on its next successful renewal rather than on the push that follows the re-grant, and neither a re-grant nor a renewal ever moves an executor’s deadline earlier, so a shard_lease_duration reduced across a restart takes effect only once the new length outruns the deadline each executor was last told.

The lease bounds how long a lost shard keeps being served, but it cannot stop an executor that has already lost one from finishing a write it had started. That is what oplog fencing is for. Each agent’s oplog records the shard epoch allowed to write it, and every batch of entries is checked against that record atomically with the insert - in the same database transaction, or on Redis in the same script - so a refused write leaves nothing behind. An executor whose epoch is behind is turned away at the storage rather than discovering the problem later: that one agent stops there, and the worker service resumes it on the shard’s new owner. The record only ever moves forward - an executor re-granted the shard at a higher epoch takes over, and one holding a stale epoch cannot claim it back - and an absent record fences an open oplog too, because the record is written before an oplog’s first entry and removed before its last. That protection covers an oplog whose record exists. Deleting an agent removes its record and forgets the epoch with it: an executor that still has the oplog open is refused, but an oplog without a record - a new agent, a deleted agent created again, or one last written before this release - is claimed by whichever epoch opens it first, so for those only the lease stops an executor that has lost the shard. The archive layers that older oplog entries move into are fenced the same way, and so are ephemeral agents’ oplogs, which are written straight into them. Blob storage has no conditional write, so an archive layer in blob storage keeps a list of its chunks in the indexed storage, fenced like the rest, and reads only the chunks listed there; archive chunks written to blob storage before this release are not listed, and are no longer read. Fencing covers the oplog and nothing else. An executor that has lost a shard can still write that agent’s key-value records, its status and its blob payloads, none of which carry an epoch: a stale status blob can overwrite a newer one until the agent’s next status change on its new owner, while the oplog - the only record replay reads - stays correct. Nor does the fence undo what has already left the machine: an outgoing call an agent made before its first refused write has happened, and the shard’s new owner, having no record of it, makes it again, which is the same at-least-once exposure a crash between the call and its oplog entry already has. A call the executor has not started yet is refused once the fence has latched, so the window is the call in flight rather than every call after the shard moved.

Fencing is enforced by every indexed storage backend, for deleting an agent as much as for writing to it: a deletion that finds another executor has taken the agent over removes nothing, and fails with the answer that sends it to the new owner. A deletion under way when its shard moves may be finished by the shard’s new owner instead. The shipped configuration defaults indexed_storage to a SQLite file of its own, which serves a single executor; executors that share shards need storage they can all reach, which means PostgreSQL or Redis, because a SQLite file is written by one process. Redis is only as durable as the last write the server kept: a failover that loses the most recent writes can lose a raised epoch with them, and the previous owner is then accepted again until the record is next raised, so prefer PostgreSQL or SQLite where that window matters. Redis Cluster is not supported. KVStoreSqlite and KVStoreMultiSqlite derive their location from the key-value storage, so they require key_value_storage.type to be Sqlite or MultiSqlite respectively; any other combination fails at startup.

A shard epoch only means something against the shard manager state that minted it. A state that is wiped, replaced or restored from a backup mints epochs below the ones the oplogs already record, and the storage refuses every write at those, so the affected agents stay unavailable until their records are cleared. To clear them, stop every executor, run DELETE FROM indexed_key_epoch on the indexed storage - for MultiSqlite and KVStoreMultiSqlite, on every *-oplog-*.db file under the root directory; on Redis, delete the keys matching *oplog-epoch:* instead - and start the executors again; each oplog records its owner’s epoch when it is next opened. Run the delete only while no executor is running, because an oplog without a record accepts whichever executor opens it first.

Distributed mode talks to etcd over plaintext HTTP and without authentication. Every entry in persistence.config.endpoints must start with http://; a replica configured with anything else refuses to start. etcd therefore has to be reachable unauthenticated, either on a trusted network or through a TLS-terminating proxy running alongside the shard manager, with the endpoints pointing at that proxy.

Give persistence.config.endpoints a single load-balanced address - the Kubernetes Service in front of the etcd cluster - rather than a list of members. With a member list the client keeps a member that is down in its rotation: reads retry through it and recover, but writes do not, and a leadership campaign can spend tens of seconds landing on the dead member before it gets anywhere.

Every write to the shard state or the quota state is a new etcd revision, and etcd keeps every revision of a key until it is told to compact, so a leader that renewed leases forever would fill etcd’s backend. The leader therefore compacts the history behind the state after each of its passes, keeping the newest persistence.config.compaction_retention_revisions revisions (1000 by default). Compaction is cluster-wide: it drops that much history for every key on the etcd cluster, not only the shard manager’s. On a cluster shared with other applications set it to 0, which disables it, and leave compaction to the cluster’s operator or to etcd’s own --auto-compaction-mode.

A quota lease change is stored as one etcd transaction: the resource, the lease, and one delete for each expired lease the change reclaims. The shard manager reclaims at most 120 expired leases per change, so a transaction holds up to 122 operations, and the etcd cluster must accept at least that many. etcd’s default --max-txn-ops of 128 does. On a cluster configured below 122, a change that reclaims many expired leases at once is refused, and so is every later change to that resource.

persistence.config.request_timeout must be at most half of persistence.config.leader_lease_ttl, and startup refuses the configuration otherwise. A lease is granted and its keepalive opened back to back, each bounded by the request timeout, so a larger value could let a lease expire before it was ever renewed.

Failover in distributed mode costs a graceful stop almost nothing: on SIGTERM or Ctrl-C a leader releases its lease and a standby takes over within milliseconds, and a standby stopped the same way leaves the election queue at once. A leader that is killed or partitioned releases nothing, so it holds its lease until etcd expires it and a standby is elected between two thirds of persistence.config.leader_lease_ttl and the full value (10s by default) after that leader’s last renewal. On top of that comes the new leader’s own startup, whose initial executor health check is capped at 15s. A broken etcd connection is not a failover: an etcd member restart or a raft leader change ends the keepalive stream and the leader reconnects it, so only etcd declaring the lease gone, or a full lease period with no acknowledged renewal, ends a leadership.

During that window the shard manager’s gRPC port is closed on every replica. A worker service does not ride it out on a warm routing table cache: the first executor-level error it hits discards the cached table, and the worker service is then blocked until a leader binds its gRPC port again. A worker executor that starts inside the window retries its registration for about 10s in total with the default shard_manager.retries and exits when that budget runs out, so raise shard_manager.retries.max_attempts on the executors if their startup has to survive a shard manager failover.

A registration is persisted before it is acknowledged, so an executor that receives a successful registration response holds a lease the next leader will find. A registration that could not be persisted - because the leader lost its lease part-way through the write, for instance - is answered with an error instead, and the executor retries it within its shard_manager.retries budget.

In distributed mode a standby does not open its gRPC port (grpc.port) until it is elected, and that is what routes traffic to the leader. Configure probes accordingly: liveness against the HTTP port (http_port), readiness against the gRPC port. Wiring readiness to HTTP would place every standby in the Service’s endpoints, and clients would get connection-refused on every request that did not land on the leader. Over HTTP a standby and a stuck replica look the same, and the metrics close that gap only partly. shard_manager_is_leader separates the leader from every non-leader - the sum across replicas should always be 1 - but it reads 0 for a healthy standby and for a wedged one alike. shard_manager_campaign_attempt_failures_total is what tells those apart: rising means the replica cannot reach etcd and is retrying, flat means it is simply waiting for the leadership.

Component Compilation Service

golem-component-compilation-service is a sidecar service responsible for compiling components. The service uses Blob storage as data storage.

Golem Router

golem-router is an Nginx proxy for Golem APIs.

Deployment Variants

For deployment variants, see the following sections:

Docker Kubernetes Golem Cloud