Fixed cluster

A fixed cluster runs each Golem service from its own image on Docker Hub , with a fixed number of worker executors. All services share one Postgres database and one S3 bucket. This page describes the configuration that every platform needs. For the steps on a platform, see VM or AWS ECS. The production-cluster example contains all settings of this page in a Docker Compose file.

Services

Every service reads a TOML file in its image, and GOLEM__<FIELD> environment variables override it. For a nested field, join the names with __: GOLEM__DB__CONFIG__HOST sets db.config.host. A list is written in TOML syntax, for example GOLEM__ROUTE_RESOLVER__TRUSTED_INGRESS_ADDRESSES='["10.0.0.5/32"]'. The Deployment page links the configuration file and the environment variables of each service.

Storage

Postgres

Each store has its own schema in the same database. These are the schema names that the all-in-one server uses, so a database can move between the two:

Each CONFIG takes HOST, PORT, DATABASE, USERNAME, PASSWORD, MAX_CONNECTIONS and SCHEMA. The session storage also needs PENDING_LOGIN_EXPIRATION (for example 1h) and CLEANUP_INTERVAL (for example 5m). With the sessions in Postgres, the cluster needs no Redis.

S3

The registry service, the worker service, the component compilation service and the worker executor use the same blob storage. With one bucket:

GOLEM__BLOB_STORAGE__TYPE=S3
GOLEM__BLOB_STORAGE__CONFIG__REGION=us-east-1
GOLEM__BLOB_STORAGE__CONFIG__OBJECT_PREFIX=""
GOLEM__BLOB_STORAGE__CONFIG__NAMESPACE_PREFIXES=true
GOLEM__BLOB_STORAGE__CONFIG__COMPILATION_CACHE_BUCKET=golem
GOLEM__BLOB_STORAGE__CONFIG__CUSTOM_DATA_BUCKET=golem
GOLEM__BLOB_STORAGE__CONFIG__OPLOG_PAYLOAD_BUCKET=golem
GOLEM__BLOB_STORAGE__CONFIG__COMPRESSED_OPLOG_BUCKETS='["golem"]'
GOLEM__BLOB_STORAGE__CONFIG__INITIAL_AGENT_FILES_BUCKET=golem
GOLEM__BLOB_STORAGE__CONFIG__COMPONENTS_BUCKET=golem
GOLEM__BLOB_STORAGE__CONFIG__FILESYSTEM_SNAPSHOTS_BUCKET=golem
GOLEM__BLOB_STORAGE__CONFIG__RETRIES__MAX_ATTEMPTS=3
GOLEM__BLOB_STORAGE__CONFIG__RETRIES__MIN_DELAY=100ms
GOLEM__BLOB_STORAGE__CONFIG__RETRIES__MAX_DELAY=1s
GOLEM__BLOB_STORAGE__CONFIG__RETRIES__MULTIPLIER=3.0
GOLEM__BLOB_STORAGE__CONFIG__RETRIES__MAX_JITTER_FACTOR=0.15

For an S3-compatible service, add GOLEM__BLOB_STORAGE__CONFIG__AWS_ENDPOINT_URL and, if the service needs it, GOLEM__BLOB_STORAGE__CONFIG__AWS_PATH_STYLE=true. The S3 client reads its credentials from the standard AWS sources, such as AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY or the role of the task. See Blob storage on S3.

Secrets

The images do not check their secrets, and their defaults are public. Set these on every replica of the service:

The registry service’s default login is GitHub OAuth2 with placeholder credentials. Set GOLEM__LOGIN__TYPE=Disabled to allow only the tokens of the initial accounts, or configure GOLEM__LOGIN__CONFIG__GITHUB__*.

Plans

The registry service creates the default plan from initial_plans.default at every start, and updates the plan when the values change. The defaults of the image limit applications, components, storage and uploads; set GOLEM__INITIAL_PLANS__DEFAULT__<LIMIT> to change them.

Built-in plugins and tools

At every start, the registry service provisions Golem’s built-in plugins and tools from its cache, and downloads from the golem-builtins releases  on GitHub each one that the cache does not have or that does not match its checksum. If a download fails, the registry service does not start. In the image, the cache is /app/builtin-artifacts, inside the container, so every new registry container downloads them again, which takes about a minute and needs outgoing HTTPS connections to GitHub. To keep the cache, mount a persistent volume at /app/builtin-artifacts. The volume must be writable by uid 65534, the user of the image; see User and Permissions.

Shard manager

Run one shard manager replica in SQL mode (GOLEM__PERSISTENCE__TYPE=Postgres), and configure the platform to stop the old replica before it starts the new one. Two shard managers on one database can run the same agent twice. See One shard manager.

While the shard manager restarts, the executors keep their shards until their lease runs out (1 minute by default), but new executors cannot register. Raise GOLEM__SHARD_MANAGER__RETRIES__MAX_ATTEMPTS on the executors, so that an executor that starts during a shard manager restart retries its registration until the shard manager is back.

etcd mode (GOLEM__PERSISTENCE__TYPE=Etcd) allows more than one replica, but only one of them, the leader, opens its gRPC port. Use it only where the platform sends traffic to the ready replica alone, such as Kubernetes with a readiness probe on the gRPC port. Platforms that replace a task that fails its health check, such as ECS, would replace every standby. etcd must run on volumes and on the private network, because the shard manager talks to it over plain HTTP. See Shard Manager.

Worker executors

Set the number of executors with the replica count of the platform. Each executor registers with the shard manager when it starts and gets a share of the shards. A stopped executor deregisters and its shards move to the other executors. An executor that disappears without stopping loses its shards when its lease runs out.

The shard manager records the source IP address of each executor’s registration, and the worker service and the shard manager dial the executor’s gRPC port at that address. On a network that translates source addresses, set GOLEM__GRPC__ADVERTISED_IP on each executor to its own reachable IP address. On an IPv6 network, set GOLEM__BIND_ADDRESS=:: on every service. See Network Addresses.

The executor’s agent files are in temporary directories by default, and nothing of an executor needs a volume. To keep agent files across executors, see Filesystem snapshots.

TLS between the services

The services talk to each other over gRPC. Each gRPC server can require TLS, and each gRPC client can use it. The services always use mutual TLS: a server with TLS enabled requires a client certificate that its client CA signed, and a client checks the server certificate against its server CA.

Every service has a server setting grpc.tls (GOLEM__GRPC__TLS__TYPE). Every client of a service has its own setting <section>.tls, with the same keys on both sides. The sections are:

Set TYPE to Enabled and give the settings of the table below. Each certificate, key and CA comes from a PEM file (the _file setting) or from an inline value (the PEM in standard Base64 on one line), not both. A service refuses a configuration that sets both, or neither, for an item.

The environment name of a setting is GOLEM__GRPC__TLS__CONFIG__SERVER_CERT_FILE on the server, and for example GOLEM__REGISTRY_SERVICE__TLS__CONFIG__CLIENT_CERT_FILE on the client of the registry service. Without server_domain_name, a client checks the server certificate against the host of the address it connects to.

# In the pod template of a service that serves gRPC and calls the registry service
containers:
  - name: golem-worker-service
    env:
      - name: GOLEM__GRPC__TLS__TYPE
        value: Enabled
      - name: GOLEM__GRPC__TLS__CONFIG__SERVER_CERT_FILE
        value: /etc/golem-grpc-tls/tls.crt
      - name: GOLEM__GRPC__TLS__CONFIG__SERVER_KEY_FILE
        value: /etc/golem-grpc-tls/tls.key
      - name: GOLEM__GRPC__TLS__CONFIG__CLIENT_CA_CERT_FILE
        value: /etc/golem-grpc-tls/ca.crt
      - name: GOLEM__REGISTRY_SERVICE__TLS__TYPE
        value: Enabled
      - name: GOLEM__REGISTRY_SERVICE__TLS__CONFIG__CLIENT_CERT_FILE
        value: /etc/golem-grpc-tls/tls.crt
      - name: GOLEM__REGISTRY_SERVICE__TLS__CONFIG__CLIENT_KEY_FILE
        value: /etc/golem-grpc-tls/tls.key
      - name: GOLEM__REGISTRY_SERVICE__TLS__CONFIG__SERVER_CA_CERT_FILE
        value: /etc/golem-grpc-tls/ca.crt
    volumeMounts:
      - name: golem-grpc-tls
        mountPath: /etc/golem-grpc-tls   # the whole Secret as a directory, never subPath
        readOnly: true
volumes:
  - name: golem-grpc-tls
    secret:
      secretName: golem-grpc-tls   # the cert-manager Certificate's Secret

Renew certificates

A service reloads renewed files without a restart. It polls the files of every _file setting every 60 seconds, which matches how often the kubelet updates a mounted Secret. When the bytes of a certificate, key or CA change, the service checks the new set and then uses it:

• A server uses the new set for every TLS handshake that starts after the reload. It requires a client certificate in every handshake, and drops a handshake that takes longer than 10 seconds.
• A client uses the new set for every new connection. A connection that is open keeps the old certificate until it closes, so a renewal does not interrupt a call or a stream.
• The check parses the certificate, the key and the CA, and refuses a set in which the key does not belong to the certificate, the certificate is expired or not valid yet, or every CA certificate has expired.

A set that fails the check is not used. The service goes on with the old one, logs the error “Changed TLS files cannot be used; keeping the current certificate” once for each distinct set of files, and counts it in golem_tls_certificate_reload_failures_total. It checks the same files again on every poll. A successful reload logs “TLS certificate reloaded” with the subject, the end of validity and the SHA-256 fingerprint.

A client whose files cannot be loaded at startup, for example because the Secret is not mounted yet, starts anyway. Its calls fail with a TLS configuration error, and it tries to load the files again, at most every 60 seconds, until they load. A server refuses to start with a bad set.

Two metrics help to watch the renewal. golem_tls_certificate_not_after_seconds{use} is the Unix time at which the certificate in use expires; alert when it comes close to the current time. golem_tls_certificate_reload_failures_total{use} rises when changed files cannot be used. The use label names the user of the certificate: grpc server, <target> client for a gRPC client (for example registry client or worker_executor client), and etcd client for the shard manager’s etcd connection. See Metrics.

Mount the Secret as a directory, not with subPath: with subPath the files in the pod never change. An inline value changes only with a restart.

Ingress

The management API is split between two services. A reverse proxy or load balancer sends the management host to them by path:

Two of the worker service paths are WebSocket connections: /v1/agents/invoke-agent-session and /v1/components/<component>/workers/<agent>/connect.

Every other host goes to the worker service’s custom request port, 9006, which serves the HTTP API and MCP deployments of that host. Reserve the management host with GOLEM__DOMAIN_REGISTRATION__RESERVED_DOMAINS='["golem.example.com"]' on the registry service, so that no environment can register it.

To give the worker service the public scheme and host, set GOLEM__ROUTE_RESOLVER__TRUSTED_INGRESS_ADDRESSES to the address range of the proxy that connects to it (/32 for one address). The proxy must replace the client’s X-Forwarded-Proto and X-Forwarded-Host with one value each. A request from a trusted address without both headers fails. A single proxy needs a fixed address: give it a static IP address, or run it next to the worker service and trust 127.0.0.1/32. For an ingress that runs as several pods, trust its pod range and admit only the ingress with a network policy; see Configuring ingress. See the production checklist.

Moving from the all-in-one server

A fixed cluster can take over the database and the bucket of an all-in-one server on Postgres:

1. Stop the all-in-one server.

2. Give the cluster the same database, schemas, bucket, namespace_prefixes, secrets and admin token.

3. Give the registry service the IDs of the initial accounts and of the plan of the all-in-one server. The two configurations use different IDs, and the registry service does not start when an account with the same email address has another ID:

   GOLEM__INITIAL_PLANS__DEFAULT__PLAN_ID=e808bd76-a6ab-4090-ade4-8447b8e8550f
   GOLEM__INITIAL_ACCOUNTS__ROOT__ID=51de7d7d-f286-49aa-b79a-96022f7e2df9
   GOLEM__INITIAL_ACCOUNTS__ROOT__PLAN_ID=e808bd76-a6ab-4090-ade4-8447b8e8550f
   GOLEM__INITIAL_ACCOUNTS__BUILTIN_PLUGIN_OWNER__ID=b0a654af-d67f-4d73-a824-cf75e122bfc0
   GOLEM__INITIAL_ACCOUNTS__BUILTIN_PLUGIN_OWNER__PLAN_ID=e808bd76-a6ab-4090-ade4-8447b8e8550f
   GOLEM__INITIAL_ACCOUNTS__BUILTIN_TOOL_OWNER__PLAN_ID=e808bd76-a6ab-4090-ade4-8447b8e8550f
   GOLEM__INITIAL_ACCOUNTS__MARKETING__PLAN_ID=e808bd76-a6ab-4090-ade4-8447b8e8550f

4. Start the cluster with the shard manager in Postgres mode on the golem_shard_manager schema, so that the shard epochs continue.

The default plan then takes the limits of the cluster’s configuration.