All-in-one server

golem server run runs every Golem service in one process: the registry service, the shard manager, the worker service, the component compilation service and the worker executor. Without a configuration file it is the local development server. With a configuration file it runs the single node and managed Postgres deployment modes.

This page describes the configuration of the golem binary and of the golemservices/golem image, which runs the binary. For the steps on a platform, see VM, Railway, Render, Fly.io or AWS ECS.

Listeners

One port

Set router.management_host (GOLEM__ROUTER__MANAGEMENT_HOST) to the host name of the management API, for example golem.example.com. The main API port then routes each request by its host: requests for the management host reach the management API, and requests for every other host reach the HTTP API and MCP deployments of that host. So one port serves everything, and a reverse proxy or a platform that routes one port per service can send all host names to it. The server also adds the management host to registry_service.domain_registration.reserved_domains, so that no environment can register it as a deployment domain.

The custom request and MCP ports still serve the deployments. Leave them on 127.0.0.1 when the main API port serves everything.

Internal listeners

The services also bind gRPC and metrics listeners on ports that the operating system selects, each on the bind_address of its service. They do not authenticate their callers. Keep every bind_address at 127.0.0.1 unless a proxy on another machine must reach the listener, and then expose only the three ports above.

Configuration

golem server run --config <file> reads a TOML file. Every field has a default, so the file contains only the fields that it changes. GOLEM__<SECTION>__<FIELD> environment variables override the file, for example GOLEM__REGISTRY_SERVICE__INITIAL_ACCOUNTS__ROOT__TOKEN. Each section has the shape of the configuration of its service:

golem --dump-config-default-toml prints the complete default configuration, followed by a commented example that keeps all state in Postgres and the blobs in S3. golem --dump-config-default-env-var prints the same as environment variables. The server sets the internal ports and hosts itself, and refuses a file that sets them. See Configuration File for all rules.

A field whose value selects a variant with type takes its settings from config. When the file selects a variant other than the default, give all settings of that variant: the defaults of the default variant do not apply to it. The examples on this page give all settings that have no default.

Secrets

The default configuration uses public secrets. Replace all of them:

Keep the secrets out of the TOML file, in an environment file that only the server can read:

umask 077
HMAC_KEY=$(openssl rand -base64 32)
cat > golem.env <<EOF
GOLEM__REGISTRY_SERVICE__INITIAL_ACCOUNTS__ROOT__TOKEN=$(openssl rand -hex 32)
GOLEM__WORKER_SERVICE__WEBHOOK_CALLBACK_HANDLER__HMAC_KEY=$HMAC_KEY
GOLEM__WORKER_EXECUTOR__AGENT_WEBHOOKS_SERVICE__HMAC_KEY=$HMAC_KEY
GOLEM__WORKER_SERVICE__INVOCATION_SESSION_TOKENS__ACTIVE_KEY__ID=key-1
GOLEM__WORKER_SERVICE__INVOCATION_SESSION_TOKENS__ACTIVE_KEY__KEY=$(openssl rand -base64 32)
EOF

The server refuses to bind any listener to an address that is not loopback while one of these settings has its default value, and lists the settings in the error. A reverse proxy on the same host makes a loopback listener public too, and the server cannot detect it, so replace the secrets in that case as well. When no initial account has the well-known local admin token, the server deletes that token from the registry at startup.

Single node

In mode 1 the state stays in the default SQLite databases under data_dir, so put data_dir on a volume that outlives the process. Put the blobs in S3: the same blob_storage in all four services that have one. This example uses one bucket, golem:

data_dir = "/var/lib/golem"
 
[registry_service.blob_storage]
type = "S3"
 
[registry_service.blob_storage.config]
region = "us-east-1"
object_prefix = ""
namespace_prefixes = true
compilation_cache_bucket = "golem"
custom_data_bucket = "golem"
oplog_payload_bucket = "golem"
compressed_oplog_buckets = ["golem"]
initial_agent_files_bucket = "golem"
components_bucket = "golem"
filesystem_snapshots_bucket = "golem"
 
[registry_service.blob_storage.config.retries]
max_attempts = 3
min_delay = "100ms"
max_delay = "1s"
multiplier = 3.0
max_jitter_factor = 0.15
 
# Repeat the three sections above for worker_service, worker_executor and
# component_compilation_service: [worker_service.blob_storage],
# [worker_service.blob_storage.config], [worker_service.blob_storage.config.retries], and so on.
 
[router]
management_host = "golem.example.com"
 
[worker_service.route_resolver]
trusted_ingress_addresses = ["127.0.0.1"]

• namespace_prefixes = true lets all blob kinds share one bucket. Choose it before the first start and do not change it later. See One bucket for all blobs.
• For an S3-compatible service, add aws_endpoint_url = "https://<endpoint>" and, if the service needs it, aws_path_style = true to each blob_storage.config.
• The S3 client reads its credentials from the standard AWS sources: AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in the environment file, or the role of the machine.
• management_host makes the main API port serve the HTTP API and MCP deployments of every other host. See One port.
• trusted_ingress_addresses makes the worker service take the public scheme and host from the X-Forwarded-Proto and X-Forwarded-Host headers of a reverse proxy on the same host. A request from a trusted address without both headers fails, so a request sent directly to the custom request port from that host does too.

Managed Postgres

In mode 2 the state goes to Postgres too. Add the database to the file of the single node. Each store has its own schema, and the server rejects two stores that share one. Use these schema names: the per-service images use the same, so the database can move to a fixed cluster later.

[registry_service.db]
type = "Postgres"
 
[registry_service.db.config]
host = "db.example.internal"
port = 5432
database = "golem"
username = "golem"
password = "<password>"
max_connections = 10
schema = "golem_registry_service"
 
[shard_manager.persistence]
type = "Postgres"
 
[shard_manager.persistence.config]
host = "db.example.internal"
port = 5432
database = "golem"
username = "golem"
password = "<password>"
max_connections = 10
schema = "golem_shard_manager"
 
[worker_executor.key_value_storage]
type = "Postgres"
 
[worker_executor.key_value_storage.config]
host = "db.example.internal"
port = 5432
database = "golem"
username = "golem"
password = "<password>"
max_connections = 10
schema = "golem_worker_executor"
 
[worker_executor.indexed_storage]
type = "Postgres"
 
[worker_executor.indexed_storage.config]
host = "db.example.internal"
port = 5432
database = "golem"
username = "golem"
password = "<password>"
max_connections = 10
schema = "golem_worker_executor_indexed"
drop_prefix_delete_batch_size = 1024
 
[worker_executor.scheduler_storage]
type = "Postgres"
 
[worker_executor.scheduler_storage.config]
host = "db.example.internal"
port = 5432
database = "golem"
username = "golem"
password = "<password>"
max_connections = 10
schema = "golem_worker_executor_scheduler"
 
[worker_service.gateway_session_storage]
type = "Postgres"
 
[worker_service.gateway_session_storage.config]
host = "db.example.internal"
port = 5432
database = "golem"
username = "golem"
password = "<password>"
max_connections = 10
schema = "golem_worker_service_session_store"
pending_login_expiration = "1h"
cleanup_interval = "5m"

The services create their schemas and tables at startup, so the database user must be able to create schemas in the database. To keep the password out of the file, set it with environment variables such as GOLEM__REGISTRY_SERVICE__DB__CONFIG__PASSWORD. The data_dir then holds only caches. The server runs without a volume, but then every start downloads the built-in plugins and tools again.

Built-in plugins and tools

At every start, the registry service provisions Golem’s built-in plugins and tools from its cache, builtin-artifacts in the data directory. It downloads from the golem-builtins releases  on GitHub each one that the cache does not have or that does not match its checksum, which takes about a minute. If a download fails, for example because the server cannot open HTTPS connections to GitHub, the server does not start.

Keep the data directory on a persistent volume, so that only the first start downloads. Without one, every start downloads again and needs GitHub.

Container image

The golemservices/golem image runs golem server run. It serves everything on one port, 9881, and these are its defaults:

The router listens on all addresses, so the server does not start until the secrets have secret values. Set GOLEM__ROUTER__MANAGEMENT_HOST to the host name of the management API. The image arguments are the arguments of golem, server run by default, so a --config file still works; on a platform without files, configure everything with environment variables.

/healthcheck answers 503 while the server starts and 200 once it serves requests. Use it as the health check of the platform: a start that downloads the built-in plugins and tools takes about a minute.

Volume ownership

The image runs the server as uid and gid 65534, the user golem, with HOME=/home/golem. The data directory, /var/lib/golem, must be writable by that user:

• A new named Docker volume at /var/lib/golem takes the owner of the directory in the image, so it needs nothing.
• A bind mount or a host directory must belong to uid and gid 65534: sudo chown -R 65534:65534 <dir>.
• A platform volume belongs to whoever the platform makes its owner. Each platform guide says what it needs.

Otherwise the server does not start, and the log says The data directory /var/lib/golem is not writable by uid 65534; give it to uid 65534 and gid 65534. Where the platform mounts the volume as root and cannot change its owner, build an image that gives the volume to uid 65534 at every start and then runs the server as that user:

FROM golemservices/golem:<version>
USER root
ENTRYPOINT ["/bin/sh", "-c", "chown -R 65534:65534 /var/lib/golem && exec setpriv --reuid=65534 --regid=65534 --clear-groups /usr/bin/tini -- /usr/local/bin/golem-entrypoint \"$@\"", "golem"]
CMD ["server", "run"]

The container starts as root only to change the owner, and setpriv runs the server as uid and gid 65534 with no other groups.

Environment file

The generate-env.sh script prints new secrets and the S3 configuration of every service as environment variables. With GOLEM_POSTGRES_HOST, it also prints the Postgres configuration of every store, with the schemas of the managed Postgres mode:

curl -fsSLO https://raw.githubusercontent.com/golemcloud/golem/main/docker-examples/single-node/generate-env.sh
chmod +x generate-env.sh
 
# Single node: the state in SQLite on the volume, the blobs in S3
GOLEM_S3_BUCKET=my-bucket GOLEM_S3_REGION=eu-central-1 ./generate-env.sh > golem.env
 
# Managed Postgres: the state in Postgres, the blobs in S3
GOLEM_S3_BUCKET=my-bucket GOLEM_S3_REGION=eu-central-1 \
GOLEM_POSTGRES_HOST=db.example.internal GOLEM_POSTGRES_PASSWORD=<password> \
  ./generate-env.sh > golem.env

Use the managed Postgres variables or Litestream, not both: with Litestream, the image keeps the storage of the worker executor in SQLite. For an S3-compatible service, also set GOLEM_S3_ENDPOINT and, if the service needs it, GOLEM_S3_PATH_STYLE=true. Add the credentials of the bucket, AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, unless the platform gives the container a role. Load the file into the secret store of the platform: each platform guide says how. The file contains the admin token, GOLEM__REGISTRY_SERVICE__INITIAL_ACCOUNTS__ROOT__TOKEN; keep it secret.

Litestream

In the single node mode, the state is only as durable as the volume. With GOLEM_LITESTREAM=true, the image runs Litestream , which copies the changes of the SQLite databases to S3, about once a second by default. Litestream is off by default. With a reliable persistent volume, such as a replicated cloud disk with backups, it is not needed: use it on platforms whose disk is ephemeral or belongs to one host.

With Litestream, the worker executor keeps its key-value storage and its indexed storage in one SQLite database each, instead of one database per agent. The server then has a fixed set of six databases in the data directory: registry.db, shard_manager.db, scheduler.db, gateway-sessions.db, key-value-storage.db and indexed-storage.db. The replica of <data_dir>/<name>.db is <replica_url>/<name>.db. Deleting an agent deletes rows, which Litestream replicates like any other write. Before the server starts, the image restores each of the six databases that the data directory does not have from the replica. So after the loss of a volume, attach an empty volume and start the container again: it restores the state that Litestream had copied.

• Turn Litestream on only for a new data directory. With Litestream, the server does not read the databases per agent in kv-store that it writes without Litestream, so the agents of a data directory that started without Litestream are not visible after the switch.
• Do not set the worker executor’s storage. The image sets worker_executor.key_value_storage and worker_executor.indexed_storage itself, and takes the data directory from GOLEM__DATA_DIR: do not set them, or data_dir, in a --config file or in GOLEM__ variables.
• One writer for all oplogs. One SQLite database holds the oplogs of all agents, and SQLite writes one transaction at a time. That fits a single node; for heavier load, use managed Postgres.
• The replica can be in the bucket of the blobs, under its own path, such as litestream. Do not give it the path of a blob kind, such as component_store.

Litestream limits

• One server only. Exactly one container may write to a GOLEM_LITESTREAM_REPLICA_URL. Two containers that replicate to the same replica corrupt it. Litestream is backup and restore: it gives no high availability and no failover to a second server.
• Asynchronous replication. Litestream copies the changes about once a second by default. When the container stops abruptly and its volume is lost, the newest writes, including oplog entries, can be lost. Agents then replay from a slightly earlier point of their oplog and can run side effects again. This is weaker than the durability of Postgres in the managed Postgres mode or in a fixed cluster; use those for production durability.
• Only the SQLite databases. Litestream does not replicate blobs. Keep the blobs in S3, for example in one bucket, not in the local blobs directory of the data directory.
• Not the cache of the built-in plugins and tools. Litestream does not replicate builtin-artifacts, so a restore onto a new volume downloads the built-in plugins and tools again, and that start needs HTTPS connections to GitHub.
• Restore fills in missing databases only. At startup, the image restores only the databases that are missing from the data directory. A restart on a volume that still has its data restores nothing, and an older replica never overwrites newer local data.

Connect the CLI

Add an environment for the server to the application manifest, with the admin token in an environment variable:

environments:
  prod:
    server:
      url: https://golem.example.com
      auth:
        staticToken: "{{ GOLEM_PROD_TOKEN }}"
 
httpApi:
  deployments:
    prod:
      - domain: api.example.com
        scheme: https
        agents:
          MyAgent: {}

GOLEM_PROD_TOKEN=<admin token> golem -E prod deploy

Deployments on a custom server use domain, not subdomain. Point each domain at the reverse proxy in DNS, and route it to the custom request port.

Upgrades

Replace the binary or the image and restart the process. The server migrates its databases at startup. Stop the old process before the new one starts, and upgrade every service of a deployment to the same version.