Deploy on Fly.io

Fly.io runs the all-in-one image on one Machine in the single node and managed Postgres modes, with the blobs in Tigris, Fly.io’s S3-compatible object storage. Go through the production checklist too.

What Fly.io decides

• One Machine. Run exactly one Machine: two servers on one database can run the same agent twice. Deploy with --ha=false, keep fly scale count 1, and keep the Machine running with auto_stop_machines = "off". The rolling deploy strategy, the default, replaces the Machine in place: it stops the old one before the new one starts. Do not use bluegreen or canary, which start new Machines next to the old ones.
• Volumes. A volume belongs to one Machine on one server. If that server fails, the Machine is down until the server comes back, and a lost volume loses what only the volume had. Fly.io snapshots volumes once a day and keeps the snapshots for five days by default. Turn on Litestream in mode 1, or use mode 2.
• Forwarded headers. The Fly Proxy does not document X-Forwarded-Host or the address it connects from, so the server does not trust it. The server routes by the Host header and sees http as the scheme. This affects the URLs it builds from the request, such as the address an OIDC login returns to.

Single node

fly apps create my-golem
fly volumes create golem_data --app my-golem --region fra --size 10
fly storage create --app my-golem

curl -fsSLO https://raw.githubusercontent.com/golemcloud/golem/main/docker-examples/single-node/generate-env.sh
chmod +x generate-env.sh
umask 077
GOLEM_S3_BUCKET=<bucket name> GOLEM_S3_REGION=auto \
GOLEM_S3_ENDPOINT=https://fly.storage.tigris.dev \
  ./generate-env.sh > golem.env
fly secrets import --app my-golem --stage < golem.env

app = "my-golem"
primary_region = "fra"
 
[build]
  image = "golemservices/golem:<version>"
 
[deploy]
  strategy = "rolling"
 
[env]
  GOLEM__ROUTER__MANAGEMENT_HOST = "golem.example.com"
  GOLEM_LITESTREAM = "true"
  GOLEM_LITESTREAM_REPLICA_URL = "s3://<bucket name>/litestream?endpoint=https://fly.storage.tigris.dev&region=auto"
 
[mounts]
  source = "golem_data"
  destination = "/var/lib/golem"
 
[http_service]
  internal_port = 9881
  force_https = true
  auto_stop_machines = "off"
  auto_start_machines = true
  min_machines_running = 1
 
  [[http_service.checks]]
    grace_period = "60s"
    interval = "15s"
    timeout = "5s"
    method = "GET"
    path = "/healthcheck"
    headers = { Host = "golem.example.com" }
 
[[vm]]
  size = "shared-cpu-2x"
  memory = "4gb"

fly deploy --ha=false --wait-timeout 10m
fly certs add golem.example.com --app my-golem
fly certs add api.example.com --app my-golem
fly certs add mcp.example.com --app my-golem

To upgrade, change the image tag in fly.toml and run fly deploy --ha=false --wait-timeout 10m. Before a restore after the loss of the volume, stop and remove the old Machine so that only one container writes to the replica. Keep the app’s secrets and Tigris bucket. Create a new volume named golem_data and deploy with the same configuration: the image restores the databases from the Litestream replica before the server starts. The new volume has no cache of the built-in plugins and tools, so that start downloads them again from GitHub.

Managed Postgres

Create a Fly.io Managed Postgres cluster in the region of the app, and pass the host, database, user and password of its connection URL to the script, together with the bucket:

GOLEM_S3_BUCKET=<bucket name> GOLEM_S3_REGION=auto \
GOLEM_S3_ENDPOINT=https://fly.storage.tigris.dev \
GOLEM_POSTGRES_HOST=<host> GOLEM_POSTGRES_DB=<database> \
GOLEM_POSTGRES_USER=<user> GOLEM_POSTGRES_PASSWORD=<password> \
  ./generate-env.sh > golem.env

Leave out the [mounts] section and the Litestream variables: the state is in Postgres. The single Machine and the rolling strategy still keep one server at a time.

Fixed cluster