Deploy on Railway

Railway runs the all-in-one image in the single node and managed Postgres modes. A project has three or four resources:

Go through the production checklist too.

What Railway decides

• Deploys. A service with a volume stops its old deployment before it starts the new one. That is what the one shard manager rule needs. Without a volume, Railway starts the new deployment and stops the old one only after the new one is active, and two servers then run on the same database for a while. Attach a volume in both modes, also in mode 2, where it holds only caches.
• Replicas. A service with a volume cannot have replicas. Run one server; it scales vertically.
• Volume size. Volumes are limited by the plan: 0.5 GB on the free and trial plans, 5 GB on Hobby and 50 GB on Pro. The trial volume is too small for real use.
• Backups. Enable a volume backup schedule  in the service’s Backups tab; do not assume that daily backups are enabled. A daily schedule can lose up to a day of the SQLite state. Turn on Litestream in mode 1, or use mode 2.
• Streams. Railway’s edge closes an HTTP request after 15 minutes even while data flows, and after 5 minutes without data. Server-sent event streams, such as MCP streams, therefore end after at most 15 minutes, and clients reconnect. WebSocket connections have no limit.
• Forwarded headers. Railway’s edge sets X-Forwarded-Proto and X-Forwarded-Host, but it does not document the addresses it connects from, so the server cannot trust them. The server routes by the Host header and sees http as the scheme. This affects the URLs it builds from the request, such as the address an OIDC login returns to.

Single node

curl -fsSLO https://raw.githubusercontent.com/golemcloud/golem/main/docker-examples/single-node/generate-env.sh
chmod +x generate-env.sh
umask 077
GOLEM_S3_BUCKET='${{Bucket.BUCKET}}' \
GOLEM_S3_REGION='${{Bucket.REGION}}' \
GOLEM_S3_ENDPOINT='${{Bucket.ENDPOINT}}' \
  ./generate-env.sh > golem.env
cat >> golem.env <<'EOF'
AWS_ACCESS_KEY_ID=${{Bucket.ACCESS_KEY_ID}}
AWS_SECRET_ACCESS_KEY=${{Bucket.SECRET_ACCESS_KEY}}
GOLEM__ROUTER__MANAGEMENT_HOST=golem.example.com
GOLEM_LITESTREAM=true
GOLEM_LITESTREAM_REPLICA_URL=s3://${{Bucket.BUCKET}}/litestream?endpoint=${{Bucket.ENDPOINT}}&region=${{Bucket.REGION}}
EOF

To upgrade, change the image tag of the service.

To restore after the loss of the volume, remove the active deployment and wait until it has stopped. Replace the volume at /var/lib/golem without starting an intermediate deployment, then redeploy. The image restores the databases from the Litestream replica before the server starts. The new volume has no cache of the built-in plugins and tools, so that start downloads them again from GitHub.

Managed Postgres

For a new deployment, follow the single-node steps, but add Postgres to the project before connecting the Golem image. Use the following command instead of the single-node environment-file command. Postgres.PGHOST is the database’s private host name; Golem does not need a public TCP proxy for the database.

umask 077
GOLEM_S3_BUCKET='${{Bucket.BUCKET}}' \
GOLEM_S3_REGION='${{Bucket.REGION}}' \
GOLEM_S3_ENDPOINT='${{Bucket.ENDPOINT}}' \
GOLEM_POSTGRES_HOST='${{Postgres.PGHOST}}' \
GOLEM_POSTGRES_PORT='${{Postgres.PGPORT}}' \
GOLEM_POSTGRES_DB='${{Postgres.PGDATABASE}}' \
GOLEM_POSTGRES_USER='${{Postgres.PGUSER}}' \
GOLEM_POSTGRES_PASSWORD='${{Postgres.PGPASSWORD}}' \
  ./generate-env.sh > golem.env
cat >> golem.env <<'EOF'
AWS_ACCESS_KEY_ID=${{Bucket.ACCESS_KEY_ID}}
AWS_SECRET_ACCESS_KEY=${{Bucket.SECRET_ACCESS_KEY}}
GOLEM__ROUTER__MANAGEMENT_HOST=golem.example.com
EOF

Set the management host to your generated domain, and configure PORT=9881 and RAILWAY_RUN_UID=0 as in the single-node steps. Leave out the Litestream variables: the state is in Postgres. The script configures all six stores, including the gateway session store, so no Redis service is needed. Changing these variables on a SQLite deployment does not copy its state to Postgres.

Keep the volume at /var/lib/golem: it makes Railway stop the old deployment before it starts the new one. The volume holds caches, not the authoritative state. If it is lost, stop the Golem deployment, replace the volume without an intermediate deployment, and redeploy with the same Postgres, bucket and secrets. The server downloads the built-in artifacts again and recovers the application and agent state from Postgres and the bucket. Do not replace or delete the Postgres volume during this procedure.

For backups of the database, see Railway’s point-in-time recovery . A server or database restart is not a backup restore test.

Test a Postgres backup restore

1. Enable point-in-time recovery in the Postgres service’s Backups tab. Railway creates a separate archive bucket and redeploys Postgres. Wait for the first base backup to finish. The restore window starts with that backup, not when the database was created.
2. Create known agent state and record a UTC recovery time. After that time, change the state again. Keep both expected states so that the test can distinguish recovery to the selected time from recovery to the latest state.
3. Restore to the selected time in the Backups tab. Railway creates a separate Postgres service and volume; it does not replace the source database. Wait for the restored database to finish recovery and accept connections.
4. Stop the Golem deployment and verify that it has stopped. Prevent automatic deployments while changing its variables. Point all six stores at the restored Postgres service, including the shard-manager, indexed and gateway session stores. Update their host, port, database, user and password references together. Keep the Golem volume, blob bucket and secrets unchanged; do not generate new secrets for this test.
5. Redeploy Golem and wait for its management health check. Read the agent state and compare it with the state at the recovery time. Verify that the later changes are absent, and make a new invocation that writes state. Restart the restored database and Golem, and verify that this new write is still present.
6. For a test, stop Golem again, restore all six stores’ original database references, and redeploy. Verify the source state before deleting the temporary database and its volume. For a real cutover, keep the restored database and enable PITR on it: Railway restores a database without archiving enabled.

Restore the stores together so that the shard-manager epochs and indexed-storage fencing records come from the same point in time. Do not clear fencing records to make an agent writable; see storage recovery precautions. Keep the blobs required by the restored database. Database recovery does not undo external side effects.

Fixed cluster