Deploy on AWS ECS

Amazon ECS on Fargate runs two deployment modes, both with RDS for PostgreSQL and one S3 bucket:

• Managed Postgres: one task of the all-in-one image.
• Fixed cluster: the per-service images, with a fixed number of worker executors.

Fargate cannot run the single node mode, because a service’s EBS volume is deleted when its task stops. Go through the production checklist too.

Shared infrastructure

Both modes need:

• Network. Fargate tasks use awsvpc networking: each task has its own IP address in the VPC, without address translation. Run the tasks in private subnets, and put them in one security group that allows all TCP traffic from itself.
• Database. One RDS for PostgreSQL database. The user of the services must be able to create schemas in it.
• Bucket. One S3 bucket, with namespace_prefixes = true on every service. Give the task role s3:ListBucket on the bucket and s3:GetObject, s3:PutObject and s3:DeleteObject on its objects. The S3 client takes the credentials of the task role, so no access keys are needed.
• Load balancer. An internet-facing Application Load Balancer in public subnets, with an HTTPS listener and a certificate from ACM for the management host and the deployment domains. Redirect HTTP to HTTPS. Set its idle timeout to 3600 seconds: WebSocket connections and server-sent event streams that are quiet for longer than the idle timeout are closed. The default is 60 seconds.

The load balancer keeps the Host header. It does not send X-Forwarded-Host, and its addresses change, so the services cannot trust it for forwarded headers; see Forwarded headers.

Managed Postgres with the all-in-one image

One ECS service runs one task of the golemservices/golem image, which serves the management API, the HTTP APIs and MCP on port 9881.

Add the Caddy sidecar to the task. Allow the load balancer’s security group to reach ports 8006 and 8007 of the task. Keep port 9881 private. The load balancer sends health checks with the task IP address as the host, not the management host. Without the sidecar’s health listener, Golem routes them to the public endpoint and returns 404.

The deployment settings make ECS stop the old task before it starts the new one, in a deploy and when it replaces a failed task. The default deployment (minimum healthy 100%, maximum 200%) starts the new task first, and for a while two servers would run on the same database, which can run the same agent twice. See One shard manager. Do not use the AWS access key variables: the task role gives the S3 client its credentials.

Run generate-env.sh with the bucket and the database inputs, such as GOLEM_S3_BUCKET, GOLEM_S3_REGION, GOLEM_POSTGRES_HOST, GOLEM_POSTGRES_USER and GOLEM_POSTGRES_PASSWORD. These are inputs of the script, not of the image: the task definition needs what the script prints. That is the secrets, the S3 settings of every service, and for each of the six stores GOLEM__<STORE>__TYPE=Postgres and its GOLEM__<STORE>__CONFIG__* settings, such as GOLEM__REGISTRY_SERVICE__DB__CONFIG__HOST. See Managed Postgres for the stores. Keep the secrets and the passwords in AWS Secrets Manager, and give them to the container as secrets of the task definition.

The task has no volume, so every start of a task downloads the built-in plugins and tools again: in a deploy, when ECS replaces a failed task, and at a restart. A download takes about a minute, so give the service a health check grace period, and let the task open HTTPS connections to GitHub, for example through a NAT gateway. If it cannot, the task does not start. To download only once, mount an Amazon EFS file system at /var/lib/golem through an access point whose POSIX user and root directory owner are uid and gid 65534, the user of the image. To move to a fixed cluster later, follow Moving from the all-in-one server.

Fixed cluster

Read Fixed cluster for the settings of each service. This section describes what is particular to ECS. The production-cluster Compose file has the complete environment of each service, which the task definitions can copy.

Architecture

              ┌───────────────────────────────────┐
  HTTPS ─────►│     Application Load Balancer     │
              └────┬──────────────┬──────────┬────┘
   management host │              │          │ other hosts
     by path       ▼              ▼          ▼
  ┌──────────────────┐     ┌────────────────────────┐
  │ registry service │     │ worker service         │
  │ :8083            │     │ :9005 API, :9006 apps  │
  └────────┬─────────┘     └───────────┬────────────┘
           │ Cloud Map DNS             │ gRPC to task IPs
           ▼                           ▼
  ┌──────────────────┐     ┌────────────────────────┐
  │ shard manager ×1 │◄────┤ worker executors ×N    │
  └──────────────────┘     └────────────────────────┘

  all services ──► RDS for PostgreSQL, S3 bucket

• Executor addresses. The shard manager records each executor’s real task IP address, so grpc.advertised_ip is not needed.
• Service discovery. Create a Cloud Map private DNS namespace, for example golem.internal, and register the registry service, the shard manager, the worker service and the component compilation service in it with A records and a short TTL, such as 10 seconds. The executors need no DNS name: the shard manager learns their addresses when they register.

Services

Point the hosts of the services at each other with their DNS names: GOLEM__REGISTRY_SERVICE__HOST=registry-service.golem.internal and so on, as the Compose file does with its service names.

Give the registry service a health check grace period long enough to download and inspect the built-in plugins and tools before it opens its listeners, for example 600 seconds. With no persistent cache at /app/builtin-artifacts, every replacement repeats this startup work and needs HTTPS access to GitHub. Size the grace period for a start with an empty cache, not only for a warm restart.

Persistent artifact cache

Mount Amazon EFS at /app/builtin-artifacts in the registry container to keep downloaded built-in plugins and tools across task replacements:

1. Create an encrypted EFS file system in the VPC, with a mount target in each Availability Zone where registry tasks can run. Allow TCP port 2049 on the mount targets from the registry tasks’ security group only.
2. Create an access point with POSIX user uid and gid 65534. Set its root directory to /builtin-artifacts, with owner uid and gid 65534 and permissions 0750. EFS creates this directory on the first mount.
3. Add this volume to the task definition. Replace the file system and access point IDs. Use Linux Fargate platform version 1.4.0 or later.

{
  "name": "builtin-artifacts",
  "efsVolumeConfiguration": {
    "fileSystemId": "fs-0123456789abcdef0",
    "rootDirectory": "/",
    "transitEncryption": "ENABLED",
    "authorizationConfig": {
      "accessPointId": "fsap-0123456789abcdef0",
      "iam": "DISABLED"
    }
  }
}

4. Add this entry to the registry container’s mountPoints:

{
  "sourceVolume": "builtin-artifacts",
  "containerPath": "/app/builtin-artifacts",
  "readOnly": false
}

The task volume’s rootDirectory must be / or omitted when it uses an access point; the access point selects the directory inside EFS. This configuration uses network and POSIX permissions, not task-role IAM authorization, for EFS access. Keep the mount targets private and restrict their security group as described above.

The cache holds both downloaded artifacts and their extracted metadata. Replacement tasks verify the artifacts’ checksums and reuse valid cached metadata instead of repeating the expensive extraction. The registry still validates the descriptors and provisions the built-in plugins and tools before it opens its listeners. Keep the health check grace period for an empty cache or a release with new artifacts, and retain HTTPS access to GitHub for downloads.

Shard manager

Run exactly one shard manager task, in Postgres mode. With minimum healthy 0% and maximum 100%, ECS stops the old task before it starts the new one, in a deploy and when it replaces a failed task. The default deployment (minimum healthy 100%, maximum 200%) starts the new task first, and for a while two shard managers could run the same agent twice. See One shard manager.

Do not run a second replica in etcd mode as a standby on ECS. A standby keeps its gRPC port closed, and ECS replaces a task that fails its container or load balancer health check, so it would replace the standby again and again. When the shard manager task fails, ECS starts a new one; the executors keep their shards for the length of their lease (1 minute by default) while it starts.

Set GOLEM__SHARD_MANAGER__RETRIES__MAX_ATTEMPTS on the executors high enough that an executor that starts during a shard manager restart keeps retrying its registration.

Worker executors

Change the desired count to scale. A task that ECS stops gets SIGTERM and deregisters, so its shards move at once. Give the executor task enough memory, and set GOLEM__MEMORY__SYSTEM_MEMORY_OVERRIDE to a number of bytes below the task memory if the executor detects more memory than the task has.

Load balancer rules

Point the management host and every deployment domain at the load balancer in DNS. Add these rules to the HTTPS listener, in this order:

Reserve the management host with GOLEM__DOMAIN_REGISTRATION__RESERVED_DOMAINS='["golem.example.com"]' on the registry service.

Deploy the cluster

1. Create the shared infrastructure and the Cloud Map namespace.
2. Create the task definitions and the ECS services in this order: registry service, shard manager, component compilation service, worker service, worker executors. Use the same image version for all of them.
3. Check the shard manager’s log: it logs each executor that registers, with its task IP address. Call https://golem.example.com/healthcheck, then deploy an application with the admin token, as Connect the CLI describes.

Forwarded headers

The load balancer keeps the Host header, so the server finds the deployment of each domain. It does not send X-Forwarded-Host, and its addresses change, so do not list it in the trusted ingress addresses. The server then sees the scheme of its own listener, http. This affects the URLs that it builds from the request, such as the address an OIDC login returns to.

To give the server the public scheme, run Caddy as a second container in the task that the load balancer reaches, and point the target group at Caddy’s port instead. For the all-in-one image, build the sidecar image from this Caddyfile. Replace golem.example.com with the management host:

:8006 {
	reverse_proxy 127.0.0.1:9881 {
		header_up X-Forwarded-Proto https
		header_up X-Forwarded-Host {host}
	}
}

:8007 {
	handle /healthcheck {
		reverse_proxy 127.0.0.1:9881 {
			header_up Host golem.example.com
			header_up X-Forwarded-Host golem.example.com
			header_up X-Forwarded-Proto https
		}
	}
	respond 404
}

FROM caddy:2
COPY Caddyfile /etc/caddy/Caddyfile

Caddy replaces the forwarded headers of every request with the host of the request and https, which is correct when the load balancer accepts HTTPS only. Port 8007 forwards only /healthcheck, with the management host. Keep this listener separate from port 8006, so it does not replace an application’s own /healthcheck route. Set the target group’s health check port to 8007 and its success code to 200. Allow only the load balancer’s security group to reach both Caddy ports.

The containers of a task share one network namespace, so trust 127.0.0.1: set GOLEM__WORKER_SERVICE__ROUTE_RESOLVER__TRUSTED_INGRESS_ADDRESSES='["127.0.0.1"]' on the all-in-one image.

For the worker service of a fixed cluster, use only the :8006 block, change its upstream port to 9006, and set GOLEM__ROUTE_RESOLVER__TRUSTED_INGRESS_ADDRESSES='["127.0.0.1"]'. Point the public target group at Caddy port 8006, but keep its health check on the worker service’s management port 9005 as the services table describes.