Filesystem Snapshots

A filesystem snapshot is a copy of an agent’s files that the worker executor keeps in blob storage. A snapshot record in the oplog of the agent names it. With filesystem snapshots, an agent keeps its files after a start from a snapshot record, after a manual update and after a snapshot-assisted automatic update. Snapshotting describes what agents see. This page describes how to enable and operate the feature.

Filesystem snapshots are disabled by default. A disabled executor does not save agent files. It writes a snapshot record, without a filesystem snapshot name, only when the agent’s files can come back with it: when the agent’s files are its untouched read-only initial files, with no other files and no provisioned files, and the agent did not rename or hard link any file, and did not set a given modification time on a file or directory. An agent whose files changed takes no automatic snapshot and recovers by a replay, and a snapshot-based manual update of it fails. See Availability. Ephemeral agents take no snapshots.

Requirements

• The executor runs on Linux 5.8 or later and keeps the agent filesystems on XFS storage with reflink: ManagedXfs (with project quotas) or ReflinkXfs (without project quotas). Each executor has its own XFS filesystem. See Prepare XFS storage.
• With ManagedXfs, the executor process must be able to change XFS project quotas. ReflinkXfs needs no capability.
• All executors that can run the same agents enable filesystem snapshots, and use the same blob storage and the same repository_key.
• With S3 blob storage, filesystem_snapshots_bucket is set, names a bucket other than custom_data_bucket, and the bucket exists. The executor does not create buckets. With an S3-compatible service, the service supports conditional writes (If-None-Match: * on PutObject) and CopyObject.
• With file system blob storage, all executors that can run the same agents use the same root.

Enable filesystem snapshots

mkfs.xfs -m reflink=1 -n ftype=1 /dev/<device>
mount -t xfs -o prjquota /dev/<device> /var/lib/golem/agents

/dev/<device>  /var/lib/golem/agents  xfs  prjquota  0 0

[filesystem_storage.mode]
type = "ManagedXfs"
 
[filesystem_storage.mode.config]
root = "/var/lib/golem/agents"

mkfs.xfs -m reflink=1 -n ftype=1 /dev/<device>
mount -t xfs -o noquota /dev/<device> /var/lib/golem/agents

/dev/<device>  /var/lib/golem/agents  xfs  noquota  0 0

chown <executor-user>:<executor-group> /var/lib/golem/agents

[filesystem_storage.mode]
type = "ReflinkXfs"
 
[filesystem_storage.mode.config]
root = "/var/lib/golem/agents"

openssl rand -hex 64

[blob_storage.config]
filesystem_snapshots_bucket = "filesystem-snapshots"

[filesystem_snapshots]
type = "Managed"
 
[filesystem_snapshots.config]
repository_key = "<128 hex characters>"

GOLEM__FILESYSTEM_SNAPSHOTS__TYPE=Managed
GOLEM__FILESYSTEM_SNAPSHOTS__CONFIG__REPOSITORY_KEY=<128 hex characters>

An executor that enables filesystem snapshots on storage without copy-on-write copies, Temporary or Directory, does not start. Its startup fails with this error:

filesystem snapshots require storage with copy-on-write copies (XFS with reflink)

Configuration

All settings except type are in the [filesystem_snapshots.config] section. The executor reads them only when type = "Managed". This is the full set with the default values:

[filesystem_snapshots]
type = "Managed"
 
[filesystem_snapshots.config]
repository_key = "<128 hex characters>"
max_concurrent_uploads = 4
max_concurrent_restores = 8
storage_call_deadline = "60s"
restore_reader_threads = 6
save_threads = 2
confirmation_wait = "60s"
store_check_limit = "5s"
capture_wait = "5s"
retained_periodic_snapshots = 128
retained_update_snapshots = 2
max_pending_deletes_per_agent = 1024
 
[filesystem_snapshots.config.storage_retry]
max_attempts = 5
min_delay = "2s"
max_delay = "2m"
multiplier = 4.0

Write each duration in the humantime  format, for example "500ms", "60s" or "2m". Each count and each wait must be greater than zero. The executor refuses to start with an invalid value and names the setting in the error, for example max_concurrent_restores must be greater than zero or repository_key must be 128 hex characters.

The default values of max_concurrent_restores and restore_reader_threads come from a benchmark on the AWS node type c8gd.4xlarge (16 vCPU, 32 GiB of memory). Another node type needs its own values. For example, on i8g.2xlarge (8 vCPU), the benchmark gave 4 restores with 4 reader threads each, because the CPU limited the restores before the disk did.

Retries

Each snapshot operation (a save, a restore, a delete, a copy or a listing) runs again after a run that failed in blob storage, by the policy in [filesystem_snapshots.config.storage_retry]:

When you set this section, give max_attempts, min_delay, max_delay and multiplier. With the defaults, the waits are 2 s, 8 s, 32 s and 2 minutes. When the last run fails, the operation fails. For an upload, the snapshot record then stays without a confirmation, and a start uses an older snapshot. The error texts of this section start with storage_retry., for example storage_retry.multiplier must be at least 1.

Where the snapshots are

The executor keeps the snapshots of each agent incarnation in their own location. An agent that is deleted and created again with the same id is a new incarnation, and so is the target of a fork. Each gets a new location. A re-created agent starts with an empty one. The location of a fork target holds the copy of the snapshots of its source. All incarnations of one agent share the agent part of the location.

<agent> is the agent name, with each character other than an ASCII letter, a digit, - or _ replaced by _, cut to 32 characters (agent when the name is empty), then - and the BLAKE3 hash, in hex, of <component id>/<agent name>. <incarnation id> is a UUID.

How the executor uses the settings

Captures

At a snapshot boundary, the executor first admits the snapshot. It refuses the snapshot, and the agent’s snapshot code does not run, in these cases:

• The free space or the free filesystem objects of the volume are below the targets of [filesystem_storage.pressure] (target_available_bytes and target_available_filesystem_objects), or the executor cannot read them.
• A delete of all snapshots of the agent waits or runs.
• An upload of the agent, its confirmation or its retention runs. At most one upload runs for each agent. There is an exception: when an automatic upload failed and waits for its next run, also while it waits until its own earlier writes to blob storage have settled, the new snapshot replaces it. The replaced upload never confirms its record. A start can still confirm that record while it is the newest snapshot record of the agent, as described in Uploads and confirmations.

For a manual update, the first two refusals fail the update at once. When an upload of the agent runs, the update waits for the run in flight, and then replaces the upload or waits for its end. All waits of the update end confirmation_wait after the update began. The update then fails.

Then the capture stops the agent’s file calls, waits for the open calls for at most capture_wait, and makes a reflink copy of the files into .scratch. The copy shares the data blocks of the files, so it uses no space when it is made. When the agent then writes to a shared block, that block gets its own copy. Deleted files also keep their blocks until the upload ends and the executor removes the capture.

A guest that returns from an invocation with an unflushed write to a WASI output stream keeps a file call open. Its captures stop after capture_wait, with the outcome busy.

Uploads and confirmations

The executor uploads a capture in the background. When the upload is complete, the running agent writes a confirmation record into its oplog. The stop of an agent does not wait for its upload. When the agent no longer runs at the end of the upload, the executor keeps the snapshot. The next start confirms it:

1. A start on the executor that runs the upload waits for the upload, for at most confirmation_wait, when the upload has started saving: it got its first upload slot. This happens before the start takes an account slot or memory. A terminal interrupt of the agent ends the wait.
2. Then the start checks blob storage once, for at most the rest of confirmation_wait, or for at most store_check_limit when it did not wait. It does not check when the upload on this executor ended with the outcome superseded, or when a terminal interrupt of the agent waits.
3. When blob storage holds the whole snapshot, the start writes the confirmation record itself.

A start on another executor does only steps 2 and 3. An upload that still runs when the agent moves to another executor is lost for that start, and the start uses an older snapshot.

An automatic update of a loaded agent restarts the agent on the same executor. Before the restart, when the update has no strategy yet and its newest snapshot record is not confirmed, the agent waits for the upload of that record, for at most confirmation_wait, so the update can use that record. When no upload of the record runs, the agent checks blob storage once, for at most store_check_limit. A stop of the agent, a retirement of the agent on this executor and a terminal interrupt end the wait. The wait does not count against the time limit of the unload for the restart.

When an update or a revert replaced the snapshot record during the upload, the confirmation finds no record with the name. The executor then deletes the snapshot and counts a dropped confirmation.

Restores

A start restores the filesystem snapshot of its baseline before the agent’s code runs. A restore downloads the whole tree, so a large tree makes the start slower. At most max_concurrent_restores restores run at the same time, each with restore_reader_threads reader threads. On ManagedXfs, a restored tree counts against the storage quota of the agent, as the files did before.

Retention

After an upload confirms an automatic snapshot, the executor deletes the older automatic snapshots of the agent. A confirmation that a start writes does not apply retention; the next upload that confirms applies it. Retention keeps:

• the new snapshot and the retained_periodic_snapshots - 1 newest older ones,
• the snapshots of the records that a start of the agent can select now, whatever their age,
• the snapshots of the successful and pending updates of the agent, manual and snapshot-assisted, and of its authoritative recovery baseline, whatever their age,
• each snapshot whose time is at most 2 minutes before the time of the new snapshot, or later, because the clocks of two executors can differ.

The last three kinds do not count toward retained_periodic_snapshots.

After a manual update record is written, the executor applies the same rule to the manual-update snapshots with retained_update_snapshots. It never deletes the snapshot of a successful or pending update of the agent, because a revert or a fork can make that update the baseline again. retained_update_snapshots limits only the other manual-update snapshots.

A failed retention delete is logged, and the next retention tries again.

Clean-up

The executor deletes filesystem snapshots in the background when the life of an agent ends or changes:

• Delete. After it deletes the oplog of an agent, the executor deletes all filesystem snapshots of that incarnation. It first stops the upload of the agent, and waits until every snapshot operation of the agent has ended.
• Revert. A revert first writes its revert entry. Then the executor deletes the snapshots of the records in the reverted part of the oplog, except a snapshot that a remaining record uses. When another executor took over the agent before the revert entry was written, the revert fails with OplogFenced and deletes nothing.
• Fork. The executor copies the snapshots of the source incarnation into the location of the new agent, inside blob storage, before it creates the new agent. With S3 and SQLite, the data does not pass through the executor. With file system blob storage, the executor copies the files. The executor checks the last successful snapshot-based manual update or snapshot-assisted automatic update in the part of the oplog that the fork copies. When that update names a filesystem snapshot that is not in blob storage, the fork is refused: Cannot fork worker at oplog index <index>: the filesystem snapshot <name> of the target's baseline is not in the store. An executor without filesystem snapshots copies nothing, so it refuses such a fork too. A fork attempt that does not create the new agent deletes its copy.

The clean-ups wait in a queue in the memory of the executor. At most max_concurrent_uploads clean-ups run at the same time. Retention does not use this queue. These deletes do: the deletes of a revert, the delete of all snapshots of a deleted agent, the delete of all snapshots of an incarnation that the recovery of a shard finds stale, and the delete of the copy of a fork attempt that did not create its new agent.

Only the snapshots of a revert wait in the queue by name. Retention deletes directly, and the queue does not limit it. A delete of all snapshots carries no names. One agent can have at most max_pending_deletes_per_agent snapshots of a revert waiting for deletion. The snapshots that wait keep their place. A revert gives its snapshots newest first. The executor queues them up to the bound, refuses the rest, which are the oldest snapshots of the reverted part, and counts a leaked clean-up with the operation overflow. The refused snapshots stay in the location of the agent. Retention deletes a refused automatic snapshot after the agent has made about retained_periodic_snapshots newer automatic snapshots, and a refused manual-update snapshot after more manual updates. The delete of the agent also deletes them.

The queue also has fixed bounds over all agents: 65,536 agents with waiting work, and 786,432 waiting snapshots. A revert past these bounds is also counted as overflow. When 65,536 agents have waiting work, a delete of all snapshots of an agent removes the oldest waiting revert deletes of another agent to make room, and counts them as overflow. Only when no such deletes wait is the delete of all snapshots refused, and counted as overflow.

Snapshots can stay in blob storage. They use space. They do not make a start wrong:

• A clean-up delete, or the delete of a snapshot that no confirmation record names, that fails after its retries. The executor counts it in filesystem_snapshot_leaked_cleanups_total.
• A clean-up that waited or ran when the executor shut down. The executor counts it with the operation shutdown. After a crash, nothing counts the lost clean-ups.
• The copy of a fork attempt whose outcome the executor does not know. The executor counts it with the operation fork_stage.

To find what stays, list the locations in Where the snapshots are. An incarnation whose agent no longer exists holds only snapshots that no agent uses.

Memory

Each restore holds memory for each of its reader threads, and each upload holds memory while it saves. This memory is outside the memory that the executor gives to agents. worker_memory_ratio sets the share for agents, and its default is 0.8. The rest must hold the executor itself and the uploads and restores.

The benchmark setup on the node type c8gd.4xlarge (16 vCPU, 32 GiB of memory) used a pod limit of about 28 GiB and this share for agents:

[memory]
worker_memory_ratio = 0.78

In that setup, 4 uploads and 8 restores with the default thread counts needed about 6 GiB outside the memory of the agents. Measure again for another node type or for other values of the settings.

Metrics

The worker executor exports these metrics for filesystem snapshots. The labels have fixed values and hold no agent identifiers. See Metrics for the other metrics.

Watch these values:

• Busy captures. filesystem_snapshot_capture_seconds_count{outcome="busy"} counts the snapshots that Golem skipped, or the manual updates that failed, because a guest left a file call open.
• Changed files without filesystem snapshots. On an executor without filesystem snapshots, filesystem_snapshot_capture_seconds_count{outcome="changed"} counts the snapshots that Golem skipped, or the manual updates that failed, because the agent’s files are not its initial files.
• Failed uploads. filesystem_snapshot_upload_seconds_count{outcome="failed"} counts uploads that failed after their retries.
• Deferred uploads. filesystem_snapshot_upload_seconds_count{outcome="deferred"} counts uploads that the instance that made the capture could not confirm, for example because the agent stopped or restarted, or its shard moved. This is normal for agents that stop soon after a snapshot. A later start confirms them.
• Dropped confirmations. filesystem_snapshot_dropped_confirmations_total counts uploads that an update or a revert made useless.
• Leaked clean-ups. filesystem_snapshot_leaked_cleanups_total counts clean-ups that left snapshots in blob storage. A growing overflow count means that deletes come faster than the clean-ups run. A shutdown count shows clean-ups that a shutdown cut off.
• Pending clean-ups. A filesystem_snapshot_cleanups_pending value that grows all the time means that the clean-ups do not keep up.
• Unused storage. filesystem_snapshot_failed_space_reclaims_total counts deletes that did not give back their storage yet.
• Restore time. filesystem_snapshot_restore_seconds shows how much a restore adds to the start of an agent, with the wait for a restore slot.

Logs

The executor writes these messages. Search for the quoted text.