Updating agents to newer versions of components

As described in the general Agents page, each agent runs on a specific version of the component it is based on, but it is possible to update an agent to a different version of the same component.

Automatic update

The automatic update mode works as described in the general Agents page.

golem agent update <AGENT_ID> auto --await

Golem restarts the agent at once. When the agent is loaded and an upload of its newest automatic snapshot is in progress, the agent first waits for that upload, for at most a limit that the operator sets, 60 seconds by default. A stop or an interrupt of the agent ends the wait earlier. Then the update can use that snapshot. When the update gets to the head of the update queue, Golem selects the strategy of the update and writes it into the oplog before the update runs. After that, the strategy does not change. A restart or a retry of the update uses the same strategy.

The replay continues through the source history that the agent committed before it stopped. It does not stop at the update request. So the work that the agent committed after the request stays, and an invocation that was in progress continues on the target revision. --await returns only after the update writes SuccessfulUpdate or FailedUpdate. Golem writes SuccessfulUpdate before the target revision does live work.

The snapshot-assisted strategy

When the agent has a usable automatic snapshot, the update starts from it and replays only the history after it. Golem looks at two automatic snapshot records: the last record, and the usable record before it. It selects the first record that obeys all these rules:

• The record is usable: a confirmation record follows it, or it has no filesystem snapshot name.
• The record has the current revision of the agent, and the target revision is newer than the current revision.
• Golem did not reject the record. Golem rejects a record when a start without a pending update could not load its snapshot or diverged in the replay after it, and when an update found that the store lost its snapshot.
• The record is before each snapshot-based manual update that waits in the update queue.
• The record has no filesystem snapshot name, or the executor has filesystem snapshots.
• No earlier update to the same target revision from the same source failed with UPDATE_SNAPSHOT_INCOMPATIBLE. See Failure codes.

The record can be newer than the update request. When no record obeys the rules, the strategy is a full replay. The target revision then replays all the history after the authoritative recovery baseline: the snapshot of the last successful manual update or snapshot-assisted update, or the start of the oplog.

How the update restores the agent’s files

An application snapshot does not hold the agent’s files. So the update uses the application snapshot of a record only together with the files of the same record:

1. Golem restores the filesystem snapshot that the record names. A record without a filesystem snapshot name gives the initial files of the source revision, because Golem writes such a record only while the agent’s files are its initial files.
2. The target revision loads the application snapshot of the record.
3. The target revision replays the history after the record. During this replay, the agent sees the configuration, the environment and the initial files of the source revision, as during a full replay.
4. At the update point, Golem applies the initial files of the target revision, as a full replay does. If the target revision changes the declaration of an initial file at a path where the agent wrote, changed or deleted a file, the update fails. The rule is the same as for a manual update.

After a successful update, the selected record is the authoritative recovery baseline. A later start can use a newer automatic snapshot. When the start cannot use that snapshot, it restores the files of the selected record, loads its application snapshot and replays the history after it, in the same order as the update. It never replays the history before the selected record. Golem does not delete the filesystem snapshot of the selected record while the successful update stays in the oplog.

A revert that removes a successful or failed outcome but keeps the update request makes the same request pending again, with the same strategy.

The snapshot-assisted strategy limits the replayed history to the history after the selected record. It does not limit the time of the update: the restore of the files, the load, the replay, the compilation of the target revision and the wait for resources take time.

When an automatic update fails

When an attempt fails, Golem writes one FailedUpdate and starts the agent again on the source revision. This is also true when a full replay diverges at a host call: the update fails with UPDATE_REPLAY_FAILED, and the agent runs on its source revision. There is one exception. A full replay starts from the authoritative recovery baseline. When the filesystem snapshot of that baseline is lost, does not restore, or needs filesystem snapshots that the executor does not have, the start of the agent fails with a visible cause, as every start from that baseline does, and Golem writes no FailedUpdate. In the same attempt, Golem does not try another snapshot record, and it does not change to a full replay. The details of some failures start with a code that tells you what to do next. See Failure codes.

Some failures do not fail the update. When the component service does not answer, when blob storage has a temporary error, or when the executor has no free memory for the agent, Golem writes no FailedUpdate. The update stays pending, and Golem tries the start again with the same strategy.

While the start replays the history for a pending automatic update, the agent keeps the status that it had before the restart. An idle agent keeps the status Idle, so Golem ignores an interrupt of the agent during that replay. Golem writes no Interrupted entry and no FailedUpdate, and the update completes. You do not need to resume the agent. When another executor takes over the agent during the replay, that executor runs the update attempt again.

Manual snapshot-based update

It is also possible to manually implement a pair of functions for saving and loading an agent’s state into an arbitrary byte array. This is useful when making significant changes to the implementation of an agent, such that the automatic replay based update mechanism is no longer working.

snapshot: {
  save() {
    const bytes = new Uint8Array(8)
    new DataView(bytes.buffer).setFloat64(0, this.value)
    return bytes
  },
  load(bytes, _context) {
    return {
      value: new DataView(
        bytes.buffer,
        bytes.byteOffset,
        bytes.byteLength,
      ).getFloat64(0),
    }
  },
}

import { Effect, Ref } from "effect"
import { Snapshot } from "@golemcloud/effect-golem"
 
snapshotting: Snapshot.custom({
  policy: Snapshot.policy.manual,
})
 
// In .implement({ ... }):
snapshot: {
  save: (state) =>
    Ref.get(state).pipe(
      Effect.map((value) => {
        const bytes = new Uint8Array(8)
        new DataView(bytes.buffer).setFloat64(0, value)
        return bytes
      }),
    ),
  restore: (bytes, _context) =>
    Ref.make(
      new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength)
        .getFloat64(0),
    ),
}

async fn save_snapshot(&self) -> Result<Vec<u8>, String> {
    Ok(self.value.to_le_bytes().to_vec())
}
 
async fn load_snapshot(
    bytes: Vec<u8>,
    _context: golem_rust::agentic::SnapshotRestoreContext,
) -> Result<Self, String> {
    let arr: [u8; 4] = bytes
        .try_into()
        .map_err(|_| "Expected a 4-byte snapshot".to_string())?;
    Ok(Self {
        value: u32::from_le_bytes(arr),
    })
}

import golem.runtime.SnapshotRestoreContext
import golem.runtime.annotations.agentImplementation
 
import scala.concurrent.Future
 
@agentImplementation()
final class CounterImpl(private val name: String) extends Counter {
  private var value: Int = 0
 
  // Counter methods ...
 
  def saveSnapshot(): Future[Array[Byte]] = Future.successful {
    val buffer = java.nio.ByteBuffer.allocate(4)
    buffer.putInt(value)
    buffer.array()
  }
}
 
object CounterImpl {
  def loadSnapshot(
    bytes: Array[Byte],
    context: SnapshotRestoreContext
  ): Future[CounterImpl] = Future.successful {
    val instance = new CounterImpl(context.identity[String](0))
    instance.value = java.nio.ByteBuffer.wrap(bytes).getInt()
    instance
  }
}

pub impl @agents.Snapshottable for Counter with save_snapshot(self) {
  let arr: Array[Byte] = []
  for i = 7; i >= 0; i = i - 1 {
    arr.push(((self.value >> (i * 8)).to_int() & 0xff).to_byte())
  }
  Bytes::from_array(arr)
}
 
pub fn Counter::load_snapshot(
  bytes: Bytes,
  context: @agents.SnapshotRestoreContext,
) -> Result[Counter, String] {
  if bytes.length() != 8 {
    return Err("Expected an 8-byte snapshot")
  }
  let mut value: UInt64 = 0
  for i = 0; i < 8; i = i + 1 {
    value = value | (bytes[i].to_uint64() << ((7 - i) * 8))
  }
  let name: String = context.identity(0) catch {
    error => return Err(error.to_string())
  }
  Ok({ name, value })
}

After making a change that requires a manual update, you need to build and deploy the updated component. Then, to update a specific agent, run the following command:

golem agent update <AGENT_ID> manual

Alternatively, you can update all agents when deploying the updated component:

golem deploy --update-agents manual

When a manual update fails

The target revision loads the snapshot when the agent starts on it. Golem writes SuccessfulUpdate only after the load is complete. When the load returns an error, Golem writes one FailedUpdate and starts the agent again on its source revision. This is also true when the agent exits during the load: the details of the FailedUpdate say that the agent exited during the snapshot load.

When an interrupt stops the load, for example a suspension of the agent or a move of the agent to another executor, Golem writes no FailedUpdate and no SuccessfulUpdate. The update stays pending, and the next start of the agent loads the snapshot again.

Failure codes

When an update fails, Golem writes a FailedUpdate entry. For the causes below, the details of the entry start with a code. The cause of the failure follows the code and its text. golem agent get <AGENT_ID> shows the details in the update history of the agent, and golem agent oplog <AGENT_ID> shows them as the error of the FAILED UPDATE entry. In each case, the agent stays on its source revision.

Other failures have no code. Their details give the cause, for example a change of the agent mode in the target revision, a conflict of the initial files at the update point, or a failure of the target revision to start.