Control the retry policy

Using Golem’s retry mechanism

Golem applies retry policies to retryable agent failures. The selected policy controls delays and retry limits. Permanent errors, permission failures, stack overflows, and deterministic traps outside atomic regions are not retried.

If the maximum number of retries is reached, the agent will be marked as failed and no further invocations will be possible on it.

During recovery, Golem replays the oplog and reuses results already recorded for preceding durable operations. Raising an unhandled exception does not cause those earlier operations to execute again.

Host retry policies govern retryable Golem operations and agent failures. They do not retry an arbitrary typed failure inside an SDK’s local effect abstraction; for example, Retry.withPolicy propagates Effect.fail.

Customizing the retry policy

The retry policy which controls the maximum number of retries and the exponential backoff is a global configuration of the Golem servers, but it can be customized for each agent.

TypeScript

The @golemcloud/golem-ts-sdk package exports the withRetryPolicy function to temporarily install a named retry policy for a block of code:


import {withRetryPolicy, type NamedRetryPolicy} from "@golemcloud/golem-ts-sdk"
 
// Durations are expected as nanoseconds
const policy: NamedRetryPolicy = {
  name: "temporary-policy",
  priority: 1000,
  predicate: {
    nodes: [{tag: "pred-true"}],
  },
  policy: {
    nodes: [
      {
        tag: "count-box",
        val: {
          maxRetries: 3,
          inner: 1,
        },
      },
      {
        tag: "clamp-delay",
        val: {
          minDelay: 100_000_000n, // 100 milliseconds
          maxDelay: 2_000_000_000n, // 2 seconds
          inner: 2,
        },
      },
      {
        tag: "exponential",
        val: {
          baseDelay: 100_000_000n,
          factor: 1.5,
        },
      },
    ],
  },
}
 
const result: string = withRetryPolicy(policy, () => {
  // this closure runs with the custom retry policy
  return "hello"
})

The policy passed to withRetryPolicy is a NamedRetryPolicy, which contains a name, priority, predicate, and policy tree:


export type RetryPredicate = {
  nodes: PredicateNode[];
};
 
export type RetryPolicy = {
  nodes: PolicyNode[];
};
 
export type NamedRetryPolicy = {
  name: string;
  priority: number;
  predicate: RetryPredicate;
  policy: RetryPolicy;
};

Effect TypeScript

The Effect SDK provides a typed retry-policy DSL and Retry.withPolicy, which installs a policy for one Effect. It restores the previous policy with the same name when its scope closes, or removes the temporary policy if none existed:


import { Duration, Effect } from "effect"
import { Retry } from "@golemcloud/effect-golem"
 
const policy = Retry.NamedPolicy.named(
  "temporary-policy",
  Retry.Policy.exponential(Duration.millis(100), 1.5)
    .clamp(Duration.millis(100), Duration.seconds(2))
    .maxRetries(3),
).priority(1000)
 
const program = Retry.withPolicy(
  policy,
  Effect.succeed("hello"),
)

Use Retry.setPolicy(policy) instead when the policy should remain installed on the durable agent until it is overwritten or removed.

Rust

The golem_rust crate exports the with_retry_policy function to temporarily install a named retry policy for a block of code:


use golem_rust::bindings::golem::api::retry::{
    ClampConfig, CountBoxConfig, ExponentialConfig, PolicyNode, PredicateNode,
};
use golem_rust::retry::{with_retry_policy, NamedRetryPolicy, RetryPolicy, RetryPredicate};
 
let policy = NamedRetryPolicy {
    name: "temporary-policy".to_string(),
    priority: 1000,
    predicate: RetryPredicate {
        nodes: vec![PredicateNode::PredTrue],
    },
    policy: RetryPolicy {
        nodes: vec![
            PolicyNode::CountBox(CountBoxConfig {
                max_retries: 3,
                inner: 1,
            }),
            PolicyNode::ClampDelay(ClampConfig {
                min_delay: 100_000_000u64, // 100 milliseconds
                max_delay: 2_000_000_000u64, // 2 seconds
                inner: 2,
            }),
            PolicyNode::Exponential(ExponentialConfig {
                base_delay: 100_000_000u64,
                factor: 1.5,
            }),
        ],
    },
};
 
with_retry_policy(policy, || {
    "hello".to_string()
})

The policy passed to with_retry_policy is a NamedRetryPolicy, which contains a name, priority, predicate, and policy tree:


pub struct RetryPredicate {
    pub nodes: Vec<PredicateNode>,
}
 
pub struct RetryPolicy {
    pub nodes: Vec<PolicyNode>,
}
 
pub struct NamedRetryPolicy {
    pub name: String,
    pub priority: u32,
    pub predicate: RetryPredicate,
    pub policy: RetryPolicy,
}

Scala

In Scala, golem.Guards exposes withRetryPolicy to temporarily install a named retry policy for a Future-returning block:


import golem.Guards
import golem.host.js.JsNamedRetryPolicy
 
import scala.concurrent.Future
import scala.scalajs.js
 
val policy = js.Dynamic
  .literal(
    "name" -> "temporary-policy",
    "priority" -> 1000,
    "predicate" -> js.Dynamic.literal(
      "nodes" -> js.Array(
        js.Dynamic.literal("tag" -> "pred-true")
      )
    ),
    "policy" -> js.Dynamic.literal(
      "nodes" -> js.Array(
        js.Dynamic.literal(
          "tag" -> "count-box",
          "val" -> js.Dynamic.literal(
            "maxRetries" -> 3,
            "inner" -> 1
          )
        ),
        js.Dynamic.literal(
          "tag" -> "clamp-delay",
          "val" -> js.Dynamic.literal(
            "minDelay" -> js.BigInt("100000000"),
            "maxDelay" -> js.BigInt("2000000000"),
            "inner" -> 2
          )
        ),
        js.Dynamic.literal(
          "tag" -> "exponential",
          "val" -> js.Dynamic.literal(
            "baseDelay" -> js.BigInt("100000000"),
            "factor" -> 1.5
          )
        )
      )
    )
  )
  .asInstanceOf[JsNamedRetryPolicy]
 
val result: Future[String] =
  Guards.withRetryPolicy(policy) {
    Future.successful("hello")
  }

The policy passed to Guards.withRetryPolicy is a JsNamedRetryPolicy facade:


import golem.host.js.{JsNamedRetryPolicy, JsRetryPolicyTree, JsRetryPredicate}
 
import scala.scalajs.js
 
@js.native
sealed trait JsNamedRetryPolicy extends js.Object {
  def name: String = js.native
  def priority: Double = js.native
  def predicate: JsRetryPredicate = js.native
  def policy: JsRetryPolicyTree = js.native
}

MoonBit

Assuming the golemcloud/golem_sdk/api package is imported as @api, use the high-level policy builders and with_named_policy:


let policy = @api.NamedPolicy::named(
  "temporary-policy",
  @api.Policy::exponential(@api.Duration::millis(100), 1.5)
    .clamp(@api.Duration::millis(100), @api.Duration::seconds(2))
    .max_retries(3),
).priority(1000)
 
let result : String = @api.with_named_policy(policy, fn() {
  // this callback runs with the custom retry policy
  "hello"
}) catch {
  error => raise error
}

Named retry policies in golem.yaml

Retry policies can also be defined declaratively in the application manifest (golem.yaml). These named retry policies are created in the environment during deployment, making them available to all agents without requiring code changes.


retryPolicyDefaults:
  local:
    default-retry:
      priority: 10
      predicate: true
      policy:
        countBox:
          maxRetries: 3
          inner:
            exponential:
              baseDelay: { secs: 1, nanos: 0 }
              factor: 2.0

Manifest policy types reference

The following policy types can be used to construct retry policy trees:

Type	Fields	Description
`"immediate"`	—	Retry immediately
`"never"`	—	Never retry
`periodic`	`{ secs, nanos }`	Fixed delay between retries
`exponential`	`{ baseDelay, factor }`	Exponentially increasing delay
`fibonacci`	`{ first, second }`	Fibonacci-sequence delays
`countBox`	`{ maxRetries, inner }`	Limit total retry count
`timeBox`	`{ limit, inner }`	Limit total retry time
`clamp`	`{ minDelay, maxDelay, inner }`	Clamp delay range
`addDelay`	`{ delay, inner }`	Add fixed delay to inner policy
`jitter`	`{ factor, inner }`	Add random jitter
`filteredOn`	`{ predicate, inner }`	Apply only when predicate matches
`andThen`	`[policy1, policy2]`	Sequential composition
`union`	`[policy1, policy2]`	Union of two policies
`intersect`	`[policy1, policy2]`	Intersection of two policies

Manifest predicates reference

Predicates control when a retry policy is applied. The following predicate types are available:

Type	Fields	Description
`"true"`	—	Always matches
`"false"`	—	Never matches
`propEq`	`{ property, value }`	Property equals value
`propNeq`	`{ property, value }`	Property not equal
`propGt` / `propGte`	`{ property, value }`	Greater than / greater or equal
`propLt` / `propLte`	`{ property, value }`	Less than / less or equal
`propExists`	string	Property exists
`propIn`	`{ property, values }`	Property in set
`propMatches`	`{ property, pattern }`	Glob match
`propStartsWith`	`{ property, prefix }`	Starts with prefix
`propContains`	`{ property, substring }`	Contains substring
`and`	`[pred1, pred2]`	Logical AND
`or`	`[pred1, pred2]`	Logical OR
`not`	predicate	Logical NOT

Predicate values in the manifest are strings, integers, or booleans. For example:


predicate:
  propEq:
    property: status-code
    value: 503