Control durability guarantees from Python

Golem provides a set of functions components can call to control details of the durable execution engine.

Adding as a dependency

To use Golem's runtime API, add all the WIT files from https://github.com/golemcloud/golem-wit/tree/main/wit/deps (opens in a new tab) in the project's wit/deps directory, and then importing this interface to the component's world:

world example {
    import golem:api/host@1.1.0;
    // ...
}

The golem-cli new examples for Python automatically create the wit/deps directory for you, except the py-actor-minimal one.

General concepts

The library allows controlling four main aspects of the durable execution engine: the current persistence level, the idempotence mode, defining atomic regions and changing retry policies (discussed in the next page).

All these features are regional - they can be changed for a section of the code within a single exported function.

As in Python we are directly invoking the runtime API's functions, these regions must be enforced by the user manually.

Persistence level

The persistence level can be one of the following:

Level	Description
`PersistenceLevel_PersistNothing`	Turns off persistence for a section. In case the worker is recovered or restarted, all the side-effecting functions will be reexecuted
`PersistenceLevel_PersistRemoteSideEffects`	Persists all the side-effects that are affecting the outside world. In case of recovery the side-effects won't be reexecuted and the persisted results will be used.
`PersistenceLevel_Smart`	The default setting; Let Golem decide what to persist to optimize performance

To change the persistence level for a section of the code, use the get_oplog_persistence_level function to get the current one, and the set_oplog_persistence_level function to set a new value:

import py_example
import py_example.imports
import py_example.imports.types
import py_example.imports.host
import py_example.exports
import py_example.exports.api
from py_example.imports.host import get_oplog_persistence_level, set_oplog_persistence_level, PersistenceLevel_PersistNothing
 
old = get_oplog_persistence_level()
try:
    set_oplog_persistence_level(PersistenceLevel_PersistNothing())
    # ...
finally:
    set_oplog_persistence_level(old)

Idempotence mode

Golem assumes that HTTP requests are idempotent by default. This means that in case of a failure, if the system cannot determine whether the request successfully reached the target or not, it will be retried. This behavior can be changed using the set_idempotence_mode function:

import py_example
import py_example.imports
import py_example.imports.types
import py_example.imports.host
import py_example.exports
import py_example.exports.api
from py_example.imports.host import get_idempotence_mode, set_idempotence_mode
 
old = get_idempotence_mode()
try:
    set_idempotence_mode(false)
    # ...
finally:
    set_idempotence_mode(true)

With disabled idempotence mode, in case Golem cannot determine if the request was sent or not, it won't retry it but the worker will fail.

Atomic regions

By default side effects are persisted and retried one by one. It is possible to group them together into atomic regions, in which case the execution is retried for some reason (the worker failed or interrupted within the region), all the side effects will be reexecuted.

To define an atomic region, use the mark_begin_operation and mark_end_operation functions:

import py_example
import py_example.imports
import py_example.imports.types
import py_example.imports.host
import py_example.exports
import py_example.exports.api
from py_example.imports.host import mark_begin_operation, mark_end_operation
 
begin = mark_begin_operation()
try:
    # ...
finally:
    mark_end_operation(begin)

Commit oplog

The oplog_commit function waits until the oplog is committed to its persistent storage. The function takes a single argument, replicas, with the desired number of storage replicas the worker's journal is replicated to. The function will block until the oplog is committed to the specified number of replicas, or, if this number is larger than the available number of replicas, until it is written to all the replicas.

HTTP client Retries