Golem Host functions
Golem provides three WIT packages of host functions for accessing Golem-specific functionality on top of the standard WebAssembly system interfaces.
These three packages are:
golem:api@1.1.6
- provides access to various aspects of Golem for the workersgolem:rpc@0.2.0
- defines the types and functions serving the Worker to Worker communication in Golemgolem:durability@1.2.0
- provides an API to implement custom durability in libraries
Please check the language specific guidelines to learn the best way to work with these APIs in your language of choice:
This page provides an overview of all the exported functions and types in a language-agnostic way.
Types
An oplog index points to a specific instruction in the worker's history:
package golem:api@1.1.6;
interface host {
/// An index into the persistent log storing all performed operations of a worker
type oplog-index = u64;
}
The worker id uniquely identifies a worker in the Golem system:
package golem:rpc@0.2.0;
interface types {
/// Represents a Golem worker
record worker-id {
component-id: component-id,
worker-name: string
}
}
Components are identified by a component id which is in fact a UUID:
package golem:rpc@0.2.0;
interface types {
/// Represents a Golem component
record component-id {
uuid: uuid,
}
/// UUID
record uuid {
high-bits: u64,
low-bits: u64
}
}
Component versions are unsigned integers:
package golem:api@1.1.6;
interface host {
/// Represents a Golem component's version
type component-version = u64;
}
Promises
Types
A promise is identified by a promise-id
type consisting of the owner worker's identifier and a number that points to
the create promise function in the worker's history:
package golem:api@1.1.6;
interface host {
/// A promise ID is a value that can be passed to an external Golem API to complete that promise
/// from an arbitrary external source, while Golem workers can await for this completion.
record promise-id {
worker-id: worker-id,
oplog-idx: oplog-index,
}
}
Functions
The following functions define the promise API:
Function name | Definition | Description |
---|---|---|
create-promise | func() -> promise-id | Creates a new promise and returns its ID |
await-promise | func(promise-id) -> list<u8> | Suspends execution until the given promise gets completed, and returns the payload passed to the promise completion. |
complete-promise | func(promise-id, list<u8>) | Completes the given promise with the given payload. Returns true if the promise was completed, false if the promise was already completed. The payload is passed to the worker that is awaiting the promise. |
delete-promise | func(promise-id) -> () | Deletes the given promise. |
Worker metadata
Types
Worker metadata is described by the following WIT types:
package golem:api@1.1.6;
interface host {
record worker-metadata {
worker-id: worker-id,
args: list<string>,
env: list<tuple<string, string>>,
status: worker-status,
component-version: u64,
retry-count: u64
}
enum worker-status {
/// The worker is running an invoked function
running,
/// The worker is ready to run an invoked function
idle,
/// An invocation is active but waiting for something (sleeping, waiting for a promise)
suspended,
/// The last invocation was interrupted but will be resumed
interrupted,
/// The last invocation failed and a retry was scheduled
retrying,
/// The last invocation failed and the worker can no longer be used
failed,
/// The worker exited after a successful invocation and can no longer be invoked
exited,
}
}
Functions
The following functions define the worker metadata API:
Function name | Definition | Description |
---|---|---|
get-worker-metadata | func(worker-id) -> option<worker-metadata> | Returns the metadata of a worker |
get-self-metadata | func() -> worker-metadata | Returns the metadata of the worker that calls this function |
Worker enumeration
The worker enumeration API allows listing all the workers belonging to a specific component. This is a slow operation and should be used for maintenance tasks and not for the core business logic.
Worker enumeration is a WIT resource, providing a method that returns a page of workers each time it is called:
package golem:api@1.1.6;
interface host {
resource get-workers {
constructor(component-id: component-id, filter: option<worker-any-filter>, precise: bool);
get-next: func() -> option<list<worker-metadata>>;
}
}
Once get-next
returns none
, the enumeration is done.
There are two parameters for customizing the worker enumeration:
Filters
An optional filter
parameter can be passed to the worker enumeration, with the following definition:
package golem:api@1.1.6;
interface host {
record worker-any-filter {
filters: list<worker-all-filter>
}
}
The worker-any-filter
matches workers that satisfy any of the listed filters.
package golem:api@1.1.6;
interface host {
record worker-all-filter {
filters: list<worker-property-filter>
}
}
A worker-all-filter
matches workers only that satisfy all of the listed filters.
package golem:api@1.1.6;
interface host {
variant worker-property-filter {
name(worker-name-filter),
status(worker-status-filter),
version(worker-version-filter),
created-at(worker-created-at-filter),
env(worker-env-filter)
}
}
The worker-name-filter
matches workers by one of their properties, such as the worker name, status, version, creation date or environment variables.
Each of these variants take a filter record holding a value and a comparator:
package golem:api@1.1.6;
interface host {
record worker-name-filter {
comparator: string-filter-comparator,
value: string
}
record worker-status-filter {
comparator: filter-comparator,
value: worker-status
}
record worker-version-filter {
comparator: filter-comparator,
value: u64
}
record worker-created-at-filter {
comparator: filter-comparator,
value: u64
}
record worker-env-filter {
name: string,
comparator: string-filter-comparator,
value: string
}
}
Where filter-comparator
and string-filter-comparator
are defined as:
package golem:api@1.1.6;
interface host {
enum filter-comparator {
equal,
not-equal,
greater-equal,
greater,
less-equal,
less
}
enum string-filter-comparator {
equal,
not-equal,
like,
not-like
}
}
Precise enumeration
The precise
flag switches the worker enumeration into a more costly variant which guarantees that every worker metadata returned is the latest one available at the time the result was generated.
When precise
is set to false
, the enumeration returns the last cached worker metadata available for each worker, which may be lagging behind the actual state of the workers.
Transactions and persistence control
Golem's transaction API allows customizing the execution engine's durability and transactional behaviors. These are low level functions, which are wrapped by SDKs providing a higher level transaction API for each supported language.
Types
Retry policy is defined by the following record:
package golem:api@1.1.6;
interface host {
/// Configures how the executor retries failures
record retry-policy {
/// The maximum number of retries before the worker becomes permanently failed
max-attempts: u32,
/// The minimum delay between retries (applied to the first retry)
min-delay: duration,
/// The maximum delay between retries
max-delay: duration,
/// Multiplier applied to the delay on each retry to implement exponential backoff
multiplier: f64,
/// The maximum amount of jitter to add to the delay
max-jitter-factor: option<f64>
}
}
It is possible to switch between persistence modes:
package golem:api@1.1.6;
interface host {
/// Configurable persistence level for workers
variant persistence-level {
persist-nothing,
persist-remote-side-effects,
smart
}
}
Functions
The following functions define the transaction API:
Function name | Definition | Description |
---|---|---|
get-oplog-index | func() -> oplog-index | Returns the current position in the persistent op log |
set-oplog-index | func(oplog-idx: oplog-index) -> () | Makes the current worker travel back in time and continue execution from the given position in the persistent op log. |
oplog-commit | func(replicas: u8) -> () | Blocks the execution until the oplog has been written to at least the specified number of replicas, or the maximum number of replicas if the requested number is higher. |
mark-begin-operation | func() -> oplog-index | Marks the beginning of an atomic operation. In case of a failure within the region selected by mark-begin-operation and mark-end-operation , the whole region will be reexecuted on retry. The end of the region is when mark-end-operation is called with the returned oplog-index. |
mark-end-operation | func(begin: oplog-index) -> () | Commits this atomic operation. After mark-end-operation is called for a given index, further calls with the same parameter will do nothing. |
get-retry-policy | func() -> retry-policy | Gets the current retry policy associated with the worker |
set-retry-policy | func(policy: retry-policy) -> () | Overrides the current retry policy associated with the worker. Following this call, get-retry-policy will return the new retry policy. |
get-oplog-persistence-level | func() -> persistence-level | Gets the worker's current persistence level. |
set-oplog-persistence-level | func(new-persistence-level: persistence-level) -> () | Sets the worker's current persistence level. This can increase the performance of execution in cases where durable execution is not required. |
get-idempotence-mode | func() -> bool | Gets the worker's current idempotence mode. |
set-idempotence-mode | func(idempotent: bool) -> () | Sets the current idempotence mode. The default is true. True means side-effects are treated idempotent and Golem guarantees at-least-once semantics. In case of false the executor provides at-most-once semantics, failing the worker in case it is not known if the side effect was already executed. |
generate-idempotency-key | func() -> uuid | Generates a new idempotency key. |
Update
The update API enables workers to trigger automatic or manual update of other workers.
Types
Automatic and manual updates are distinguished by the following enum:
package golem:api@1.1.6;
interface host {
/// Describes how to update a worker to a different component version
enum update-mode {
/// Automatic update tries to recover the worker using the new component version
/// and may fail if there is a divergence.
automatic,
/// Manual, snapshot-based update uses a user-defined implementation of the `save-snapshot` interface
/// to store the worker's state, and a user-defined implementation of the `load-snapshot` interface to
/// load it into the new version.
snapshot-based
}
}
Functions
The following function defines the update API:
Function name | Definition | Description |
---|---|---|
update-worker | func(worker-id: worker-id, target-version: component-version, mode: update-mode) -> () | Initiates an update attempt for the given worker. The function returns immediately once the request has been processed, not waiting for the worker to get updated. |
Oplog search and query
The oplog
interface in golem:api
provides functions to search and query the worker's persisted oplog.
The interface defines a big variant
data type called oplog-entry
:
package golem:api@1.1.6;
interface host {
variant oplog-entry {
/// The initial worker oplog entry
create(create-parameters),
/// The worker invoked a host function
imported-function-invoked(imported-function-invoked-parameters),
/// The worker has been invoked
exported-function-invoked(exported-function-invoked-parameters),
/// The worker has completed an invocation
exported-function-completed(exported-function-completed-parameters),
/// Worker suspended
suspend(datetime),
/// Worker failed
error(error-parameters),
/// Marker entry added when get-oplog-index is called from the worker, to make the jumping behavior
/// more predictable.
no-op(datetime),
/// The worker needs to recover up to the given target oplog index and continue running from
/// the source oplog index from there
/// `jump` is an oplog region representing that from the end of that region we want to go back to the start and
/// ignore all recorded operations in between.
jump(jump-parameters),
/// Indicates that the worker has been interrupted at this point.
/// Only used to recompute the worker's (cached) status, has no effect on execution.
interrupted(datetime),
/// Indicates that the worker has been exited using WASI's exit function.
exited(datetime),
/// Overrides the worker's retry policy
change-retry-policy(change-retry-policy-parameters),
/// Begins an atomic region. All oplog entries after `BeginAtomicRegion` are to be ignored during
/// recovery except if there is a corresponding `EndAtomicRegion` entry.
begin-atomic-region(datetime),
/// Ends an atomic region. All oplog entries between the corresponding `BeginAtomicRegion` and this
/// entry are to be considered during recovery, and the begin/end markers can be removed during oplog
/// compaction.
end-atomic-region(end-atomic-region-parameters),
/// Begins a remote write operation. Only used when idempotence mode is off. In this case each
/// remote write must be surrounded by a `BeginRemoteWrite` and `EndRemoteWrite` log pair and
/// unfinished remote writes cannot be recovered.
begin-remote-write(datetime),
/// Marks the end of a remote write operation. Only used when idempotence mode is off.
end-remote-write(end-remote-write-parameters),
/// An invocation request arrived while the worker was busy
pending-worker-invocation(pending-worker-invocation-parameters),
/// An update request arrived and will be applied as soon the worker restarts
pending-update(pending-update-parameters),
/// An update was successfully applied
successful-update(successful-update-parameters),
/// An update failed to be applied
failed-update(failed-update-parameters),
/// Increased total linear memory size
grow-memory(grow-memory-parameters),
/// Created a resource instance
create-resource(create-resource-parameters),
/// Dropped a resource instance
drop-resource(drop-resource-parameters),
/// Adds additional information for a created resource instance
describe-resource(describe-resource-parameters),
/// The worker emitted a log message
log(log-parameters),
/// The worker's has been restarted, forgetting all its history
restart(datetime),
/// Activates a plugin
activate-plugin(activate-plugin-parameters),
/// Deactivates a plugin
deactivate-plugin(deactivate-plugin-parameters),
/// Revert a worker to a previous state
revert(revert-parameters),
/// Cancel a pending invocation
cancel-invocation(cancel-invocation-parameters),
/// Start a new span in the invocation context
start-span(start-span-parameters),
/// Finish an open span in the invocation context
finish-span(finish-span-parameters),
/// Set an attribute on an open span in the invocation context
set-span-attribute(set-span-attribute-parameters),
}
}
and two resources for querying a worker's oplog.
- the
get-oplog
resource enumerates through all entries of the oplog - the
search-oplog
resource accepts a search expression and only returns the matching entries
Both resources, once constructed, provide a get-next
function that returns a chunk of oplog entries. Repeatedly calling this function goes through the whole data set, and eventually returns none
.
Durability
The golem:durability
package contains an API that can be leveraged by libraries to provide a custom durability implementation for their own API. This is the same interface that Golem uses under the hood to make the WASI interfaces durable. Golem applications are not supposed to use this package directly.
Types
The durable-function-type
is a variant that categorizes a durable function in the following way:
package golem:api@1.1.6;
interface host {
variant durable-function-type {
/// The side-effect reads from the worker's local state (for example local file system,
/// random generator, etc.)
read-local,
/// The side-effect writes to the worker's local state (for example local file system)
write-local,
/// The side-effect reads from external state (for example a key-value store)
read-remote,
/// The side-effect manipulates external state (for example an RPC call)
write-remote,
/// The side-effect manipulates external state through multiple invoked functions (for example
/// a HTTP request where reading the response involves multiple host function calls)
///
/// On the first invocation of the batch, the parameter should be `None` - this triggers
/// writing a `BeginRemoteWrite` entry in the oplog. Followup invocations should contain
/// this entry's index as the parameter. In batched remote writes it is the caller's responsibility
/// to manually write an `EndRemoteWrite` entry (using `end_function`) when the operation is completed.
write-remote-batched(option<oplog-index>)
}
}
The durable-execution-state
record provides information about the current execution state, and can be queried using the current-durable-execution-state
function:
package golem:api@1.1.6;
interface host {
record durable-execution-state {
is-live: bool,
persistence-level: persistence-level,
}
}
Here the is-live
field indicates whether the executor is currently replaying a worker's previously persisted state or side effects should be executed. The persistence-level
is a user-configurable setting that can turn off persistence for certain sections of the code.
The persisted-durable-function-invocation
is a record holding all the information about one persisted durable function. This should be used during replay to simulate the side effect instead of actually running it.
Functions
The durability API consists of a couple of low-level functions that must be called in a correct way to make it work.
The logic to be implemented is the following, in pseudocode:
observe-function-call("interface", "function")
state = current-durable-execution-state()
if state.is-live {
result = perform-side-effect(input)
persist-typed-durable-function-invocation("function", encode(input), encode(result), durable-function-type)
} else {
// Execute the side effect
persisted = read-persisted-durable-function-invocation()
result = decode(persisted.response)
}
The input
and result
values must be encoded into value-and-type
, the dynamic value representation from the golem:rpc
package.
In cases when a durable function's execution interleaves with other calls, the begin-durable-function
and end-durable-function
calls can be used to mark the beginning and end of the operation.
Invocation context
Golem associates an invocation context with each invocation, which contains various information depending on how the exported function was called. This context gets inherited when making further invocations via worker-to-worker communication, and it is also possible to define custom spans and associate custom attributes to it.
The spans are not automatically sent to any tracing system but they can be reconstructed from the oplog, for example using oplog processor plugins, to provide real-time tracing information.
To get the current invocation context, use the current-context
host function:
package golem:api@1.1.6;
/// Invocation context support
interface context {
current-context: func() -> invocation-context;
}
The invocation-context
itself is a resource with various methods for querying attributes of the invocation context:
method | description |
---|---|
trace-id | Returns the trace ID associated with the context, coming from either an external trace header or generated at the edge of Golem |
span-id | Returns the span ID associated with the context |
parent | Returns the parent invocation context, if any |
get-attribute | Gets an attribute from the context by key |
get-attributes | Gets all attributes from the context |
get-attribute-chain | Gets all values of a given attribute from the current and parent contexts |
get-attribute-chains | Get all attributes and their previous values |
trace-context-headers | Gets the W3C Trace Context headers associated with the current invocation context |
Custom attributes can only be set on custom spans. First start a new span using start-span
package golem:api@1.1.6;
/// Invocation context support
interface context {
/// Starts a new `span` with the given name, as a child of the current invocation context
start-span: func(name: string) -> span;
}
and then use the span
resource's methods:
method | description |
---|---|
started-at | Returns the timestamp when the span was started |
set-attribute | Sets an attribute on the span |
set-attributes | Sets multiple attributes on the span |
finish | Ends the current span |
Dropping the resource is equivalent to calling finish
on the span.
The custom spans are pushed onto the invocation context stack, so whenever an RPC call or HTTP call is made, their parent span(s) will include the user-defined custom spans as well as the rest of the invocation context.