Invocation Context

Golem associates an invocation context with each agent invocation. This invocation context consists of a stack of spans, each having a collection of attributes. Currently only string attributes are supported.

The invocation context can be queried and manipulated (by adding custom spans with custom attributes) using the SDK.

By default, the invocation context’s spans are not used by Golem for anything; they are just information propagated among agents to be used by either the agents or oplog processor plugins installed to them. It is possible to reconstruct all information about invocation context from the oplog, so oplog processor plugins can be used, for example, to provide live traces of agents through OpenTelemetry.

Span lifecycle metadata is embedded in durable operation Start, End, and Cancelled records. This includes the opening context and timestamp, applied attribute updates, and the completed or cancelled terminal. A logical span can outlive the invocation that opened it and keeps that origin trace context when a later invocation uses or closes it. A failed, interrupted, or retrying attempt is not by itself a terminal for the logical span.

The OTLP exporter restores this recorded context for spans and logs. Cross-trace relationships are links rather than invalid cross-trace parents. Deleting an agent provides no terminal record to the exporter, so deletion does not manufacture a successful completion. Fork and revert adjust pending span state to retained history, but cannot retract telemetry already accepted by a collector.

Automatic invocation context spans

Incoming HTTP requests

The root spans of an invocation contain information depending on how that invocation was started. When invoking through the invocation REST API, W3C Trace Context headers are automatically processed to set the root trace-id and parent span-id.

When using custom APIs, the same headers are also taken into account by default.

In both cases, when the invocation’s trigger is an HTTP request, a span is added with the following attributes:

request.method
request.uri
request.remote_addr

Invocations

When an invocation reaches an agent, a new span is created containing the following attributes:

name: span’s name (invoke-exported-function)
idempotency_key: unique identifier of the invocation
function_name: the fully qualified function name
worker_id: agent’s ID (consists of component ID and agent name)
agent_type: the agent type of this agent
agent_parameters: constructor parameter values of the agent

Outgoing HTTP requests

Outgoing HTTP requests are also represented by their own automatic spans with the following attributes:

name: span’s name (outgoing-http-request)
request.method: the outgoing request’s HTTP method
request.uri: the outgoing request’s URI

RPC calls

RPC calls on the caller side are also represented by custom spans. First, a span is created when the underlying RPC resource is initialized, with the following attributes:

name: span’s name (rpc-connection)
target_worker_id: the target agent’s ID

Within this span, a new span is created for each remote invocation, with the following attributes:

name: span’s name (rpc-invocation)
function_name: the invoked function’s fully qualified name
idempotency_key: the unique identifier of the invocation

The connection span can remain open and retain its original trace across multiple agent invocations. An RPC retry does not complete the logical operation until a recorded completion or cancellation occurs.

Note: some attribute names such as worker_id and target_worker_id use legacy “worker” terminology for backwards compatibility.