Observability

Golem 1.5 includes built-in observability through the OpenTelemetry protocol. The built-in golem-otlp-exporter plugin exports agent traces, logs, and metrics to any OTLP-compatible collector.

Setting up the OTLP plugin

Enable the OTLP exporter by adding it to your golem.yaml:

components:
  my-app:ts-main:
    templates: ts
    plugins:
      - name: golem-otlp-exporter
        version: 1.5.0
        parameters:
          endpoint: "http://localhost:4318"
          signals: "traces,logs,metrics"

The following plugin configuration keys are available:

With agent-type, service.name is the agent type only. Constructor parameters and a phantom-agent instance ID are excluded, while golem.agent.id still identifies the full agent. This keeps service grouping stable and avoids unbounded service names.

Custom spans

import { startSpan, currentContext } from 'golem:api/context@1.5.0';
 
const span = startSpan('my-operation');
span.setAttribute('env', { tag: 'string', val: 'production' });
// ... do work ...
span.finish();

import { Effect } from "effect"
 
const operation = Effect.gen(function* () {
  yield* Effect.annotateCurrentSpan("env", "production")
  // ... do work ...
}).pipe(Effect.withSpan("my-operation"))

use golem_rust::bindings::golem::api::context;
 
let span = context::start_span("my-operation");
span.set_attribute("env", &context::AttributeValue::String("production".to_string()));
// ... do work ...
span.finish();

import golem.host.ContextApi
 
val span = ContextApi.startSpan("my-operation")
span.setAttribute("env", ContextApi.AttributeValue.StringValue("production"))
// ... do work ...
span.finish()

@context.with_span(
  "my-operation",
  attributes=[("env", "production")],
  fn(span) {
    // ... do work ...
    span.set_attribute("step", "processing")
  },
)

Trace context propagation

Trace and span IDs are automatically propagated from inbound HTTP requests through code-first routes, and outgoing HTTP requests include standard W3C trace headers (traceparent / tracestate).

Span openings, attribute changes, and completed or cancelled terminals are recorded with durable operations. A span may remain open across invocations and retains the trace context from where it started; closing it from a later invocation does not move it into that invocation’s trace. Failed or retrying attempts do not close a surviving logical span.

TypeScript diagnostics_channel integration

For TypeScript agents, invocation context is wired to node:diagnostics_channel, enabling integration with Node.js tracing utilities:

import { tracingChannel } from 'node:diagnostics_channel';
 
const dc = tracingChannel('my-operation');
const result = dc.traceSync(
  () => { return 42; },
  { method: 'GET', url: '/api/data' }
);

For Effect TypeScript agents, prefer Effect’s tracing APIs; the SDK wires them to Golem invocation context automatically. For host context details or W3C propagation headers, use Tracing.currentContext or Tracing.traceContextHeaders:

import { Effect } from "effect"
import { Tracing } from "@golemcloud/effect-golem"
 
const context = Effect.gen(function* () {
  const snapshot = yield* Tracing.currentContext
  const headers = yield* Tracing.traceContextHeaders
  return { snapshot, headers }
})

Logs

When log exporting is enabled via the signals parameter, stdout/stderr output and dedicated log calls are forwarded to the OTLP collector. Standard logging APIs work out of the box — console.log in TypeScript, println! in Rust, and so on. Exported logs use the trace context recorded with the log rather than the invocation current when the exporter processes it.

P3 standard output is buffered. Its context is captured when the host emits the buffered chunk, which can be later than the guest write and can use a different current span.

Metrics

The following metrics are exported per agent:

Each metric resource includes service.name, golem.agent.id, golem.component.id, and golem.component.version. Collector backends may normalize dotted metric names to underscores.

Metric data-point dimensions are intentionally bounded. Active resource and memory values are gauges; individual resource IDs, stream IDs, and payload values are not data-point attributes. The agent identity remains a resource attribute.

Delivery and history semantics

OTLP delivery is best effort. Once a source batch is accepted, exporter lifecycle state advances even if the collector rejects a request. Traces, logs, and metrics are attempted independently, so one failed signal does not prevent attempts for the others. A later batch is not a reliable retry of failed collector delivery, and a collector may observe duplicates after an ambiguous request or crash.

Exactly-once oplog batch delivery to a plugin is not exactly-once delivery to the collector. Switching the plugin instance can currently lose in-memory state for spans opened before the switch; continuity across plugin instances is deferred to GOL-667. Do not switch instances while relying on open-span continuity.

Fork and revert update pending state to match retained history, but OpenTelemetry cannot retract telemetry a collector already accepted. Source deletion sends no terminal lifecycle signal to the exporter, so it does not fabricate a successful close for open spans.

Oplog processor plugins

The OTLP exporter is a built-in oplog processor plugin. Custom oplog processor plugins can be written in Rust by implementing the process-oplog-entries interface. Golem guarantees exactly-once delivery of oplog entries to plugins. This guarantee ends at the plugin boundary; OTLP collector delivery is described above.

Plugins can be installed per-component and activated or deactivated per-agent via the CLI:

golem agent activate-plugin --plugin-name my-plugin 'MyAgent("my-agent")'
golem agent deactivate-plugin --plugin-name my-plugin 'MyAgent("my-agent")'

For more details on plugin management, see the How-To Guides.