Enabling OpenTelemetry for a Go Agent

The golem-otlp-exporter is a built-in plugin that exports agent telemetry (traces, logs, metrics) to any OTLP-compatible collector via OTLP/HTTP. It runs on the platform side (as an oplog processor), so enabling it is a manifest concern — the configuration is the same regardless of the agent’s language. No plugin installation is needed; just enable it in golem.yaml.

Step 1 — Enable the Plugin in golem.yaml

Add the plugin to the component (or agent) that should emit telemetry:

components:
  my-app:service:
    plugins:
      - name: golem-otlp-exporter
        version: "1.5.8"
        parameters:
          endpoint: "http://localhost:4318"
          signals: "traces,logs,metrics"

Plugin Parameters

Step 2 — Deploy

golem deploy --yes

After deployment, newly created agents from this component automatically send telemetry to the configured collector.

What Gets Exported

Traces

Spans are created automatically for:

• Agent invocations
• RPC calls to other agents
• Outgoing HTTP requests

Trace and span IDs propagate from inbound HTTP requests (via code-first routes) and are included in outgoing HTTP request headers automatically.

Custom spans

The github.com/golemcloud/golem/sdks/go/golem/invocation package starts your own spans and reads the invocation context:

span := invocation.StartSpan("charge")
defer span.Finish()
span.SetAttribute("order", orderID)
 
ctx := invocation.CurrentContext()
slog.Info("charging", "trace", ctx.TraceID(), "span", ctx.SpanID())

• A custom span is a child of the current context; RPC and HTTP calls made while it is open inherit it.
• ctx.Attribute(key, inherited), Attributes, AttributeChain, AttributeChains and Parent read the span stack; TraceContextHeaders() returns the W3C headers as an http.Header.
• invocation.AllowForwardingTraceContextHeaders(false) stops adding trace headers to outgoing HTTP requests and returns the previous setting.
• A span that is never finished ends when it is garbage collected; natively (in go test) the context is empty and spans record nothing.

Logs

When logs is included in signals, all log output is forwarded to the OTLP collector. Use the standard library log/slog — the runtime routes it through the host logging channel and, in turn, to the collector. See the golem-logging-go skill for full logging guidance.

import "log/slog"
 
slog.Info("hello from Go")
slog.Info("processing step", "step", 1)

The default logging level is Info, so slog.Debug entries are filtered out unless you configure the logging handler with logging.Options{Level: slog.LevelDebug} (see golem-logging-go).

Metrics

When metrics is included in signals, exported metrics include the following non-exhaustive list:

These are emitted OTLP names. A backend such as Prometheus may normalize dots to underscores and add suffixes; use that backend’s mapping when writing queries.

Each metric includes service.name, golem.agent.id, golem.component.id, and golem.component.version attributes.

Export Semantics

• Durable Start, End, and Cancelled metadata drives span lifecycle. Long-lived spans retain their origin trace across invocations; failed or retrying attempts do not prematurely close the logical span. Logs use their recorded trace context.
• Stream summaries use bounded event/outcome metric labels and aggregate item counts; they do not emit a span per item. Active resource and memory values are gauges, and metric labels avoid resource IDs and payload values.
• In agent-type service-name mode, service.name excludes constructor parameters and phantom instance IDs; golem.agent.id remains the full identity.
• Collector export is best effort. Accepted source state advances when a send fails, and traces, logs, and metrics are still attempted independently. Exactly-once plugin batch delivery is not exactly-once collector delivery.
• Fork/revert cannot retract telemetry already accepted by a collector. Source deletion does not provide a terminal signal, so no successful close is invented. Switching plugin instances can lose open-span state.

Local Observability Stack

The Golem repository includes a ready-made Docker Compose setup at docker-examples/otlp-collector/:

docker compose -f docker-examples/otlp-collector/docker-compose.yml up -d

This starts:

• OTel Collector on port 4318 (OTLP/HTTP)
• Jaeger on http://localhost:16686  (traces)
• Prometheus on http://localhost:9090  (metrics)
• Loki via Grafana (logs)
• Grafana on http://localhost:3000  (admin/admin)

Configure the plugin with endpoint: "http://localhost:4318" to use this stack.

Per-Environment Configuration

Use presets to vary the endpoint across environments:

Use the environment selector app-env:production. A plain production custom preset can become the default custom preset when it is the only one, applying the production endpoint in every environment.

components:
  my-app:service:
    plugins:
      - name: golem-otlp-exporter
        version: "1.5.8"
        parameters:
          endpoint: "http://localhost:4318"
          signals: "traces,logs,metrics"
    presets:
      app-env:production:
        pluginsMergeMode: replace
        plugins:
          - name: golem-otlp-exporter
            version: "1.5.8"
            parameters:
              endpoint: "https://otel.prod.example.com:4318"
              headers: "x-api-key={{ OTLP_API_KEY }}"
              signals: "traces,logs,metrics"

Parameter values support Jinja-style {{ VAR_NAME }} substitution resolved against the host machine’s environment variables at deploy time.

Key Points

• Built-in — no golem plugin register needed, just add to golem.yaml.
• Deploy required — run golem deploy after adding the plugin configuration.
• Trace context propagates automatically through HTTP routes and RPC calls.
• Plugin can be activated/deactivated per agent with golem agent activate-plugin / golem agent deactivate-plugin.

Related Guides