Logs

Golem server logs

Every Golem service emits structured logs with with configurable format and level. The default configuration for our dockerized builds is to log to the container's standard output using JSON format. Each service supports the same set of configuration keys controlling the logging behavior, which can be overwritten using environment variables.

The default log level and filtering can be configured using the RUST_LOG environment variable, as described in the tracing-subscriber (opens in a new tab) crate's documentation. The simplest way to use this environment variable is to set a global log level, for example RUST_LOG=debug.

The Golem specific config keys are the following:

[tracing]
console = false
dtor_friendly = false
file_name = "worker-executor.log"
file_truncate = true
 
[tracing.file]
ansi = false
compact = false
enabled = false
json = true
json_flatten = true
json_flatten_span = true
pretty = false
span_events_active = false
span_events_full = false
without_time = false
 
[tracing.stdout]
ansi = true
compact = false
enabled = true
json = false
json_flatten = true
json_flatten_span = true
pretty = false
span_events_active = false
span_events_full = false
without_time = false

Overwriting config

To overwrite any of the above config keys using environment variables, use the GOLEM_ prefix and the __ separator for levels. For example to disable logging to the standard output, and instead enable logging to a file, set the following two variables:

GOLEM_TRACING__FILE__ENABLED=true
GOLEM_TRACING__STDOUT__ENABLED=false

Configurable options

The top-level tracing block allows setting the file name and truncate behavior for file logging, as well as enabling connection to tokio-console (opens in a new tab).

For both file and stdout logging, a set of boolean flags control the format of the emitted log lines:

Flag	Description
`ansi`	Use ANSI colors for the output
`compact`	Use the compact formatter.
`enabled`	Enables logging to file or stdout
`json`	Use JSON logging
`json_flatten`	Flatten event metadata
`json_flatten_span`	Flatten nested span fields
`pretty`	Use the pretty formatter
`span_events_active`	Emit one event per enter/exit of a span
`span_events_full`	Emit one event at all points (new, enter, exit, drop)
`without_time`	Do not emit timestamps for the log entries

Golem agent logs

In TypeScript agents, use the standard functions of console to log messages. console.log writes to the agent's standard output, while console.debug, console.info, etc. are writing log entries with the specific log level.

Getting agent logs

Agent logs (both stdout/err and WASI logging entries) are persisted for each agent, and they can be watched in live by using the connect API. There are three ways to connect to an agent:

Using golem
Using the connect WebSocket API
Using the Golem Console (opens in a new tab) (Golem Cloud only)

To connect to a agent's log with golem, use either the golem agent stream command or the --stream flag when making an invocation. See the CLI documentation for more details.

Metrics Invocation Context