Tracing in the Buildkite agent

Using Datadog APM

If you are looking to use Datadog's Application Performance Monitoring (APM) tracing with a Buildkite agent, Using Datadog APM section of Buildkite Pipelines' Datadog integration documentation.

Using OpenTelemetry tracing

Before starting the Buildkite agent, install and configure an OpenTelemetry Collector. Learn more about this from OpenTelemetry's Install the Collector page of their documentation.

Once the Collector is up and running, start the Buildkite agent with:

buildkite-agent start --tracing-backend opentelemetry

This will enable OpenTelemetry tracing, and start sending traces to an OpenTelemetry Collector.

The Buildkite agent's OpenTelemetry implementation uses the OTLP gRPC exporter to export trace information. This means that there must be a Collector capable of ingesting OTLP gRPC traces accessible by the Buildkite agent. By default, the Buildkite agent will export trace information to https://localhost:4317, but this can be overridden by passing in an environment variable OTEL_EXPORTER_OTLP_ENDPOINT containing an updated endpoint for the Collector when the agent is started. Once traces are being sent, you can view the internal state of the collector by visiting the TraceZ debug interface:

http://localhost:55679/debug/tracez

This interface shows active and sampled spans and is helpful for troubleshooting your OpenTelemetry trace pipeline.

To set the OpenTelemetry service name, provide the --tracing-service-name example-buildkite-agent. The default service name when not specified is buildkite-agent.

If using the OpenTelemetry Tracing Notification Service, you can provide the --tracing-propagate-traceparent flag to propagate traces from the Buildkite control plane, and through to your Agent trace spans.

Learn more about configuring the OpenTelemetry integration with Buildkite Pipelines from the OpenTelemetry integrations page.

Trace context propagation

Starting from Buildkite agent version v3.100, when a Buildkite agent executes a command (build script, hook, plugin, and so on), the current trace context is automatically propagated to the child process using environment variables. This enables distributed tracing across job boundaries, and your build scripts can continue the trace started by the agent or the Buildkite Pipelines backend.

The agent serializes the trace context into multiple formats for compatibility with various tracing libraries:

The environment variable names follow the OpenTelemetry Environment Variable Carriers specification.

To continue the trace in your build script, configure your tracing library to extract context from the environment variables. For example, with the OpenTelemetry SDK, you can read the TRACEPARENT variable and create a child span that links back to the agent's span.

Sending OpenTelemetry traces to Honeycomb

To send traces to Honeycomb, in addition to starting the Buildkite agent with the --tracing-backend opentelemetry option, you also need to add the following environment variables. The API token provided by Honeycomb will need to be replaced in the OTEL_EXPORTER_OTLP_HEADERS below.

# this is the same as --tracing-backend opentelemetry
export BUILDKITE_TRACING_BACKEND="opentelemetry"
# service name is configurable
export OTEL_SERVICE_NAME="buildkite-agent"
# the agent only supports GRPC transport
export OTEL_EXPORTER_OTLP_PROTOCOL="grpc"
# the GRPC transport requires a port to be specified in the URL
export OTEL_EXPORTER_OTLP_ENDPOINT="https://api.honeycomb.io:443"
# authentication of traces is done via the API key in this header
export OTEL_EXPORTER_OTLP_HEADERS="x-honeycomb-team=xxxxx"

Exporting job logs as OpenTelemetry logs

From Buildkite agent version v3.135.0, the agent can export job log output as OpenTelemetry log records, in addition to the regular Buildkite job log. Job log export is opt-in. When enabled, job output is emitted as OpenTelemetry log records, and the normal Buildkite job log stream that appears in the Buildkite dashboard is unchanged.

Exported records reuse the OTLP exporter configuration that the agent already reads for OpenTelemetry tracing. You can send job logs to the same collector or backend that receives agent traces.

Enabling job log export

Job log export is independent of the tracing backend, so you can turn it on with or without --tracing-backend opentelemetry. Start the agent with the --job-logs-otlp flag:

buildkite-agent start --job-logs-otlp

You can also enable it with the BUILDKITE_JOB_LOGS_OTLP environment variable:

export BUILDKITE_JOB_LOGS_OTLP=true

Configuring the endpoint

Job log records are sent using the standard OpenTelemetry OTLP exporter environment variables. The agent reads the log-specific OTEL_EXPORTER_OTLP_LOGS_* variables first, then falls back to the generic OTEL_EXPORTER_OTLP_* variables, matching the OpenTelemetry SDK environment variable specification.

The service name defaults to buildkite-agent. When OpenTelemetry tracing is also enabled, it follows the --tracing-service-name flag.

The following example sends job logs to a local collector over HTTP, using the http/protobuf protocol recommended by the OpenTelemetry specification:

export BUILDKITE_JOB_LOGS_OTLP=true
export OTEL_EXPORTER_OTLP_ENDPOINT="http://otel-collector:4318"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
buildkite-agent start

What each record contains

Each record carries a line of job output as its body, with the same [REDACTED] markers as the Buildkite job log. Records are built from the already-redacted job log stream, so they inherit the job log redaction automatically, including secrets split across separate writes and secrets added mid-job through the Job API. Lines longer than the maximum record size of 64 KiB are split across multiple records. Both child-process output and the bootstrap control output, such as section headers, prompts, comments, and warnings, are exported, so the records match the downloadable Buildkite job log.

Each record is emitted at INFO severity and is timestamped with the arrival of the line's first byte, matching the start-of-line semantics of the Buildkite job log. The following resource attributes are set on every record:

The following attributes identify the source job on every record:

Correlating logs with traces

When the agent also runs with OpenTelemetry tracing enabled, each log record is correlated with the trace of the job that produced it. Every record carries the trace ID and the span ID of the nearest enclosing phase or hook span at the time the line is emitted, falling back to the root job span. This lets you pivot from a trace span to the log lines it produced in a backend that supports log-to-trace correlation.

Records carry the enclosing phase or hook span rather than a more specific child operation span. When tracing is disabled, records are still exported, but without trace correlation.