Agent experiments

Buildkite frequently introduces new experimental features to the agent. Use the --experiment flag to opt-in to them and test them out:

buildkite-agent start --experiment experiment1 --experiment experiment2

Or you can set them in your agent configuration file:

experiment="experiment1,experiment2"

If an experiment doesn't exist, no error will be raised.

Please note that there is a likely chance that these experiments we will be removed or changed. Therefore, using them should be at your own risk, and without the expectation that these experiments will work in future.

Available experiments

Agent API

This exposes a local API for interacting with the agent process. ...with primitives that can be used to solve local concurrency problems (such as multiple agents handling some shared local resource).

The API is exposed using a Unix Domain Socket. The path to the socket is not available using a environment variable - rather, there is a single (configurable) path on the system.

To use this feature, set experiment="agent-api" in your agent configuration.

Interpolation prefers runtime env

When interpolating the pipeline level environment block, a pipeline level environment variable could take precedence over environment variables depending on the ordering. This may contravene Buildkite's documentation that suggests the Job runtime environment takes precedence over that defined by combining environment variables defined in a pipeline.

We previously made this the default behaviour of the agent (as of v3.63.0) but have since reverted it.

To use this feature, set experiment="interpolation-prefers-runtime-env" in your agent configuration.

Legacy post hook order

This experiment is an escape hatch that reverts to the v3 execution order of post-checkout and post-command hooks.

In Agent v3, hooks of any kind would run in the same order as one another (for plugins, the order in which plugins are specified for a step). In v4, multiple post-checkout, post-command, or pre-exit hooks execute in reverse order. This change makes it easier for multiple plugins and hooks to compose.

For example, suppose a step specifies two plugins A and B, and there are also agent and repository hooks. Under version 3, each hook type would execute in the same order:

  • agent pre-checkout
  • (pre-checkout is not possible for repository hooks)
  • plugin A pre-checkout
  • plugin B pre-checkout
  • (checkout)
  • agent post-checkout
  • repository post-checkout
  • plugin A post-checkout
  • plugin B post-checkout
  • agent pre-command
  • repository pre-command
  • plugin A pre-command
  • plugin B pre-command
  • (command)
  • agent post-command
  • repository post-command
  • plugin A post-command
  • plugin B post-command
  • agent pre-exit
  • repository pre-exit
  • plugin A pre-exit
  • plugin B pre-exit

Under version 4, the execution order is (key differences in bold):

  • agent pre-checkout
  • (pre-checkout is not possible for repository hooks)
  • plugin A pre-checkout
  • plugin B pre-checkout
  • (checkout)
  • plugin B post-checkout
  • plugin A post-checkout
  • repository post-checkout
  • agent post-checkout
  • agent pre-command
  • repository pre-command
  • plugin A pre-command
  • plugin B pre-command
  • (command)
  • plugin B post-command
  • plugin A post-command
  • repository post-command
  • agent post-command
  • plugin B pre-exit
  • plugin A pre-exit
  • repository pre-exit
  • agent pre-exit

To use this feature, set experiment="legacy-post-hook-order" in your agent configuration.

PTY raw

Set PTY to raw mode, to avoid mapping LF (\n) to CR,LF (\r\n) in job command output. These extra newline characters are normally not noticed, but can make raw logs appear double-spaced in some circumstances.

We run commands in a PTY mostly (entirely?) so that the program detects a PTY and behaves like it's running in a terminal, using ANSI escapes to provide colours, progress meters etc. But we don't need the PTY to modify the stream. (Or do we? That's why this is an experiment)

To use this feature, set experiment="pty-raw" in your agent configuration.

Zip plugins

Allows plugins to be downloaded as zip archives instead of being cloned from a Git repository. This is useful for plugins hosted as zip files on HTTP(S) URLs.

To use this feature, set experiment="zip-plugins" in your agent configuration.

Replaced experiments

The following experiments were replaced by supported agent options.

Descending spawn priority

Replaced in v4.0.0-beta.1 by --spawn-with-priority descending.

Kubernetes exec

Replaced in v3.74.0 by --kubernetes-exec.

Removed experiments

The following experiments were removed without becoming part of the default agent behavior.

Allow artifact path traversal

Removed in v4.0.0-beta.1 because the path traversal behavior was insecure.