Agent experiments

Buildkite frequently introduces new experimental features to the agent. Use the --experiment flag to opt-in to them and test them out:

buildkite-agent start --experiment experiment1 --experiment experiment2

Or you can set them in your agent configuration file:

experiment="experiment1,experiment2"

If an experiment doesn't exist, no error will be raised.

Please note that there is a likely chance that these experiments we will be removed or changed. Therefore, using them should be at your own risk, and without the expectation that these experiments will work in future.

Available experiments

Agent API

This exposes a local API for interacting with the agent process. ...with primitives that can be used to solve local concurrency problems (such as multiple agents handling some shared local resource).

The API is exposed using a Unix Domain Socket. The path to the socket is not available using a environment variable - rather, there is a single (configurable) path on the system.

To use this feature, set experiment="agent-api" in your agent configuration.

Allow artifact path traversal

Uploaded artifacts include a relative path used by the artifact download tool to download the artifact to a suitable location relative to the destination path. In most circumstances the relative paths generated by artifact upload won't contain .. components, and so will always be downloaded at or inside the destination path.

However, it is possible to upload artifacts using glob patterns containing one or more .. components, which may be preserved in the artifact path. It is also possible for a user to call the Agent REST API directly in order to upload artifacts with arbitrary paths.

Leaving this experiment disabled prevents .. components in artifact paths from traversing up from the destination path. Enabling this experiment permits the less-secure behaviour of allowing artifact paths containing .. to traverse up the destination path.

For example, if an artifact was uploaded with the path ../../foo.txt, then the command:

buildkite-agent artifact download '*.txt' .

has a different effect depending on this experiment:

  • With allow-artifact-path-traversal disabled, foo.txt is downloaded to ./foo.txt.
  • With allow-artifact-path-traversal enabled, foo.txt is downloaded to ../../foo.txt.

To use this feature, set experiment="allow-artifact-path-traversal" in your agent configuration.

Descending spawn priority

To use this feature, set experiment="descending-spawn-priority" in your agent configuration.

Interpolation prefers runtime env

When interpolating the pipeline level environment block, a pipeline level environment variable could take precedence over environment variables depending on the ordering. This may contravene Buildkite's documentation that suggests the Job runtime environment takes precedence over that defined by combining environment variables defined in a pipeline.

We previously made this the default behaviour of the agent (as of v3.63.0) but have since reverted it.

To use this feature, set experiment="interpolation-prefers-runtime-env" in your agent configuration.

Normalised upload paths

Artifacts found by buildkite-agent artifact upload will be uploaded using URI/Unix-style paths, even on Windows. This changes the URLs that artifacts uploaded from Windows agents are stored at, but to one which is URI-compatible.

Artifact names displayed in Buildkite's web UI, as well as in the API, are changed by this.

Take buildkite-agent artifact upload coverage\report.xml as an example:

  • By default, and without this experiment, this file is uploaded to s3://example/coverage\report.xml.
  • With this experiment enabled, it would be s3://example/coverage/report.xml.

Status: a major improvement for Windows compatibility, we'd like this to be the standard behaviour in 4.0. 👍👍

To use this feature, set experiment="normalised-upload-paths" in your agent configuration.

Override zero exit on cancel

If the job is cancelled, and the exit status of the process is 0, it is overridden to be 1 instead.

When cancelling a job, the agent signals the process, which typically causes it to exit with a non-zero status code. On Windows this is not true - the process exits with code 0 instead, which makes the job appear to be successful. (It successfully exited, no?) By overriding the status to 1, a cancelled job should appear as a failure, regardless of the OS the agent is running on.

To use this feature, set experiment="override-zero-exit-on-cancel" in your agent configuration.

Propagate agent config vars

To use this feature, set experiment="propagate-agent-config-vars" in your agent configuration.

PTY raw

Set PTY to raw mode, to avoid mapping LF (\n) to CR,LF (\r\n) in job command output. These extra newline characters are normally not noticed, but can make raw logs appear double-spaced in some circumstances.

We run commands in a PTY mostly (entirely?) so that the program detects a PTY and behaves like it's running in a terminal, using ANSI escapes to provide colours, progress meters etc. But we don't need the PTY to modify the stream. (Or do we? That's why this is an experiment)

To use this feature, set experiment="pty-raw" in your agent configuration.

Resolve commit after checkout

After repository checkout, resolve BUILDKITE_COMMIT to a commit hash. This makes BUILDKITE_COMMIT useful for builds triggered against non-commit-hash refs such as HEAD.

Status: broadly useful, we'd like this to be the standard behaviour in 4.0. 👍👍

To use this feature, set experiment="resolve-commit-after-checkout" in your agent configuration.