buildkite-agent pipeline

The Buildkite Agent's pipeline command allows you to add and replace build steps in the running build. The steps are defined using YAML or JSON and can be read from a file or streamed from the output of a script.

See the Defining your pipeline steps guide for a step-by-step example and list of step types.

Uploading pipelines

Usage

buildkite-agent pipeline upload [file] [options...]

Description

Allows you to change the pipeline of a running build by uploading either a YAML (recommended) or JSON configuration file. If no configuration file is provided, the command looks for the file in the following locations:

buildkite.yml
buildkite.yaml
buildkite.json
.buildkite/pipeline.yml
.buildkite/pipeline.yaml
.buildkite/pipeline.json
buildkite/pipeline.yml
buildkite/pipeline.yaml
buildkite/pipeline.json

You can also pipe build pipelines to the command allowing you to create scripts that generate dynamic pipelines. The configuration file has a limit of 500 steps per file. Configuration files with over 500 steps must be split into multiple files and uploaded in separate steps.

Example

$ buildkite-agent pipeline upload
$ buildkite-agent pipeline upload my-custom-pipeline.yml
$ ./script/dynamic_step_generator | buildkite-agent pipeline upload

Options

`--replace` #	Replace the rest of the existing pipeline with the steps uploaded. Jobs that are already running are not removed. Environment variable: `$BUILDKITE_PIPELINE_REPLACE`
`--job value` #	The job that is making the changes to its build Environment variable: `$BUILDKITE_JOB_ID`
`--dry-run` #	Rather than uploading the pipeline, it will be echoed to stdout Environment variable: `$BUILDKITE_PIPELINE_UPLOAD_DRY_RUN`
`--format value` #	In dry-run mode, specifies the form to output the pipeline in. Must be one of: json,yaml (default: "json") Environment variable: `$BUILDKITE_PIPELINE_UPLOAD_DRY_RUN_FORMAT`
`--no-interpolation` #	Skip variable interpolation into the pipeline prior to upload Environment variable: `$BUILDKITE_PIPELINE_NO_INTERPOLATION`
`--reject-secrets` #	When true, fail the pipeline upload early if the pipeline contains secrets Environment variable: `$BUILDKITE_AGENT_PIPELINE_UPLOAD_REJECT_SECRETS`
`--jwks-file value` #	Path to a file containing a JWKS. Passing this flag enables pipeline signing Environment variable: `$BUILDKITE_AGENT_JWKS_FILE`
`--jwks-key-id value` #	The JWKS key ID to use when signing the pipeline. Required when using a JWKS Environment variable: `$BUILDKITE_AGENT_JWKS_KEY_ID`
`--signing-aws-kms-key value` #	The AWS KMS key identifier which is used to sign pipelines. Environment variable: `$BUILDKITE_AGENT_AWS_KMS_KEY`
`--debug-signing` #	Enable debug logging for pipeline signing. This can potentially leak secrets to the logs as it prints each step in full before signing. Requires debug logging to be enabled Environment variable: `$BUILDKITE_AGENT_DEBUG_SIGNING`
`--agent-access-token value` #	The access token used to identify the agent Environment variable: `$BUILDKITE_AGENT_ACCESS_TOKEN`
`--endpoint value` #	The Agent API endpoint (default: "`https://agent.buildkite.com/v3`") Environment variable: `$BUILDKITE_AGENT_ENDPOINT`
`--no-http2` #	Disable HTTP2 when communicating with the Agent API. Environment variable: `$BUILDKITE_NO_HTTP2`
`--debug-http` #	Enable HTTP debug mode, which dumps all request and response bodies to the log Environment variable: `$BUILDKITE_AGENT_DEBUG_HTTP`
`--no-color` #	Don't show colors in logging Environment variable: `$BUILDKITE_AGENT_NO_COLOR`
`--debug` #	Enable debug mode. Synonym for `--log-level debug`. Takes precedence over `--log-level` Environment variable: `$BUILDKITE_AGENT_DEBUG`
`--log-level value` #	Set the log level for the agent, making logging more or less verbose. Defaults to notice. Allowed values are: debug, info, error, warn, fatal (default: "notice") Environment variable: `$BUILDKITE_AGENT_LOG_LEVEL`
`--experiment value` #	Enable experimental features within the buildkite-agent Environment variable: `$BUILDKITE_AGENT_EXPERIMENT`
`--profile value` #	Enable a profiling mode, either cpu, memory, mutex or block Environment variable: `$BUILDKITE_AGENT_PROFILE`
`--redacted-vars value` #	Pattern of environment variable names containing sensitive values (default: "_PASSWORD", "_SECRET", "_TOKEN", "_PRIVATE_KEY", "_ACCESS_KEY", "_SECRET_KEY", "_CONNECTION_STRING") Environment variable*: `$BUILDKITE_REDACTED_VARS`

Pipeline format

The pipeline can be written as YAML or JSON, but YAML is more common for its readability. There are three top level properties you can specify:

agents - A map of agent characteristics such as os or queue that restrict what agents the command will run on
env - A map of environment variables to apply to all steps
steps - A list of build pipeline steps

Insertion order

Steps are inserted immediately following the job performing the pipeline upload. Note that if you perform multiple uploads from a single step, they can appear to be in reverse order, because the later uploads are inserted earlier in the pipeline.

Environment variable substitution

The pipeline upload command supports environment variable substitution using the syntax $VAR and ${VAR}.

For example, the following pipeline substitutes a number of Buildkite's default environment variables into a trigger step:

- trigger: "app-deploy"
  label: ":rocket: Deploy"
  branches: "main"
  async: true
  build:
    message: "${BUILDKITE_MESSAGE}"
    commit: "${BUILDKITE_COMMIT}"
    branch: "${BUILDKITE_BRANCH}"

If you want an environment variable to be evaluated at run-time (for example, using the step's environment variables) make sure to escape the $ character using $$ or \$. For example:

- command: "deploy.sh $$SERVER"
  env:
    SERVER: "server-a"

Escaping the $ character

If you need to prevent substitution, you can escape the $ character by using $$ or \$.

For example, using $$USD and \$USD will both result in the same value: $USD.

Disabling interpolation

You can disable interpolation with the --no-interpolation flag, which was added in v3.1.1.

Requiring environment variables

You can set required environment variables using the syntax ${VAR?}. If VAR is not set, the pipeline upload command will print an error and exit with a status of 1.

For example, the following step will cause the pipeline upload to error if the SERVER environment variable has not been set:

- command: "deploy.sh \"${SERVER?}\""

You can set a custom error message after the ? character. For example, the following prints the error message SERVER: is not set. Please specify a server if the environment variable has not been set:

- command: "deploy.sh \"${SERVER?is not set. Please specify a server}\""

Default, blank, and missing values

If an environment variable has not been set it will evaluate to a blank string. You can set a fallback value using the syntax ${VAR:-default-value}.

For example, the following step will run the command deploy.sh staging:

- command: "deploy.sh \"${SERVER:-staging}\""

Environment Variables	Syntax	Result
	`"${SERVER:-staging}"`	`"staging"`
`SERVER=""`	`"${SERVER:-staging}"`	`"staging"`
`SERVER="staging-5"`	`"${SERVER:-staging}"`	`"staging-5"`

If you need to substitute environment variables containing empty strings, you can use the syntax ${VAR-default-value} (notice the missing :).

Environment Variables	Syntax	Result
	`"${SERVER-staging}"`	`"staging"`
`SERVER=""`	`"${SERVER-staging}"`	`""`
`SERVER="staging-5"`	`"${SERVER-staging}"`	`"staging-5"`

Extracting character ranges

You can substitute a subset of characters from an environment variable by specifying a start and end range using the syntax ${VAR:start:end}.

For example, the following step will echo the first 7 characters of the BUILDKITE_COMMIT environment variable:

- command: "echo \"Short commit is: ${BUILDKITE_COMMIT:0:7}\""

If the environment variable has not been set, the range will return a blank string.