buildkite-agent pipeline

The Buildkite Agent’s pipeline command allows you to add and replace build steps in the running build. The steps are defined using YML or JSON and can be read from a file or streamed from the output of a script.

See the Defining Your Pipeline Steps guide for a step-by-step example and list of step types.

Uploading pipelines

$ buildkite-agent pipeline upload --help
Usage:

   buildkite-agent pipeline upload <file> [arguments...]

Description:

   Allows you to change the pipeline of a running build by uploading either a
   YAML (recommended) or JSON configuration file. If no configuration file is
   provided, the command looks for the file in the following locations:

   - buildkite.yml
   - buildkite.yaml
   - buildkite.json
   - .buildkite/pipeline.yml
   - .buildkite/pipeline.yaml
   - .buildkite/pipeline.json

   You can also pipe build pipelines to the command allowing you to create
   scripts that generate dynamic pipelines.

Example:

   $ buildkite-agent pipeline upload
   $ buildkite-agent pipeline upload my-custom-pipeline.yml
   $ ./script/dynamic_step_generator | buildkite-agent pipeline upload

Options:

   --replace                   Replace the rest of the existing pipeline with the steps uploaded. Jobs that are already running are not removed. [$BUILDKITE_PIPELINE_REPLACE]
   --job value                 The job that is making the changes to it's build [$BUILDKITE_JOB_ID]
   --no-interpolation          Skip variable interpolation the pipeline when uploaded [$BUILDKITE_PIPELINE_NO_INTERPOLATION]
   --agent-access-token value  The access token used to identify the agent [$BUILDKITE_AGENT_ACCESS_TOKEN]
   --endpoint value            The Agent API endpoint (default: "https://agent.buildkite.com/v3") [$BUILDKITE_AGENT_ENDPOINT]
   --no-color                  Don't show colors in logging [$BUILDKITE_AGENT_NO_COLOR]
   --debug                     Enable debug mode [$BUILDKITE_AGENT_DEBUG]
   --debug-http                Enable HTTP debug mode, which dumps all request and response bodies to the log [$BUILDKITE_AGENT_DEBUG_HTTP]

Pipeline format

The pipeline can be written as YAML or JSON, but YAML is more common for its readability. There are two top level properties you can specify:

Environment variable substitution

The pipeline upload command supports environment variable substitution using the syntax $VAR and ${VAR}.

For example, the following pipeline substitutes a number of Buildkite’s default environment variables into a trigger step:

- trigger: "app-deploy"
  label: ":rocket: Deploy"
  branches: "master"
  async: true
  build:
    message: "${BUILDKITE_MESSAGE}"
    commit: "${BUILDKITE_COMMIT}"
    branch: "${BUILDKITE_BRANCH}"

If you want an environment variable to be evaluated at run-time (for example, using the step’s environment variables) make sure to escape the $ character using $$ or \$. For example:

- command: "deploy.sh $$SERVER"
  env:
    SERVER: "server-a"

Insertion order

Steps are inserted immediately following the job performing the pipeline upload. Note that if you perform multiple uploads from a single step, they can appear to be in reverse order, because the later uploads are inserted earlier in the pipeline.

Escaping the $ character

If you need to prevent substitution, you can escape the $ character by using $$ or \$.

For example, using $$USD and \$USD will both result in the same value: $USD.

Disabling interpolation

You can disable interpolation with the --no-interpolation flag, which was added in v3.1.1.

Requiring environment variables

You can set required environment variables using the syntax ${VAR?}. If VAR is not set, the pipeline upload command will print an error and exit with a status of 1.

For example, the following step will cause the pipeline upload to error if the SERVER environment variable has not been set:

- command: "deploy.sh \"${SERVER?}\""

You can set a custom error message after the ? character. For example, the following prints the error message SERVER: is not set. Please specify a server if the environment variable has not been set:

- command: "deploy.sh \"${SERVER?is not set. Please specify a server}\""

Default, blank and missing values

If an environment variable has not been set it will evaluate to a blank string. You can set a fallback value using the syntax ${VAR:-default-value}.

For example, the following step will run the command deploy.sh staging:

- command: "deploy.sh \"${SERVER:-staging}\""
Environment VariablesSyntaxResult
"${SERVER:-staging}""staging"
SERVER="""${SERVER:-staging}""staging"
SERVER="staging-5""${SERVER:-staging}""staging-5"

If you need to substitute environment variables containing empty strings, you can use the syntax ${VAR-default-value} (notice the missing :).

Environment VariablesSyntaxResult
"${SERVER-staging}""staging"
SERVER="""${SERVER-staging}"""
SERVER="staging-5""${SERVER-staging}""staging-5"

Extracting character ranges

You can substitute a subset of characters from an environment variable by specifying a start and end range using the syntax ${VAR:start:end}.

For example, the following step will echo the first 7 characters of the BUILDKITE_COMMIT environment variable:

- command: "echo \"Short commit is: ${BUILDKITE_COMMIT:0:7}\""

If the environment variable has not been set, the range will return a blank string.