buildkite-agent pipeline

The Buildkite Agent’s pipeline command allows you to add and replace build steps in the running build. The steps are defined using YML or JSON and can be read from a file or streamed from the output of a script.

See the Defining Your Pipeline Steps guide for a step-by-step example and list of step types.

Uploading pipelines

Usage:

   buildkite-agent pipeline upload <file> [arguments...]

Description:

   Allows you to change the pipeline of a running build by uploading either a
   JSON or YAML configuration file. If no configuration file is provided,
   we look for the file in the following locations:

   - .buildkite/pipeline.yml
   - .buildkite/pipeline.json

   You can also pipe build pipelines to the command, allowing you to create scripts
   that generate dynamic pipelines.

Example:

   $ buildkite-agent pipeline upload
   $ buildkite-agent pipeline upload my-custom-steps.json
   $ ./script/dynamic_step_generator | buildkite-agent pipeline upload

Options:

   --replace                                    Replace the rest of the existing pipeline with the steps uploaded. Jobs that are already running are not removed. [$BUILDKITE_PIPELINE_REPLACE]
   --job                                        The job that is making the changes to it's build [$BUILDKITE_JOB_ID]
   --agent-access-token                         The access token used to identify the agent [$BUILDKITE_AGENT_ACCESS_TOKEN]
   --endpoint "https://agent.buildkite.com/v3"  The Agent API endpoint [$BUILDKITE_AGENT_ENDPOINT]
   --no-color                                   Don't show colors in logging [$BUILDKITE_AGENT_NO_COLOR]
   --debug                                      Enable debug mode [$BUILDKITE_AGENT_DEBUG]
   --debug-http                                 Enable HTTP debug mode, which dumps all request and response bodies to the log [$BUILDKITE_AGENT_DEBUG_HTTP]

Pipeline format

The pipeline can be written as YAML or JSON, but YAML is more common for its readability. There are two top level properties you can specify:

Environment variable substitution

In Buildkite Agent versions 3.0 and above, the pipeline upload command supports environment variable substitution using the syntax $VAR and ${VAR}.

For example, the following pipeline substitutes a number of Buildkite’s default environment variables into a trigger step:

- trigger: "app-deploy"
  label: ":rocket: Deploy"
  branches: "master"
  async: true
  build:
    message: "${BUILDKITE_MESSAGE}"
    commit: "${BUILDKITE_COMMIT}"
    branch: "${BUILDKITE_BRANCH}"

If you want an environment variable to be evaluated at run-time (for example, using the step’s environment variables) make sure to escape the $ character using $$ or \$. For example:

- command: "deploy.sh $$SERVER"
  env:
    SERVER: "server-a"

Buildkite Agent 2.x support

If you are unable to upgrade your agent to version 3.0 or above, it is possible (but not recommended) to emulate the agent’s environment variable substitution using bash:

echo $(eval echo $(cat pipeline.yml)) | buildkite-agent pipeline upload

Escaping the $ character

If you need to prevent substitution, you can escape the $ character by using $$ or \$.

For example, using $$USD and \$USD will both result in the same value: $USD.

Requiring environment variables

You can set required environment variables using the syntax ${VAR?}. If VAR is not set, the pipeline upload command will print and error and exit with a status of 1.

For example, the following step will cause the pipeline upload to error if the SERVER environment variable has not been set:

- command: "deploy.sh \"${SERVER?}\""

You can set a custom error message after the ? character. For example, the following prints the error message SERVER: is not set. Please specify a server if the environment variable has not been set:

- command: "deploy.sh \"${SERVER?is not set. Please specify a server}\""

Default, blank and missing values

If an environment variable has not been set it will evaluate to a blank string. You can set a fallback value using the syntax ${VAR:-default-value}.

For example, the following step will run the command deploy.sh staging:

- command: "deploy.sh \"${SERVER:-staging}\""
Environment VariablesSyntaxResult
"${SERVER:-staging}""staging"
SERVER="""${SERVER:-staging}""staging"
SERVER="staging-5""${SERVER:-staging}""staging-5"

If you need to substitute environment variables containing empty strings, you can use the syntax ${VAR-default-value} (notice the missing :).

Environment VariablesSyntaxResult
"${SERVER-staging}""staging"
SERVER="""${SERVER-staging}"""
SERVER="staging-5""${SERVER-staging}""staging-5"

Extracting character ranges

You can substitute a subset of characters from an environment variable by specifying a start and end range using the syntax ${VAR:start:end}.

For example, the following step will echo the first 7 characters of the BUILDKITE_COMMIT environment variable:

- command: "echo \"Short commit is: ${BUILDKITE_COMMIT:0:7}\""

If the environment variable has not been set, the range will return a blank string.