Defining Your Pipeline Steps

Pipeline steps can be defined visually in Buildkite or, more powerfully, in code using a pipeline.yml file.

Defining your pipeline steps in a pipeline.yml file gives you access to more configuration options than the web interface, and allows you to version, audit and review your build pipelines alongside your source code.

Getting started

After creating a pipeline, you can add steps in the Buildkite Pipeline Settings. In the 'Steps' section, click the ➕ to add a step. You can create your entire pipeline by adding steps in this way.

To use a pipeline.yml file, click the ➕ to add a step and select “Read steps from repository”:

Screenshot of the 'Read steps from repository' menu item

Your pipeline will now show a new step labeled :pipeline: that runs a buildkite-agent pipeline upload command:

Screenshot of the pipeline upload step

When this step runs on an agent, it adds the steps you defined in .buildkite/pipeline.yml to your pipeline. For example, here is a pipeline file with a single step:

steps:
  - label: Example Test
    command: echo "Hello!"

The above example is a command step, which will run the code echo "Hello!" and print "Hello!" to the build's log.

To test your .buildkite/pipeline.yml file, commit your pipeline.yml file and push it to your repository. If you've set up webhooks, this will automatically create a new build. You can also create a new build using the 'New Build' button on the pipeline page.

Screenshot of the build passing with pipeline upload step first, and then the example step

If your pipeline has a lot of steps and you have multiple agents available to run them, they will automatically run at the same time. If your steps rely on running in sequence, you can use wait steps to ensure that the steps before the 'wait' are completed before subsequent steps are run.

When each step is run by an agent, it will be run with a clean checkout of the pipeline's repository. If your commands or scripts rely on the output from previous steps, you will need to either combine them into a single script or use artifacts to pass data between steps. This enables any step to be picked up by any agent and run steps concurrently to speed up your build.

Example pipeline

Here’s a more complete example based on our Buildkite Agent’s build pipeline. It contains script commands, wait steps, block steps, and automatic artifact uploading:

steps:
  - label: ':hammer: Tests'
    command: 'scripts/tests.sh'
    env:
      BUILDKITE_DOCKER_COMPOSE_CONTAINER: app

  - wait

  - label: ':package: Package'
    command: 'scripts/build-binaries.sh'
    artifact_paths: 'pkg/*'
    env:
      BUILDKITE_DOCKER_COMPOSE_CONTAINER: app

  - wait

  - label: ':debian: Publish'
    command: 'scripts/build-debian-packages.sh'
    artifact_paths: 'deb/**/*'
    branches: 'master'
    agents:
      queue: 'deploy'

  - block: ':shipit: Release'
    branches: 'master'

  - label: ':github: Release'
    command: 'scripts/build-github-release.sh'
    artifact_paths: 'releases/**/*'
    branches: 'master'

  - wait

  - label: ':whale: Update images'
    command: 'scripts/release-docker.sh'
    branches: 'master'
    agents:
      queue: 'deploy'

Step types

There are four different step types you can use in your Buildkite pipelines. Click through each step type for detailed usage documentation:

Customizing the pipeline upload path

By default the pipeline upload step reads your pipeline definition from .buildkite/pipeline.yml in your repository. You can specify a different file path by adding it as the first argument:

Screenshot of a custom pipeline file upload

A common use for custom file paths is when separating test and deployment steps into two completely separate pipelines, both using the same repository URL. For example, your test pipeline’s upload command could be:

buildkite-agent pipeline upload .buildkite/pipeline.yml

And your deployment pipeline’s upload command could be:

buildkite-agent pipeline upload .buildkite/pipeline.deploy.yml

For a list of all command line options, see the buildkite-agent pipeline upload documentation.

Dynamic pipelines

Because the pipeline upload step runs on your agent machine, you can generate pipelines dynamically using scripts from your source code. This provides you with the flexiblity to structure your pipelines as you wish.

The following example generates a list of parallel test steps based upon the test/* directories within your repository:

#!/bin/bash

# exit immediately on failure, or if an undefined variable is used
set -eu

# begin the pipeline.yml file
echo "steps:"

# add a new command step to run the tests in each test directory
for test_dir in test/*/; do
  echo "  - command: \"run_tests "${test_dir}"\"
done

To use this script, you'd save it to .buildkite/pipeline.sh inside your repository, ensure it is executable, and then update your pipeline upload step to use the new script:

.buildkite/pipeline.sh | buildkite-agent pipeline upload

When a build is run it will execute the script and pipe its output to the pipeline upload command, dynamically adding build steps to the end of the build pipeline.

Migrating existing steps to a pipeline.yml

If you have a pipeline that is defined in the web interface and would like to migrate it to a pipeline.yml without any interruption to builds, add a command step and wait step at the start of your pipeline:

For agents running Linux, macOS or other Unixes, the command is:

[[ -f ".buildkite/pipeline.yml" ]] && buildkite-agent pipeline upload --replace || true

Or, if your Agent runs Windows:

CMD /k if exist ".buildkite/pipeline.yml" buildkite-agent pipeline upload .buildkite/pipeline.yml --replace

This line of code will check if the pipeline file exists and, if it does, execute the buildkite agent pipeline upload --replace command to replace any steps that were defined via the web interface with those defined in the pipeline.yml.

To ensure that your existing steps don't run alongside the check, we include a wait step after it.

Your steps should then look something like this:

Screenshot of the above described pipeline upload command, before some existing steps

Further documentation

You can also create pipelines using the buildkite-agent command line tool. See the buildkite-agent pipeline documentation for a full list of the available parameters.