Defining Your Pipeline Steps

Pipeline steps can be defined visually in Buildkite or, more powerfully, in code using a pipeline.yml file.

Defining your pipeline steps in a pipeline.yml file gives you access to more configuration options than the web interface, and allows you to version, audit and review your build pipelines alongside your source code.

Getting started

Create a pipeline from the Pipelines page of Buildkite using the ➕ button.

Required fields are 'name', 'repository', and 'steps'. You can leave the 'steps' with only the default empty command step to get started, the steps can be edited at any time from your pipeine settings page.

Screenshot of the 'New Pipeline' setup form

You can set up webhooks at this point, but this step is optional. These webhook setup instructions can be found in pipeline settings on your specific repository provider page.

You can also use the REST API to create a pipeline and define your steps. See the Pipelines API for further information.

Adding steps

There are two ways to add steps to your pipeline: with the visual editor in Buildkite or with a pipeline.yml file. You can add steps immediately during your initial pipeline setup, or you can edit them on the pipeline settings page.

Visual pipeline editor

Add steps to your pipeline using the ➕ dropdown button. You can add command, wait, and block steps here.

Command steps can be configured using the expandable sections on the right of the step settings.

For more complex steps, like trigger steps, or those utilizing plugins or advanced configuration, you'll need to use a pipeline.yml file.

Pipeline.yml file

Before getting started with a pipeline.yml file, you'll need to tell Buildkite where to find your steps.

In your Buildkite pipeline settings, add a new step using the ➕ button and choose 'read steps from repository'.

Screenshot of the 'Read steps from repository' menu item

Your pipeline will now show a new step labeled :pipeline: that runs a buildkite-agent pipeline upload command:

Screenshot of the pipeline upload step

When you run a build, this step will add any steps defined in the pipeline.yml file to your Buildkite pipeline.

Create a file called pipeline.yml in a directory called .buildkite in your repository. This file will describe your pipeline steps in YAML, and will be versioned alongside your code.

The following example YAML defines a pipeline with one command step that will echo 'Hello' into your build log:

steps:
  - label: Example Test
    command: echo "Hello!"

With the above example code in a pipeline.yml file, commit and push the file up to your repository. If you have set up webhooks, this will automatically create a new build. You can also create a new build using the 'New Build' button on the pipeline page.

Screenshot of the build passing with pipeline upload step first, and then the example step

For more example steps and detailed configuration options, see the example pipeline.yml below, or the step type specific documentation: command steps, wait steps, block steps, and trigger steps.

If your pipeline has a lot of steps and you have multiple agents available to run them, they will automatically run at the same time. If your steps rely on running in sequence, you can separate them with wait steps to ensure that the steps before the 'wait' are completed before subsequent steps are run.

When each step is run by an agent, it will be run with a clean checkout of the pipeline's repository. If your commands or scripts rely on the output from previous steps, you will need to either combine them into a single script or use artifacts to pass data between steps. This enables any step to be picked up by any agent and run steps concurrently to speed up your build.

Example pipeline

Here’s a more complete example based on our Buildkite Agent’s build pipeline. It contains script commands, wait steps, block steps, and automatic artifact uploading:

steps:
  - label: ':hammer: Tests'
    command: 'scripts/tests.sh'
    env:
      BUILDKITE_DOCKER_COMPOSE_CONTAINER: app

  - wait

  - label: ':package: Package'
    command: 'scripts/build-binaries.sh'
    artifact_paths: 'pkg/*'
    env:
      BUILDKITE_DOCKER_COMPOSE_CONTAINER: app

  - wait

  - label: ':debian: Publish'
    command: 'scripts/build-debian-packages.sh'
    artifact_paths: 'deb/**/*'
    branches: 'master'
    agents:
      queue: 'deploy'

  - block: ':shipit: Release'
    branches: 'master'

  - label: ':github: Release'
    command: 'scripts/build-github-release.sh'
    artifact_paths: 'releases/**/*'
    branches: 'master'

  - wait

  - label: ':whale: Update images'
    command: 'scripts/release-docker.sh'
    branches: 'master'
    agents:
      queue: 'deploy'

Step types

There are four different step types you can use in your Buildkite pipelines. Click through each step type for detailed usage documentation:

Customizing the pipeline upload path

By default the pipeline upload step reads your pipeline definition from .buildkite/pipeline.yml in your repository. You can specify a different file path by adding it as the first argument:

Screenshot of a custom pipeline file upload

A common use for custom file paths is when separating test and deployment steps into two completely separate pipelines, both using the same repository URL. For example, your test pipeline’s upload command could be:

buildkite-agent pipeline upload .buildkite/pipeline.yml

And your deployment pipeline’s upload command could be:

buildkite-agent pipeline upload .buildkite/pipeline.deploy.yml

For a list of all command line options, see the buildkite-agent pipeline upload documentation.

Dynamic pipelines

Because the pipeline upload step runs on your agent machine, you can generate pipelines dynamically using scripts from your source code. This provides you with the flexiblity to structure your pipelines as you wish.

The following example generates a list of parallel test steps based upon the test/* directories within your repository:

#!/bin/bash

# exit immediately on failure, or if an undefined variable is used
set -eu

# begin the pipeline.yml file
echo "steps:"

# add a new command step to run the tests in each test directory
for test_dir in test/*/; do
  echo "  - command: \"run_tests "${test_dir}"\"
done

To use this script, you'd save it to .buildkite/pipeline.sh inside your repository, ensure it is executable, and then update your pipeline upload step to use the new script:

.buildkite/pipeline.sh | buildkite-agent pipeline upload

When the build is run it will execute the script and pipe the output to the pipeline upload command. The upload command will insert the steps from the script into the build immediately after the upload step.

In the below pipeline.yaml example, when the build runs it will execute the .buildkite/pipeline.sh script, then the test steps from the script will be added to the build before the wait step and command step. After the test steps have run, the wait and command step will run.

steps:
  - command: .buildkite/pipeline.sh | buildkite-agent pipeline upload
    label: :pipeline: Upload
  - wait
  - command: "other-script.sh"
    label: Run other operations

Migrating existing steps to a pipeline.yml

If you have a pipeline that is defined in the web interface and would like to migrate it to a pipeline.yml without any interruption to builds, add a command step and wait step at the start of your pipeline:

For agents running Linux, macOS or other Unixes, the command is:

[[ -f ".buildkite/pipeline.yml" ]] && buildkite-agent pipeline upload --replace || true

Or, if your Agent runs Windows:

CMD /k if exist ".buildkite/pipeline.yml" buildkite-agent pipeline upload .buildkite/pipeline.yml --replace

This line of code will check if the pipeline file exists and, if it does, execute the buildkite agent pipeline upload --replace command to replace any steps that were defined via the web interface with those defined in the pipeline.yml.

To ensure that your existing steps don't run alongside the check, we include a wait step after it.

Your steps should then look something like this:

Screenshot of the above described pipeline upload command, before some existing steps

Cloning a pipeline

When creating a new pipeline, you can take a shortcut if you want to set up the new pipeline with the same steps as an existing pipeline.

Using the ?clone URL parameter, you can prefill the new pipeline page with the steps from another pipeline. It will not copy any other fields such as environment variables or repository information.

The below example URL will copy the steps from the 'My Llamas Pipeline' into the New Pipeline page:

https://buildkite.com/organizations/acme-inc/pipelines/new?clone=my-llamas-pipeline

Further documentation

You can also upload pipelines from the command line using the buildkite-agent command line tool. See the buildkite-agent pipeline documentation for a full list of the available parameters.