Defining Your Pipeline Steps

You define your build pipeline steps in your pipeline settings or, more powerfully, via a pipeline.yml file.

Defining your pipeline steps in a pipeline.yml file gives you access to more configuration options than is available in the web interface, and allows you to version, audit and review your build pipelines alongside your source code.

Getting started

In your pipeline settings, click the ➕ to add a step and choose “Read steps from repository”:

Screenshot of the 'Read steps from repository' menu item

Your pipeline steps will now show a new :pipeline: step that runs a buildkite-agent pipeline upload command:

Screenshot of the pipeline upload step

When this step runs on an agent, it reads in the pipeline file from .buildkite/pipeline.yml in your repository. For example, here is a pipeline file with a single step that prints “Hello!” to the build log:

steps:
  - label: Example Test
    command: echo "Hello!"

To test your .buildkite/pipeline.yml file, commit the pipeline file and push it to your repository to create a new build:

Screenshot of the build passing with pipeline upload step first, and then the example step

Example pipeline

Here’s a more complete example based on our own Buildkite Agent’s build pipeline. It contains script commands, wait steps, block steps, and artifact uploading:

steps:
  - label: ':hammer: Tests'
    command: 'scripts/tests.sh'
    env:
      BUILDKITE_DOCKER_COMPOSE_CONTAINER: app

  - wait

  - label: ':package: Package'
    command: 'scripts/build-binaries.sh'
    artifact_paths: 'pkg/*'
    env:
      BUILDKITE_DOCKER_COMPOSE_CONTAINER: app

  - wait

  - label: ':debian: Publish'
    command: 'scripts/build-debian-packages.sh'
    artifact_paths: 'deb/**/*'
    branches: 'master'
    agents:
      queue: 'deploy'

  - block: ':shipit: Release'
    branches: 'master'

  - label: ':github: Release'
    command: 'scripts/build-github-release.sh'
    artifact_paths: 'releases/**/*'
    branches: 'master'

  - wait

  - label: ':whale: Update images'
    command: 'scripts/release-docker.sh'
    branches: 'master'
    agents:
      queue: 'deploy'

Step types

There are four different step types you can use in your Buildkite pipelines. See each step type for full documentation:

Customizing the pipeline file path

By default the pipeline upload step reads your pipeline definition from .buildkite/pipeline.yml in your repository. You can specify a different file path by adding it as the first argument:

Screenshot of a custom pipeline file upload

A common use for custom file paths is when separating test and deployment steps into two completely separate pipelines, both using the same repository URL. For example, your test pipeline’s upload command could be:

buildkite-agent pipeline upload .buildkite/pipeline.yml

And your deployment pipeline’s upload command could be:

buildkite-agent pipeline upload .buildkite/pipeline.deploy.yml

For a list of all command line options, see the buildkite-agent pipeline upload documentation.

Dynamic pipelines

Because the pipeline upload step runs on your agent machine, you can generate pipelines dynamically using scripts from your source code. This provides you with the flexiblity to structure your pipelines as you wish.

The following example generates a list of parallel test steps based upon the test/* directories within your repository:

#!/bin/bash

# exit immediately on failure, or if an undefined variable is used
set -eu

# begin the pipeline.yml file
echo "steps:"

# add a new command step to run the tests in each test directory
for test_dir in test/*/; do
  echo "  - command: \"run_tests "${test_dir}"\"
done

To use this script, you'd save it to .buildkite/pipeline.sh inside your repository, ensure it is executable, and then update your pipeline upload step to use the new script:

.buildkite/pipeline.sh | buildkite-agent pipeline upload

When a build is run it will execute the script and pipe its output to the pipeline upload command, dynamically adding build steps to the end of the build pipeline.

Migrating existing steps to a pipeline.yml

If you have a pipeline that is defined in the web interface and would like to migrate it to a pipeline.yml without any interruption to builds, you can add the following command step at the start of your pipeline:

[[ -f ".buildkite/pipeline.yml" ]] && buildkite-agent pipeline upload --replace || true

This line of code will check if the pipeline file exists and, if it does, execute the buildkite agent pipeline upload --replace command to replace any steps that were defined via the web interface with those defined in the pipeline.yml.

Further documentation

See the buildkite-agent pipeline documentation for a full list of options.