Defining Your Pipeline Steps

Pipeline steps are defined in YAML and are either stored in Buildkite or in your repository using a pipeline.yml file.

Defining your pipeline steps in a pipeline.yml file gives you access to more configuration options and environment variables than the web interface, and allows you to version, audit and review your build pipelines alongside your source code.

Getting started

Create a pipeline from the Pipelines page of Buildkite using the ➕ button.

Required fields are 'name', and 'repository'.

Screenshot of the 'New Pipeline' setup form

You can set up webhooks at this point, but this step is optional. These webhook setup instructions can be found in pipeline settings on your specific repository provider page.

Both the REST API and GraphQL API can be used to create a pipeline programmatically. See the Pipelines REST API and the GraphQL API for details and examples.

Adding steps

There are two ways to define steps in your pipeline: using the YAML step editor in Buildkite or with a pipeline.yml file. The web steps visual editor is still available if you haven't migrated to YAML Steps but will be deprecated in the future.

If you have not yet migrated to YAML Steps, you can do on your pipeline's settings page. See the Migrating to YAML Steps guide for more information about the changes and the migration process.

However you add steps to your pipeline, keep in mind that steps may run on different agents. It is good practice to install your dependencies in the same step that you run them.

YAML steps editor

To add steps using the YAML editor, click the 'Edit Pipeline' button on the Pipeline Settings page.

Starting your YAML with the steps object, you can add as many steps as you require of each different type. Quick reference documentation and examples for each step type can be found in the sidebar on the right.

Pipeline.yml file

Before getting started with a pipeline.yml file, you'll need to tell Buildkite where it will be able to find your steps.

In the YAML steps editor, add the following YAML:

steps:
  - label: "<img class="emoji" title="pipeline" alt=":pipeline:" src="https://buildkiteassets.com/emojis/img-buildkite-64/pipeline.png" draggable="false" /> Pipeline upload"
    command: buildkite-agent pipeline upload

When you eventually run a build from this pipeline, this step will look for a directory called .buildkite containing a file named pipeline.yml. Any steps it finds inside that file will be uploaded to Buildkite and will appear during the build.

Create your pipeline.yml file in a .buildkite directory in your repo.

If you're using any tools that ignore hidden directories, you can store your pipeline.yml file either in the top level of your repository, or in a non-hidden directory called buildkite. The upload command will search these places if it doesn't find a .buildkite directory.

The following example YAML defines a pipeline with one command step that will echo 'Hello' into your build log:

pipeline.yml
steps:
  - label: "Example Test"
    command: echo "Hello!"

With the above example code in a pipeline.yml file, commit and push the file up to your repository. If you have set up webhooks, this will automatically create a new build. You can also create a new build using the 'New Build' button on the pipeline page.

Screenshot of the build passing with pipeline upload step first, and then the example step

For more example steps and detailed configuration options, see the example pipeline.yml below, or the step type specific documentation: command steps, wait steps, block steps, input steps, and trigger steps.

If your pipeline has more than one step and you have multiple agents available to run them, they will automatically run at the same time. If your steps rely on running in sequence, you can separate them with wait steps. This will ensure that any steps before the 'wait' are completed before steps after the 'wait' are run.

When a step is run by an agent, it will be run with a clean checkout of the pipeline's repository. If your commands or scripts rely on the output from previous steps, you will need to either combine them into a single script or use artifacts to pass data between steps. This enables any step to be picked up by any agent and run steps in parallel to speed up your build.

Example pipeline

Here’s a more complete example based on the Buildkite Agent’s build pipeline. It contains script commands, wait steps, block steps, and automatic artifact uploading:

pipeline.yml
steps:
  - label: ":hammer: Tests"
    command: scripts/tests.sh
    env:
      BUILDKITE_DOCKER_COMPOSE_CONTAINER: app

  - wait

  - label: ":package: Package"
    command: scripts/build-binaries.sh
    artifact_paths: "pkg/*"
    env:
      BUILDKITE_DOCKER_COMPOSE_CONTAINER: app

  - wait

  - label: ":debian: Publish"
    command: scripts/build-debian-packages.sh
    artifact_paths: "deb/**/*"
    branches: "master"
    agents:
      queue: "deploy"

  - block: ":shipit: Release"
    branches: "master"

  - label: ":github: Release"
    command: scripts/build-github-release.sh
    artifact_paths: "releases/**/*"
    branches: "master"

  - wait

  - label: ":whale: Update images"
    command: scripts/release-docker.sh
    branches: "master"
    agents:
      queue: "deploy"

Step types

There are four different step types you can use in your Buildkite pipelines. Click through each step type for detailed usage documentation:

Customizing the pipeline upload path

By default the pipeline upload step reads your pipeline definition from .buildkite/pipeline.yml in your repository. You can specify a different file path by adding it as the first argument:

steps:
  - label: "<img class="emoji" title="pipeline" alt=":pipeline:" src="https://buildkiteassets.com/emojis/img-buildkite-64/pipeline.png" draggable="false" /> Pipeline upload"
    command: buildkite-agent pipeline upload .buildkite/deploy.yml

A common use for custom file paths is when separating test and deployment steps into two separate pipelines. Both pipeline.yml files are stored in the same repo and both Buildkite pipelines use the same repo URL. For example, your test pipeline’s upload command could be:

buildkite-agent pipeline upload .buildkite/pipeline.yml

And your deployment pipeline’s upload command could be:

buildkite-agent pipeline upload .buildkite/pipeline.deploy.yml

For a list of all command line options, see the buildkite-agent pipeline upload documentation.

Dynamic pipelines

Because the pipeline upload step runs on your agent machine, you can generate pipelines dynamically using scripts from your source code. This provides you with the flexibility to structure your pipelines however you require.

The following example generates a list of parallel test steps based upon the test/* directory within your repository:

pipeline.sh
#!/bin/bash

# exit immediately on failure, or if an undefined variable is used
set -eu

# begin the pipeline.yml file
echo "steps:"

# add a new command step to run the tests in each test directory
for test_dir in test/*/; do
  echo "  - command: \"run_tests "${test_dir}"\""
done

To use this script, you'd save it to .buildkite/pipeline.sh inside your repository, ensure it is executable, and then update your pipeline upload step to use the new script:

.buildkite/pipeline.sh | buildkite-agent pipeline upload

When the build is run it will execute the script and pipe the output to the pipeline upload command. The upload command will insert the steps from the script into the build immediately after the upload step.

In the below pipeline.yml example, when the build runs it will execute the .buildkite/pipeline.sh script, then the test steps from the script will be added to the build before the wait step and command step. After the test steps have run, the wait and command step will run.

pipeline.yml
steps:
  - command: .buildkite/pipeline.sh | buildkite-agent pipeline upload
    label: ":pipeline: Upload"
  - wait
  - command: "other-script.sh"
    label: "Run other operations"

Cloning a pipeline

When creating a new pipeline, you can take a shortcut if you want to set up the new pipeline with the same steps as an existing pipeline.

Using the ?clone URL parameter, you can prefill the new pipeline page with the steps from another pipeline. It will not copy any other fields such as environment variables or repository information.

The below example URL will copy the steps from the 'My Llamas Pipeline' into the New Pipeline page:

https://buildkite.com/organizations/acme-inc/pipelines/new?clone=my-llamas-pipeline

Further documentation

You can also upload pipelines from the command line using the buildkite-agent command line tool. See the buildkite-agent pipeline documentation for a full list of the available parameters.