1. Resources
  2. /
  3. Plugins
  4. /
  5. buildpipe-buildkite-plugin

Buildpipe

A Buildkite plugin to dynamically generate pipelines. Especially useful for monorepos where you want to create dependencies between projects.

Example

initial_pipeline.yml

steps:
  - label: ":buildkite:"
    plugins:
      - jwplayer/buildpipe#v0.10.3:
          dynamic_pipeline: dynamic_pipeline.yml

dynamic_pipeline.yml

env:
  GLOBAL_ENV_VAR: test
projects:
 - label: project1
   path: project1/  # changes in this directory will trigger steps for project1
   skip:
     - deploy*  # skip steps with label matching deploy* (e.g. deploy-prd)
     - test
   env:
     PROJECT_ENV_VAR: project1variable
 - label: project2
   skip: test
   path:
      - project2/
      - project1  # you can trigger a project using multiple paths
 - label: project3
   skip:
     - deploy-stg
   path: project3/somedir/  # subpaths can also be triggered
steps:  # the same schema as regular buildkite pipeline steps
  - label: test
    env:
      BUILDPIPE_SCOPE: project  # this variable ensures a test step is generated for each project
    command:
      - cd $$BUILDPIPE_PROJECT_PATH  # BUILDPIPE_PROJECT_PATH is set by buildpipe
      - make test
  - wait
  - label: build
    branches: "master"
    env:
      BUILDPIPE_SCOPE: project
    command:
      - cd $$BUILDPIPE_PROJECT_PATH
      - make build
      - make publish-image
    agents:
      - queue=build
  - wait
  - label: tag
    branches: "master"
    command:
      - make tag-release
  - wait
  - label: deploy-stg
    branches: "master"
    concurrency: 1
    concurrency_group: deploy-stg
    env:
      BUILDPIPE_SCOPE: project
    command:
      - cd $$BUILDPIPE_PROJECT_PATH
      - make deploy-staging
  - wait
  - block: ":rocket: Release!"
    branches: "master"
  - wait
  - label: deploy-prd
    branches: "master"
    concurrency: 1
    concurrency_group: deploy-prd
    env:
      BUILDPIPE_SCOPE: project
    command:
      - cd $$BUILDPIPE_PROJECT_PATH
      - make deploy-prod

The above pipelines specify the following:

  • There are three projects to track in the repository.
  • The env variable BUILDPIPE_SCOPE: project tells buildpipe to generate a step for each project if that project changed.
  • The skip option will skip any step label matching deploy*.
  • The env variable BUILDPIPE_PROJECT_PATH is created by buildpipe as the project’s path. If multiple paths are specified for a project, it’s the first path.
  • There is a global environment variable defined (GLOBAL_ENV_VAR). This variable will be added to every step in the pipeline.
  • There is also a project scoped environment variable defined (PROJECT_ENV_VAR). That variable will be added to all steps of the project where it was defined (project1).

Full working example

For a full working example, check out Buildkite Monorepo Example.

Configuration

Plugin

OptionRequiredTypeDefaultDescription
default_branchNostringmasterDefault branch of repository
diff_prNostringOverride command for non-default branch (see below for a better explanation of the defaults)
diff_defaultNostringOverride command for default branch (see below for a better explanation of the defaults)
dynamic_pipelineYesstringThe name including the path to the pipeline that contains all the actual steps
log_levelNostringINFOThe Level of logging to be used by the python script underneath; pass DEBUG for verbose logging if errors occur

Project schema

OptionRequiredTypeDefaultDescriptionEnvironment variable
labelYesstringProject labelBUILDPIPE_PROJECT_LABEL
pathYesarrayThe path(s) that specify changes to a projectBUILDPIPE_PROJECT_PATH
skipNoarrayExclude steps that have labels that match the rule
envNodictionaryDefine environment variable on a project scope

Other useful things to note:

  • Option skip makes use of Unix shell-style wildcards (Look at .gitignore files for inspiration)
  • If multiple paths are specified, the environment variable BUILDPIPE_PROJECT_PATH will be the first path.
  • Environment variables are available in the pipeline step.

Environment Variables

Since version 0.9.2, there is the option to define environment variables on different scope levels in the pipeline. Buildkite already supports environment variables defined on a step scope, but buildpipe adds the ability to define global and also project specific environment variables.

Project env vars

env:
  THIS_IS_A_GLOBAL_ENV_VAR: global
  GLOBAL_ENV_VAR_2: another_global_value
projects:
  - label: project1
    path: project1/
    env:
      THIS_IS_A_PROJECT_ENV_VAR: project_scoped
      PROJECT_ENV_VAR_2: second_env_var_value
  - label: project2
    ...
steps:
  - label: step1
    ...

Specifying build projects

Since version 0.10.0, you can manually specify which projects you wish to build using the BUILDKITE_PLUGIN_BUILDPIPE_BUILD_PROJECTS environment variable. If this environemnt variable is provided, it will take precedence over any existing git diff. You can specify the build projects in one of two ways:

  1. Provide a comma-separated list of build projects (identified by the project label attribute). For example:
BUILDKITE_PLUGIN_BUILDPIPE_BUILD_PROJECTS="project1,project2"
  1. Build all projects in a Buildpipe configuration:
BUILDKITE_PLUGIN_BUILDPIPE_BUILD_PROJECTS="*"

diff_ commands

Depending on your merge strategy, you might need to use different diff command.

Buildpipe assumes you are using a merge strategy on the default branch, which is assumed to be master.

The command for the non-default branch (e.g. when you have a PR up) is:

git log --name-only --no-merges --pretty=format: origin..HEAD

The command for the default branch you merge to is currently:

git log -m -1 --name-only --pretty=format: $BUILDKITE_COMMIT

Requirements

Only curl is required to download the binary.

Installing buildpipe

You can also install buildpipe in the agent bootstrap script:

curl -Lf -o /usr/local/bin/buildpipe https://github.com/jwplayer/buildpipe-buildkite-plugin/releases/download/v${BUILDPIPE_VERSION}/buildpipe-linux \
  && chmod +x /usr/local/bin/buildpipe

Your initial pipeline would need to pass the options as environment variables with prefix BUILDKITE_PLUGIN_BUILDPIPE_. For example:

steps:
  - label: "buildpipe"
    command: buildpipe
    env:
      BUILDKITE_PLUGIN_BUILDPIPE_DYNAMIC_PIPELINE: path/to/dynamic_pipeline.yml
      BUILDKITE_PLUGIN_BUILDPIPE_LOG_LEVEL: debug

Troubleshooting

Buildpipe is incorrectly showing project as changed

Buildkite doesn’t by default do clean checkouts. To enable clean checkouts set the BUILDKITE_CLEAN_CHECKOUT environment variable. An example is to modify the pre-checkout hook, .buildkite/hooks/pre-checkout:

#!/bin/bash
set -euo pipefail

echo '--- :house_with_garden: Setting up pre-checkout'

export BUILDKITE_CLEAN_CHECKOUT="true"

Testing

make test

License

Apache v2

The plugins listed on this webpage are provided for informational purposes only. They have not undergone any formal security review or assessment. While we strive to provide useful resources, we cannot guarantee the safety, reliability, or integrity of these plugins. Users are strongly advised to conduct their own security evaluations before downloading, installing, or using any plugin. By using these plugins, you acknowledge and accept any risks associated with their use. We disclaim any liability for any harm or damages arising from the use of the plugins listed.

Start turning complexity into an advantage

Create an account to get started with a 30-day free trial. No credit card required.

Buildkite Pipelines

Platform

  1. Pipelines
  2. Pipeline templates
  3. Public pipelines
  4. Test Engine
  5. Package Registries
  6. Mobile Delivery Cloud
  7. Pricing

Hosting options

  1. Self-hosted agents
  2. Mac hosted agents
  3. Linux hosted agents

Resources

  1. Docs
  2. Blog
  3. Changelog
  4. Webinars
  5. Plugins
  6. Case studies
  7. Events
  8. Migration Services
  9. Comparisons

Company

  1. About
  2. Careers
  3. Press
  4. Brand assets
  5. Contact

Solutions

  1. Replace Jenkins
  2. Workflows for AI/ML
  3. Testing at scale
  4. Monorepo mojo
  5. Bazel orchestration

Legal

  1. Terms of Service
  2. Acceptable Use Policy
  3. Privacy Policy
  4. Subprocessors
  5. Service Level Agreement

Support

  1. System status
  2. Forum
© Buildkite Pty Ltd 2025