Buildkite Agent Plugins

Plugins are only available in the beta 3.0 version of the agent

Plugins provide a way to extend the functionality that Buildkite natively offers. Another way to look at plugins is as a external repository of Hooks that are referenced from your pipeline.yml file, downloaded and run as part of your builds. They can be hosted on GitHub, BitBucket or a private git repository and are automatically fetched as required.

Buildkite offers a collection of useful plugins to get you started, including a docker-compose plugin and a plugin for secure secrets storage on Amazon S3.

Using a plugin

Plugins are only available for pipelines expressed in YAML, so all of these examples use that syntax.

Plugins are defined per-step and are automatically fetched from where they are hosted, based on the name of the project. See Plugin sources for how you can host your own.

An example of what a plugin might look like in your code:

steps:
  - plugins:
      detect-clowns:
        emojis: 🤡

This example will install the (hypothetical) buildkite-plugins/detect-clowns-buildkite-plugin when your pipeline is evaluated and run for this build step.

A real-life example is the docker-compose plugin. It provides an integration with docker-compose that assists with running your builds within a docker environment:

steps:
  # Prebuild the app image, upload it to a registry for later steps
  - name: "Docker Build"
    plugins:
      docker-compose#v1.3.2:
        build: app
        image-repository: index.docker.io/org/repo

  - wait

  # Use the app image built above to run concurrent tests
  - name: "Docker Test %n"
    command: test.sh
    parallelism: 25
    plugins:
      docker-compose#v1.3.2:
        run: app

In the 2.x agent, this approach was only possible by either using the docker environment variables, or writing a custom script.

Plugin sources

If you refer to a plugin just by name (say detect-clowns), it defaults to https://github.com/buildkite-plugins/detect-clowns-buildkite-plugin. If you used myorg/detect-clowns it would be https://github.com/myorg/detect-clowns-buildkite-plugin.

You can also use a fully qualified git repository name like https://github.com/buildkite-plugins/detect-clowns-buildkite-plugin.git or git@github.com:foo/bar.git#v1.3, which lets you refer to your own private git repositories. Branches, tags and commits are all valid after the #.

We recommend that plugins be pinned to a specific version to prevent behaviour changing unexpectedly. This is done with a git tag of vX.X.X.

Developing a plugin

The best place to get started is the Buildkite Plugins repository, or third-party plugins with the buildkite-agent tag.

We mostly write our plugins as shell scripts, so we've created a plugin tester docker image that simplifies writing BATS tests and getting your plugin tested (so meta).

Anatomy of a plugin

Plugins are essentially a collection of hooks with some metadata. The agent parses the plugin configuration and passes it on to the plugin in a slightly normalized form, both as JSON or environment variables.

This means that you can write a plugin as a shell script, or as a binary.

A minimal plugin has a plugin.json at it's root:

{
  "name": "docker",
  "description": "Runs your build steps in Docker containers",
  "author": "@buildkite",
  "public": true,
  "requirements": ["docker"]
}

And then hook scripts under hooks/. A good example is the docker plugin, which runs a command within a docker container.

The pipeline invocation looks like:

steps:
  - command: yarn install && yarn run test
    plugins:
      docker#v1.0.0:
        image: "node:7"
        workdir: /app

And then the logic of the (simplified) command hook script in hooks/command:

#!/bin/bash
set -euo pipefail

echo "--- Running ${BUILDKITE_COMMAND} in ${BUILDKITE_PLUGIN_DOCKER_IMAGE}"

docker run -it --rm \
  --volume "$PWD:${BUILDKITE_PLUGIN_DOCKER_WORKDIR}" \
  --workdir"${BUILDKITE_PLUGIN_DOCKER_WORKDIR}" \
  "${BUILDKITE_PLUGIN_DOCKER_IMAGE}" bash -c "${BUILDKITE_COMMAND}"

The agent parses the plugin block and passes the parameters through to the script as environment variables in the form BUILDKITE_PLUGIN_{NAME}_{KEY}, or if a list is provided then BUILDKITE_PLUGIN_{NAME}_{KEY}_{IDX}.

This lets a script supplement the standard environment variables that the agent provides to any script with structured configuration from the plugin definition.

Plugins are also able to offer command and environment hooks, which aren't allowed in per-project hooks. This lets plugins change how commands are executed entirely.

Security and Reliability

One issue with plugins is that they introduce a third-party dependency in your pipeline. They provide a new way to introduce executable code into your pipeline beyond scripts in your repository and access to the pipeline.yml.

The practices that we follow are:

  • Always pin to a particular version of the plugin, e.g. docker-compose#287293c4. You can refer to any git ref. The agent will check this out the first time it encounters it and then re-use that checked out code.

  • Follow the Unofficial Bash Strict Mode and check your work with shellcheck.

  • Apply special care in plugins to avoid executing arbitrary code passed in by config. As these are just environment variables, anything that can set environment variables can mess with plugin configuration.

For particularly high-security setups, you can disable plugins selectively by removing them from the BUILDKITE_PLUGINS variable in your environment hook.

We love feedback!

Agent plugins are still new, and we are evolving them as we learn more. We'd love feedback on how you use them and how we might make them better.