Securing your Buildkite Agent

In cases where a Buildkite Agent is being deployed into a sensitive environment there are a few default settings and techniques which may be adjusted.

Using environment hooks for secrets

The agent’s global environment hook is a convenient place for making secrets available to the build scripts. Once the environment hook exports the secret as an environment variable it will then be available to the build script. For example:

#!/bin/bash
set -euo pipefail

echo '--- :house_with_garden: Setting up the environment'

export GITHUB_RELEASE_ACCESS_KEY='xxx'

If you want to control which build steps have which secrets exposed you can check the value of $BUILDKITE_COMMAND before exporting it, for example:

#!/bin/bash
set -euo pipefail

echo '--- :house_with_garden: Setting up the environment'

if [[ "${BUILDKITE_COMMAND}" == "script/release" ]]; then
  export RELEASE_KEY=`vault get release-key`
fi

Disabling automatic ssh-keyscan

By default the agent will automatically accept the Git SSH host using the ssh-keyscan command when doing the first checkout on a new agent host. The agent runs a similar command to this:

ssh-keyscan "<host>" >> "~/.ssh/known_hosts"

If you choose to disable this functionality, you'll need to manually perform your first checkout, or ensure the SSH fingerprint of your source code host is already present on your build machine.

This operation can be disabled by setting no-ssh-keyscan in one of the following ways:

  • Environment variable: BUILDKITE_NO_SSH_KEYSCAN=true
  • Command line flag: --no-ssh-keyscan
  • Configuration setting: no-ssh-keyscan=true

Disabling command eval

By default the agent allows you to run any command on the build server (e.g. make test). You can disable command evaluation and allow only the execution of scripts (with no ability to pass command line flags). Disabling command evaluation will also disable plugin execution.

Once disabled your build steps will need to be checked into your repository as scripts, and the only way to pass arguments is via environment variables.

To disable command line evaluation use one of the following settings:

  • Environment variable: BUILDKITE_NO_COMMAND_EVAL=1
  • Command line flag: --no-command-eval
  • Configuration setting: no-command-eval=true

Custom hooks

If you have a custom 'command' hook, using no-command-eval will have no effect on your command execution. See Whitelisting and Custom bootstrap scripts for examples of how to completely lock down your agent from arbitrary code execution.

Disabling plugins

As plugins execute in the same way as local hooks, they can pose a potential security risk. If you're using third party plugins, you'll be executing the third party's code on your agent.

You can disable plugins with the command line flag: --no-plugins.

Disabling local hooks

If you have configured your agent to do very specific things with global hooks (i.e. run commands within a Docker container or a chroot environment), you probably don't want them overriden with local ones. Disabling local hooks will also disable plugins.

You can disable local hooks with the command line flag: --no-local-hooks.

Whitelisting

You can use the agent’s environment hook to whitelist repositories, commands, plugins, or any other logic you wish. The environment hook will be run before any source code is checked out onto the machine.

For example, the following environment hook allows only a single file from a single repository to be executed on the agent machine:

#!/bin/bash
set -euo pipefail

if [[ "${BUILDKITE_REPO}" != "git@server:repo.git" ]]; then
  echo "Repository not allowed: ${BUILDKITE_REPO}"
  exit 1
fi

if [[ "${BUILDKITE_COMMAND}" != "some-script.sh" ]]; then
  echo "Command not allowed: ${BUILDKITE_COMMAND}"
  exit 1
fi

if [[ "${BUILDKITE_PLUGINS:-}" != "" ]]; then
  echo "Plugins not allowed: ${BUILDKITE_PLUGINS}"
  exit 1
fi

Whitelisting Plugins

Per above, whitelisting can be done with the agent’s environment hook. It may be desirable to whitelist a set of plugins that agents are allowed to execute.

See the buildkite/buildkite-plugin-whitelister for an example of checking requested plugins against a whitelist.

Customizing the bootstrap

The Buildkite Agent comes with a default bootstrap script, but can be configured to run your own. Providing your own bootstrap provides the highest level of security and control of your agent machine. You can use it to customize your agent, sanitize command output, and implement your own security logic.

The Buildkite Agent is separated into a daemon executable and a bootstrap executable. The daemon is responsible for communicating with the Buildkite API and executing the bootstrap. The bootstrap is responsible checking out source code, calling hooks, running commands, and uploading build artifacts. The bootstrap is passed environment variables by the daemon process, and has its output streams and exit status captured.

For example, the following custom bootstrap will print out the environment variables passed by the main agent process, print a hello world, and exit with a failure status:

#!/bin/bash
set -euo pipefail

env

echo "Hello world"

exit 1

Forcing clean checkouts

By default Buildkite will reuse (after cleaning) a previous checkout. This may not be safe if building commits from untrusted sources (e.g. 3rd party pull requests). To force a clean checkout every time, set BUILDKITE_CLEAN_CHECKOUT=true in the environment.

Disallowing hooks in the repository

The default bootstrap script will execute hooks from the agent configuration (global) and the repository (local). Running local hooks may be undesirable if you are building commits from untrusted sources (e.g. 3rd party pull requests).

Local hooks can be disabled by setting BUILDKITE_NO_LOCAL_HOOKS=true in the environment.

If local hooks are disabled and one is in the checkout, the job will fail.

NOTE: If building untrusted commits, you should also contain the build scripts and anything else that may be influenced by the repository contents within chroots, containers, VMs, etc as is appropriate for your needs.