Managing Pipeline Secrets

When you need to use secret values in your pipelines, there are some best practices you should follow to ensure they stay safely within your infrastructure and are never stored in, or sent to, Buildkite.

Using a secrets storage service

A best practice for secret storage is to use your own secrets storage service, such as AWS Secrets Manager or Hashicorp Vault.

There are also various Buildkite Plugins that integrate reading and exposing secrets to your build steps using secrets storage services.

Storing secrets in environment hooks

If you don’t use a secrets storage service, the recommended place to store secrets is in agent environment hooks. Agent hooks are stored on your agent machine, are only accessible by you, and can conditionally expose secrets to your pipeline steps.

For example, say you had the following deploy command step in a pipeline:

steps:
  - command: scripts/trigger-github-deploy
    id: trigger-github-deploy

This step runs the scripts/trigger-github-deploy script from the repository, and creates a GitHub Deployment using a GitHub personal access token. The script has the following source code:

#!/bin/bash
set -euo pipefail

curl \
  --header "Authorization: token $GITHUB_MY_APP_DEPLOYMENT_ACCESS_TOKEN" \
  --request POST \
  --data "{\"ref\": \"$BUILDKITE_COMMIT\"}" \
  https://api.github.com/repos/my-org/my-app/deployments

To set up the GITHUB_MY_APP_DEPLOYMENT_ACCESS_TOKEN secret for the step, you would create an environment agent hook on your agent machine to conditionally export it. For example:

#!/bin/bash
set -euo pipefail

if [[ "$BUILDKITE_PIPELINE_SLUG" == "my-app" ]]; then
  if [[ "$BUILDKITE_STEP_IDENTIFIER" == "trigger-github-deploy" ]]; then
    export GITHUB_MY_APP_DEPLOYMENT_ACCESS_TOKEN="bd0fa963610b..."
  fi
fi

Adding conditional checks, such as the pipeline slug and the step identifier, helps to limit accidental use and discloure of secrets.

Storing secrets with the Elastic CI Stack for AWS

To store secrets when using the Elastic CI Stack for AWS, place them inside your stack’s encrypted S3 bucket. Unlike regular agent hooks, the Elastic CI Stack’s env hooks are per-pipeline.

For example, to expose a GITHUB_MY_APP_DEPLOYMENT_ACCESS_TOKEN environment variable to a step with identifier trigger-github-deploy, you would create the following env file on your local development machine:

#!/bin/bash
set -euo pipefail

if [[ "$BUILDKITE_STEP_IDENTIFIER" == "trigger-github-deploy" ]]; then
  export GITHUB_MY_APP_DEPLOYMENT_ACCESS_TOKEN="bd0fa963610b..."
fi

You then upload the env file, encrypted, into the secrets S3 bucket with the following command:

# Upload the env
aws s3 cp --acl private --sse aws:kms env "s3://elastic-ci-stack-my-stack-secrets-bucket/my-app/env"
# Remove the original file
rm env

See the Elastic CI Stack for AWS readme for more information and examples.

Anti-pattern: Storing secrets in your pipeline settings

You should never store secrets in the pipeline settings in the Buildkite web user interface. Not only does this expose the secret value to Buildkite, but pipeline settings are often returned in REST and GraphQL API payloads.

Never store secret values in your Buildkite pipeline settings.

Anti-pattern: Storing secrets in your pipeline.yml

You should never store secrets in the env block at the top of your pipeline steps, whether it's in a pipeline.yml file or the YAML Pipeline Steps (beta).

env:
  # Security risk! The secret will be sent to and stored by Buildkite, and
  # be available in the "Uploaded Pipelines" list in the job’s Timeline tab.
  GITHUB_MY_APP_DEPLOYMENT_ACCESS_TOKEN: "bd0fa963610b..."

steps:
  - command: scripts/trigger-github-deploy

Never store secrets in the env section of your pipeline.

Anti-pattern: Referencing secrets in your pipeline YAML

You should never refer to secrets directly in your pipeline.yml file, as they may be interpolated during the pipeline upload and sent to Buildkite. For example:

steps:
  # Security risk! The environment variable containing the secret will be
  # interpolated into the YML file and then sent to Buildkite.
  - command: |
      curl \
        --header "Authorization: token $GITHUB_MY_APP_DEPLOYMENT_ACCESS_TOKEN" \
        --request POST \
        --data "{\"ref\": \"$BUILDKITE_COMMIT\"}" \
        https://api.github.com/repos/my-org/my-app/deployments

Referencing secrets in your steps risks them being interpolated, uploaded to Buildkite, and shown in plain text in the "Uploaded Pipelines" list in the job’s Timeline tab.

To prevent the risk of interpolation, it is recommended that you replace the command block with a script in your repository, for example:

steps:
  - command: scripts/trigger-github-deploy

Use build scripts instead of command blocks for steps that use secrets.

If you must define your script in your steps, you can prevent interpolation by using the $$ syntax:

steps:
  # By using $$ the value of the secret is never sent to Buildkite. This is
  # still not best practice, as it's easy to forget the additional $ character
  # and expose the secret.
  - command: |
      curl \
        --header "Authorization: token $$GITHUB_MY_APP_DEPLOYMENT_ACCESS_TOKEN" \
        --request POST \
        --data "{\"ref\": \"$$BUILDKITE_COMMIT\"}" \
        https://api.github.com/repos/my-org/my-app/deployments