Extending Buildkite with plugins: HashiCorp Vault

When it comes to CI/CD products, extensibility is key. Every team has a slightly different toolset or way of working, so the best products let you adapt them to suit your needs. This ensures you and your team get the best possible experience from a tool.

Buildkite supports extensibility in many ways, and plugins are a major part of this. In this post, we'll describe how plugins work in Buildkite and introduce you to the Vault secrets plugin. The Vault secrets plugin is the recommended way to integrate with HashiCorp Vault and a great option for protecting your secrets.

What is a plugin?

Plugins are small self-contained pieces of functionality you add to pipelines to customize Buildkite to your specific workflow. They modify the behavior of jobs in a pipeline using one or more of the job lifecycle hooks. Each hook modifies a different part of the job lifecycle, for example:

• Setting up the environment.
• Checking out the code.
• Running commands.
• Handling artifacts.
• Cleaning up the environment.

Plugins can be open source and available for anyone to use or private and kept in a repository only your organization and agents can access. If you keep them private, they can be in a centralized place or the same repository as your source code.

Using plugins

Rather than a web-based plugin management system like other CI/CD tools, Buildkite lets you manage plugins directly in pipeline definitions. This makes Buildkite plugins more decentralized and allows for easier version control.

You use plugins in pipeline command steps to access a library of commands or perform actions. To add a plugin to a command step, use the plugins attribute in your pipeline definition. The plugins attribute accepts an array so you can add multiple plugins to the same step.

For example, the following pipeline runs test.sh inside an app service container using the Docker Compose plugin:

steps:
  - command: test.sh
    plugins:
      - docker-compose#v4.14.0:
          run: app

This is equivalent to running:

docker-compose run app test.sh

Spotlight: Vault secrets plugin

HashiCorp Vault is an identity-based secrets and encryption management system, and Buildkite is an Enterprise Verified Premier Partner of HashiCorp. You can use Vault in your pipelines to manage secrets securely. A secret in CI/CD is anything you want to tightly control access to, such as API keys, passwords, and certificates.

The Vault secrets plugin is developed and maintained by Buildkite as the recommended way to use Vault in your pipelines. It aims to provide a general solution most teams can use to integrate with Vault.

The plugin allows agents to authenticate to Vault and acquire pipeline secrets while running a job. It also enables you to create more granular policies for access by agents and pipelines, making it easier to have a better security posture.

The plugin wraps Vault's CLI and allows you to configure multiple authentication methods to acquire a token, including AppRole, AWS, and JWT.

For example, the following pipeline accesses Vault using AppRole authentication:

steps:
  - command: ./run_build.sh
    plugins:
      - vault-secrets#v1.1.0:
          server: "https://my-vault-server"
          path: secret/buildkite
          auth:
            method: "approle"
            role-id: "my-role-id"
            secret-env: "VAULT_SECRET_ID"

The order of operations goes like the following:

1. Authenticate to Vault. This gets a token scoped to the appropriate policies using the configured authentication method.
2. Check Vault for any secrets using different paths:
   • env
   • environment
   • private_ssh_key
   • id_rsa_github
   • git-credentials
3. Download the secrets, and add them to the job environment.

To learn more or contribute to the plugin, see the Vault secrets plugin on GitHub. You can also use this demo repository to quickly get a local Vault environment running.

Finding plugins

Many common problems or workflows already have dedicated plugins from Buildkite or the Buildkite community. Check the Plugins directory if you want to look for an existing solution.

If you can't find an existing plugin that does what you need, check out the documentation to write your own. Buildkite plugins are written in Bash and loosely coupled with Buildkite, making them more maintainable and less prone to compatibility issues than in other CI/CD tools.