The Buildkite agent
The Buildkite agent is a small, reliable and cross-platform build runner that makes it easy to run automated builds on your own self-hosted or Buildkite's hosted infrastructure. The agent's main responsibilities are polling buildkite.com for work, running a build's jobs, reporting back the status code and output log of the job, and uploading the job's artifacts.
This page provides Buildkite organization administrators with an overview of the differences between self-hosted and Buildkite hosted agents, how the Buildkite agent works, the agent's lifecycle, how to customize the agent's functionality with hooks, and the agent's command line usage.
If you're new to Buildkite Pipelines, run through the Getting started with Pipelines tutorial, which will initially set you up to run Buildkite hosted agents. From there, you can decide whether to continue using Buildkite hosted agents, or set yourself up to run self-hosted agents.
Self-hosted and Buildkite hosted agents compared
The following table lists key feature differences between self-hosted and Buildkite hosted agents. If you are looking to establish, expand or modify your Buildkite agent infrastructure, this table should help you choose which path or paths to take.
In summary though:
-
Self-hosted agents are suitable when your organization has any of the following requirements:
- You need full control over your agent infrastructure.
- Your agents need a lot of customization.
- You operate under strict security conditions that requires your source code and CI/CD build runners (the agents) to be managed on premises or in your own cloud-based infrastructure.
Buildkite hosted agents is a fully-managed platform that offers fast and specialized CI/CD build runners, which work well under default conditions. This option lets you get up and running rapidly to build your projects.
| Feature | Self-hosted agents | Buildkite hosted agents |
|---|---|---|
|
Infrastructure management |
You provision, scale, and maintain your own servers |
Fully managed by Buildkite |
|
Supported platforms |
||
|
Agent lifecycle |
Persistent or ephemeral (your choice) |
Always ephemeral, destroyed after each job |
|
Scaling |
Manual or through tools such as the Elastic CI Stack for AWS or Agent Stack for Kubernetes |
Automatic, scales on demand |
|
Agent version updates |
You manage upgrades across your fleet |
Managed by Buildkite |
|
Configuration |
Full control through configuration file, environment variables, and command-line flags |
Configured through queue settings in the Buildkite interface |
|
All agent and job lifecycle hooks supported |
Job lifecycle hooks supported through custom base images |
|
|
Cache volumes |
You manage your own caching |
Built-in NVMe-backed cache volumes at no extra cost |
|
Docker builds |
You configure Docker on your agents |
Remote Docker builders with layer caching (Enterprise only) |
|
Container registry |
You manage your own registries |
Built-in internal container registry |
|
Network access |
Full control over networking |
US East Coast data centers with configurable network security |
|
Experimental features |
Access to agent experiments |
Not available |
How it works
The agent works by polling Buildkite's agent API over HTTPS. There is no need to forward ports or provide incoming firewall access, and the agents can be run across any number of machines and networks.
The agent starts by registering itself with Buildkite, and once registered it's placed into your organization's agents pool. The agent periodically polls the Buildkite platform, looking for new work, waiting to accept an available job.
After accepting a build job the agent will execute the command, streaming back the build script's output and then posting the final exit status.
Whilst the job is running you can use the buildkite-agent meta-data command to set and get build-wide meta-data, and buildkite-agent artifact for fetching and retrieving binary build-wide artifacts. These two commands allow you to have completely isolated build jobs (similar to a 12 factor web application) but have access to shared state and data storage across any number of machines and networks.
Job routing
By default, a pipeline's jobs run on the first available agent associated with the relevant queues that the pipeline's cluster is set to. Agents associated with a queue are ordered for selection by how recently these agents successfully completed a job. This takes advantage of warm caches to guarantee the fastest run time possible.
Learn more about how Buildkite routes jobs to queues in the Queues overview page.
Agent lifecycle
The agent goes through several stages during its operation, from starting up and registering with Buildkite, through to polling for and running jobs, and shutting down. For details on signal handling, exit codes, and troubleshooting common lifecycle issues, see the Agent lifecycle page.
Customizing with hooks
The agent's behavior can be customized using hooks, which are shell scripts that exist on your build machines or in each pipeline's code repository. Hooks can be used to set up secrets as well as overriding default behavior. See the hooks documentation for full details.
Command line usage
The Buildkite agent has a command line interface (CLI) that lets you interact with and control the agent through the command line. For a complete reference of all available commands, see the Command-line reference.