Running a build's jobs in parallel is a way to decrease your build’s total running time. This guide will show you how to use multiple agents and job parallelism to increase the speed of your builds.
On this page:
Running multiple agents
There are two techniques for scaling your build agents: horizontally across multiple machines, or vertically on a single machine. For the sake of simplicity, the following example shows vertically scaling your agents on a single machine. For details on horizontal scaling, which allows for running hundreds or thousands of build agents, see the auto-scaling your build agents section.
The steps for running multiple agents are slightly different for each platform. Automated installers and detailed instructions can be found in the installation section. For this example we'll use a Ubuntu-based build agent to start 5 instances of the Buildkite agent:
# After running the standard install instructions... # Stop and disable the default buildkite-agent service sudo systemctl stop buildkite-agent && sudo systemctl disable buildkite-agent # Create a systemd template sudo cp /lib/systemd/system/buildkite-agent.service /etc/systemd/system/buildkite-agent@.service # Start five agents using the systemd template we created above sudo systemctl enable --now buildkite-agent@1 sudo systemctl enable --now buildkite-agent@2 sudo systemctl enable --now buildkite-agent@3 sudo systemctl enable --now buildkite-agent@4 sudo systemctl enable --now buildkite-agent@5
To run the same step in parallel over all 5 of the agents, we can set the
parallelism field for a single build step:
Update the name of the step to use
%n, like the example below. This will substitute a number in at runtime so that you can differentiate the different build jobs:
Now that the pipeline is configured, create a new build:
If you inspect the first job’s environment variables you’ll find:
BUILDKITE_PARALLEL_JOB environment variable stores the index of each parallel job created from a parallel build step, starting from 0. For a build step with
parallelism: 5, the value would be 0, 1, 2, 3, and 4 respectively.
BUILDKITE_PARALLEL_JOB_COUNT environment variable stores the total number of jobs created from this step for this build.
You can use these two environment variables to divide your application’s tests between the different jobs.
The following libraries have built-in support the
BUILDKITE_PARALLEL_JOB_COUNT environment variables:
Knapsack is a ruby gem for automatically dividing your tests between parallel jobs, as well as making sure each job runs in comparable time. It supports RSpec, Cucumber, and Minitest.
Shardy McShardFace is an npm package for dividing your tests between parallel jobs. it shards as evenly as possible, uneven splits will end up in the tail shards, supports sharding fewer items than the parallelism count, and distributes items into shards based on a given seed for a random number generator to provide random, but stable distribution. See the documentation.
You can safely run multiple build jobs on a single machine, as the agent runs each build in its own checkout directory. You’ll still need to ensure your application supports running in parallel on the same machine, and doesn't try to write to any shared resources at the same time (such as modifying the same database at the same time).
One convenient way of achieving build job isolation is to use the agent’s built in Docker Compose support which will run each job inside a set of completely isolated Docker containers.
Auto-scaling your build agents
In addition to the Elastic CI Stack for AWS (which has built-in support for auto-scaling) we provide a number of APIs and tools you can use to auto-scale your own build agents:
- GraphQL API allows you to efficiently fetch your organization’s scheduled jobs count, agents count, and details about each agent.
- Pipelines REST API and Agents API you’re able to fetch each pipeline’s job count, and information about each agent.
- Agent priorities allow you to define which agents are assigned work first, such as high performance ephemeral agents.
- Agent queues allow you to divide your agent pools into separate groups for scaling and performance purposes.
- buildkite-metrics tool allow you to collect your organization’s Buildkite metrics and report them to AWS CloudWatch and StatsD.
Using these tools you can automate your build infrastructure, scale your agents based on demand, and massively reduce build times using job parallelism.