Parallel Builds

Running a build's jobs in parallel is a way to decrease your build’s total running time. This guide will show you how to use multiple agents and job parallelism to increase the speed of your builds.

Running multiple agents

The steps for running multiple agents are slightly different for each platform. Automated installers and detailed instructions can be found in the installation section. For this example we'll use a Ubuntu-based build agent to start 5 instances of the Buildkite agent:

# After running the standard install instructions...

# Stop and disable the default buildkite-agent service
sudo systemctl stop buildkite-agent && sudo systemctl disable buildkite-agent

# Create a systemd template
sudo cp /lib/systemd/system/buildkite-agent.service /etc/systemd/system/buildkite-agent@.service

# Start five agents using the systemd template we created above
sudo systemctl enable --now buildkite-agent@1
sudo systemctl enable --now buildkite-agent@2
sudo systemctl enable --now buildkite-agent@3
sudo systemctl enable --now buildkite-agent@4
sudo systemctl enable --now buildkite-agent@5

Parallel Jobs

To run the same step in parallel over all 5 of the agents, we can set the parallelism field for a single build step:

steps:
  - command: "tests.sh"
    parallelism: 5

Update the name of the step to use %n, like the example below. This will substitute a number in at runtime so that you can differentiate the different build jobs:

steps:
  - command: "tests.sh"
    label: "Test %n"
    parallelism: 5

Now that the pipeline is configured, create a new build:

If you inspect the first job’s environment variables you’ll find:

BUILDKITE_PARALLEL_JOB=0
BUILDKITE_PARALLEL_JOB_COUNT=5

The BUILDKITE_PARALLEL_JOB environment variable stores the job number starting from 0. In this case each job’s value would be 0, 1, 2, 3, and 4 respectively.

The BUILDKITE_PARALLEL_JOB_COUNT environment variable stores the total number of jobs created from this step for this build.

You can use these two environment variables to divide your application’s tests between the different jobs.

Libraries

The following libraries have built-in support the BUILDKITE_PARALLEL_JOB and BUILDKITE_PARALLEL_JOB_COUNT environment variables:

• Knapsack
  Knapsack is a ruby gem for automatically dividing your tests between parallel jobs, as well as making sure each job runs in comparable time. It supports RSpec, Cucumber, and Minitest.

• Knapsack Pro
  A commercially supported version of Knapsack that provides a hosted service for test timing data and additional job distribution modes. See the documentation and step-by-step tutorial for setup instructions and example pipelines.

• Shardy McShardFace
  Shardy McShardFace is an npm package for dividing your tests between parallel jobs. it shards as evenly as possible, uneven splits will end up in the tail shards, supports sharding fewer items than the parallelism count, and distributes items into shards based on a given seed for a random number generator to provide random, but stable distribution. See the documentation.

Isolated jobs

You can safely run multiple build jobs on a single machine, as the agent runs each build in its own checkout directory. You’ll still need to ensure your application supports running in parallel on the same machine, and doesn't try to write to any shared resources at the same time (such as modifying the same database at the same time).

One convenient way of achieving build job isolation is to use the agent’s built in Docker Compose support which will run each job inside a set of completely isolated Docker containers.

Auto-scaling your build agents

In addition to the Elastic CI Stack for AWS (which has built-in support for auto-scaling) we provide a number of APIs and tools you can use to auto-scale your own build agents:

• GraphQL API allows you to efficiently fetch your organization’s scheduled jobs count, agents count, and details about each agent.
• Pipelines REST API and Agents API you’re able to fetch each pipeline’s job count, and information about each agent.
• Agent priorities allow you to define which agents are assigned work first, such as high performance ephemeral agents.
• Agent queues allow you to divide your agent pools into separate groups for scaling and performance purposes.
• buildkite-metrics tool allow you to collect your organization’s Buildkite metrics and report them to AWS CloudWatch and StatsD.

Using these tools you can automate your build infrastructure, scale your agents based on demand, and massively reduce build times using job parallelism.