Parallel Builds

Running a build's jobs in parallel is a simple technique to decrease your build’s total running time. This guide will show you how to use multiple agents and job parallelism to increase the speed of your builds.

Running multiple agents

The steps for running multiple agents are slightly different for each platform. Automated installers and detailed instructions can be found in the installation section. For this example we'll use a Ubuntu-based build agent to start 5 instances of the Buildkite agent:

# After running the standard install instructions...

sudo cp /etc/init/buildkite-agent.conf /etc/init/buildkite-agent-2.conf
sudo service buildkite-agent-2 start

sudo cp /etc/init/buildkite-agent.conf /etc/init/buildkite-agent-3.conf
sudo service buildkite-agent-3 start

sudo cp /etc/init/buildkite-agent.conf /etc/init/buildkite-agent-4.conf
sudo service buildkite-agent-4 start

sudo cp /etc/init/buildkite-agent.conf /etc/init/buildkite-agent-5.conf
sudo service buildkite-agent-5 start

Parallel Jobs

To run the same step in parallel over all 5 of the agents, we can set the parallelism field for a single build step:

Update the name of the step to use %n, like the example below. This will substitute a number in at runtime so that you can differentiate the different build jobs:

In pipeline.yml format, this step would look like:

steps:
  - command: "test.sh"
    label: "Tests %n"
    parallelism: 5

Now that the pipeline is configured, create a new build:

If you inspect the first job’s environment variables you’ll find:

BUILDKITE_PARALLEL_JOB=0
BUILDKITE_PARALLEL_JOB_COUNT=5

The BUILDKITE_PARALLEL_JOB environment variable stores the job number starting from 0. In this case each job’s value would be 0, 1, 2, 3, and 4 respectively.

The BUILDKITE_PARALLEL_JOB_COUNT environment variable stores the total number of jobs created from this step for this build.

You can use these two environment variables to divide your application’s tests between the different jobs.

Libraries

The following libraries have built-in support the BUILDKITE_PARALLEL_JOB and BUILDKITE_PARALLEL_JOB_COUNT environment variables:

• Knapsack
  Knapsack is a ruby gem for automatically dividing your tests between parallel jobs, as well as making sure each job runs in comparable time. It supports RSpec, Cucumber, and Minitest.

• Knapsack Pro
  A commercially supported version of Knapsack that provides a hosted service for test timing data and additional job distribution modes. See the documentation and step-by-step tutorial for setup instructions and example pipelines.

Isolated jobs

You can safely run multiple build jobs on a single machine, as the agent runs each build in it's own checkout directory. You’ll still need to ensure your application supports running in parallel on the same machine, and doesn't try to write to any shared resources at the same time (such as modifying the same database at the same time).

One convenient way of achieving build job isolation is to use the agent’s built in Docker Compose support which will run each job inside a set of completely isolated Docker containers.

Auto-scaling your build agents

In addition to the Elastic CI Stack for AWS (which has built-in support for auto-scaling) we provide a number of APIs and tools you can use to auto-scale your own build agents:

• GraphQL API allows you to efficiently fetch your organization’s scheduled jobs count, agents count, and details about each agent.
• Pipelines REST API and Agents API you’re able to fetch each pipeline’s job count, and information about each agent.
• Agent priorities allow you to define which agents are assigned work first, such as high performance emphemeral agents.
• Agent queues allow you to divide your agent pools into separate groups for scaling and performance purposes.
• buildkite-metrics tool allow you to collect your organization’s Buildkite metrics and report them to AWS CloudWatch and StatsD.

Using these tools you can automate your build infrastructure, scale your agents based on demand, and massively reduce build times using job parallelism.