Buildkite Rails Docker Parallel Pipeline Example
This repository is an example Buildkite pipeline that demonstrates how to run a large number of parallel test jobs for a Rails application using Docker and Knapsack.
See the full Getting Started Guide for step-by-step instructions on how to get this running.
How it works
In the pipeline configuration file the parallelism property for the test step is set to 20. When a build, the step will appear 20 times in the pipeline, each with different environment variables exposed so you can divvy up your test suite accordingly. You can set the parallelism as high as you need to bring down your build times.
This example:
- Uses Docker Compose to orchestrate the environment (Redis, Postgres, Memcached)
- Runs tests in parallel across 20 containers using Buildkite’s
parallelismfeature - Leverages Knapsack for even test distribution
- Is suitable for use with autoscaling environments like the Buildkite AWS Stack
Key files:
docker-compose.yml: defines app servicesDockerfile: sets up the Ruby environment and database waiterspipeline.yml: configures a parallel test stepscripts/ci/parallel_specs.sh: runs specs in parallel
Parallel test execution
In pipeline.yml, the test step is configured with:
parallelism: 20This causes the step to run 20 times in parallel. Each instance is exposed to environment variables like BUILDKITE_PARALLEL_JOB and BUILDKITE_PARALLEL_JOB_COUNT, which can be used to split the test suite dynamically.
Alternate setups
💡 Not using Docker? Try the Rails Parallel Example
💡 Want dynamic splitting/allocation? See the Knapsack Pro version
Further Reading
See the parallelizing builds guide for more information to create parallelized and distributed builds with Buildkite.
See the Containerized Builds with Docker guide for information on how to use Docker Compose with Buildkite pipelines.
Consult the Knapsack documentation for configuring your database and dependent services to support running parallel steps on the one machine.
License
See LICENSE.md (MIT)
