Configuring test splitting

Buildkite maintains its open source Test Engine Client (bktec) tool. This tool uses your Test Engine test suite data to intelligently partition tests throughout your test suite into multiple sets, such that each set of tests runs in parallel across your agents. This results in a test plan, where a test plan defines which tests are run on which agents. Currently, the Test Engine Client tool only supports RSpec and Jest.

Dependencies

The Test Engine Client relies on execution timing data captured by the test collectors from previous builds to partition your tests evenly across your agents. Therefore, you will need to configure the Ruby test collector for your test suite if you are running RSpec, and JavaScript test collector if you are running Jest.

Installation

The Test Engine Client is supported on both Linux and macOS with 64-bit ARM and AMD architectures. You can install the client using the following installers:

If you need to install this tool on a system without an installer listed above, you'll need to perform a manual installation using one of the binaries from Test Engine Client's releases page. Once you have the binary, make it executable in your pipeline.

Using the Test Engine Client

Once you have downloaded the Test Engine Client (bktec) binary and it is executable in your pipeline, you'll need to configure some additional environment variables for the Test Engine Client to function. You can then update your pipeline step to call bktec instead of calling RSpec to run your tests. Learn more about how to do this in Update the pipeline step.

Configure environment variables

The Test Engine Client tool uses a number of predefined and mandatory environment variables, as well as several optional ones for either RSpec or Jest.

Predefined environment variables

By default, the following predefined environment variables are available to your testing environment and do not need any further configuration. If, however, you use Docker or some other type of containerization tool to run your tests, and you wish to use these predefined environment variables in these tests, you may need to expose these environment variables to your containers.

`BUILDKITE_BUILD_ID #`	The UUID of the pipeline build. The Test Engine Client uses this UUID along with `BUILDKITE_STEP_ID` to uniquely identify the test plan.
`BUILDKITE_JOB_ID #`	The UUID of the job in the pipeline's build.
`BUILDKITE_ORGANIZATION_SLUG #`	The slug of your Buildkite organization.
`BUILDKITE_PARALLEL_JOB #`	The index number of a parallel job created from a parallel build step. Ensure you configure `parallelism` in your pipeline definition. Learn more about parallel build steps in Concurrency and parallelism.
`BUILDKITE_PARALLEL_JOB_COUNT #`	The total number of parallel jobs created from a parallel build step. Ensure you configure `parallelism` in your pipeline definition. Learn more about parallel build steps in Concurrency and parallelism.
`BUILDKITE_STEP_ID #`	The UUID of the step group in the pipeline build. The Test Engine Client uses this UUID along with `BUILDKITE_BUILD_ID` to uniquely identify the test plan.

Mandatory environment variables

The following mandatory environment variables must be set.

`BUILDKITE_TEST_ENGINE_API_ACCESS_TOKEN #`	Buildkite API access token with `read_suites`, `read_test_plan`, and `write_test_plan` scopes. You can create an API access token from Personal Settings > API Access Tokens in the Buildkite interface. To avoid this token being tied to any one employee or person, ideally, this token should be created for a bot user account within your Buildkite organization.
`BUILDKITE_TEST_ENGINE_RESULT_PATH #`	The path to store the test result. The Test Engine Client uses this environment variable to tell the runner where to store the test result. The Test Engine Client reads the test result after each test run for retries and verification. For RSpec, the result is generated using the `--format json` and `--out` CLI options, while for Jest, it is generated using the `--json` and `--outputFile` options. We have included these options in the default test command for RSpec and Jest. If you need to customize your test command, make sure to append the CLI options to save the result to a file. Please refer to the `BUILDKITE_TEST_ENGINE_TEST_CMD` environment variable for more details. The Test Engine Client will not delete the file after running the test, however it will be deleted by Buildkite Agent as part of the build lifecycle.
`BUILDKITE_TEST_ENGINE_RUNNER #`	The test runner to use for running tests. Currently `rspec` and `jest` are supported.
`BUILDKITE_TEST_ENGINE_SUITE_SLUG #`	The slug of your Test Engine test suite. You can find the suite slug in the url for your test suite. For example, the slug for the url: `https://buildkite.com/organizations/my-organization/analytics/suites/my-suite` is `my-suite`.

Optional RSpec environment variables

The following optional RSpec environment variables can also be used to configure the Test Engine Client's behavior.

`BUILDKITE_TEST_ENGINE_DEBUG_ENABLED #` Default: `-`	A flag to enable more verbose logging.
`BUILDKITE_TEST_ENGINE_RETRY_CMD #` Default: `-`	The command to retry the failed tests. The Test Engine Client will replace the `{{testExamples}}` placeholder with the failed tests. If not set, the client will use the same command defined in `BUILDKITE_TEST_ENGINE_TEST_CMD`.
`BUILDKITE_TEST_ENGINE_RETRY_COUNT #` Default: `0`	The number of retries. The Test Engine Client runs the test command defined in `BUILDKITE_TEST_ENGINE_TEST_CMD` and retries only the failed tests up to `BUILDKITE_TEST_ENGINE_RETRY_COUNT` times, using the retry command defined in `BUILDKITE_TEST_ENGINE_RETRY_CMD`.
`BUILDKITE_TEST_ENGINE_SPLIT_BY_EXAMPLE #` Default: `-`	A flag to enable split by example. When this option is `true`, the Test Engine Client will split the execution of slow test files over multiple partitions.
`BUILDKITE_TEST_ENGINE_TEST_CMD #` Default: `bundle exec rspec --format progress --format json --out {{resultPath}} {{testExamples}}`	The test command to run your tests. The Test Engine Client will replace the `{{testExamples}}` placeholder with the test plan. It is necessary to include `--format json --out {{resultPath}}` in the test command, because the Test Engine Client needs to read the result after each test run. Please refer to the `BUILDKITE_TEST_ENGINE_RESULT_PATH` environment variable for more details.
`BUILDKITE_TEST_ENGINE_TEST_FILE_EXCLUDE_PATTERN #` Default: `-`	The glob pattern to exclude certain test files or directories. The exclusion will be applied after discovering the test files using a pattern configured with `BUILDKITE_TEST_ENGINE_TEST_FILE_PATTERN`. This option accepts the pattern syntax supported by the zzglob library.
`BUILDKITE_TEST_ENGINE_TEST_FILE_PATTERN #` Default: `spec/*/_spec.rb`	The glob pattern to discover test files. You can exclude certain test files or directories from the discovered test files using a pattern that can be configured with `BUILDKITE_TEST_ENGINE_TEST_FILE_EXCLUDE_PATTERN`. This option accepts the pattern syntax supported by the zzglob library.

Optional Jest environment variables

The following optional Jest environment variables can also be used to configure the Test Engine Client's behavior.

`BUILDKITE_TEST_ENGINE_DEBUG_ENABLED #` Default: `-`	A flag to enable more verbose logging.
`BUILDKITE_TEST_ENGINE_RETRY_CMD #` Default: `yarn test --testNamePattern '{{testNamePattern}}' --json --testLocationInResults --outputFile {{resultPath}}`	The command to retry the failed tests. The Test Engine Client will replace the `{{testNamePattern}}` placeholder with the failed tests. It is necessary to include `--json --testLocationInResults --outputFile {{resultPath}}` in the command, because the Test Engine Client needs to read the result after each test run. Please refer to the `BUILDKITE_TEST_ENGINE_RESULT_PATH` environment variable for more details.
`BUILDKITE_TEST_ENGINE_RETRY_COUNT #` Default: `0`	The number of retries. The Test Engine Client runs the test command defined in `BUILDKITE_TEST_ENGINE_TEST_CMD` and retries only the failed tests up to `BUILDKITE_TEST_ENGINE_RETRY_COUNT` times, using the retry command defined in `BUILDKITE_TEST_ENGINE_RETRY_CMD`.
`BUILDKITE_TEST_ENGINE_TEST_CMD #` Default: `yarn test {{testExamples}} --json --testLocationInResults --outputFile {{resultPath}}`	The test command to run your tests. The Test Engine Client will replace and populate the `{{testExamples}}` placeholder with the test plan. It is necessary to include `--json --testLocationInResults --outputFile {{resultPath}}` in the command, because the Test Engine Client needs to read the result after each test run. Please refer to the `BUILDKITE_TEST_ENGINE_RESULT_PATH` environment variable for more details.
`BUILDKITE_TEST_ENGINE_TEST_FILE_EXCLUDE_PATTERN #` Default: `node_modules`	The glob pattern to exclude certain test files or directories. The exclusion will be applied after discovering the test files using a pattern configured with `BUILDKITE_TEST_ENGINE_TEST_FILE_PATTERN`. This option accepts the pattern syntax supported by the zzglob library.
`BUILDKITE_TEST_ENGINE_TEST_FILE_PATTERN #` Default: `/{__tests__//,.spec,*.test}.{ts,js,tsx,jsx}`	The glob pattern to discover test files. You can exclude certain test files or directories from the discovered test files using a pattern that can be configured with `BUILDKITE_TEST_ENGINE_TEST_FILE_EXCLUDE_PATTERN`. This option accepts the pattern syntax supported by the zzglob library.

Update the pipeline step

With the environment variables configured, you can now update your pipeline step to run the Test Engine Client instead of running RSpec, or Jest directly. The following example pipeline step demonstrates how to partition your RSpec test suite across 10 nodes.

pipeline.yml

steps:
  - name: "RSpec"
    command: bktec
    parallelism: 10
    env:
      BUILDKITE_TEST_ENGINE_API_ACCESS_TOKEN: your-secret-token
      BUILDKITE_TEST_ENGINE_RESULT_PATH: tmp/rspec-result.json
      BUILDKITE_TEST_ENGINE_SUITE_SLUG: my-suite
      BUILDKITE_TEST_ENGINE_RUNNER: rspec

API rate limits

There is a limit on the number of API requests that the Test Engine Client can make to the server. This limit is 10,000 requests per minute per Buildkite organization. When this limit is reached, the Test Engine Client will pause and wait until the next minute is reached before retrying the request. This rate limit is independent of the REST API rate limits, and only applies to the Test Engine Client's interactions with the Test Splitting API.