Configuring the Test Engine Client

Buildkite maintains its open source Test Engine Client (bktec) tool. Currently, the bktec tool supports RSpec, Jest, Cypress, PlayWright, and Pytest, pytest-pants, Go, and cucumber testing frameworks.

If your testing framework is not supported, get in touch via support@buildkite.com or submit a pull request.

Dependencies

bktec relies on execution timing data captured by the test collectors from previous builds to partition your tests evenly across your agents. Therefore, you will need to configure the test collector for your test framework.

Installation

bktec is supported on both Linux and macOS with 64-bit ARM and AMD architectures. You can install the client using the following installers:

If you need to install this tool on a system without an installer listed above, you'll need to perform a manual installation using one of the binaries from Test Engine Client's releases page. Once you have the binary, make it executable in your pipeline.

Using bktec

Once you have downloaded the bktec binary and it is executable in your pipeline, you'll need to configure some additional environment variables for bktec to function. You can then update your pipeline step to call bktec run instead of calling RSpec to run your tests. Learn more about how to do this in Update the pipeline step.

Configure environment variables

bktec uses a number of predefined and mandatory environment variables, as well as several optional ones for either RSpec or Jest.

Predefined environment variables

By default, the following predefined environment variables are available to your testing environment and do not need any further configuration. If, however, you use Docker or some other type of containerization tool to run your tests, and you wish to use these predefined environment variables in these tests, you may need to expose these environment variables to your containers.

`BUILDKITE_BUILD_ID #`	The UUID of the pipeline build. The Test Engine Client uses this UUID along with `BUILDKITE_STEP_ID` to uniquely identify the test plan.
`BUILDKITE_JOB_ID #`	The UUID of the job in the pipeline's build.
`BUILDKITE_ORGANIZATION_SLUG #`	The slug of your Buildkite organization.
`BUILDKITE_PARALLEL_JOB #`	The index number of a parallel job created from a parallel build step. Ensure you configure `parallelism` in your pipeline definition. Learn more about parallel build steps in Concurrency and parallelism.
`BUILDKITE_PARALLEL_JOB_COUNT #`	The total number of parallel jobs created from a parallel build step. Ensure you configure `parallelism` in your pipeline definition. Learn more about parallel build steps in Concurrency and parallelism.
`BUILDKITE_STEP_ID #`	The UUID of the step group in the pipeline build. The Test Engine Client uses this UUID along with `BUILDKITE_BUILD_ID` to uniquely identify the test plan.

Mandatory environment variables

The following mandatory environment variables must be set.

`BUILDKITE_TEST_ENGINE_API_ACCESS_TOKEN #`	Buildkite API access token with `read_suites`, `read_test_plan`, and `write_test_plan` scopes. You can create an API access token from Personal Settings > API Access Tokens in the Buildkite interface. To avoid this token being tied to any one employee or person, ideally, the token should be created for a bot user account within your Buildkite organization.
`BUILDKITE_TEST_ENGINE_RESULT_PATH #`	The path to store the test result. The Test Engine Client uses this environment variable to tell the runner where to store the test result. The Test Engine Client reads the test result after each test run for retries and verification. For RSpec, the result is generated using the `--format json` and `--out` CLI options, while for Jest, it is generated using the `--json` and `--outputFile` options. We have included these options in the default test command for RSpec and Jest. If you need to customize your test command, make sure to append the CLI options to save the result to a file. Please refer to the `BUILDKITE_TEST_ENGINE_TEST_CMD` environment variable for more details. The Test Engine Client will not delete the file after running the test, however it will be deleted by Buildkite Agent as part of the build lifecycle.
`BUILDKITE_TEST_ENGINE_TEST_RUNNER #`	The test runner to use for running tests. Currently `rspec`, `jest`, `cypress`, `playwright`, `pytest`, `pytest-pants`, `gotest`, and `cucumber` are supported.
`BUILDKITE_TEST_ENGINE_SUITE_SLUG #`	The slug of your Test Engine test suite. You can find the suite slug in the url for your test suite. For example, the slug for the url: `https://buildkite.com/organizations/my-organization/analytics/suites/my-suite` is `my-suite`.

Optional RSpec environment variables

The following optional RSpec environment variables can also be used to configure bktec's behavior.

`BUILDKITE_TEST_ENGINE_DEBUG_ENABLED #` Default: `-`	A flag to enable more verbose logging.
`BUILDKITE_TEST_ENGINE_RETRY_CMD #` Default: `-`	The command to retry the failed tests. The Test Engine Client will replace the `{{testExamples}}` placeholder with the failed tests. If not set, the client will use the same command defined in `BUILDKITE_TEST_ENGINE_TEST_CMD`.
`BUILDKITE_TEST_ENGINE_RETRY_COUNT #` Default: `0`	The number of retries. The Test Engine Client runs the test command defined in `BUILDKITE_TEST_ENGINE_TEST_CMD` and retries only the failed tests up to `BUILDKITE_TEST_ENGINE_RETRY_COUNT` times, using the retry command defined in `BUILDKITE_TEST_ENGINE_RETRY_CMD`.
`BUILDKITE_TEST_ENGINE_SPLIT_BY_EXAMPLE #` Default: `-`	A flag to enable split by example. When this option is `true`, the Test Engine Client will split the execution of slow test files over multiple partitions.
`BUILDKITE_TEST_ENGINE_TEST_CMD #` Default: `bundle exec rspec --format progress --format json --out {{resultPath}} {{testExamples}}`	The test command to run your tests. The Test Engine Client will replace the `{{testExamples}}` placeholder with the test plan. It is necessary to include `--format json --out {{resultPath}}` in the test command, because the Test Engine Client needs to read the result after each test run. Please refer to the `BUILDKITE_TEST_ENGINE_RESULT_PATH` environment variable for more details.
`BUILDKITE_TEST_ENGINE_TEST_FILE_EXCLUDE_PATTERN #` Default: `-`	The glob pattern to exclude certain test files or directories. The exclusion will be applied after discovering the test files using a pattern configured with `BUILDKITE_TEST_ENGINE_TEST_FILE_PATTERN`. This option accepts glob pattern syntax.
`BUILDKITE_TEST_ENGINE_TEST_FILE_PATTERN #` Default: `spec/*/_spec.rb`	The glob pattern to discover test files. You can exclude certain test files or directories from the discovered test files using a pattern that can be configured with `BUILDKITE_TEST_ENGINE_TEST_FILE_EXCLUDE_PATTERN`. This option accepts glob pattern syntax.

Optional Jest environment variables

The following optional Jest environment variables can also be used to configure bktec's behavior.

`BUILDKITE_TEST_ENGINE_DEBUG_ENABLED #` Default: `-`	A flag to enable more verbose logging.
`BUILDKITE_TEST_ENGINE_RETRY_CMD #` Default: `yarn test --testNamePattern '{{testNamePattern}}' --json --testLocationInResults --outputFile {{resultPath}}`	The command to retry the failed tests. The Test Engine Client will replace the `{{testNamePattern}}` placeholder with the failed tests. It is necessary to include `--json --testLocationInResults --outputFile {{resultPath}}` in the command, because the Test Engine Client needs to read the result after each test run. Please refer to the `BUILDKITE_TEST_ENGINE_RESULT_PATH` environment variable for more details.
`BUILDKITE_TEST_ENGINE_RETRY_COUNT #` Default: `0`	The number of retries. The Test Engine Client runs the test command defined in `BUILDKITE_TEST_ENGINE_TEST_CMD` and retries only the failed tests up to `BUILDKITE_TEST_ENGINE_RETRY_COUNT` times, using the retry command defined in `BUILDKITE_TEST_ENGINE_RETRY_CMD`.
`BUILDKITE_TEST_ENGINE_TEST_CMD #` Default: `yarn test {{testExamples}} --json --testLocationInResults --outputFile {{resultPath}}`	The test command to run your tests. The Test Engine Client will replace and populate the `{{testExamples}}` placeholder with the test plan. It is necessary to include `--json --testLocationInResults --outputFile {{resultPath}}` in the command, because the Test Engine Client needs to read the result after each test run. Please refer to the `BUILDKITE_TEST_ENGINE_RESULT_PATH` environment variable for more details.
`BUILDKITE_TEST_ENGINE_TEST_FILE_EXCLUDE_PATTERN #` Default: `node_modules`	The glob pattern to exclude certain test files or directories. The exclusion will be applied after discovering the test files using a pattern configured with `BUILDKITE_TEST_ENGINE_TEST_FILE_PATTERN`. This option accepts glob pattern syntax.
`BUILDKITE_TEST_ENGINE_TEST_FILE_PATTERN #` Default: `/{__tests__//,.spec,*.test}.{ts,js,tsx,jsx}`	The glob pattern to discover test files. You can exclude certain test files or directories from the discovered test files using a pattern that can be configured with `BUILDKITE_TEST_ENGINE_TEST_FILE_EXCLUDE_PATTERN`. This option accepts glob pattern syntax.

Update the pipeline step

With the environment variables configured, you can now update your pipeline step to run bktec instead of running RSpec, or Jest directly. The following example pipeline step demonstrates how to partition your RSpec test suite across 10 nodes.

pipeline.yml

steps:
  - name: "RSpec"
    command: bktec run
    parallelism: 10
    env:
      BUILDKITE_TEST_ENGINE_API_ACCESS_TOKEN: your-secret-token
      BUILDKITE_TEST_ENGINE_RESULT_PATH: tmp/rspec-result.json
      BUILDKITE_TEST_ENGINE_SUITE_SLUG: my-suite
      BUILDKITE_TEST_ENGINE_TEST_RUNNER: rspec

API rate limits

There is a limit on the number of API requests that bktec can make to the server. This limit is 10,000 requests per minute per Buildkite organization. When this limit is reached, bktec will pause and wait until the next minute is reached before retrying the request. This rate limit is independent of the REST API rate limits, and only applies to the Test Engine Client's interactions with the Test Splitting API.

Dynamic parallelism

Usually the parallelism value is hard coded in the bktec pipeline step. However, from version 2.0.0, it is possible to run bktec with a dynamic parallelism value based on a target time for the test run. A common use case for this is test selection, where feature branch builds only run a subset of tests relevant to the changes being made.

Dynamic parallelism is supported using the bktec plan command. When used with the --max-parallelism and --target-time flags (see list of bktec plan flags for more information), bktec generates a test plan and estimates the parallelism required to achieve the specified target build time. bktec then uploads a dynamic pipeline using the specified pipeline template.

In the following example, the test-selection.sh script is assumed to generate a list of test files, one per line, relevant to the changes in a feature branch.

pipeline.yml

steps:
  - name: "Test selection"
    command: test-selection.sh > selected-files.txt

  - wait: ~

  - name: "Dynamic pipeline"
    key: "dynamic-pipeline"
    command: bktec plan --max-parallelism 10 --target-time 2m --files selected-files.txt --pipeline-upload .buildkite/dynamic-pipeline-template.yml

In this example pipeline, bktec uploads a dynamic pipeline using .buildkite/dynamic-pipeline-template.yml by invoking buildkite agent pipeline upload. Learn more about the bktec plan additional environment variables generated during pipeline uploads.

These variables can be used in the template file provided to the --pipeline-upload flag, where you can use environment variable substitution to obtain their values.

.buildkite/dynamic-pipeline-template.yml

steps:
- command: "bktec run --plan-identifier ${BUILDKITE_TEST_ENGINE_PLAN_IDENTIFIER}"
  name: "bktec run"
  depends_on: "dynamic-pipeline"
  parallelism: ${BUILDKITE_TEST_ENGINE_PARALLELISM}

bktec plan flags

The bktec plan command supports the following flags, which controls the behavior of the dynamic parallelism test plan. Each flag's value alternatively can be supplied using an environment variable.

`--max-parallelism`	The maximum allowed parallelism for a dynamic parallelism test plan. Environment variable: `$BUILDKITE_TEST_ENGINE_MAX_PARALLELISM`
`--target-time`	Target duration for each node, for example, `2m30s`. The test planner will attempt to split the test plan into equal duration buckets of this duration and calculate the optimum parallelism to achieve this, up to the value supplied to `--max-parallelism` Environment variable: `$BUILDKITE_TEST_ENGINE_TARGET_TIME`
`--files`	Path to a file containing a newline separated list of test file names to be executed. Environment variable: `$BUILDKITE_TEST_ENGINE_FILES`

bktec plan additional environment variables

The bktec plan command generates the following additional environment variables when uploading the pipeline.

`BUILDKITE_TEST_ENGINE_PLAN_IDENTIFIER`	The identifier of the test plan generated by `bktec plan`.
`BUILDKITE_TEST_ENGINE_PARALLELISM`	The parallelism estimated by the test planner to achieve the requested target build time.