# Installing and using the client

This page provides instructions on how to install the Test Engine Client ([bktec](https://github.com/buildkite/test-engine-client)) using [installers](#installation) provided by Buildkite, as well as [configure and use bktec](#using-bktec).

## Installation

bktec is supported on both Linux ([Debian](#installation-debian) and [Red Hat](#installation-red-hat)) and [macOS](#installation-macos), as well as in [Docker](#installation-docker), for 64-bit ARM and AMD architectures. You can install the client using the following installers.
If you need to install this tool on a system without an installer listed below, you'll need to perform a manual installation using one of the binaries from [Test Engine Client's releases page](https://github.com/buildkite/test-engine-client/releases/latest). Once you have the binary, make it executable in your pipeline.

### Debian

1. Ensure you have curl and gpg installed first:

    ```shell
    apt update && apt install curl gpg -y
    ```

1. Install the registry signing key:

    ```shell
    curl -fsSL "https://packages.buildkite.com/buildkite/test-engine-client-deb/gpgkey" | gpg --dearmor -o /etc/apt/keyrings/buildkite_test-engine-client-deb-archive-keyring.gpg
    ```

1. Configure the registry:

    ```shell
    echo -e "deb [signed-by=/etc/apt/keyrings/buildkite_test-engine-client-deb-archive-keyring.gpg] https://packages.buildkite.com/buildkite/test-engine-client-deb/any/ any main\ndeb-src [signed-by=/etc/apt/keyrings/buildkite_test-engine-client-deb-archive-keyring.gpg] https://packages.buildkite.com/buildkite/test-engine-client-deb/any/ any main" > /etc/apt/sources.list.d/buildkite-buildkite-test-engine-client-deb.list
    ```

1. Install the package:

    ```shell
    apt update && apt install bktec
    ```

### Red Hat

1. Configure the registry:

    ```shell
    echo -e "[test-engine-client-rpm]\nname=Test Engine Client - rpm\nbaseurl=https://packages.buildkite.com/buildkite/test-engine-client-rpm/rpm_any/rpm_any/\$basearch\nenabled=1\nrepo_gpgcheck=1\ngpgcheck=0\ngpgkey=https://packages.buildkite.com/buildkite/test-engine-client-rpm/gpgkey\npriority=1" > /etc/yum.repos.d/test-engine-client-rpm.repo
    ```

2. Install the package:

    ```shell
    dnf install -y bktec
    ```

### macOS

The Test Engine Client can be installed using [Homebrew](https://brew.sh) with [Buildkite tap formulae](https://github.com/buildkite/homebrew-buildkite). To install, run:

```shell
brew tap buildkite/buildkite && brew install buildkite/buildkite/bktec
```

### Docker

You can run the Test Engine Client inside a Docker container using the official image in [Docker Hub](https://hub.docker.com/r/buildkite/test-engine-client/tags).

To run the client using Docker:

```shell
docker run buildkite/test-engine-client
```

Or, to add the Test Engine Client binary to your Docker image, include the following in your Dockerfile:

```dockerfile
COPY --from=buildkite/test-engine-client /usr/local/bin/bktec /usr/local/bin/bktec
```

### Buildkite plugin

The [Tests Buildkite plugin](https://buildkite.com/resources/plugins/buildkite-plugins/tests-buildkite-plugin/) installs bktec automatically by default. Manual installation of bktec is not required when using the plugin.

Before downloading, the plugin checks whether bktec is already available on the `PATH`. If bktec is already installed and no specific `client-version`, `client-os`, or `client-arch` is configured, the plugin uses the existing binary and skips the download.

Set `install-client: false` in the plugin configuration to skip automatic installation and manage bktec yourself:

```yaml
steps:
  - label: "RSpec"
    command: bktec run
    plugins:
      - tests#v1.0.0:
          test-runner: rspec
          install-client: false
```
{: codeblock-file="pipeline.yml"}

Skipping automatic installation is useful when bktec is pre-installed in your build environment, such as through a shared Docker image or a package manager on your agent.

## Using bktec

Buildkite maintains its open source Test Engine Client ([bktec](https://github.com/buildkite/test-engine-client)) tool. Currently, the bktec tool supports [RSpec](/docs/pipelines/configure/tests/test-collection/ruby-collectors#rspec-collector), [Jest](/docs/pipelines/configure/tests/test-collection/javascript-collectors#configure-the-test-framework-jest), [Cypress](/docs/pipelines/configure/tests/test-collection/javascript-collectors#configure-the-test-framework-cypress), [PlayWright](/docs/pipelines/configure/tests/test-collection/javascript-collectors#configure-the-test-framework-playwright), and [Pytest](/docs/pipelines/configure/tests/test-collection/python-collectors#pytest-collector), pytest-pants, [Go](/docs/pipelines/configure/tests/test-collection/golang-collectors), and cucumber testing frameworks.

If your testing framework is not supported, get in touch through support@buildkite.com or submit a pull request.

Once you have [installed the bktec binary](#installation) and it is executable in your pipeline, you'll need to [configure some additional environment variables](#using-bktec-configure-environment-variables) for bktec to function. You can then [update your pipeline step](#using-bktec-update-the-pipeline-step) to call `bktec run` instead of calling RSpec to run your tests.

### Configure environment variables

bktec uses a number of [predefined](#predefined-environment-variables) and [mandatory](#mandatory-environment-variables) environment variables, as well as several optional ones for either [RSpec](#optional-rspec-environment-variables) or [Jest](#optional-jest-environment-variables).

<h4 id="predefined-environment-variables">Predefined environment variables</h4>

By default, the following predefined environment variables are available to your testing environment and do not need any further configuration. If, however, you use Docker or some other type of containerization tool to run your tests, and you wish to use these predefined environment variables in these tests, you may need to expose these environment variables to your containers.



| BUILDKITE_BUILD_ID # | The UUID of the pipeline build. The Test Engine Client uses this UUID along with `BUILDKITE_STEP_ID` to uniquely identify the test plan. |
| --- | --- |
| BUILDKITE_JOB_ID # | The UUID of the job in the pipeline's build. |
| BUILDKITE_ORGANIZATION_SLUG # | The slug of your Buildkite organization. |
| BUILDKITE_PARALLEL_JOB # | The index number of a parallel job created from a parallel build step. Ensure you configure `parallelism` in your pipeline definition. Learn more about parallel build steps in [Concurrency and parallelism](https://buildkite.com/docs/pipelines/controlling-concurrency#concurrency-and-parallelism). |
| BUILDKITE_PARALLEL_JOB_COUNT # | The total number of parallel jobs created from a parallel build step. Ensure you configure `parallelism` in your pipeline definition. Learn more about parallel build steps in [Concurrency and parallelism](https://buildkite.com/docs/pipelines/controlling-concurrency#concurrency-and-parallelism). |
| BUILDKITE_STEP_ID # | The UUID of the step group in the pipeline build. The Test Engine Client uses this UUID along with `BUILDKITE_BUILD_ID` to uniquely identify the test plan. |



<h4 id="mandatory-environment-variables">Mandatory environment variables</h4>

The following mandatory environment variables must be set.



| BUILDKITE_TEST_ENGINE_RESULT_PATH # | The path to store the test result. The Test Engine Client uses this environment variable to tell the runner where to store the test result. The Test Engine Client reads the test result after each test run for retries and verification. For RSpec, the result is generated using the `--format json` and `--out` CLI options, while for Jest, it is generated using the `--json` and `--outputFile` options. We have included these options in the default test command for RSpec and Jest. If you need to customize your test command, make sure to append the CLI options to save the result to a file. Please refer to the `BUILDKITE_TEST_ENGINE_TEST_CMD` environment variable for more details. The Test Engine Client will not delete the file after running the test, however it will be deleted by the Buildkite agent as part of the build lifecycle. |
| --- | --- |
| BUILDKITE_TEST_ENGINE_TEST_RUNNER # | The test runner to use for running tests. Currently `rspec`, `jest`, `cypress`, `playwright`, `pytest`, `pytest-pants`, `gotest`, and `cucumber` are supported. |
| BUILDKITE_TEST_ENGINE_SUITE_SLUG # | The slug of your Test Engine test suite. You can find the suite slug in the url for your test suite. For example, the slug for the url: `https://buildkite.com/organizations/my-organization/analytics/suites/my-suite` is `my-suite`. |



<h4 id="optional-environment-variables">Authentication environment variables</h4>

The following optional environment variables control authentication for `bktec` and test collection in the configured runner.



| BUILDKITE_TEST_ENGINE_API_ACCESS_TOKEN # | Buildkite API access token with `read_suites`, `read_test_plan`, and `write_test_plan` scopes. You can create an [API access token](https://buildkite.com/user/api-access-tokens) from **Personal Settings** > **API Access Tokens** in the Buildkite interface. To avoid this token being tied to any one employee or person, ideally, the token should be created for a bot user account within your Buildkite organization. If blank, `bktec` version 2.6.0 and above will generate a temporary [OIDC token](/docs/pipelines/configure/tests/test-collection/oidc) for authentication. |
| --- | --- |
| BUILDKITE_ANALYTICS_TOKEN # | Buildkite Test Engine suite token which is used by the [test collector](/docs/pipelines/configure/tests/test-collection) to upload test results to Test Engine. If blank, `bktec` version 2.6.0 and above will generate a temporary [OIDC token](/docs/pipelines/configure/tests/test-collection/oidc) for authentication. |
| BUILDKITE_BUILDKITE_TEST_ENGINE_OIDC_LIFETIME # | Specify the lifetime of OIDC tokens generated by `bktec`. Ensure that this lifetime exceeds the duration of your longest expected test run. Default `2h0m0s`. |
| BUILDKITE_BUILDKITE_TEST_ENGINE_OIDC # | Set to `false` to disable OIDC token generation in `bktec`. |



<h4 id="optional-rspec-environment-variables">Optional RSpec environment variables</h4>

The following optional RSpec environment variables can also be used to configure bktec's behavior.



| BUILDKITE_TEST_ENGINE_DEBUG_ENABLED # **Default**: `-` | A flag to enable more verbose logging. |
| --- | --- |
| BUILDKITE_TEST_ENGINE_RETRY_CMD # **Default**: `-` | The command to retry the failed tests. The Test Engine Client will replace the `{{testExamples}}` placeholder with the failed tests. If not set, the client will use the same command defined in `BUILDKITE_TEST_ENGINE_TEST_CMD`. |
| BUILDKITE_TEST_ENGINE_RETRY_COUNT # **Default**: `0` | The number of retries. The Test Engine Client runs the test command defined in `BUILDKITE_TEST_ENGINE_TEST_CMD` and retries only the failed tests up to `BUILDKITE_TEST_ENGINE_RETRY_COUNT` times, using the retry command defined in `BUILDKITE_TEST_ENGINE_RETRY_CMD`. |
| BUILDKITE_TEST_ENGINE_SPLIT_BY_EXAMPLE # **Default**: `-` | A flag to enable split by example. When this option is `true`, the Test Engine Client will split the execution of slow test files over multiple partitions. |
| BUILDKITE_TEST_ENGINE_TEST_CMD # **Default**: `bundle exec rspec --format progress --format json --out {{resultPath}} {{testExamples}}` | The test command to run your tests. The Test Engine Client will replace the `{{testExamples}}` placeholder with the test plan. It is necessary to include `--format json --out {{resultPath}}` in the test command, because the Test Engine Client needs to read the result after each test run. Please refer to the `BUILDKITE_TEST_ENGINE_RESULT_PATH` environment variable for more details. |
| BUILDKITE_TEST_ENGINE_TEST_FILE_EXCLUDE_PATTERN # **Default**: `-` | The glob pattern to exclude certain test files or directories. The exclusion will be applied after discovering the test files using a pattern configured with `BUILDKITE_TEST_ENGINE_TEST_FILE_PATTERN`. _This option accepts [glob pattern syntax](/docs/pipelines/configure/glob-pattern-syntax)._ |
| BUILDKITE_TEST_ENGINE_TEST_FILE_PATTERN # **Default**: `spec/**/*_spec.rb` | The glob pattern to discover test files. You can exclude certain test files or directories from the discovered test files using a pattern that can be configured with `BUILDKITE_TEST_ENGINE_TEST_FILE_EXCLUDE_PATTERN`. _This option accepts [glob pattern syntax](/docs/pipelines/configure/glob-pattern-syntax)._ |



<h4 id="optional-jest-environment-variables">Optional Jest environment variables</h4>

The following optional Jest environment variables can also be used to configure bktec's behavior.



| BUILDKITE_TEST_ENGINE_DEBUG_ENABLED # **Default**: `-` | A flag to enable more verbose logging. |
| --- | --- |
| BUILDKITE_TEST_ENGINE_RETRY_CMD # **Default**: `yarn test --testNamePattern '{{testNamePattern}}' --json --testLocationInResults --outputFile {{resultPath}}` | The command to retry the failed tests. The Test Engine Client will replace the `{{testNamePattern}}` placeholder with the failed tests. It is necessary to include `--json --testLocationInResults --outputFile {{resultPath}}` in the command, because the Test Engine Client needs to read the result after each test run. Please refer to the `BUILDKITE_TEST_ENGINE_RESULT_PATH` environment variable for more details. |
| BUILDKITE_TEST_ENGINE_RETRY_COUNT # **Default**: `0` | The number of retries. The Test Engine Client runs the test command defined in `BUILDKITE_TEST_ENGINE_TEST_CMD` and retries only the failed tests up to `BUILDKITE_TEST_ENGINE_RETRY_COUNT` times, using the retry command defined in `BUILDKITE_TEST_ENGINE_RETRY_CMD`. |
| BUILDKITE_TEST_ENGINE_TEST_CMD # **Default**: `yarn test {{testExamples}} --json --testLocationInResults --outputFile {{resultPath}}` | The test command to run your tests. The Test Engine Client will replace and populate the `{{testExamples}}` placeholder with the test plan. It is necessary to include `--json --testLocationInResults --outputFile {{resultPath}}` in the command, because the Test Engine Client needs to read the result after each test run. Please refer to the `BUILDKITE_TEST_ENGINE_RESULT_PATH` environment variable for more details. |
| BUILDKITE_TEST_ENGINE_TEST_FILE_EXCLUDE_PATTERN # **Default**: `node_modules` | The glob pattern to exclude certain test files or directories. The exclusion will be applied after discovering the test files using a pattern configured with `BUILDKITE_TEST_ENGINE_TEST_FILE_PATTERN`. _This option accepts [glob pattern syntax](/docs/pipelines/configure/glob-pattern-syntax)._ |
| BUILDKITE_TEST_ENGINE_TEST_FILE_PATTERN # **Default**: `**/{__tests__/**/*,*.spec,*.test}.{ts,js,tsx,jsx}` | The glob pattern to discover test files. You can exclude certain test files or directories from the discovered test files using a pattern that can be configured with `BUILDKITE_TEST_ENGINE_TEST_FILE_EXCLUDE_PATTERN`. _This option accepts [glob pattern syntax](/docs/pipelines/configure/glob-pattern-syntax)._ |



### Update the pipeline step

With the environment variables configured, you can now update your pipeline step to run bktec instead of running RSpec, or Jest directly. The following example pipeline step demonstrates how to partition your RSpec test suite across 10 nodes.

```
steps:
  - name: "RSpec"
    command: bktec run
    parallelism: 10
    env:
      BUILDKITE_TEST_ENGINE_RESULT_PATH: tmp/rspec-result.json
      BUILDKITE_TEST_ENGINE_SUITE_SLUG: my-suite
      BUILDKITE_TEST_ENGINE_TEST_RUNNER: rspec
```
{: codeblock-file="pipeline.yml"}

## API rate limits

There is a limit on the number of API requests that bktec can make to the server. This limit is 10,000 requests per minute per Buildkite organization. When this limit is reached, bktec will pause and wait until the next minute is reached before retrying the request. This rate limit is independent of the [REST API rate limits](/docs/apis/rest-api/limits), and only applies to the Test Engine Client's interactions with the Test Splitting API.

## Dynamic parallelism

Usually the `parallelism` value is hard coded in the bktec pipeline step. However, from version 2.0.0, it is possible to run bktec with a dynamic `parallelism` value based on a target time for the test run. A common use case for this is test selection, where feature branch builds only run a subset of tests relevant to the changes being made.

Dynamic parallelism is supported using the `bktec plan` command. When used with the `--max-parallelism` and `--target-time` flags (see list of [bktec plan flags](#dynamic-parallelism-bktec-plan-flags) for more information), bktec generates a test plan and estimates the `parallelism` required to achieve the specified target build time. bktec then [uploads a dynamic pipeline](/docs/agent/cli/reference/pipeline) using the specified pipeline template.

In the following example, the `test-selection.sh` script is assumed to generate a list of test files, one per line, relevant to the changes in a feature branch.

```
steps:
  - name: "Test selection"
    command: test-selection.sh > selected-files.txt

  - wait: ~

  - name: "Dynamic pipeline"
    key: "dynamic-pipeline"
    command: bktec plan --max-parallelism 10 --target-time 2m --files selected-files.txt --pipeline-upload .buildkite/dynamic-pipeline-template.yml
```
{: codeblock-file="pipeline.yml"}

In this example pipeline, bktec uploads a dynamic pipeline using `.buildkite/dynamic-pipeline-template.yml` by invoking `buildkite agent pipeline upload`. Learn more about the [bktec plan additional environment variables](#dynamic-parallelism-bktec-plan-additional-environment-variables) generated during pipeline uploads.

These variables can be used in the template file provided to the `--pipeline-upload` flag, where you can use [environment variable substitution](/docs/agent/cli/reference/pipeline#environment-variable-substitution) to obtain their values.

```
steps:
- command: "bktec run --plan-identifier ${BUILDKITE_TEST_ENGINE_PLAN_IDENTIFIER}"
  name: "bktec run"
  depends_on: "dynamic-pipeline"
  parallelism: ${BUILDKITE_TEST_ENGINE_PARALLELISM}
```
{: codeblock-file=".buildkite/dynamic-pipeline-template.yml"}

### bktec plan flags

The `bktec plan` command supports the following flags, which controls the behavior of the dynamic parallelism test plan. Each flag's value alternatively can be supplied using an environment variable.



| `--max-parallelism` | The maximum allowed parallelism for a dynamic parallelism test plan. **Environment variable:** `$BUILDKITE_TEST_ENGINE_MAX_PARALLELISM` |
| --- | --- |
| `--target-time` | Target duration for each node, for example, `2m30s`. The test planner will attempt to split the test plan into equal duration buckets of this duration and calculate the optimum parallelism to achieve this, up to the value supplied to `--max-parallelism` **Environment variable:** `$BUILDKITE_TEST_ENGINE_TARGET_TIME` |
| `--files` | Path to a file containing a newline separated list of test file names to be executed. **Environment variable:** `$BUILDKITE_TEST_ENGINE_FILES` |



### bktec plan additional environment variables

The `bktec plan` command generates the following additional environment variables when uploading the pipeline.



| `BUILDKITE_TEST_ENGINE_PLAN_IDENTIFIER` | The identifier of the test plan generated by `bktec plan`. |
| --- | --- |
| `BUILDKITE_TEST_ENGINE_PARALLELISM` | The parallelism estimated by the test planner to achieve the requested target build time. |