Manual test selection

Set up manual test selection

The recommended setup uses two steps. The first step generates the list of tests to run and saves it as an artifact. The second step downloads the list and passes it to bktec run, so that every parallel job uses the same list.

1. Create a .buildkite/select-tests.sh script that writes the tests to run to tests-to-run.txt, one path per line. The following example selects the RSpec spec files changed on the current branch. Replace the git diff command with your own selection logic:

   #!/usr/bin/env bash
   set -euo pipefail
   
   base_branch="${BUILDKITE_PULL_REQUEST_BASE_BRANCH:-main}"
   git fetch origin "${base_branch}"
   
   # Select the spec files that were added or changed on this branch
   git diff --name-only --diff-filter=d --relative \
     "origin/${base_branch}...HEAD" -- '*_spec.rb' > tests-to-run.txt
   

   List paths relative to the directory that bktec runs in, without a location prefix. If bktec runs in a subdirectory, such as backend in a monorepo, replace --relative with --relative=backend.

2. Create a .buildkite/run-selected-tests.sh script that downloads the list and runs bktec with the manual selection strategy:

   #!/usr/bin/env bash
   set -euo pipefail
   
   buildkite-agent artifact download tests-to-run.txt .
   
   if ! grep -q '[^[:space:]]' tests-to-run.txt; then
     echo "No tests selected for this build"
     exit 0
   fi
   
   "${BUILDKITE_TEST_ENGINE_CLIENT_PATH:-bktec}" run \
     --selection-strategy manual \
     --selection-param "files=$(cat tests-to-run.txt)"
   

   Keep the double quotes around the --selection-param value, so that the newlines between paths are preserved.

3. Add both steps to your pipeline.yml file. Use the Tests Buildkite plugin on the test step to install bktec, authenticate with OIDC, and upload results:

   steps:
     - label: "Select tests"
       key: "select-tests"
       command: ".buildkite/select-tests.sh"
       artifact_paths: "tests-to-run.txt"
   
     - label: "Run selected tests"
       depends_on: "select-tests"
       command: ".buildkite/run-selected-tests.sh"
       parallelism: 10
       plugins:
         - tests#v1.0.0:
             test-runner: rspec
             result-path: tmp/rspec-result.json
   

If none of the listed paths match a test that bktec discovers, bktec runs no tests. When fewer tests are selected than there are parallel jobs, the remaining jobs exit without running tests. To size the step to the selected tests, use dynamic parallelism.

Use manual selection with dynamic parallelism

To size the test step to the selected tests, pass the same selection flags to bktec plan, and set a maximum parallelism and target time. bktec creates the test plan, then uploads the test step with the parallelism needed to reach the target time. Learn more in Dynamic parallelism.

steps:
  - label: "Select tests"
    key: "select-tests"
    command: ".buildkite/select-tests.sh"
    artifact_paths: "tests-to-run.txt"

  - label: "Plan selected tests"
    key: "plan-selected-tests"
    depends_on: "select-tests"
    command: ".buildkite/plan-selected-tests.sh"
    plugins:
      - tests#v1.0.0:
          test-runner: rspec
          result-path: tmp/rspec-result.json
          max-parallelism: 10
          target-time: 2m

The planning script runs bktec plan with the selected tests:

#!/usr/bin/env bash
set -euo pipefail

buildkite-agent artifact download tests-to-run.txt .

if ! grep -q '[^[:space:]]' tests-to-run.txt; then
  echo "No tests selected for this build"
  exit 0
fi

"${BUILDKITE_TEST_ENGINE_CLIENT_PATH:-bktec}" plan \
  --selection-strategy manual \
  --selection-param "files=$(cat tests-to-run.txt)" \
  --pipeline-upload .buildkite/selected-tests-template.yml

The pipeline template runs the .buildkite/run-selected-tests.sh script from the setup steps with the plan that bktec plan created:

steps:
  - label: "Run selected tests"
    command: ".buildkite/run-selected-tests.sh"
    depends_on: "plan-selected-tests"
    parallelism: ${BUILDKITE_TEST_ENGINE_PARALLELISM}
    plugins:
      - tests#v1.0.0:
          test-runner: rspec
          result-path: tmp/rspec-result.json
          plan-identifier: ${BUILDKITE_TEST_ENGINE_PLAN_IDENTIFIER}

Check the selection in the job log

bktec prints a planning summary at the start of each job, showing the selection this job requested and the selection that Test Engine applied:

Requested
  Selection strategy: manual
    files = <3 nonblank entries; 92 bytes>
  Parallelism: 10 (fixed)

Selection summary
  Applied strategy: manual
  Selected: 3 of 412 test selectors (0.7%)
  Estimated compute: 72.4s of 2304s (3.1%)
  Candidate timing coverage: 96%

Test Engine caches each test plan, so retrying a job reuses the original selection, even if the list has changed. The summary then shows Using existing plan. Start a new build to select tests again.

Git metadata

When a selection strategy is set, bktec also sends git metadata with the test plan request, including the full git diff against the base branch, which contains your source changes. Manual selection doesn't need this metadata. To turn it off, set BUILDKITE_TEST_ENGINE_COLLECT_GIT_METADATA to false in the pipeline-level env.

Review selection in Orchestration

Each build page's Tests tab includes an Orchestration page, which shows how bktec selected and split the tests in each test plan used in the build. To open it, select the build's Tests tab, then select Orchestration. A test plan appears once a job using it finishes.

The estimated time saved rows on the Test selection card only appear when at least half of the candidates have timing history. Estimated test wall-clock time saved only appears for steps with a fixed parallelism.

When at least one test was selected, the panel also shows a Selection map of the suite's test files, with the selected files in purple. Plans that mix test files or selectors with individual tests show Coverage map unavailable for plans with mixed test formats instead.