Test Collector Buildkite Plugin

A Buildkite plugin for uploading JSON or JUnit files to Buildkite Test Engine ✨

Options

These are all the options available to configure this plugin’s behaviour.

Required

files (string or array of strings)

One or more patterns of files to upload to Test Engine, relative to the root of the searching path (./ by default). May contain * to match any number of characters of any type (unlike shell expansions, it will match / and . if necessary). Can be either a single pattern in a string or any number of them in an array.

format (string)

Format of the file.

Only the following values are allowed: junit, json

Optional

api-token-env-name (string)

Name of the environment variable that contains the Test Engine API token.

Default value: BUILDKITE_ANALYTICS_TOKEN

api-url (string)

Full URL for the API to upload to. Defaults to https://analytics-api.buildkite.com/v1/uploads

base-path (string)

Where to search for files to upload. Defaults to the working directory .

branches (string)

String containing a regex to only do an upload in branches that match it (using the case-insensitive bash =~ operator against the BUILDKITE_BRANCH environment variable).

For example:

• prod will match any branch name that contains the substring prod
• ^stage- will match all branches that start with stage-
• -ISSUE-[0-9]*$ will match branches that end with ISSUE-X (where X is any number)

Important: you may have to be careful to escape special characters like $ during pipeline upload

debug (boolean)

Print debug information to the build output.

Default value: false.

Can also be enabled with the environment variable BUILDKITE_ANALYTICS_DEBUG_ENABLED.

exclude-branches (string)

String containing a regex avoid doing an upload in branches that match it (using the case-insensitive bash =~ operator against the BUILDKITE_BRANCH environment variable ).

For example:

• prod will exclude any branch name that contains the substring prod
• ^stage- will exclude all branches that start with stage-
• -SECURITY-[0-9]*$ will exclude branches that end with SECURITY-X (where X is any number)

Important:

• you may have to be careful to escape special characters like $ during pipeline upload
• exclusion of branches is done after the inclusion (through the branches option)

follow-symlinks (boolean)

By default the plugin will not follow symlinked folders, set this option to true to do so. This will add the -L option to the find command used to get the files to upload.

missing-error (integer)

Exit code of the hook when no files are found to upload as long as the step’s command has not failed already.

This should allow you to use a special exit code to soft-fail on when no files to upload. Warning: make sure that the exit code used can not be generated by your command to avoid hiding actual step failures.

Default value: 1

tags (array of strings)

A list of tags to apply to all test results in the upload, in the format key=value.

For example:

tags:
  - "arch=arm64"
  - "language.version=1.2.3"

timeout (number)

Maximum number of seconds to wait for each file to upload before timing out.

Default value: 30

annotation-link(boolean)

Adds an annotation to the build run with a link to the uploaded report.

Default value: false

upload-concurrency(number)

The number of concurrent file uploads to perform to the Buildkite Test Engine API.

Default value: 1

Requirements

This plugin requires jq for parsing JSON data. If jq is not found on the agent, sed will be used as a fallback. Ensure that sed is also available to handle scenarios where jq cannot be used.

Fallback Behavior

If jq is unavailable, the plugin will attempt to parse the results using sed. This ensures that the plugin remains functional even if the preferred JSON parser is missing.

Examples

Upload a JUnit file

To upload a JUnit file to Test Engine from a build step:

steps:
  - label: "🔨 Test"
    command: "make test"
    plugins:
      - test-collector#v1.11.0:
          files: "test/junit-*.xml"
          format: "junit"

Upload a JSON file

To upload a JSON file to Test Engine from a build step:

steps:
  - label: "🔨 Test"
    command: "make test"
    plugins:
      - test-collector#v1.11.0:
          files:
            - "test-data-*.json"
          format: "json"

Using build artifacts

You can also use build artifacts generated in a previous step:

steps:
  # Run tests and upload 
  - label: "🔨 Test"
    command: "make test --junit=tests-N.xml"
    artifact_paths: "tests-*.xml"

  - wait

  - label: "🔍 Test Engine"
    command: buildkite-agent artifact download "tests-*.xml" .
    plugins:
      - test-collector#v1.11.0:
          files: "tests-*.xml"
          format: "junit"

Branch filtering

Only upload on the branches that end with -qa

steps:
  - label: "🔨 Test"
    command: "make test"
    plugins:
      - test-collector#v1.11.0:
          files: "test-data-*.json"
          format: "json"
          branches: "-qa$"

Do not upload on the branch that is exactly named legacy:

steps:
  - label: "🔨 Test"
    command: "make test"
    plugins:
      - test-collector#v1.11.0:
          files: "test-data-*.json"
          format: "json"
          exclude-branches: "^legacy$"

Only upload on branches that start with stage- but do not contain hotfix

steps:
  - label: "🔨 Test"
    command: "make test"
    plugins:
      - test-collector#v1.11.0:
          files: "test-data-*.json"
          format: "json"
          branches: "^stage-"
          exclude-branches: "hotfix"

⚒ Developing

You can use the bk cli to run the pipeline locally:

bk local run

:warning: bk local run was removed on v3. You can use v2 and get the binaries from https://github.com/buildkite/cli/releases/tag/v2.0.0

Or if you want to run just the tests, you can use the docker Plugin Tester:

docker run --rm -ti -v "${PWD}":/plugin buildkite/plugin-tester:latest

👩‍💻 Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/buildkite-plugins/test-collector-buildkite-plugin

📜 License

The package is available as open source under the terms of the MIT License.