Tags

The tags feature is currently in public preview. Test Engine customers can tag executions with an additional 10 custom tags beyond the included core tags.

Tags is a Test Engine feature that adds dimensions to test execution metadata so tests and executions can be better filtered, aggregated, and compared in Test Engine visualizations.

Tagging can be used to observe aggregated data points—for example, to observe aggregated performance across several tests, and (optionally) narrow the dataset further based on specific constraints.

Tags are key:value pairs containing two parts:

  • The tag's key is the identifier, which can only exist once on each test, and is case sensitive.
  • The tag's value is the specific data or information associated with the key.

Core tags

The following core tags are vital to helping you understand and improve the performance of your test suite. These tags are included in the managed tests price.

Where possible, Test Engine will automatically ingest this data on your behalf.

Tag key Use case
Tag key build.id Use case Filtering and aggregating based on the build identifier.
Tag key build.job_id Use case Filtering and aggregating based on the job identifier.
Tag key build.step_id Use case Filtering and aggregating based on the step identifier.
Tag key cloud.provider Use case Filtering and aggregating based on your cloud provider to compare cloud provider performance and reliability in your test suite.
Example: aws vs gcp.
Tag key cloud.region Use case Filtering and aggregating based on your cloud region to compare region performance and reliability in your test suite.
Example: us-east-1 vs us-east-2.
Tag key code.file.path Use case Filtering and aggregating based on the file path or subsection of the file path.
Tag key collector.name Use case Filtering and aggregating based on the Test Engine collector you are using. Useful when onboarding or updating your Test Engine collector.
Tag key collector.version Use case Filtering and aggregating based on the Test Engine collector version you are using. Useful when onboarding or updating your Test Engine collector.
Tag key host.arch Use case Filtering and aggregating based on the architecture to compare architecture performance and reliability in your test suite.
Example: arm64 vs x86_64.
Tag key host.type Use case Filtering and aggregating based on the instance type to compare instance performance and reliability in your test suite.
Example: m4.large vs m5.large.
Tag key language.name Use case Filtering and aggregating based on the programming language to compare language performance and reliability in your test suite.
Example: python vs javascript.
Tag key language.version Use case Filtering and aggregating based on the language version to compare version performance and reliability in your test suite.
Example: 3.0.2 vs 2.5.3.
Tag key scm.branch Use case Filtering and aggregating based on source code branch to compare branch performance and reliability. For example you might be rolling out a new dependency and you are testing this in a branch.
Tag key scm.commit_sha Use case Filtering and aggregating based on commit_sha to compare specific commit performance and reliability.
Tag key test.framework.name Use case Filtering and aggregating based on testing framework to compare performance and reliability.
Tag key test.framework.version Use case Filtering and aggregating based on testing framework version to compare performance and reliability.

Custom tags

In addition to the core tags, you can tag executions with your own custom tags.

Defining tags

Test Engine has the following tagging requirements:

  • Up to 10 tags may be specified at the upload level (applying to all executions), per upload.
  • Up to 10 tags may be specified on each execution.

Tag keys

  • Must not be blank.
  • Must begin with a letter, and may contain letters, numbers, underscores, hyphens and periods.
  • Must be less than 64 bytes of UTF-8 text.

Tag values

  • Must not be blank.
  • Must be less than 128 bytes of UTF-8 text.

Tagging methods

Tags may be assigned using the following collection methods:

Usage

After you have assigned tags at the test collection level, start using them to filter and group your test results. Tags are used in the following areas of the Buildkite Platform.

Test execution drawer

On the test page, you can open the execution drawer by selection an execution.

This presents all the tags which have been applied to the test execution.

Screenshot of test page with execution drawer open displaying execution tags available for filtering and aggregtion

Group by tag

Grouping by tag on the test page breaks down the test reliability and duration (p50, p95), so that you can compare performance across the tag values.

Screenshot of test page with a group by tag aggregation applied breaking down metrics by architecture

Filter by tag

Filtering by tag on the test page will constrain all executions for the test which match the filter conditions.

Screenshot of test page with a tag filter applied restricting executions to just those that ran on t3.large in ruby

Filtering by tag on the test index page will constrain all tests to those that had executions matching the conditions of the filter. In the following case all tests that ran on the t3.large instance type.

Screenshot of test index with a tag filter applied restricting tests to just those running on t3.large instances

You can filter by tag using the @ prefix to denote that you want to filter by tag. For example:

@host.type:t3.large

To filter tests by tags in Pipelines, select the Tests tab in either the job or build interface and then select the Add filter filter button.

Screenshot of filtering tests on a pipeline build