Test Engine glossary

Dimensions

In the context of Test Engine, dimensions are structured data, consisting of tags, which can be used to filter or group (that is, aggregate) test executions. Dimensions are added to test executions using the tags feature, which you can learn more about in Tags.

Execution

An execution is an instance of a single test, which is generated as part of a run. An execution tracks several aspects of a test, including its result (passed, failed, skipped, other), duration (time), and dimensions (that is, tags).

Flaky test

A flaky test is a test that produce inconsistent or unreliable results, despite being run in the same code and environment. Flaky tests are usually identified following a number of runs which are executed as part of pipeline builds, such as those of a Buildkite pipeline.

Learn more about flaky tests in Flaky test management.

Managed test

A managed test refers to any test (within all test suites of a Buildkite organization) that can be uniquely identified by its combination of test suite, scope, and name of the test.

For example, each of the following three tests are unique managed tests:

• Test Suite 1 - here.is.scope.one - Login Test name

• Test Suite 1 - here.is.another.scope - Login Test name

• Test Suite 2 - here.is.scope.one - Login Test name

Test Engine uses managed tests to track key areas test runs, and for billing purposes.

Learn more about managed tests in Usage and billing.

Quarantine

Quarantine is a classification applied to a test that, based on the state of the test, changes how Test Engine executes that test as part of a run. When a test is quarantined, and its test state is flagged as:

• mute, the test is executed as part of the run, but its failure does not cause the pipeline build to fail, allowing the test's metadata to still be collected.

• skip, the test is not be executed as part of the run, which can allow pipeline builds to execute more rapidly and can reduce costs, but no data is recorded from the test.

Learn more about quarantining tests in Test state and quarantine.

Run

A run is the execution of one or more tests in a test suite. A run is sometimes referred to as a test run, bearing in mind that a single test run usually involves the execution of multiple tests. A Test Engine run is analogous to a Pipeline build.

Scope

A scope is a mechanism that can be implemented to differentiate between two or more identically named tests. For example, the following hypothetical tests have the same name as they both test the process of a user logging into a product platform. However, one of these applies to this test being done on a mobile device, while the other applies to a desktop web setting. Therefore, a scope can be used to differentiate between these two tests.

A test's scope is used in determining a managed test's uniqueness.

Tag

A tag is a key:value pair containing two parts:

• The tag's key is the identifier, which can only exist once on each test, and is case sensitive.
• The tag's value is the specific data or information associated with the key.

In Test Engine, tags add dimensions to test execution metadata, so that tests and their executions can be better filtered, aggregated, and compared in Test Engine visualizations.

Tagging can be used to observe aggregated data points—for example, to observe aggregated performance across several tests, and (optionally) narrow the dataset further based on specific constraints.

Learn more about tags in the Tags topic.

Test

A test is an individual piece of code that runs as part of an application's or component's (for example, a library's) building process (which can be automated by Pipelines), to ensure that a specific area of the application or component functions as expected.

Test collection

Test collection is the process of collecting test data from a development project. Test collection may consist of one or more test collectors configured within a development project, or make use other methods based on common standards such as JUnit XML or JSON to collect tests.

While a development project's test runners (such RSpec or Jest) are typically configured with their respective test collectors, the JUnit XML or JSON test collection mechanisms can be used to collect test data from multiple test runners.

Test collector

A test collector is a dedicated open source source library (developed by Buildkite) that can be implemented into your development project, to collect test data from a test runner within your project.

Buildkite offers a number of test collectors for a range of languages and their test runners.

Test runner

A test runner is a synonymous term used for a test framework, which is typically a code library that can be integrated into a development project to facilitate the implementation of tests for that project.

Test state

A test state is a configurable flag that can be applied to a test (typically flaky tests), which quarantines the test and affects how the test is executed as part of a run.

The following test state flags for quarantining tests are supported:

• mute, the test is executed as part of the run, but its failure does not cause the pipeline build to fail, allowing the test's metadata to still be collected.

• skip, the test is not be executed as part of the run, which can allow pipeline builds to execute more rapidly and can reduce costs, but no data is recorded from the test.

Learn more about test states in Test state and quarantine.

Test suite

A test suite is a collection of tests, which is managed through Buildkite Test Engine. A test suite is sometimes abbreviated to suite.

In a development project configured with of one or more test runners, it is usually typical to configure a separate test suite each of the project's test runners.