Usage and billing

Test Engine is designed to optimize your test suites through the management of your tests.

Managed tests

Each and every test that can be uniquely identified by its combination of test suite, scope, and name, is a managed test, which in turn is used for billing purposes in Test Engine.

For example, each of the following three tests are unique managed tests:

  • Test Suite 1 - here.is.scope.one - Login Test name
  • Test Suite 1 - here.is.another.scope - Login Test name
  • Test Suite 2 - here.is.scope.one - Login Test name

Test Engine conducts the following on each managed test:

For billing purposes, Buildkite measures usage by calculating the number of managed tests that have executed (run) at least once each day, and then bills based on the 90th percentile of this usage for the month. This billing method ensures that occasional spikes in usage, such as those caused by refactoring, don't result in excessive charges.

Be aware that if a specific managed test has run multiple times on a specific day, then this only counts once towards the usage measurement for that day.

Test executions

Some legacy Buildkite plans measure usage based on the total number of times a test was executed (run).

You can find the test execution details for a run at the bottom of the run page, and your organization's total usage in Settings.

Test executions run page

Usage page

The Usage page is available on every Buildkite plan, and shows a breakdown of all billable usage for your organization including managed tests and test executions.

The managed tests usage page graphs the maximum number of unique tests per day over the organization's billing periods. This page includes a breakdown of usage by suite and a CSV download of usage over the period.

The test executions usage page graphs the total executions over the organization's billing periods. This page includes a breakdown of usage by suite and a CSV download of usage over the period.

Your organization's usage is also accessible in the GraphQL API.