Usage and billing

Test Engine is designed to optimize your test suites through the management of your tests.

Managed tests

Buildkite bills Test Engine customers by number of managed tests. See the Buildkite Pricing page for plan-level details.

Each and every test that can be uniquely identified by its combination of test suite, scope, and name, is a managed test.

For example, each of the following three tests are unique managed tests:

  • Test Suite 1 - here.is.scope.one - Login Test name
  • Test Suite 1 - here.is.another.scope - Login Test name
  • Test Suite 2 - here.is.scope.one - Login Test name

Test Engine conducts the following on each managed test:

  • Tracks its history
  • Maintains its state (for example, Enterprise plan customers can quarantine tests by disabling them under certain conditions)
  • Attributes ownership by team

For billing purposes, Buildkite measures usage by calculating the number of managed tests that have executed (run) at least once each day, and then bills based on the 90th percentile of this usage for the month. This billing method ensures that occasional spikes in usage, such as those caused by refactoring, don't result in excessive charges.

Executed managed tests are only charged once per day

If a specific managed test has run multiple times on a specific day, then this only counts once towards the usage measurement for that day.

Test executions

Personal and legacy plans only

This section is only applicable to Buildkite Test Engine customers on the Personal and paid legacy plans.

If you are on the Personal plan, your first 50,000 test executions are free, after which, you will need to upgrade to the Pro or Enterprise plan to continue using Test Engine. For customers on the Pro or Enterprise plan, usage is billed per managed test.

Customers on legacy paid plans may still be billed per individual test execution, which sum to the total number of times a test was executed (test execution count). However, this approach is no longer used on current and new Buildkite Pro or Enterprise plans. Instead, see Managed tests for details about the current billing approach for these plans.

You can find the test execution details for a run at the top of the run page, and your organization's total usage in Settings.

Test executions run page

Usage page

The Usage page is available on every Buildkite plan, and shows a breakdown of all billable usage for your organization including managed tests and test executions.

The managed tests usage page graphs the maximum number of unique tests per day over the organization's billing periods. This page includes a breakdown of usage by suite and a CSV download of usage over the period.

The test executions usage page graphs the total executions over the organization's billing periods. This page includes a breakdown of usage by suite and a CSV download of usage over the period.