Usage and billing
Test Engine is designed to optimize your test suites through the management of your tests.
Managed tests
Each and every test that can be uniquely identified by its combination of test suite, scope, and name, is a managed test, which in turn is used for billing purposes in Test Engine.
For example, each of the following three tests are unique managed tests:
- Test Suite 1 - here.is.scope.one - Login Test name
- Test Suite 1 - here.is.another.scope - Login Test name
- Test Suite 2 - here.is.scope.one - Login Test name
Test Engine conducts the following on each managed test:
- Tracks its history
- Calculates its flakiness
- Maintains its state (for example, Enterprise plan customers can quarantine tests by disabling them under certain conditions)
- Attributes ownership by team
For billing purposes, Buildkite measures usage by calculating the number of managed tests that have executed (run) at least once each day, and then bills based on the 90th percentile of this usage for the month. This billing method ensures that occasional spikes in usage, such as those caused by refactoring, don't result in excessive charges.
Executed managed tests are only charged once per day
If a specific managed test has run multiple times on a specific day, then this only counts once towards the usage measurement for that day.
Test executions
Some legacy Buildkite plans measure usage based on the total number of times a test was executed (run).
You can find the test execution details for a run at the bottom of the run page, and your organization's total usage in Settings.
Usage page
The Usage page is available on every Buildkite plan, and shows a breakdown of all billable usage for your organization including managed tests and test executions.
The managed tests usage page graphs the maximum number of unique tests per day over the organization's billing periods. This page includes a breakdown of usage by suite and a CSV download of usage over the period.
The test executions usage page graphs the total executions over the organization's billing periods. This page includes a breakdown of usage by suite and a CSV download of usage over the period.
Your organization's usage is also accessible in the GraphQL API.