Buildkite Test Engine

Deliver confidently with fast and reliable tests

Test suite reliability
89.73%
Disabled
Flaky
Flaky assigned to
Mobile team
Mobile app push notifications deliver on schedule
Owners: Mobile team
main
./spec/mobile_app_push_notifications_deliver_on_schedule_spec.rb
github↗
💻
:ruby:
Protagonist
Rails monolith
Runs
1,254/week
📱
:swift:
Mobile app
Runs
473/week

Flaky tests? React in real time

Keep the critical path clear

Eliminate wasteful retries and keep merge queues moving by automatically categorizing and quarantining problematic tests, including flaky test detection.

Assign team ownership

Map tests to teams and automatically assign ownership when troublesome test cases are detected.

An example CODEOWNERS file window with a foreground visual showing assigned flaky tests

Being able to deliver software to production quickly and consistently is a critical capability for Persona Identities. Flaky tests were significantly impacting our engineering productivity; Buildkite's Test Engine cut flakey test disruptions by 50% in only four weeks, which translated into substantial productivity gains.

Ian Chesal
Head of Infrastructure at Persona Identities

Find your slowest tests, fast

Understand test performance

Identify and optimize slow or poor performing tests using timing data analysis from all test runs.

Use timing data to split work

Group tests into parallel jobs based on timing data—using the built-in intelligent sorting or your own rules for distributing tests.

Before

10m
3m
2m
1m

Waiting on slowest node

After

4m
4m
4m
4m

Optimal partitioning reduces wait time

Optimize tests on any platform

Get test insights from any CI/CD tool: Buildkite Pipelines, GitHub Actions, Jenkins, CircleCI, and more.

Buildkite, CircleCI, Jenkins, and GitHub Actions logos

Real-time insights and analytics

Get a comprehensive view of all tests

Understand what’s running and what needs attention across your test suites, all from one place. Unit tests, integration tests, end-to-end tests—see analysis of your regression testing health in a single view.

💻
:ruby:
Protagonist
Rails monolith
Runs
0/week
📱
:swift:
Mobile app
Runs
0/week
:react:
:jest:
Marketing site
Runs
0/week

Report on progress

Automate reports on the status of unreliable or troublesome tests across teams on a configurable cadence.

A digest email report on top 5 flaky tests

Deep tracing and instrumentation

Automatic tracing helps you see why tests are slow or behaving differently between executions.

HTTP
GET http://selenium:4444/wd/hub/session
349ms
SQL
SELECT session_id FROM sessions WHERE
8ms
Rendered OTP Screen
HTTP
GET http://selenium:4444/wd/hub/session
25ms
HTTP
POST http://selenium:4444/wd/hub/session
8ms
SQL
SELECT session_id FROM sessions WHERE
8ms
SQL
INSERT INTO otp_attempts (user_id, created_at
8ms

Frequently asked questions

Got a question that’s not on our list? Want a demo? Just want to chat? Get in touch.

Flaky tests are automated tests that produce inconsistent or unreliable results, despite being run on the same code and environment. They cause frustration, decrease confidence in testing, and waste time while you investigate whether the failure is due to a genuine bug.

By default, Test Engine detects flaky tests by surfacing when the same test is run multiple times on the same commit SHA with different results. The tests might run multiple times within a single build or across different builds. Either way, they are detected as flaky if they report both passed and failed results.

Test Engine also offers several other flaky test detection configurations based on those used by top software teams.

Yes! Use our supported collectors for RSpec, Jest, Cypress, pytest, Swift, and more—or easily implement your own collector. Or, use Test Engine with JUnit XML-formatted test results (the industry standard for test result metadata).

Buildkite Test Engine uniquely identifies each test by its combination of test suite, scope, and name. These “managed tests” are used to power Test Engine's analytics, as well as to provide a stable basis for billing purposes.

We use a 90th percentile (P90) billing method for managed tests. This approach encourages continuous improvement of your testing practices while maintaining predictable costs.

To accomplish this, Buildkite measures usage by calculating the number of managed tests that have executed (run) at least once each day, and at month's end, disregard the top 10%. By disregarding the highest usage days, you can freely add new tests, experiment with different testing strategies, or temporarily increase test coverage without worrying about billing spikes.

Yes! All new organizations get a 30-day free trial of the Pro plan to try the key features. See Pricing for all the details on the plans.

Buildkite provides an SLA of 99.95% uptime and a status page to track any incidents.

Resources

Get started with real time test management

Create an account to get started with a 30-day free trial. No credit card required.

Buildkite Test Engine

Platform

  1. Pipelines
  2. Pipeline templates
  3. Public pipelines
  4. Test Engine
  5. Package Registries
  6. Mobile Delivery Cloud
  7. Pricing

Hosting options

  1. Self-hosted agents
  2. Mac hosted agents
  3. Linux hosted agents

Resources

  1. Docs
  2. Blog
  3. Changelog
  4. Webinars
  5. Plugins
  6. Case studies
  7. Events

Company

  1. About
  2. Careers
  3. Press
  4. Brand assets
  5. Contact

Solutions

  1. Replace Jenkins
  2. Workflows for AI/ML

Support

  1. System status
  2. Forum