Getting started

👋 Welcome to Buildkite Test Engine! You can use Test Engine to help you track and analyze the test steps automated through CI/CD using either Buildkite Pipelines or another CI/CD application.

This tutorial helps you understand Buildkite Test Engine's fundamentals, by guiding you through the creation of a new Test Engine test suite, and then cloning and running a simple example Ruby project to generate test results that are collected and reported through this test suite. Note that Buildkite Test Engine supports other languages and test runners too.

Before you start

To complete this tutorial, you'll need:

A Buildkite account. If you don't have one already, create a 30-day free trial account.
Git, to clone the Ruby project example.
Ruby—macOS users can also install Ruby with Homebrew.
- Once Ruby is installed, open a terminal or command prompt and install the RSpec testing framework using the following command:
```
gem install rspec
```

Create a test suite

To begin creating a new test suite:

Select Test Suites in the global navigation to access the Test Suites page.
Select New test suite.
On the Identify, track and fix problematic tests page, enter an optional Application name. For example, RSpec test suites.
Enter a mandatory Test suite name. For example, My RSpec example test suite.
Enter the Default branch name, which is the default branch that Test Engine shows trends for, and can be changed any time. For example (and usually), main.
Enter an optional Suite emoji, using emoji syntax. For example, :ruby: for a ruby emoji representing the Ruby language.
Select Set up suite.
If your Buildkite organization has the teams feature enabled, select the relevant Teams to be granted access to this test suite, followed by Continue.

The new test suite's Complete test suite setup page is displayed, requesting you to configure your test collector within your development project.

Keep this web page open.

Clone the Ruby example test suite project

Then, clone the Ruby example test suite project:

Open a terminal or command prompt, and run the following command:
```
git clone git@github.com:buildkite/ruby-example-test-suite.git
```
Change directory (cd) into the ruby-example-test-suite directory.
(Optional) Run the following rspec command to test that RSpec test runner executes successfully:
```
rspec
```
After about a minute, the command output should display something similar to:
```
disabled tests
......................

Finished in 1 minute 4.06 seconds (files took 0.29045 seconds to load)
22 examples, 0 failures
```

Configure your Ruby project with its test collector

Next, configure your Ruby project's RSpec test runner with its Buildkite test collector:

Install the buildkite-test_collector gem by running the following gem command:
```
gem install buildkite-test_collector
```

Add the following lines of code to your project's spec_helper.rb file:

require 'buildkite/test_collector'

Buildkite::TestCollector.configure(hook: :rspec)

The top of this file should look similar to:

require 'yaml'
require 'json'
require 'buildkite/test_collector'

Buildkite::TestCollector.configure(hook: :rspec)

begin
  skip_data = File.read('skipped.json')
  skip = JSON.parse(skip_data)
rescue
  skip = []
end

...

Run RSpec (again) to send your test data to Test Engine

Back on the Complete test suite setup page, copy the Test Suite API token value.
At your terminal/command prompt, run the following rspec command (with additional environment variables) to execute the RSpec test runner and send its execution data back to your Test Engine test suite:
```
BUILDKITE_ANALYTICS_TOKEN=<api-token-value> BUILDKITE_ANALYTICS_MESSAGE="My first test run" rspec
```
where:
- <api-token-value> is the value of the Test Suite API token value you copied in the previous step. This value can typically be pasted without any quotation marks.
- BUILDKITE_ANALYTICS_MESSAGE is an environment variable, which is usually used for a source control (Git) commit message, and is presented in a run of your Buildkite test suite. However, in this scenario, this environment variable and its value are being used to describe the test run (or build). Learn more about these types of environment variables, which are available to other CI/CD providers (that is, those other than Buildkite Pipelines, CircleCI or GitHub Actions), as well as containers, and manually run builds such as this rspec execution command above.
The command output should display something similar to:
```
disabled tests
......................

Finished in 1 minute 4.06 seconds (files took 0.25227 seconds to load)
22 examples, 0 failures
```
Back in Test Engine, your test suite should now be displayed, showing its Runs tab, with a summary of details from the last execution of the RSpec test runner in the previous step. The final result should indicate My first test run (obtained from the value of BUILDKITE_ANALYTICS_MESSAGE in the previous step) with a status of PASSED.

If this page indicates Still processing data after a while, refresh your browser page to display the results. If the status indicates PENDING, wait a little longer until the final result appears.

Next steps

That's it! You've successfully created a test suite, configured your Ruby project with a test collector, and executed the project's test runner to send its test data to your test suite. 🎉

Learn more about:

How to configure test collection for other test runners.
CI environment variables that test collectors (and other test collection mechanisms) provide to your Buildkite test suites, when your test runs are automated through CI/CD.
How to work with test suites in Buildkite Test Engine.