Ruby collectors

RSpec collector

RSpec is a behavior-driven development library for Ruby. If you're already using RSpec for your tests, add the buildkite-test_collector gem to your code to collect your test results into your Test Engine dashboard.

Before you start, make sure RSpec runs with access to CI environment variables.

1. Create a new branch:

   git checkout -b install-buildkite-test-engine
   

2. Add buildkite-test_collector to your Gemfile in the :test group:

   group :test do
     gem "buildkite-test_collector"
   end
   

3. Run bundle to install the gem and update your Gemfile.lock:

   bundle
   

4. Add the Test Engine code to your application in spec/spec_helper.rb, and set the BUILDKITE_ANALYTICS_TOKEN securely on your agent or agents. Please ensure gems that patch Net::HTTP, like httplog and sniffer, are required before buildkite/test_collector to avoid conflicts.

   require "buildkite/test_collector"
   
   Buildkite::TestCollector.configure(hook: :rspec)
   

5. Commit and push your changes:

   $ git add .
   $ git commit -m "Install and set up Buildkite Test Engine"
   $ git push
   

Once you're done, in your Test Engine dashboard, you'll see analytics of test executions on all branches that include this code.

If you don't see branch names, build numbers, or commit hashes in the Test Engine UI, then see CI environments to learn more about exporting your environment to the collector.

Troubleshooting allow_any_instance_of errors

If you're using RSpec and seeing errors related to allow_any_instance_of that look like this:

Failure/Error: allow_any_instance_of(Object).to receive(:sleep)
       Using `any_instance` to stub a method (sleep) that has been defined on a prepended module (Buildkite::TestCollector::Object::CustomObjectSleep) is not supported.

You can fix them by being more specific in your stubbing by replacing allow_any_instance_of(Object).to receive(:sleep) with allow_any_instance_of(TheClassUnderTest).to receive(:sleep).

Troubleshooting test grouping issues

RSpec supports anonymous test cases—tests which are automatically named based on the subject and/or inputs to the expectations within the test. However, this can lead to unstable test names across different test runs, incorporating elements such as object IDs, database IDs, timestamps, and more.

As a consequence, each test is assigned a new identity per run within Test Engine. This poses a challenge for using the Test Engine product effectively, as historical data across tests becomes difficult to track and analyze.

To mitigate this issue and ensure the reliability of Test Engine, it's advisable to provide explicit and stable descriptions for each test case within your RSpec test suite. By doing so, you can maintain consistency in test identification across multiple runs, enabling better tracking and analysis of test performance over time.

minitest collector

minitest provides a complete suite of testing facilities supporting TDD, BDD, mocking, and benchmarking.

If you're already using minitest for your tests, add the buildkite-test_collector gem to your code to collect your test results into your Test Engine dashboard.

1. Create a new branch:

   git checkout -b install-buildkite-collector
   

2. Add buildkite-test_collector to your Gemfile in the :test group:

   group :test do
     gem "buildkite-test_collector"
   end
   

3. Run bundle to install the gem and update your Gemfile.lock:

   bundle
   

4. Add the Test Engine code to your application in test/test_helper.rb, and set the BUILDKITE_ANALYTICS_TOKEN securely on your agent or agents. Please ensure gems that patch Net::HTTP, like httplog and sniffer, are required before buildkite/test_collector to avoid conflicts.

   require "buildkite/test_collector"
   
   Buildkite::TestCollector.configure(hook: :minitest)
   

5. Commit and push your changes:

   git add .
   git commit -m "Install and set up Buildkite Test Engine"
   git push
   

Once you're done, in your Test Engine dashboard, you'll see analytics of test executions on all branches that include this code.

If you don't see branch names, build numbers, or commit hashes in the Test Engine UI, then see CI environments to learn more about exporting your environment to the minitest collector.

Adding annotation spans

This gem allows adding custom annotations to the span data sent to Buildkite using the .annotate method. For example:

Buildkite::TestCollector.annotate("Visiting login")

This would appear like so:

This is particularly useful for tests that generate a lot of span data such as system/feature tests. You can find all annotations under Span timeline at the bottom of every test execution page.

Tagging duplicate test executions with a prefix/suffix

For builds that execute the same test multiple times it's possible to tag each test execution with a prefix/suffix describing the test environment. This is useful when running a test suite against multiple versions of Ruby or Rails. The prefix/suffix is set using these environment variables:

BUILDKITE_ANALYTICS_EXECUTION_NAME_PREFIX
BUILDKITE_ANALYTICS_EXECUTION_NAME_SUFFIX

When viewing a test every execution is displayed including its corresponding prefix/suffix. For example: