Python collectors

pytest collector

pytest is a testing framework for Python. If you're already using pytest, then you can install buildkite-test-collector to collect test results into your Test Engine dashboard.

Before you start, make sure pytest runs with access to CI environment variables.

To get started with buildkite-test-collector:

1. In your CI environment, set the BUILDKITE_ANALYTICS_TOKEN environment variable securely to your Buildkite Test Engine API token.

2. Add buildkite-test-collector to your list of dependencies. Some examples:

   • If you're using a requirements.txt file, add buildkite-test-collector on a new line.

   • If you're using a setup.py file, add buildkite-test-collector to the extras_require argument, like this:

     extras_require={"dev": ["pytest", "buildkite-test-collector"]}

   • If you're using Pipenv, run pipenv install --dev buildkite-test-collector.

   If you're using another tool, see your dependency management system's documentation for help.

3. Commit and push your changes:

   $ git add .
   $ git commit -m "Install and set up Buildkite Test Engine"
   $ git push
   

Once you're done, in your Test Engine dashboard, you'll see analytics of test executions on all branches that include this code.

If you don't see branch names, build numbers, or commit hashes in Test Engine, then read CI environments to learn more about exporting your environment to the collector.

Upload custom tags for test executions

You can group test executions using custom tags to compare metrics across different dimensions, such as:

• Language versions
• Cloud providers
• Instance types
• Team ownership
• and more

We offer a tagging solution based on pytest custom markers.

Upload-level tags

In your conftest.py file, you can use pytest global hook to tag all your test executions in a centralized way.

import pytest
import sys

def pytest_itemcollected(item):
  # add execution tag to all tests
  item.add_marker(pytest.mark.execution_tag("test.framework.name", "pytest"))
  item.add_marker(pytest.mark.execution_tag("test.framework.version", pytest.__version__))
  item.add_marker(pytest.mark.execution_tag("cloud.provider", "aws"))
  item.add_marker(pytest.mark.execution_tag("language.version", sys.version))

Execution-level tags

For more granular control, you can programmatically or statically add tags to target individual tests.

To do it statically, targeting a single test or module:

import pytest

 @pytest.mark.execution_tag("team", "frontend")
 def test_add():
     assert 1 + 1 == 2

To do it programmatically, for example:

import pytest
import sys

def pytest_itemcollected(item):
  # You can use the rich data provided by pytest to selectively add execution tag to tests
  if "e2e" in item.location[0]:
    item.add_marker(pytest.mark.execution_tag("type", "browser"))