Importing JSON

If a test collector is not available for your test framework, you can upload tests results directly to the Test Analytics API or write your own test collector. You can upload JSON-formatted test results (described in this page) or JUnit XML.

How to import JSON in Buildkite

It's possible to import JSON (or JUnit files) to Buildkite Test Analytics with or without the help of a plugin.

Using a plugin

To import JSON-formatted test results to Test Analytics using Test Collector plugin from a build step:

pipeline.yml
steps:
  - label: "🔨 Test"
    command: "make test"
    plugins:
      - test-collector#v1.0.0:
          files: "test-data-*.json"
          format: "json"

See more configuration information in the Test Collector plugin README.

Using the plugin is the recommended way as it allows for a better debugging process in case of an issue.

An equivalent of Buildkite Test Collector plugin for GitHub Action and CircleCI Orb is in the works. Stay tuned!

Without a plugin

If for some reason you cannot or do not want to use the Test Collector plugin, or if you are looking to implement your own integration, another approach is possible.

To import JSON-formatted test results in Buildkite, make a POST request to https://analytics-api.buildkite.com/v1/uploads with a multipart/form-data.

For example, to import the contents of a JSON-formatted test results (test-results.json):

  1. Securely set the Test Analytics token environment variable (BUILDKITE_ANALYTICS_TOKEN).

  2. Run the following curl command:

    curl \
      -X POST \
      -H "Authorization: Token token=\"$BUILDKITE_ANALYTICS_TOKEN\"" \
      -F "data=@test-results.json" \
      -F "format=json" \
      -F "run_env[CI]=buildkite" \
      -F "run_env[key]=$BUILDKITE_BUILD_ID" \
      -F "run_env[url]=$BUILDKITE_BUILD_URL" \
      -F "run_env[branch]=$BUILDKITE_BRANCH" \
      -F "run_env[commit_sha]=$BUILDKITE_COMMIT" \
      -F "run_env[number]=$BUILDKITE_BUILD_NUMBER" \
      -F "run_env[job_id]=$BUILDKITE_JOB_ID" \
      -F "run_env[message]=$BUILDKITE_MESSAGE" \
      https://analytics-api.buildkite.com/v1/uploads
    

To learn more about passing through environment variables to run_env-prefixed fields, see CI Environments.

How to import JSON in CircleCI

To import JSON-formatted test results, make a POST request to https://analytics-api.buildkite.com/v1/uploads with a multipart/form-data body, including as many of the following fields as possible in the request body:

For example, to import the contents of a test-results.json file in a CircleCI pipeline:

  1. Securely set the Test Analytics token environment variable (BUILDKITE_ANALYTICS_TOKEN).

  2. Run the following curl command:

    curl \
    -X POST \
    -H "Authorization: Token token=\"$BUILDKITE_ANALYTICS_TOKEN\"" \
    -F "data=@test-results.json" \
    -F "format=json" \
    -F "run_env[CI]=circleci" \
    -F "run_env[key]=$CIRCLE_WORKFLOW_ID-$CIRCLE_BUILD_NUM" \
    -F "run_env[number]=$CIRCLE_BUILD_NUM" \
    -F "run_env[branch]=$CIRCLE_BRANCH" \
    -F "run_env[commit_sha]=$CIRCLE_SHA1" \
    -F "run_env[url]=$CIRCLE_BUILD_URL" \
    https://analytics-api.buildkite.com/v1/uploads
    

To learn more about passing through environment variables to run_env-prefixed fields, see CI Environments.

How to import JSON in GitHub Actions

To import JSON-formatted test results, make a POST request to https://analytics-api.buildkite.com/v1/uploads with a multipart/form-data body, including as many of the following fields as possible in the request body:

For example, to import the contents of a test-results.json file in a GitHub Actions pipeline run:

  1. Securely set the Test Analytics token environment variable (BUILDKITE_ANALYTICS_TOKEN).

  2. Run the following curl command:

    curl \
    -X POST \
    -H "Authorization: Token token=\"$BUILDKITE_ANALYTICS_TOKEN\"" \
    -F "data=@test-results.json" \
    -F "format=json" \
    -F "run_env[CI]=github_actions" \
    -F "run_env[key]=$GITHUB_ACTION-$GITHUB_RUN_NUMBER-$GITHUB_RUN_ATTEMPT" \
    -F "run_env[number]=$GITHUB_RUN_NUMBER" \
    -F "run_env[branch]=$GITHUB_REF" \
    -F "run_env[commit_sha]=$GITHUB_SHA" \
    -F "run_env[url]=https://github.com/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID" \
    https://analytics-api.buildkite.com/v1/uploads
    

To learn more about passing through environment variables to run_env-prefixed fields, see CI Environments.

JSON test results data reference

JSON test results data is made up of an array of one or more test result objects. A test result object contains an overall result and metadata. It also contains a history object, which is a summary of the duration of the test run. Within the history object, detailed span objects record the highest resolution details of the test run.

Schematically, the JSON test results data is like this:

Or in a simplified code view:

[
  {
    /* Test result object */
    "history": {
      /* history object */
      "children": [
        /* span objects */
      ]
    }
  },
  { /* Test result object */ },
]

Test result objects

A test result represents a single test run.

Key Type Description Examples
Key id (required) Type UUIDv4 string Description

A unique identifier for this test result. If a test execution with this UUID already exists in the Test Analytics database, this result is ignored.

Examples

1b70486f-ca5f-4e6d-beb8-6347b2e49278

Key scope Type string Description

A group or topic for the test

Examples

Student.isEnrolled()

Key name Type string Description

A name or description for the test

Examples

Manager.isEnrolled() returns_boolean

Key identifier (required) Type string Description

A unique identifier for the test. If your test runner supports it, use the identifier needed to rerun this test

Examples

Manager.isEnrolled#returns_boolean

Key location Type string Description

The file and line number where the test originates, separated by a colon (:)

Examples

./tests/Manager/isEnrolled.js:32

Key file_name Type string Description

The file where the test originates

Examples

./tests/Manager/isEnrolled.js

Key result Type string (passed, failed, skipped or unknown) Description

The outcome of the test

Examples

failed

Key failure_reason Type string Description

If applicable, a short summary of why the test failed

Examples

Expected Boolean, got Object

Key failure_expanded Type array of hashes Description

A more detailed explanation of why the test failed

Examples
Key history (required) Type history object Description

Read History objects

Examples

Example:

{
  "id": "95f7e024-9e0a-450f-bc64-9edb62d43fa9",
  "scope": "Analytics::Upload associations",
  "name": "fails",
  "identifier": "./spec/models/analytics/upload_spec.rb[1:1:3]",
  "location": "./spec/models/analytics/upload_spec.rb:24",
  "file_name": "./spec/models/analytics/upload_spec.rb",
  "result": "failed",
  "failure_reason": "Failure/Error: expect(true).to eq false",
  "failure_expanded": [
  {
      "expanded": [
      "  expected: false",
      "       got: true",
      "",
      "  (compared using ==)",
      "",
      "  Diff:",
      "  @@ -1 +1 @@",
      "  -false","  +true"
      ],
      "backtrace": [
      "./spec/models/analytics/upload_spec.rb:25:in `block (3 levels) in \u003ctop (required)\u003e'","./spec/support/log.rb:17:in `run'",
      "./spec/support/log.rb:66:in `block (2 levels) in \u003ctop (required)\u003e'",
      "./spec/support/database.rb:19:in `block (2 levels) in \u003ctop (required)\u003e'",
      "/Users/abc/Documents/rspec-buildkite-analytics/lib/rspec/buildkite/analytics/uploader.rb:153:in `block (2 levels) in configure'",
      "-e:1:in `\u003cmain\u003e'"
      ]
  }
  ],
  "history": {
    /* history object */
  }
}

History objects

A history object represents the overall duration of the test run and contains detailed span data, more finely recording the test run.

Key Type Description
Key start_at (required) Type number Description

A monotonically increasing number

Key end_at Type number Description

A monotonically increasing number

Key duration (required) Type number Description

How long the test took to run

Key children Type array of span objects Description

Read Span objects

Example:

{
  "start_at": 347611.724809,
  "end_at": 347612.451041,
  "duration": 0.726232000044547,
  "children": [
    /* span objects */
  ]
}

Span objects

Span objects represent the finest duration resolution of a test run. It represents, for example, the duration of an individual database query within a test.

Key Type Description
Key section (required) Type string (http, sql, sleep or annotation) Description

A section category for this span

Key start_at Type number Description

A monotonically increasing number

Key end_at Type number Description

A monotonically increasing number

Key duration (required) Type number Description

How long the span took to run

Example:

{
  "section": "sql",
  "start_at": 347611.734956,
  "end_at": 347611.735647,
  "duration": 0.0006910000229254365
}