Importing JSON

How to import JSON in Buildkite

It's possible to import JSON (or JUnit files) to Buildkite Test Engine with or without the help of a plugin.

Using a plugin

To import JSON-formatted test results to Test Engine using Test Collector plugin from a build step:

steps:
  - label: "🔨 Test"
    command: "make test"
    plugins:
      - test-collector#v1.0.0:
          files: "test-data-*.json"
          format: "json"

See more configuration information in the Test Collector plugin README.

Using the plugin is the recommended way as it allows for a better debugging process in case of an issue.

Without a plugin

If for some reason you cannot or do not want to use the Test Collector plugin, or if you are looking to implement your own integration, another approach is possible.

To import JSON-formatted test results in Buildkite, make a POST request to https://analytics-api.buildkite.com/v1/uploads with a multipart/form-data.

For example, to import the contents of a JSON-formatted test results (test-results.json):

1. Securely set the Test Engine token environment variable (BUILDKITE_ANALYTICS_TOKEN).

2. Run the following curl command:

   curl \
     -X POST \
     -H "Authorization: Token token=\"$BUILDKITE_ANALYTICS_TOKEN\"" \
     -F "data=@test-results.json" \
     -F "format=json" \
     -F "run_env[CI]=buildkite" \
     -F "run_env[key]=$BUILDKITE_BUILD_ID" \
     -F "run_env[url]=$BUILDKITE_BUILD_URL" \
     -F "run_env[branch]=$BUILDKITE_BRANCH" \
     -F "run_env[commit_sha]=$BUILDKITE_COMMIT" \
     -F "run_env[number]=$BUILDKITE_BUILD_NUMBER" \
     -F "run_env[job_id]=$BUILDKITE_JOB_ID" \
     -F "run_env[message]=$BUILDKITE_MESSAGE" \
     https://analytics-api.buildkite.com/v1/uploads
   

To learn more about passing through environment variables to run_env-prefixed fields, see the Buildkite or Other CI providers (including manually) on the CI environments page.

A single file can have a maximum of 5000 test results, and if that limit is exceeded then the upload request will fail. To upload more than 5000 test results for a single run upload multiple smaller files with the same run_env[key].

How to import JSON in CircleCI

To import JSON-formatted test results, make a POST request to https://analytics-api.buildkite.com/v1/uploads with a multipart/form-data body, including as many of the following fields as possible in the request body:

For example, to import the contents of a test-results.json file in a CircleCI pipeline:

1. Securely set the Test Engine token environment variable (BUILDKITE_ANALYTICS_TOKEN).

2. Run the following curl command:

   curl \
   -X POST \
   -H "Authorization: Token token=\"$BUILDKITE_ANALYTICS_TOKEN\"" \
   -F "data=@test-results.json" \
   -F "format=json" \
   -F "run_env[CI]=circleci" \
   -F "run_env[key]=$CIRCLE_WORKFLOW_ID-$CIRCLE_BUILD_NUM" \
   -F "run_env[number]=$CIRCLE_BUILD_NUM" \
   -F "run_env[branch]=$CIRCLE_BRANCH" \
   -F "run_env[commit_sha]=$CIRCLE_SHA1" \
   -F "run_env[url]=$CIRCLE_BUILD_URL" \
   https://analytics-api.buildkite.com/v1/uploads
   

To learn more about passing through environment variables to run_env-prefixed fields, see CI environments > CircleCI page section.

A single file can have a maximum of 5000 test results, and if that limit is exceeded then the upload request will fail. To upload more than 5000 test results for a single run upload multiple smaller files with the same run_env[key].

How to import JSON in GitHub Actions

To import JSON-formatted test results, make a POST request to https://analytics-api.buildkite.com/v1/uploads with a multipart/form-data body, including as many of the following fields as possible in the request body:

For example, to import the contents of a test-results.json file in a GitHub Actions pipeline run:

1. Securely set the Test Engine token environment variable (BUILDKITE_ANALYTICS_TOKEN).

2. Run the following curl command:

   curl \
   -X POST \
   -H "Authorization: Token token=\"$BUILDKITE_ANALYTICS_TOKEN\"" \
   -F "data=@test-results.json" \
   -F "format=json" \
   -F "run_env[CI]=github_actions" \
   -F "run_env[key]=$GITHUB_ACTION-$GITHUB_RUN_NUMBER-$GITHUB_RUN_ATTEMPT" \
   -F "run_env[number]=$GITHUB_RUN_NUMBER" \
   -F "run_env[branch]=$GITHUB_REF" \
   -F "run_env[commit_sha]=$GITHUB_SHA" \
   -F "run_env[url]=https://github.com/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID" \
   https://analytics-api.buildkite.com/v1/uploads
   

To learn more about passing through environment variables to run_env-prefixed fields, see CI environments > GitHub Actions page section.

A single file can have a maximum of 5000 test results, and if that limit is exceeded then the upload request will fail. To upload more than 5000 test results for a single run upload multiple smaller files with the same run_env[key].

JSON test results data reference

JSON test results data is made up of an array of one or more "test result" objects. A test result object contains an overall result and metadata. It also contains a history object, which is a summary of the duration of the test run. Within the history object, detailed span objects record the highest resolution details of the test run.

Schematically, the JSON test results data is like this:

• Test results
  • History
    • Spans
      • Detail

Or in a simplified code view:

[
  {
    /* Test result object */
    "history": {
      /* history object */
      "children": [
        /* span objects */
      ]
    }
  },
  { /* Test result object */ },
]

Test result objects

A test result represents a single test run.

Example:

{
  "id": "95f7e024-9e0a-450f-bc64-9edb62d43fa9",
  "scope": "Analytics::Upload associations",
  "name": "fails",
  "location": "./spec/models/analytics/upload_spec.rb:24",
  "file_name": "./spec/models/analytics/upload_spec.rb",
  "result": "failed",
  "failure_reason": "Failure/Error: expect(true).to eq false",
  "failure_expanded": [
    /* failure_expanded object */
  ],
  "history": {
    /* history object */
  }
}

Failure expanded objects

A failure expanded array contains extra details about the failed test.

Example:

{
  "expanded": [
    "  expected: false",
    "       got: true",
    "",
    "  (compared using ==)",
    "",
    "  Diff:",
    "  @@ -1 +1 @@",
    "  -false","  +true"
  ],
  "backtrace": [
    "./spec/models/analytics/upload_spec.rb:25:in `block (3 levels) in <top (required)>'","./spec/support/log.rb:17:in `run'",
    "./spec/support/log.rb:66:in `block (2 levels) in <top (required)>'",
    "./spec/support/database.rb:19:in `block (2 levels) in <top (required)>'",
    "/Users/abc/Documents/rspec-buildkite-analytics/lib/rspec/buildkite/analytics/uploader.rb:153:in `block (2 levels) in configure'",
    "-e:1:in `<main>'"
  ]
}

History objects

A history object represents the overall duration of the test run and contains detailed span data, more finely recording the test run.

Example:

{
  "start_at": 347611.724809,
  "end_at": 347612.451041,
  "duration": 0.726232000044547,
  "children": [
    /* span objects */
  ]
}

Span objects

Span objects represent the finest duration resolution of a test run. It represents, for example, the duration of an individual database query within a test.

Example:

{
  "section": "sql",
  "start_at": 347611.734956,
  "end_at": 347611.735647,
  "duration": 0.0006910000229254365
  "detail": {
    ...
  }
}

Detail objects

Detail objects contains additional information about the span.

HTTP Example:

{
  "detail": {
    method: "POST",
    url: "https://example.com",
    lib: "curl"
  }
}

SQL Example:

{
  "detail": {
    query: "SELECT * FROM ..."
  }
}

Test result format

The following JSON code block shows an example of how your JSON test results should be formatted, so that these results can be successfully uploaded to Test Engine.

[
  {
    "id": "95f7e024-9e0a-450f-bc64-9edb62d43fa10",
    "scope": "Analytics::Upload associations",
    "name": "fails",
    "location": "./spec/models/analytics/upload_spec.rb:24",
    "file_name": "./spec/models/analytics/upload_spec.rb",
    "result": "failed",
    "failure_reason": "Failure/Error: expect(true).to eq false",
    "failure_expanded": [],
    "history": {
      "start_at": 347611.724809,
      "end_at": 347612.451041,
      "duration": 0.726232000044547,
      "children": [
        {
          "section": "http",
          "start_at": 347611.734956,
          "end_at": 347611.735647,
          "duration": 0.0006910000229254365,
          "detail": {
            "method": "POST",
            "url": "https://example.com",
            "lib": "curl"
          }
        }
      ]
    }
  },
  {
    "id": "56f6e013-8e9a-340f-bc53-8edb51d32fa09",
    "scope": "Analytics::Upload associations",
    "name": "passes",
    "location": "./spec/models/analytics/upload_spec.rb:56",
    "file_name": "./spec/models/analytics/upload_spec.rb",
    "result": "passed",
    "history": {
      "start_at": 347611.724809,
      "end_at": 347612.451041,
      "duration": 0.726232000044547,
      "children": [
        {
          "section": "http",
          "start_at": 347611.734956,
          "end_at": 347611.735647,
          "duration": 0.0006910000229254365,
          "detail": {
            "method": "GET",
            "url": "https://example.com",
            "lib": "curl"
          }
        }
      ]
    }
  }
]