Build exports
Build exports is only available on an Enterprise plan, which has a build retention period of 12 months.
If you need to retain build data beyond the retention period in your Buildkite plan, you can export the data to your own Amazon S3 bucket.
If you don't configure an S3 bucket, Buildkite stores the build data for 12 months in case you need it. You cannot access this build data through the API or Buildkite dashboard, but you can request the data by contacting support.
How it works
Builds older than the build retention limit are automatically exported as JSON to the S3 bucket you have configured. If you haven't configured a bucket for build export, Buildkite stores that build data as JSON in our own S3 bucket for a further 12 months in case you need it. The following diagram outlines this process.
Buildkite exports each build as multiple gzipped JSON files, which include the following data:
buildkite/build-exports/org={UUID}/date={YYYY-MM-DD}/pipeline={UUID}/build={UUID}/
βββ annotations.json.gz
βββ artifacts.json.gz
βββ build.json.gz
βββ step-uploads.json.gz
βββ jobs/
βββ job-{UUID}.json.gz
βββ job-{UUID}.log
The files are stored in the following formats:
- Annotations
- Artifacts
-
Builds (but without
jobs
, as they are stored in separate files) - Jobs (as would be embedded in a Build via the REST API)
Configure build exports
To configure build exports for your organization, you'll need to prepare an S3 bucket before enabling exports in the Buildkite dashboard.
Prepare your S3 bucket
- Read and understand Security best practices for Amazon S3.
- Your bucket must be located in Amazon's
us-east-1
region. - Your bucket must have a policy allowing cross-account access as described here and demonstrated in the example belowΒΉ.
- Allow Buildkite's AWS account
032379705303
tos3:GetBucketLocation
. - Allow Buildkite's AWS account
032379705303
tos3:PutObject
keys matchingbuildkite/build-exports/org=YOUR-BUILDKITE-ORGANIZATION-UUID/*
. - Do not allow AWS account
032379705303
tos3:PutObject
keys outside that prefix.
- Allow Buildkite's AWS account
- Your bucket should use modern S3 security features and configurations, for example (but not limited to):
- Block public access to prevent accidental misconfiguration leading to data exposure.
- ACLs disabled with bucket owner enforced to ensure your AWS account owns the objects written by Buildkite.
-
Server-side data encryption (
SSE-S3
is enabled by default but you may want to considerSSE-KMS
). - S3 Versioning to help recover objects from accidental deletion or overwrite.
- You may want to use Amazon S3 Lifecycle to manage storage class and object expiry.
ΒΉ Your S3 bucket policy should look like this, with YOUR-BUCKET-NAME-HERE
and
YOUR-BUILDKITE-ORGANIZATION-UUID
substituted with your details:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "BuildkiteGetBucketLocation",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::032379705303:root"
},
"Action": "s3:GetBucketLocation",
"Resource": "arn:aws:s3:::YOUR-BUCKET-NAME-HERE"
},
{
"Sid": "BuildkitePutObject",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::032379705303:root"
},
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::YOUR-BUCKET-NAME-HERE/buildkite/build-exports/org=YOUR-BUILDKITE-ORGANIZATION-UUID/*"
}
]
}
Your Buildkite Organization ID (UUID) can be found on the settings page described in the next section.
Enable build exports
To enable build exports:
- Navigate to your organization's pipeline settings.
- In the Export Build Data to S3 section, enter your Amazon S3 bucket name to use.
- Select Enable Export.