Amazon S3 storage

Before you start

Before you can start linking your private Amazon S3 storage to Buildkite Package Registries, you will need to have created your own empty Amazon S3 bucket.

Learn more about:

• Amazon S3 from the main Amazon S3 page, as well as the Amazon S3 documentation.

• How to create an Amazon S3 bucket from Amazon's Getting started with Amazon S3 guide.

Link your private Amazon S3 bucket to Buildkite Package Registries

To link your private Amazon S3 bucket to Package Registries:

1. As a Buildkite organization administrator, select Settings in the global navigation to access the Organization Settings page.

2. In the Packages section, select Private Storage Link to open its page.

3. Select Add private storage link start the private storage configuration process.

4. On the Provide your storage's details (page) > Step 1: Select your storage provider, select AWS.

5. In Step 2: Create or locate your AWS S3 bucket, select Open AWS to open the list of Amazon S3 buckets in your AWS account, to either retrieve your existing empty S3 bucket, or create a new one if you haven't already done so.

   Note: If you are not already signed in to your AWS account, you may need to navigate to the area listing your S3 buckets.

6. Back on the Buildkite interface, in Step 3: Enter your AWS S3 bucket details, specify the Region (for example, us-east-1) and Bucket name for your Amazon S3 bucket, then select Continue.

7. On the next Authorize Buildkite in AWS page, select Launch Stack to open the Quick create stack page in the AWS CloudFormation interface.

8. Ensure the the following fields are populated with the correct information:

   • Template URL—should be based on:
     https://packages-public-assets.s3.amazonaws.com/cf-templates/byo-storage-bucket-policy-yyyymmdd.yml
   • Stack name—buildkitePackagesProvisioning by default, but can be changed if another CloudFormation stack of the same name exists in your AWS account.
   • BucketName—the name of your Amazon S3 bucket (specified on the previous Provide your bucket's details page in Buildkite).
   • KeyPrefix—{org.uuid}/, where {org.uuid} is the UUID of your Buildkite organization.
   • IAM role - optional—specify any IAM role name or ARN to restrict the actions that can be performed on your CloudFormation stack in your S3 bucket.

9. Select Create stack to begin creating the CloudFormation stack for your Amazon S3 bucket.

10. Once the stack is created, return to the Buildkite interface and select Run diagnostic to verify that Buildkite Package Registries can do the following with packages in your Amazon S3 private storage:

    • publish (PUT)
    • download (GET)
    • tag (PUT)
    • delete (DELETE)

11. Once the Diagnostic Result page indicates a Pass for each of these diagnostic tests, select Create Private Storage Link to complete this linking process.

You are returned to the Private Storage Link page, where you can:

• Set the default Buildkite Package Registries storage for your Buildkite organization.

• Set the storage independently for each of your Buildkite registries.

Deleting packages

When deleting a package, Buildkite Package Registries does not delete the associated objects from your storage. Instead, Package Registries marks them for deletion using Amazon S3 object tags. Learn more about these object tags in Amazon's Categorizing your storage using tags documentation.

An object tagged for deletion by Package Registries has the following key value pair:

Set the expiration on objects from your Amazon S3 bucket by adding an S3 Lifecycle configuration that filters on these object tags. For example, to remove objects 30 days after they're tagged, you can implement the following rule:

{
  "Rules": [
    {
      "ID": "BuildkiteDeleteExpired",
      "Status": "Enabled",
      "Filter": {
        "Tag": {
          "Key": "buildkite:deleted",
          "Value": "*"
        }
      },
      "Expiration": { "Days": 30 }
    }
  ]
}

Learn more about filter syntax in these lifecycle rules from Amazon's Lifecycle configuration elements > Filter element documentation.