S3 Cache Buildkite Plugin

Save and restore cache to and from AWS S3.

Example

Add the following to your pipeline.yml:

steps:
  - command: npm install && npm test
    plugins:
      - peakon/s3-cache#v2.3.1:
          id: CACHE_IDENTIFIER # optional, default: none
          aws_profile: aws-profile-name # optional, default: none
          restore_dry_run: false # set it to "true" to only check if cacheKey is present on S3 (no download / restoring)
          save:
            - key: 'v1-node-modules-{{ checksum("package-lock.json") }}' # required
              paths: ["node_modules"] # required, array of strings
              when: on_success # optional, one of {always, on_success, on_failure}, default: on_success
              overwrite: false # optional, set true to overwrite cache on S3 even if object already exists
          restore:
            - keys:
                - 'v1-node-modules-{{ checksum "package-lock.json" }}'
                - "v1-node-modules-" # will load latest cache starting with v1-node-modules- (not yet implemented)

Configuration

Prerequisites

Make sure to set BUILDKITE_PLUGIN_S3_CACHE_BUCKET_NAME=your-cache-bucket-name before using this plugin.

Plugin

You can specify either save or restore or both of them for a single pipeline step.

Checking if cache was successfully restored

In some cases you may need to build a conditional logic in the build command based on the results of cache restore operation (for example, to avoid re-generating the cache which already exists and was restored successfully).

To support this use-case, this plugin exports environment variables that can be used during a command step. The feature is opt-in and requires id to be specified in plugin configuration.

For example, this step generates a cache of node_modules (which is then used by all jobs that need it):

steps:
  - command: '[ ! "${BUILDKITE_PLUGIN_S3_CACHE_npm_0_KEY_0_HIT}" =~ ^(true)$ ] && npm install'
    plugins:
      - peakon/s3-cache#2.2.1:
          id: npm
          restore_dry_run: true # This saves runtime, but doesn't check for integrity
          restore:
            - keys: ['v1-node-modules-{{ checksum "package-lock.json" }}']
          save:
            - key: 'v1-node-modules-{{ checksum "package-lock.json" }}'
              paths: ["node_modules"]

Supported functions

• checksum 'filename(s)' - sha256 hash of filename(s) support glob expansion

• epoch - time in seconds since Unix epoch (in UTC)

• .Environment.SOME_VAR - a value of environment variable SOME_VAR

AWS profiles

You can specify a custom AWS profile to be used by AWS CLI

• in pipeline YAML (aws_profile: profile_name)
• via BUILDKITE_PLUGIN_S3_CACHE_AWS_PROFILE environment variable (e.g. inside agent environment hook).

Developing

To run the tests:

docker compose run --rm tests

Contributing

1. Fork the repo
2. Make the changes
3. Run the tests
4. Commit and push your changes
5. Send a pull request