Monorepo State Buildkite Plugin

A support plugin for the official monorepo-diff Buildkite plugin — keeps monorepo sub-pipelines from re-running when nothing they own has changed.

It tracks each service’s last successful build commit in an S3-compatible bucket, then computes the changed files since that commit so monorepo-diff triggers only the sub-pipelines that actually need to run. If a service’s last build failed, it is diffed from its previous success — so it re-runs until it passes, even without new changes. (Both the official monorepo-diff#… and the chronotc/monorepo-diff#… plugin refs are recognised.)

The heavy lifting is a small Go binary (bms, “Buildkite Monorepo State”) delivered by a downloader-shim hook — no runtime on the agent beyond git.

📖 Full step-by-step guide: docs/USAGE.md

Why S3

S3 is the universal object-storage contract. The same backend reaches AWS S3, MinIO, Cloudflare R2, and Google Cloud Storage (S3 interop) by changing only an endpoint URL. Credentials come from the AWS SDK default chain, so the aws-assume-role-with-web-identity plugin, OIDC, instance profiles, and env credentials all work with no access keys in your pipeline YAML.

How it works

Phase	Where	What happens
diff	upload step pre-command	The plugin runs first: bms diff reads each watched service’s last-success hash from S3 and computes the changed files (first run / missing history → all files). The result is published for the whole build (see below) and can also be fed to monorepo-diff via diff: "bms diff".
set	sub-pipeline post-command	On success only, bms set records the current commit as that service’s last-success hash.

The environment hook downloads bms and puts it on PATH before any command runs, so monorepo-diff can call bms diff regardless of plugin order.

Published diff output (monorepo-level reuse)

In diff mode the pre-command hook computes the changed files once and exposes them so any step can reuse the result without recomputing:

• Environment variables for the current step’s command and later hooks:
  • MONOREPO_STATE_CHANGED_FILES — newline-separated list.
  • MONOREPO_STATE_CHANGED_COUNT — number of changed files.
• Build meta-data, readable from any step in the pipeline: buildkite-agent meta-data get monorepo-state:changed-files (key configurable via metadata-key).

So monorepo-diff’s diff: can reuse the published value instead of recomputing:

      - chronotc/monorepo-diff#v1.2.0:
          diff: "buildkite-agent meta-data get monorepo-state:changed-files"

Usage

Upload step — compute changes and trigger sub-pipelines:

steps:
  - plugins:
      - hasithaishere/monorepo-state#v0.1.0:
          bucket: my-monorepo-state
          region: ap-southeast-1
          config-path: ".buildkite/pipeline.yml"   # or watch-paths: [...]
      - chronotc/monorepo-diff#v1.2.0:
          diff: "bms diff"
          watch:
            - path: service/app-server
              config: { trigger: app-server-pipeline }

Sub-pipeline — record success:

steps:
  - command: make deploy
    plugins:
      - hasithaishere/monorepo-state#v0.1.0:
          bucket: my-monorepo-state
          region: ap-southeast-1
          service-path: service/app-server

After all sub-pipelines pass, record the pipeline-wide baseline once with main-pipeline: true (see examples/save-step.yml).

Configuration

Option	Type	Default	Description
bucket	string	required	S3-compatible bucket holding state objects.
region	string	us-east-1	Bucket region. R2 wants auto.
endpoint-url	string	—	Custom endpoint for MinIO / R2 / GCS interop.
prefix	string	buildkite/store	Key prefix for all state objects.
force-path-style	boolean	false	Use path-style addressing (required for MinIO).
config-path	string	—	Pipeline YAML to extract monorepo-diff watch paths from.
watch-paths	string | list	—	Service paths to diff (alternative to config-path).
forced-deploy-paths	string | list	—	Bypass state/git and emit these paths verbatim.
metadata-key	string	monorepo-state:changed-files	Build meta-data key the pre-command hook publishes changed files under.
service-path	string	—	Service to save state for (save mode).
main-pipeline	boolean	false	Save the pipeline-wide baseline (save mode).
version	string	the plugin’s released version	Override the bms binary release to download. Defaults to the version baked into the checked-out plugin (kept in sync by release-please), so pinning #vX.Y.Z fetches the matching binary. If the version can’t be resolved or downloaded, the build fails.
download	boolean	true	false → use a bms already on PATH.

Provide exactly one of watch-paths / config-path for diff, and exactly one of service-path / main-pipeline for save.

State layout

<prefix>/<org>/<pipeline>/last-success-hash/<service-path>
<prefix>/<org>/<pipeline>/last-success-hash/main-pipeline

Backend examples

• AWS S3 — just bucket + region; credentials from the SDK chain / OIDC.
• MinIO — endpoint-url: https://minio.example.com, force-path-style: true, region: us-east-1; creds via AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY.
• Cloudflare R2 — endpoint-url: https://<account>.r2.cloudflarestorage.com, region: auto; R2 API token as AWS creds.
• Google Cloud Storage (interop) — endpoint-url: https://storage.googleapis.com with an HMAC key as AWS creds.

Development

go test ./...                       # unit tests
shellcheck hooks/* lib/*.bash       # lint hooks
goreleaser build --snapshot --clean # cross-compile locally

Releases are cut by GoReleaser on a tag, publishing bms_<os>_<arch> binaries and checksums.txt that the shim verifies.

Contributing

Contributions are welcome! Please read CONTRIBUTING.md for the development setup, coding guidelines, and PR checklist. In short:

• Keep Go logic in internal/* and the binary thin; hooks stay in Bash.

• bms diff must write only the changed-file list to stdout (logs to stderr).

• Read config through internal/config (env-driven, hermetically testable) — never parse plugin YAML by hand.

• Add tests, then make sure these pass before opening a PR:

  go vet ./...
  go test ./...
  shellcheck -x hooks/* lib/*.bash
  bats tests/

GitHub Actions runs the same checks on every pull request. Use the issue and pull request templates when reporting bugs, requesting features, or submitting changes. Found a security issue? Please follow SECURITY.md rather than opening a public issue.

License

MIT — see LICENSE.