Buildkite Code Review Pipeline Example

This example demonstrates an AI-powered code review pipeline built with Buildkite dynamic pipelines and Claude Code. Adding a label to a GitHub PR triggers an AI agent that reviews the pull request and posts feedback.

How it works

1. You add a buildkite-review label to a GitHub PR
2. GitHub sends a webhook to Buildkite, which starts a build
3. The first step evaluates the webhook payload: if it’s not a label event, the build exits early
4. A TypeScript handler reads the payload, validates the PR, and posts an acknowledgement comment
5. The handler uses the Buildkite SDK to dynamically generate a code review step and uploads it with buildkite-agent pipeline upload
6. That step launches Claude Code in a Docker container to review the PR and post feedback

The handler pattern

The core of the handler is short: read the webhook payload from build metadata, evaluate whether to act, and generate a step with the Buildkite SDK:

// 1. Read the webhook payload that Buildkite stored as build metadata
const payload = JSON.parse(
  execSync("buildkite-agent meta-data get buildkite:webhook").toString(),
);

// 2. Evaluate the condition — right event, right label?
if (payload.action !== "labeled" || payload.label.name !== process.env.TRIGGER_ON_LABEL) {
  process.exit(0);
}

// 3. Generate a step with the Buildkite SDK and pipe it into `pipeline upload`
const pipeline = new Pipeline();
pipeline.addStep({ label: ":mag: Review the PR", command: "scripts/claude.sh" });
execSync("buildkite-agent pipeline upload", { input: pipeline.toYAML() });

The real handler also validates the PR exists and posts an acknowledgement comment between steps 2 and 3. See scripts/handler.ts.

The key Buildkite features at play:

• buildkite-agent pipeline upload: adding steps to a running build based on runtime conditions
• buildkite-agent meta-data: reading webhook payloads stored as build metadata
• @buildkite/buildkite-sdk: programmatically generating pipeline YAML in TypeScript
• Buildkite webhooks: triggering builds from external events
• Buildkite Hosted Models: proxying LLM requests through Buildkite’s model provider endpoint

What’s interesting about this?

Like the self-healing pipeline example, this pipeline has no fixed steps. The webhook payload determines whether anything runs at all, and what runs is generated programmatically at build time. This is a different flavour of the same dynamic pipelines pattern: instead of remediating a failure, it’s performing an intelligent review.

Setup

To run this yourself, you’ll need:

• A Buildkite account
• A GitHub repository with a Buildkite pipeline configured to receive webhooks
• An Anthropic API key (or use Buildkite Hosted Models)
• Docker installed on your Buildkite agent

1. Fork this repo
2. Create a Buildkite pipeline pointing to your fork with webhook support enabled
3. Configure a GitHub webhook to send pull_request events with the labeled action to Buildkite
4. Set up the required secrets: GITHUB_TOKEN and BUILDKITE_API_TOKEN
5. Add the buildkite-review label to any PR

Known limitations

• The handler assumes the Buildkite org slug and pipeline slug match the GitHub org and repo name. This won’t always be the case, so you may need to configure these separately.

Credits

Originally built by Grant Colegate and Christian Nunciato as a demo for AWS re:Invent.

License

See LICENSE (MIT)