Buildkite Agent Stack for Kubernetes Improvements: Easier to Adopt, Scale, and Maintain

Over the past few months, we’ve made a series of improvements to the Buildkite Agent Stack for Kubernetes, focused on simpler setup, improved efficiency, and reliable scaling. Bringing us to version 0.29.2, these updates are the result of our sustained engineering investment in our Kubernetes integration, designed to help platform teams reduce overheads and ship faster with greater confidence.

What’s improved

• Simpler setup: You can now deploy a stack with just a single cluster-scoped agent token. Previously-required configuration is no longer required: the GraphQL token, organization slug, and cluster UUID are no longer required or used. Default queue behavior is also streamlined: the stack now defaults to the cluster's default queue, rather than a queue called kubernetes.
• Better scaling under load: The internals of the controller have been reworked to more efficiently discover and handle large volumes of jobs, even in high-concurrency environments. Job environment variables are now passed between job containers directly, reducing per-job Kubernetes object sizes. Fixes to tag matching and queue routing ensure that job lifecycles are handled correctly by the controller.
• More secure by default: GraphQL tokens, which have many capabilities that are unnecessary for running a stack, are no longer used. Everything now runs on the REST API using scoped agent tokens, and we’ve started internal testing of scoped secret support for more secure isolation across pipelines and environments.
• Greater configurability: Helm chart values now expose settings for things like job name prefixes, controller env vars, resource limits, and pod annotations, giving teams more flexibility to tune the stack for their environment with out-of-the-box controls.
• Better errors and observability: Error messages for Kubernetes job creation failures now include the job spec in YAML form, making it easier to diagnose misconfigurations. Prometheus metrics have been updated to support monitoring stack performance. Stack-level job failures are now reported to Buildkite using a new stack_error signal reason.

Recent releases

These improvements have been rolling out since March, with highlights in the following versions:

• v0.29.2: Fixes a permissions issue within job pods, upgrades Buildkite Agent to v3.100.1.
• v0.29.0: Introduces cluster-only agent support, removes the queue=kubernetes default tag, adds better error output and tag matching logic.
• v0.26.0 through v0.28.2: Improvements to cleanup, resource management, default queue handling, plugin mounts, GraphQL removal, and REST-only job fetch.

If you're running the Buildkite Agent Stack for Kubernetes, now’s a great time to upgrade. These changes simplify how you manage agents, reduce the number of moving parts, and make builds more predictable at scale. To get started, check out the official documentation.