Control where your retries run with Retry Agent Affinity

A common frustration when running in CI is that agent instances can become corrupted in such a way where they are no longer able to consistently run jobs. This can lead to retries failing, which can be difficult to diagnose and fix. Within Buildkite, by default, jobs are run on the agent that has most recently finished a job, which can lead to retries running on the same agent that failed the first time causing repeated failures.

With the new Retry Agent Affinity feature, you can control where your retries run. Retry agent affinity is set at the Cluster Queue level, and when set to prefer-different, will prefer to run retries on a different agent than the one that failed the first time. This can help to avoid retries failing due to repeated runs on the same corrupted agent.

To set the Retry Agent Affinity on a cluster queue, you can go into the Cluster Queue settings and select the Retry Agent Affinity option:

Alternatively, this value can be set via both the REST and graphQL APIs.

Availability

Retry agent affinity is available for all self-hosted cluster queues.

If your agents are using an acquire-job workflow such as agent-stack-k8s, the setting will have no effect, as Buildkite doesn't control job dispatch for these agents.

Retry agent affinity can't be set on hosted queues, as agents in hosted queues are ephemeral, and so retries will always run on different agents from the original job.