Queues overview

Each pipeline has the ability to separate its jobs (define by the pipeline's steps) using queues. This allows you to isolate a set of jobs or agents, or both, making sure that only specific agents will run jobs that are intended for them.

Buildkite Pipelines allows you to configure two types of queues:

  • Self-hosted, where you manage Buildkite agents in your own infrastructure.
  • Buildkite hosted queues, where Buildkite manages the agents for you as a fully-managed platform.

Learn more about how to create and manage queues in Managing queues.

Common use cases for queues include deployment agents, and pools of agents for specific pipelines or teams.

Targeting a queue from a pipeline

Target specific queues (either self-hosted or Buildkite hosted ones) using the agents attribute on your pipeline steps, or at the root level for the entire pipeline.

For example, the following pipeline would run on the priority queue as determined by the root level agents attribute (and ignores the agents running the default queue). The tests.sh build step matches only agents running on the linux-medium-x86 queue.

agents:
  queue: "priority"

steps:
  - command: echo "hello"

  - command: tests.sh
    agents:
      queue: "linux-medium-x86"

Alternative methods

Branch patterns are another way to control what work is done. You can use branch patterns to determine which pipelines and steps run based on the branch name.

Assigning a self-hosted agent to a queue

A self-hosted agent can be assigned to a self-hosted queue using an agent tag as a queue tag when the agent is started.

This configuration can be set at the command line when starting the agent, the agent's configuration file, or through an environment variable.

Agents can only be assigned to a single self-hosted queue within a cluster.

In the following example, the --tags flag of the buildkite-agent start command is used to configure this agent to listen on the linux-medium-x86 queue, which is part of the Testing cluster:

buildkite-agent start --token "TESTING-AGENT-TOKEN-VALUE" --tags "queue=linux-medium-x86"

Ensure you have already configured your cluster's agent tokens and queues

Your clusters and queues should already be configured before starting your agents to target these queues.

By default, a pipeline's jobs run on the first available agent within a queue, ordered by how recently an agent within that queue successfully completed a job. You can, however, alter this behavior by changing the priority for some or all of your agents. Learn more about this in Buildkite agent prioritization.

The default self-hosted queue

If you don't assign a self-hosted agent to a self-hosted queue by setting the agent's queue tag (for example, queue=linux-medium-x86) when it is started, the agent will automatically be assigned to your configured default self-hosted queue (for example, queue=default) and accept jobs from that queue.

Clusters without a default self-hosted queue configured

If you start a self-hosted agent without explicitly specifying an existing self-hosted queue in your cluster and a default self-hosted queue is not configured in this cluster, or your default queue is set to a Buildkite hosted queue, then your agent will fail to connect to the Buildkite platform.

You must either explicitly specify an existing self-hosted queue within in your cluster when starting the agent, or have a default self-hosted queue already configured in this cluster for the agent to connect successfully.

Setting up queues for unclustered agents

This section documents a deprecated Buildkite feature

Learn more about unclustered agents and their tokens in Working with unclustered agent tokens.

For unclustered agents, queues are configured when starting a Buildkite agent. An unclustered agent can listen on a single queue or on multiple queues. For multiple queues, add as many extra queue tags as are required.

In the following example, the --tags flag of the buildkite-agent start command is used to configure this unclustered agent to listen on both the development and testing queues:

buildkite-agent start --token "UNCLUSTERED-AGENT-TOKEN-VALUE" --tags "queue=development,queue=testing"