Controlling Concurrency

Some tasks need to be run with very strict concurrency rules to ensure they don’t collide with each other. Common examples for needing concurrency control are deployments, app releases and infrastructure tasks.

To help you control concurrency, Buildkite provides two primitives: concurrency limits and concurrency groups. While these two primitives are closely linked and interdependant, they operate at different levels.

Concurrency Limits

Concurrency limits define the number of jobs that are allowed to be running at any one time. These limits are set per-step and only apply to jobs that are based on that step.

Setting a concurrency limit of 1 on a step in your pipeline will ensure that no two jobs created from that step will run at the same time. These jobs will not run at the same time, even if there are agents available to do so.

You can add concurrency limits to steps either through Buildkite, or your pipeline.yml file. Defining concurrency in a pipeline.yml file also requires the use of the concurrency_group attribute. See Concurrency Groups for more information.

Screenshot of a step's Global Concurrency Limit field

I'm seeing an error about a missing concurrency_group_id when I run my pipeline upload

This error is caused by a missing concurrency group label. If you set a concurrency limit in a `pipeline.yml` file, you must also set a group label.

Concurrency Groups

Concurrency groups are labels that can be used to group together Buildkite jobs when applying concurrency limits. When you add a group label to a step in a pipeline.yml file, it will be available organsiation wide. These group labels are similar to agent meta-data, and are checked at job runtime to determine which jobs are allowed to run concurrently. Although they're created on individual steps, they represent concurrent access to shared resources and can be used by other pipelines.

steps:
- command: "tests.sh"
  concurrency: 1
  concurrency_group: "tests"

The names of these groups should be unique, unless they're accessing a shared resource like a deployment target. You may want to ensure that order or volume sensitive operations like deploys or API access don't happen at the same time, and run in order that they were created. For example, having multiple groups called deploy on steps that are accessing different resources will affect which jobs are able to run at the same time.

Concurrency groups guarantee that jobs will be run in the order that they were created in. Jobs inherit the creation time of their parent. Parents of jobs can be either a build or a pipeline upload job. As pipeline uploads add more jobs to the build after it has started, the jobs that they add will inherit the creation time of the pipeline upload rather than the build.

Common examples for concurrency group values include:

  • our-payment-gateway/deployment
  • terraform/update-state
  • my-mobile-app/app-store-release

When you create a step using the Buildkite web interface, you can’t set a custom concurrency group label. A label will be automatically generated for you based on that step. If you want to use group labels, you can define your steps in a pipeline.yml file.

Example: Deploy Step

The following is an example command step that ensures deployments run one at a time:

- command: 'deploy.sh'
  label: ':rocket: Deploy production'
  branches: 'master'
  agents:
    deploy: true
  concurrency: 1
  concurrency_group: 'our-payment-gateway/deploy'

If multiple builds are created with this step, each deployment job will be queued up and run one at a time in the order they were created.