Controlling Concurrency

Some tasks need to be run with very strict concurrency rules to ensure they don’t collide with each other. Common examples for needing concurrency control are deployments, app releases and infrastructure tasks.

To help you control concurrency, Buildkite provides two primitives: concurrency limits and concurrency groups. While these two primitives are closely linked and interdependent, they operate at different levels.

Concurrency Limits

Concurrency limits define the number of jobs that are allowed to run at any one time. These limits are set per-step and only apply to jobs that are based on that step.

Setting a concurrency limit of 1 on a step in your pipeline will ensure that no two jobs created from that step will run at the same time, even if there are agents available.

You can add concurrency limits to steps either through Buildkite, or your pipeline.yml file. When adding a concurrency limit, you'll also need the concurrency_group attribute so that steps in other pipelines can use it as well.

I'm seeing an error about a missing `concurrency_group_id` when I run my pipeline upload

This error is caused by a missing `concurrency_group` attribute. Add this attribute to the same step where you defined the `concurrency` attribute.

Concurrency Groups

Concurrency groups are labels that group together Buildkite jobs when applying concurrency limits. When you add a group label to a step the label becomes available to all Pipelines in that organization. These group labels are checked at job runtime to determine which jobs are allowed to run in parallel. Although concurrency groups are created on individual steps, they represent concurrent access to shared resources and can be used by other pipelines.

The following is an example command step that ensures deployments run one at a time. If multiple builds are created with this step, each deployment job will be queued up and run one after the other in the order they were created.

pipeline.yml
- command: 'deploy.sh'
  label: ':rocket: Deploy production'
  branches: 'master'
  agents:
    deploy: true
  concurrency: 1
  concurrency_group: 'our-payment-gateway/deploy'

Make sure your concurrency_group names are unique, unless they're accessing a shared resource like a deployment target.

For example, if you have two pipelines that each deploy to a different target but you give them both the concurrency_group label deploy, they will be part of the same concurrency group and will not be able to run at the same time, even though they're accessing separate deployment targets. Unique concurrency group names such as our-payment-gateway/deployment, terraform/update-state, or my-mobile-app/app-store-release, will ensure that each one is part of its own concurrency group.

Concurrency groups guarantee that jobs will be run in the order that they were created in. Jobs inherit the creation time of their parent. Parents of jobs can be either a build or a pipeline upload job. As pipeline uploads add more jobs to the build after it has started, the jobs that they add will inherit the creation time of the pipeline upload rather than the build.

Concurrency and Parallelization

Sometimes you need strict concurrency while also having jobs that would benefit from parallelization. In these situations you can use concurrency gates to control which jobs run in parallel and which jobs run one at a time.

In the following setup, only one build at a time can enter the concurrency gate, but within that gate the e2e tests can run up to three at a time, subject to Agent availability.

pipeline.yml
steps:
  - command: echo "Running unit tests"

  - command: echo "Start of concurency gate"
    concurrency_group: gate
    concurrency: 1

  - wait

  - command: echo "Running deployment"
    key: dep

  - command: echo "Running e2e tests after the deployment"
    parallelism: 3
    depends_on: dep

  - wait

  - command: echo "End of concurency gate"
    concurrency_group: gate
    concurrency: 1

  - command: echo "This and subsequent steps run independently"