Group step

A group step can contain various sub-steps, and display them in a single logical group on the Build page.

For example, you can group all of your linting steps or all of your UI test steps to keep the Build page less messy. Sub-groups and nested groups are not supported. A pipeline upload that places a group step inside another group is rejected.

If you need a sense of hierarchy across many groups (for example, when a pipeline generator script in a dynamic pipeline loops over teams and services), use flat sibling groups with a Category: Subcategory naming convention in the group label, such as Backend: Auth Tests and Backend: API Tests. To run a downstream stage after several related groups, give those groups related keys and reference them in an array under depends_on.

The group step also helps manage dependencies between a collection of steps, for example, "step X" depends_on everything in "group Y".

A group step can be defined in your pipeline settings or your pipeline.yml file.

Here is an example of using the group step:

pipeline.yml
steps:
  - group: ":lock_with_ink_pen: Security Audits"
    key: "audits"
    steps:
      - label: ":brakeman: Brakeman"
        command: ".buildkite/steps/brakeman"
      - label: ":bundleaudit: Bundle Audit"
        command: ".buildkite/steps/bundleaudit"
      - label: ":yarn: Yarn Audit"
        command: ".buildkite/steps/yarn"
      - label: ":yarn: Outdated Check"
        command: ".buildkite/steps/outdated"

This is how it's displayed in the UI:

Job groups displayed in the Buildkite UI

Only the first 50 jobs within a build header can ever be displayed in the UI, so you might not see all of your groups at all times. However, the jobs are fine and will still show up on the build page.

Jobs per upload limit

A single pipeline upload call can create at most 500 jobs by default, including all jobs across every group in that upload. Teams generating large dynamic pipelines often hit this limit. If you need to create more jobs, split the work across multiple uploads or request a higher limit. See Pipelines limits for the full set of pipeline-related platform limits.

Group step attributes

Required attributes:

group Name of the group in the UI. In YAML, if you don't want a label, pass a ~. Can also be provided in the label or name attribute if null is provided to the group attribute. If multiple are specified, group takes precedence.
Type: string or null
steps A list of steps in the group; at least 1 step is required. Allowed step types: wait, trigger, command/commands, block, input.
Type: array

Optional attributes:

allow_dependency_failure Whether to continue to run this step if any of the steps named in the depends_on attribute fail.
Default: false
depends_on A list of step or group keys that this step depends on. This step or group will only run after the named steps have completed. See managing step dependencies for more information.
Example: "test-suite"
if A boolean expression that omits the step when false. See Using conditionals for supported expressions. When set on a group step, the expression is merged into the if attribute of each step inside the group, rather than applied to the group itself. As a result, the group container is still shown in the Buildkite interface even when every step inside it is omitted by the conditional.
Example: build.message != "skip me"
key A unique string to identify the step, block, or group.
Keys can not have the same pattern as a UUID (xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx).
When set on a group step, all jobs within the group will include a group_key field in REST API builds endpoint responses with this value, allowing you to identify which jobs belong to this group.
Example: "test-suite"
Aliases: identifier, id
label The label that will be displayed in the pipeline visualization in Buildkite (name of the group in the UI). Supports emoji.
Example: ":hammer: Tests" will be rendered as "🔨 Tests"
Alias: name
notify Allows you to trigger build notifications to different services. You can also choose to conditionally send notifications based on pipeline events.
Example: "github_commit_status:"
skip Whether to skip this step or not. Passing a string provides a reason for skipping this command. Passing an empty string is equivalent to false. When set on a group step, the value is merged into the skip attribute of each step inside the group, rather than applied to the group itself. As a result, the group container is still shown in the Buildkite interface even when every step inside it is skipped.
Note: Skipped steps will be hidden in the pipeline view by default, but can be made visible by toggling the 'Skipped jobs' icon.
Example: true
Example: false
Example: "My reason"

Concurrency in groups

The concurrency, concurrency_group, and concurrency_method attributes are not group step attributes. To limit how many jobs can run in parallel for work inside a group, set these attributes on the command steps within the group, not on the group step itself. See Controlling concurrency for details.

For example, to limit each service deploy in a group to one concurrent job:

pipeline.yml
steps:
  - group: ":rocket: Deploy"
    key: "deploy"
    depends_on: "tests"
    steps:
      - label: "Deploy auth"
        command: "make deploy-auth"
        concurrency: 1
        concurrency_group: "deploy/auth"
      - label: "Deploy payments"
        command: "make deploy-payments"
        concurrency: 1
        concurrency_group: "deploy/payments"

In the example above, deploy/auth and deploy/payments are separate concurrency groups, so the auth and payments deploys can run in parallel, but two auth deploys from different builds cannot.

Agent-applied attributes

These attributes are only applied by the Buildkite agent when uploading a pipeline (buildkite-agent pipeline upload), since they require direct access to your code or repository to process correctly.

Agent-applied attributes are not accepted in pipelines set using the Buildkite interface.

if_changed

if_changed A glob pattern that omits the step from a build if it does not match any files changed in the build.
Example: "{**.go,go.mod,go.sum,fixtures/**}"
From version 3.109.0 of the Buildkite agent, if_changed also supports lists of glob patterns and include and exclude attributes.
Minimum Buildkite agent versions: 3.99 (with --apply-if-changed flag), 3.103.0 (enabled by default), 3.109.0 (expanded syntax)

For an example pipeline, demonstrating various forms of if_changed, see Using if_changed.

Parallel groups

Adding a group step to a pipeline switches the entire build to directed acyclic graph (DAG) scheduling, which you can view through the DAG tab on the build page. Instead of running steps in the order they're listed in pipeline.yml (with wait steps as barriers), Buildkite schedules each step as soon as its explicit and implicit dependencies are satisfied. This is why consecutive group steps run in parallel.

Adding a group can reorder existing steps

Because a group step puts the whole build into DAG scheduling, adding one to an existing pipeline can change the order in which other steps run. If your pipeline relies on the listed order of steps without wait or depends_on between them, add explicit dependencies before introducing a group step.

If you put two or more group steps in a YAML config file consecutively, they will run in parallel. For example:

pipeline.yml
# 1.sh and 3.sh will start at the same time.
# 2.sh will start when 1.sh finishes, and 4.sh will start
# when 3.sh finishes.
steps:
  - group: "first"
    steps:
      - command: "1.sh"
      - wait
      - command: "2.sh"
  - group: "second"
    steps:
      - command: "3.sh"
      - wait
      - command: "4.sh"

Running jobs in parallel has some limitations:

  • Parallel groups will be displayed ungrouped if the build's jobs are truncated because Buildkite doesn't currently store or calculate any information about the number of jobs in a non-parallel group.

  • If a parallel step exists within a group, parallel jobs are treated as regular jobs within a step group - so you can't have parallel groups within step groups. So, for example, a group that contains two steps each with parallel: 4 will display eight jobs in it, with no visual indication that those eight jobs are two parallel steps.

  • If a parallel job group is within a named group, the groups are handled as though the parallel group isn't there.

  • It's impossible to have a parallel job with only some of the jobs within a group, as they're all created on the same YAML step entry.

Using wait steps in job groups

You can have wait steps in a group. Such steps operate independently of other groups. For example, both groups will operate independently here, meaning d.sh won't wait on a.sh to finish. Note also that wait steps are counted in the group step total, so both Group01 and Group02 contain 3 steps.

pipeline.yml
steps:
  - group: "Group01"
    depends_on: "tests"
    steps:
      - command: "a.sh"
      - wait
      - command: "b.sh"

  - group: "Group02"
    depends_on: "tests"
    id: "toast"
    steps:
      - command: "c.sh"
      - wait
      - command: "d.sh"

  - command: "yay.sh"
    depends_on: "toast"

Group merging

If you upload a pipeline that has a group or label that matches the group of the step that uploaded it, those groups will be merged together in the Buildkite UI.

For merging to happen, all of the following must be true:

  • The step that runs buildkite-agent pipeline upload is itself inside a group step.
  • The first step in the uploaded pipeline is a group step.
  • The group or label on the uploaded group matches the group or label of the group containing the upload step.

If any of these conditions are not met, the uploaded group is added as a separate group in the Buildkite interface, even when its label matches an existing group.

Note that inside a single pipeline, groups with the same group or label will not be merged in the Buildkite UI. The same applies when multiple separate pipeline upload calls each create a group with the same name. Each upload produces its own group, and the UI shows two distinct groups with the same label. To avoid this in dynamic pipelines that run several uploads, give each generator's groups a distinct label, or consolidate the steps into a single upload.

You can't define the same key twice

Trying to create different groups or steps with the same key attribute will result in an error.

For example, you have a YAML file:

pipeline.yml
steps:
  - group: "Setup"
    steps:
      - commands:
        - "buildkite-agent pipeline upload"
        - echo "start"

And this YAML file uploads a pipeline that has a group with the same name:

pipeline.yml
steps:
  - group: "Setup"
    steps:
      - command: "docker build"

These groups will be merged into one in the UI, and the docker build step will be added to the existing group.

Similarly, if you have a YAML file:

pipeline.yml
steps:
  - group: ~
    label: "Setup"
    steps:
      - commands:
        - "buildkite-agent pipeline upload"
        - echo "start"

And this YAML file uploads a pipeline that has a group with the same label:

pipeline.yml
steps:
  - group: ~
    label: "Setup"
    steps:
      - command: echo "proceed"

These groups will be merged into one in the UI, and the echo "proceed" step will be added to the existing group.

When merging does not happen

In the following example, the buildkite-agent pipeline upload command runs from a top-level command step rather than from inside a group:

pipeline.yml
steps:
  - command: "buildkite-agent pipeline upload"

If the uploaded pipeline contains a group with the same label as an existing group on the build, merging does not happen, because the upload step is not itself inside a group:

pipeline.yml
steps:
  - group: "Setup"
    steps:
      - command: "docker build"

The uploaded Setup group is shown as a separate group in the Buildkite interface, even when another Setup group already exists on the build. To merge the groups, move the upload command into a group step with the matching label.

Example

:buildkite: Group steps An example of how to group steps in a pipeline github.com/buildkite/group-step-example