Pipelines glossary

Action

An action is part of a workflow and provides a user-defined operation that is triggered automatically when a workflow monitor enters the alarm or recover event state for a test. Actions can apply to the test itself (for example, changing its state or label), or to an external system (for example, sending a Slack notification about the test).

Learn more about actions in Alarm and recover actions.

Agent

An agent is a small, reliable, and cross-platform build runner that connects your infrastructure to Buildkite. It polls Buildkite for work, runs jobs, and reports results. You can install agents on local machines, cloud servers, or other remote machines. You need at least one agent to run builds.

To learn more, see the Agent overview.

Agent targeting

Agent targeting is how a step selects which agents can run its jobs. A step declares its requirements using the agents attribute, which holds one or more key=value tags (for example, queue=ios or os=linux). An agent only accepts a job when its own tags satisfy the step's agents query. The most common targeting tag is queue, which routes a job to a queue.

To learn more, see Defining steps and the Queues overview.

Alarm

An alarm is one of the two types of events that a workflow monitor can alert on, the other being recover. Alarm events are reported by the monitor when the alarm conditions are met. Depending on the monitor type, these alarm conditions are configurable.

Alarm actions are performed when the alarm event is reported by the monitor. Repeated occurrences of the test meeting the alarm conditions do not retrigger alarm actions.

Annotation

An annotation is rich content (Markdown, with optional HTML, images, and styling) that a step attaches to a build to add context beyond the raw job log. Common uses include test failure summaries, deployment links, and custom buttons. Annotations are created with the buildkite-agent annotate command and appear on the build page.

To learn more, see Annotations.

Artifact

An artifact is a file generated during a build. You can keep artifacts in a Buildkite-managed storage service or a third-party cloud storage service like Amazon S3, Google Cloud Storage, or Artifactory. Common uses include storing assets like logs and reports, or passing files between steps.

To learn more, see Build artifacts.

Block step

A block step pauses a build until it is manually unblocked, either in the Buildkite interface or through the API. Block steps are often used as a manual approval gate before a sensitive step, such as a production deployment.

To learn more, see Block step.

Build

A build is a single run of a pipeline. You can trigger a build in various ways, including through the dashboard, API, as the result of a webhook, on a schedule, or even from another pipeline using a trigger step.

Build matrix

A build matrix expands a single command step into multiple jobs, one for each combination of values across one or more dimensions (for example, operating system and language version). It is a concise way to run the same command across many configurations without defining each step by hand.

To learn more, see Build matrix.

Build metadata

Build metadata is a set of key/value string pairs attached to a build that any job in that build can read or write using the buildkite-agent meta-data command. It is the standard way to pass small pieces of state, such as a commit reference or an approval decision, between steps in the same build. For larger files, use artifacts instead.

To learn more, see Build meta-data.

Buildkite organization administrator

A Buildkite organization administrator is a user with full administrative control over a Buildkite organization. Organization administrators can manage teams, configure organization-level settings, control pipeline and security permissions, and access usage reports and audit logs.

To learn more, see User and team permissions.

Cluster

A cluster groups queues of agents along with pipelines. Clusters allow teams to self-manage their agent pools, let admins create isolated sets of agents and pipelines within the one Buildkite organization, and help to make agents and queues more discoverable across your organization.

To learn more, see the Clusters overview.

Concurrency group

A concurrency group is a named limit, configured on a step, that controls how many jobs sharing the same group key can run at once across builds and pipelines. Concurrency groups are commonly used to serialize access to a shared resource, such as a deployment target, without an external lock.

To learn more, see Controlling concurrency.

Dimensions

In the context of test suites, dimensions are structured data, consisting of tags, which can be used to filter or group (that is, aggregate) test executions. Dimensions are added to test executions using the tags feature, which you can learn more about in Tags.

Dynamic pipeline

Dynamic pipelines define their steps at runtime using scripts, giving you the flexibility to only run the steps relevant to particular code changes and workflows.

Dynamic pipelines are helpful when you have a complex build process that requires different steps to execute based on runtime conditions, such as the branch, the environment, or the results of previous steps. A dynamic pipeline generates its steps using a pipeline upload, where a job runs buildkite-agent pipeline upload to add steps to the current build.

To learn more, see Dynamic pipelines.

Ephemeral agent

An ephemeral agent is a Buildkite agent that only operates for the duration in which it runs a job. Such an agent is disconnected either once its job is completed, or the agent's idle time period has been reached. An ephemeral agent is created when one of the following options has been used to start the Buildkite agent:

• --acquire-job
• --disconnect-after-job
• --disconnect-after-idle-timeout

Learn more about ephemeral agents in Pause and resume an agent.

Execution

An execution is an instance of a single test, which is generated as part of a run. An execution tracks several aspects of a test, including its result (passed, failed, skipped, other), duration (time), and dimensions (that is, tags).

Flaky test

A flaky test is a test that produces inconsistent or unreliable results, despite being run in the same code and environment. Flaky tests are identified using workflows.

Learn more about flaky tests in Reduce flaky tests.

Hook

A hook is a method of customizing the behavior of Buildkite through lifecycle events. They let you run scripts at different points of the agent or job lifecycle. Using hooks, you can extend the functionality of Buildkite and automate tasks specific to your workflow and requirements.

Hooks run at named points in the agent or job lifecycle, such as environment, checkout, pre-command, command, post-command, and pre-exit. The pre-exit hook runs after the command, before the agent reports the job result, and is commonly used for cleanup tasks.

To learn more, see Hooks.

Hosted agent

A hosted agent is a Buildkite agent that runs on infrastructure managed by Buildkite, so you do not provision, scale, or maintain the machine yourself. Hosted agents are configured per queue and are available with Linux and macOS images. A self-hosted agent, by contrast, runs on infrastructure you control.

To learn more, see the Buildkite hosted agents overview.

Job

A job is the execution of a command step during a build. Jobs run the commands, scripts, or plugins defined in the step.

A job can be in various states during its lifecycle, such as pending, scheduled, running, finished, failed, canceled, and others. These states represent the execution state of the job as it progresses through the build system.

To learn more, see Job states.

Managed test

A managed test refers to any test (within all test suites of a Buildkite organization) that can be uniquely identified by its combination of test suite, scope, and name of the test.

For example, each of the following three tests are unique managed tests:

• Test Suite 1 - here.is.scope.one - Login Test name

• Test Suite 1 - here.is.another.scope - Login Test name

• Test Suite 2 - here.is.scope.one - Login Test name

Managed tests are used to track key areas of test runs, and for billing purposes.

Merge queue

Merge queue support is the integration between Buildkite Pipelines and a source control provider merge queue, such as the GitHub merge queue. When enabled, Buildkite triggers a build for each commit the merge queue proposes, so that changes are tested together in their intended merge order before they land.

To learn more, see Set up a GitHub merge queue.

Monitor

A monitor is a part of a workflow and is used to observe tests over time. Monitors help to surface valuable qualitative information about the tests in your test suite, which can be difficult to surmise from raw execution data. Monitors can report on special events (for example, a passed on retry event) or produce scores (such as, transition count score).

A single monitor watches over all the tests in your test suite (apart from those excluded by filters) and generates individual alarm and recover events for each test, which then trigger the associated alarm and recover action.

Learn more about the different monitors types in Monitors.

Notification

A notification is an outbound message that Buildkite Pipelines sends when a build event occurs, such as a build starting, passing, or failing. Notifications are configured in the pipeline with the notify attribute, or through integrations, and can be delivered to channels such as email, Slack, and webhooks.

To learn more, see Triggering notifications.

OIDC

OpenID Connect (OIDC) lets a job authenticate to an external service, such as a cloud provider, without storing long-lived secrets. A job requests a short-lived OIDC token with the buildkite-agent oidc request-token command and exchanges it for provider credentials. The token's claims, such as the organization, pipeline, and branch, let the provider apply fine-grained trust policies.

To learn more, see OpenID Connect (OIDC) in Buildkite Pipelines.

Parallelism

Parallelism runs the same command step across a number of jobs at the same time. Set the parallelism attribute on a step to the number of parallel jobs you want. Combined with test splitting, parallelism shards a long test suite across many agents.

To learn more, see Parallelizing builds.

Pipeline

A pipeline is a container for modeling and defining workflows. Pipelines contain a series of steps to achieve goals like building, testing, and deploying software.

To learn more, see the Pipeline overview.

Pipeline schedule

A pipeline schedule creates a build automatically at a recurring time, defined using cron syntax. A schedule can set the branch, commit, message, and environment used for the builds it creates.

To learn more, see Scheduled builds.

Pipeline template

A pipeline template is a reusable pipeline configuration, managed at the Buildkite organization level, that multiple pipelines can be created from. Templates let platform teams provide a consistent, governed starting point so that pipeline owners do not each maintain their own copy.

To learn more, see Pipeline templates.

Pipeline upload

A pipeline upload is the action a running job takes when it adds steps to the current build using the buildkite-agent pipeline upload command. Uploaded steps can be appended, or used to replace steps that have not yet started. Pipeline upload is the mechanism behind dynamic pipelines.

To learn more, see the buildkite-agent pipeline command reference.

Plugin

Plugins are small, self-contained pieces of extra functionality that help you customize Buildkite to your specific workflow. They modify command steps using hooks to perform actions like checking code quality, deploying to cloud services, or sending notifications.

Plugins can be open source and available for anyone to use or private for just your organization.

To learn more, see Plugins.

Quarantine

Quarantine is a classification applied to a test that, based on the state of the test, changes how that test is executed as part of a run. When a test is quarantined, and its test state is flagged as:

• muted, the test is executed as part of the run, but its failure does not cause the pipeline build to fail, allowing the test's metadata to still be collected.

• skipped, the test is not executed as part of the run, which can allow pipeline builds to execute more rapidly and can reduce costs, but no data is recorded from the test.

Learn more about quarantining tests in Test state and quarantine.

Queue

A queue defines agents on which pipeline builds can run their jobs. Queues are configured within a cluster, where each queue defines a particular group of agents, isolating a set of your pipeline's jobs and the agents they run on. Typical uses for queues include separating deployment agents and pools of agents for specific pipelines or teams.

To learn more, see the Queues overview and Manage queues pages.

Recover

A recover is one of the two types of events that a workflow monitor can alert on, the other being alarm. Recover events are hysteric, meaning that the recover event can only be reported on a test that has a previous alarm event. In such a situation, when the monitor detects that the test has met the recover conditions, a recover event is reported. Depending on the monitor type, these recover conditions can be configurable.

Recover actions are performed when the recover event is reported by the monitor. Repeated occurrences of the test meeting the recover conditions do not retrigger recover actions.

Retry

A retry re-runs a failed job. Buildkite Pipelines supports automatic retries, configured with the retry attribute on a step (for example, by exit status or when an agent is lost), and manual retries triggered by a user in the Buildkite interface or through the API.

To learn more, see Retrying jobs.

Run

A run is the execution of one or more tests in a test suite. A run is sometimes referred to as a test run, bearing in mind that a single test run usually involves the execution of multiple tests. A test suite run is analogous to a pipeline build.

Scope

A scope is a mechanism that can be implemented to differentiate between two or more identically named tests. For example, the following hypothetical tests have the same name as they both test the process of a user logging into a product platform. However, one of these applies to this test being done on a mobile device, while the other applies to a desktop web setting. Therefore, a scope can be used to differentiate between these two tests.

A test's scope is used in determining a managed test's uniqueness.

Signed pipelines

Signed pipelines cryptographically sign step definitions so that an agent only runs steps signed by a trusted key. This reduces the risk of unauthorized steps being added to a privileged build through a pipeline upload.

To learn more, see Signed pipelines.

Skip and cancel intermediate builds

Skip and cancel intermediate builds are pipeline settings that avoid running superseded builds when several commits arrive on the same branch in quick succession. Skipping prevents a queued intermediate build from starting, while canceling stops a running intermediate build, so that only the latest commit is built.

To learn more, see Skip queued intermediate builds and Cancel running intermediate builds.

Soft fail

A soft fail is a step outcome that reports failure without failing the build. Set the soft_fail attribute on a step to true, or to a list of exit statuses, so that a failing step is surfaced for visibility but does not block downstream steps. This differs from a hard failure, which fails the build. The corresponding step outcome is soft_failed, described in Step.

To learn more, see Soft failing a step.

Step

A step describes a single, self-contained task as part of a pipeline. You define a step in the pipeline configuration using one of the following step types:

• Command step: Runs one or more shell commands on one or more agents.
• Wait step: Pauses a build until all previous jobs have completed.
• Block step: Pauses a build until it's manually unblocked.
• Input step: Pauses a build until information has been collected from a user.
• Trigger step: Creates a build on another pipeline.
• Group step: Displays a group of sub-steps as one parent step.

A step can be in one of the following internal states, which the Buildkite agent can retrieve, when the step is ready to run, or is currently running:

• ignored: The step is ignored due to a conditional evaluation.
• waiting_for_dependencies: The step is waiting for its dependencies to complete.
• ready: The step is ready to run but hasn't started yet.
• running: The step is currently running.
• failing: The step is in the process of failing.
• finished: The step has completed execution — usually follows either the running or failing state.
• canceled: The step has been canceled - follows the waiting_for_dependencies, ready, running, or failing state.

Once a step's run has completed with a state of finished, the step's outcome can be one of the following states:

• neutral: The passing or failure of the step's outcome is not relevant (for example, the outcome of a wait step).
• passed: The step's outcome is considered successful.
• soft_failed: The step's outcome is considered successful, but with a warning. See Soft fail.
• hard_failed: The step's outcome is considered failed.
• errored: The step's outcome is considered failed because something happened to abort the step early.

A block or input step tracks the state of the build and its steps that ran before it, which can be failed, passed, or running.

To learn more, see Defining steps.

Tag

A tag is a key:value pair containing two parts:

• The tag's key is the identifier, which can only exist once on each test, and is case sensitive.
• The tag's value is the specific data or information associated with the key.

Within test suites, tags add dimensions to test execution metadata, so that tests and their executions can be better filtered, aggregated, and compared in test suite visualizations.

Tagging can be used to observe aggregated data points—for example, to observe aggregated performance across several tests, and (optionally) narrow the dataset further based on specific constraints.

Learn more about tags in the Tags topic.

Test

A test is an individual piece of code that runs as part of an application's or component's (for example, a library's) building process (which can be automated by Pipelines), to ensure that a specific area of the application or component functions as expected.

Test collection

Test collection is the process of collecting test data from a development project. Test collection may consist of one or more test collectors configured within a development project, or make use of other methods based on common standards such as JUnit XML or JSON to collect tests.

While a development project's test runners (such as RSpec or Jest) are typically configured with their respective test collectors, the JUnit XML or JSON test collection mechanisms can be used to collect test data from multiple test runners.

Test collector

A test collector is a dedicated open source library (developed by Buildkite) that can be implemented into your development project, to collect test data from a test runner within your project.

Buildkite offers a number of test collectors for a range of languages and their test runners.

Test Engine Client (bktec)

The Test Engine Client (bktec) is a command-line tool that splits a test suite across parallel jobs using historical timing data so that each job runs a similar share of the total test time. It is part of the Buildkite Pipelines test suites feature.

To learn more, see Speed up builds with bktec.

Test runner

A test runner is a synonymous term used for a test framework, which is typically a code library that can be integrated into a development project to facilitate the implementation of tests for that project.

Test state

A test state is a configurable flag that can be applied to a test (typically flaky tests), which quarantines the test and affects how the test is executed as part of a run.

The supported test state flags are:

• enabled: the test is in a trusted state and runs normally.
• muted: the test is quarantined—see Quarantine for the resulting run behavior.
• skipped: the test is quarantined—see Quarantine for the resulting run behavior.

Learn more about test states in Test state and quarantine.

Test suite

A test suite is a collection of tests, managed within Buildkite Pipelines through its test suites features (Test Engine). A test suite is sometimes abbreviated to suite.

In a development project configured with one or more test runners, it is typical to configure a separate test suite for each of the project's test runners.

Trigger step

A trigger step creates a build on another pipeline from within the current build. It can pass environment variables and build metadata to the triggered build, which makes it the standard way to chain pipelines together, such as a build pipeline triggering a separate deployment pipeline.

To learn more, see Trigger step.

Workflow

A workflow defines a process that's composed of a single monitor and any number of actions. A workflow enables a user to define a custom identification and management system for tests of interest in their suite. Flaky test management is a common use case for workflows.

Learn more about workflows in the Workflows overview.