A long-running job can know it will exit non-zero before its command is complete. Until now, the build stayed running until that job finished, delaying notifications, investigation, and remediation.

With promise job failure, the job can declare its expected exit status early. Buildkite applies retry and soft-fail rules, then marks the build failing if the promise represents a hard failure.

The job keeps running to complete tests, teardown, logs, artifacts, and test-result uploads. Engineers and AI agents can investigate at the same time, so you can act sooner without losing diagnostic context.

Declare a promised failure

Declare a promised failure from a running command job with Buildkite Agent v3.128.0 or later:

buildkite-agent job promise-failure 1 --reason "test_failure (2 failed after retries)"

Call it only after the job has confirmed that the failure is definitive and build-critical, including exhausting any retries managed inside your test suite.

Existing build.failing and step.failing notifications can respond earlier, and Preflight can begin investigating while the job continues.

Sibling jobs using cancel_on_build_failing can also stop work that is no longer useful.

If you use Buildkite Test Engine Client v2.9.0 or later, set BUILDKITE_TEST_ENGINE_PROMISE_FAILURE to true to declare promised failures automatically after retries are exhausted and hard test failures remain.

Find opportunities in your pipelines

Use this prompt with an AI agent that can access your Buildkite builds, jobs, logs, and test results to look for opportunities to use this functionality (use our MCP or CLI):

Analyze the last 30 days of our Buildkite pipelines for opportunities to use
promise job failure.

If the result set is large, prioritize the longest-running and most frequently
failing jobs.

Find jobs that ultimately hard-failed and where a definitive, build-critical
failure was visible well before the job finished.

Favor jobs that kept running useful work afterward, such as remaining tests,
teardown, or uploading logs, artifacts, coverage, and test results.

Exclude soft failures, failures still eligible for automatic or in-suite
retries, muted or quarantined tests, and jobs that could simply exit early
without losing useful work.

Rank candidates by estimated time saved.

For each one, report the pipeline and step, evidence of the early failure, time
between that signal and job finish, work that continued afterward, and a safe
place to declare the promise. Include risks and uncertainty.

Do not change pipeline code.

See the promise job failure documentation for setup, integrations, API details, and guidance on measuring the time saved.

To make headless CI practical for AI agents and infrastructure tools, Buildkite now exposes more setup, governance, and troubleshooting operations through REST and GraphQL.

The new coverage spans pipeline creation, organization administration, notification services, hosted agent resources, and artifacts.

It also includes the previously announced REST infrastructure-failure diagnosis improvements, which give automation more context when troubleshooting failed jobs.

Create a valid pipeline sooner

• Pipeline creation through REST now rejects requests with missing or blank step configuration, returning actionable validation feedback before a pipeline is created.
• Organization administrators can list and inspect existing GitHub, GitHub Enterprise Server, Bitbucket Server, and GitLab Self-Managed repository connections.
• New pipeline endpoints inspect, enable, and disable incoming GitHub webhook processing for organizations enrolled in expanded webhook triggers.

Authorizing a new repository provider still starts in the Buildkite interface.

Automate organization administration

• New read-only endpoints list and inspect organization audit events using cursor pagination.
• Organization administrators can read and update pipeline defaults, manage hosted agent SSH access, public pipeline creation, build exports.
• Organization administrators can read and update API IP allowlists, inactive token revocation, and restrictions on user API token creation.
• A new endpoint enables Teams for an organization.
• Organization invitations can be listed, inspected, created, and revoked, including role, SSO mode, and team assignments.
• Organization member roles and SSO requirements can be updated through PATCH /v2/organizations/{org.slug}/members/{user.uuid}.

Manage notifications and GitHub webhook processing

REST clients can list, inspect, create, update, delete, enable, and disable supported organization notification services.

Supported services include webhooks, Slack incoming webhooks, Amazon EventBridge, Datadog, and OpenTelemetry. Slack Workspace and Linear services can be managed through REST after browser-based authorization.

Configure Buildkite hosted agents from code

• Custom agent images can be listed, inspected, created, and deleted.
• Hosted-agent egress network ranges can be retrieved for firewall and network-policy automation.
• Cache volumes can be listed and deleted by tag.
• Hosted Git mirror and container cache settings can be read and updated through the clusters API.

These resources let infrastructure automation keep hosted agent images, network rules, and cache settings in sync.

Find the right artifacts faster

Build- and job-level REST artifact lists now support state and path filters before pagination. Path filters support exact matches and glob patterns, making it easier to find specific outputs in large artifact collections.

GraphQL clients can delete artifacts with the artifactDelete mutation.

API reference

• Pipeline setup: Pipelines and repository connections.
• Organization administration: Audit events, pipeline settings, API settings, invitations, members, and Teams.
• Notifications: Notification services.
• Hosted agents: Clusters, agent images, network ranges, and cache volumes.
• Artifacts: REST artifacts and artifactDelete.

For a complete method and path index, see the REST API overview and GraphQL API reference.

Buildkite Agent v3.135.0 can now send job logs to your OpenTelemetry collector. This brings CI output into the same observability backend as your application and infrastructure telemetry, making it easier to investigate slow or failing jobs without switching between tools.

When OpenTelemetry tracing is also enabled, each record is correlated with the job, phase, or hook span that produced it. In backends that support log-to-trace correlation, you can pivot from a span directly to the output that explains what happened. Tracing is optional — logs can also be exported on their own.

Enable the new OTLP log sink with --job-logs-otlp or BUILDKITE_JOB_LOGS_OTLP=true on buildkite-agent start. Your normal Buildkite job log stream is unchanged.

The agent uses the standard OTEL_EXPORTER_OTLP_LOGS_* and OTEL_EXPORTER_OTLP_* environment variables, so you can use the same collector configuration as your trace export. Both grpc (the default) and http/protobuf are supported through OTEL_EXPORTER_OTLP_LOGS_PROTOCOL.

Each log record carries:

• Body: one line from the job log stream, with the same [REDACTED] markers as the Buildkite UI and downloadable job log, including command output and agent output such as section headers, prompts, and warnings
• Trace correlation: the trace and span ID of the enclosing phase or hook span, falling back to the root job span when no phase is active. Records are uncorrelated when tracing is disabled
• Job attributes: organization slug, pipeline slug, branch, queue, agent name and ID, build ID and number, job ID, label, and step key
• Timestamp: the arrival of the line's first byte, matching the start-of-line semantics of the Buildkite job log timestamper

Agent operators opt in to job log export when starting the agent.

For configuration details and the complete record schema, see Exporting job logs as OpenTelemetry logs.

Agents no longer need to hand-roll a polling loop over get_build to find out how a build went. The Buildkite MCP Server now includes wait_for_build, a tool that waits for a build to reach a terminal state and reports the outcome.

Each call waits up to 45 seconds. If the build settles in that window, the tool returns the final build details along with the state it reached: passed, failed, canceled, skipped, not_run, or blocked. If the build is still running, the call comes back with the current state and build_elapsed_seconds, counted from the build's own start time, and the agent calls again to keep waiting. A long wait becomes a handful of cheap calls rather than a tight polling loop, and the tool tells the agent to stop after roughly ten consecutive calls and report the build as still running instead of waiting forever.

This works well ahead of get_build_failure_summary: wait for the build, and if it failed, go straight to the likely cause.

These improvements apply to both the open-source MCP server and the Remote MCP server.

The Model Context Protocol has gone stateless—and Buildkite is ready. The Buildkite MCP Server is now compliant with the MCP 2026-07-28 specification across both the open-source server and the Remote MCP Server.

The headline change transforms MCP from a bidirectional, stateful protocol into a stateless request-response protocol. This was one of the most highly requested improvements from MCP developers, unlocking better reliability and simpler horizontal scaling for servers. Protocol-level sessions and the initialization handshake are gone; instead, every request carries its protocol version and client capabilities, allowing any server instance to handle it without session affinity.

The Buildkite MCP Server also supports the new server/discover method for capability and version discovery, standardized HTTP headers for routing requests by method and tool name, and cache metadata on list responses. It no longer advertises the deprecated MCP Logging capability.

Existing clients remain supported through protocol version negotiation, while clients that support MCP 2026-07-28 can use the new protocol automatically.

The OpenTelemetry Notification Service now has a Request Log panel that is available in preview. This gives you more visibility into your trace exports and helps you diagnose connectivity, authentication, and endpoint configuration issues.

The panel shows the last 20 outbound trace export requests. Each row shows the request UUID and exported span name (for example, buildkite.job or buildkite.step). When the endpoint responds, the row also shows the HTTP status code and request duration. Expanding a row shows:

• Request: Headers and body. Custom header values, such as API keys and bearer tokens, are redacted. The OTLP protobuf body is decoded into readable JSON, with trace and span IDs shown in hexadecimal.
• Response: Headers and body, or an error message when no HTTP response is received. Protobuf responses, including any partialSuccess rejections and error messages, are decoded into readable JSON.

This feature is in preview. Contact support@buildkite.com to have it enabled for your organization.

For more information, see the OpenTelemetry Request Log docs.

On 1 September 2026 (AEST), Buildkite Agent v4 becomes the stable release. From that date, installations and upgrades that follow the latest or stable release channels will use v4.

We expect most customers to be unaffected. v4 retires functionality that has been deprecated for years and makes current, supported behaviour the default, giving the agent a cleaner baseline for what we build next. New capabilities have continued shipping in v3, so for most setups the move to v4 changes nothing about how you use the agent.

You can test v4 in beta today to confirm your setup is ready before the switch. The v3 → v4 upgrade guide covers every change, with instructions for testing, planning your upgrade, or staying on v3 if you need more time.

What might affect you

• Secrets in pipeline uploads now fail the upload. Detected secrets cause pipeline upload to fail by default. Use --allow-secrets to opt out while you update your pipeline.
• post-checkout, post-command, and pre-exit hooks now run in reverse order. In v3, agent, repository, and plugin hooks run in the same order for every phase. In v4, these three run in reverse: the last configured plugin runs first, followed by earlier plugins, repository hooks, then agent hooks. This lets cleanup unwind setup in the opposite order, but may affect hooks that depend on the current sequence. The legacy-post-hook-order experiment temporarily restores the v3 order.
• Agent configuration variables propagate into child environments. propagate-agent-config-vars is now the default, so configuration such as BUILDKITE_GIT_* and BUILDKITE_SHELL passes into containers and other child processes. These variables appear in BUILDKITE_ENV_FILE as bare names rather than KEY=value entries — update any custom hooks or tools that parse every line as an assignment.
• OpenTracing is removed in favour of OpenTelemetry. v4 removes the agent's built-in Datadog APM tracing backend and its direct DogStatsD metrics integration. To keep sending traces to Datadog, enable OpenTelemetry with --opentelemetry-tracing (or BUILDKITE_OPENTELEMETRY_TRACING) and configure the Datadog Agent or another OpenTelemetry Collector to receive OTLP traces. For metrics, enable --opentelemetry-metrics (or BUILDKITE_OPENTELEMETRY_METRICS) and send them to the same collector, or scrape the agent's Prometheus endpoint with Datadog's OpenMetrics integration.
• Some configuration options are renamed or removed. Several have been deprecated for years: --meta-data was replaced by --tags in 2017 and has worked as an alias ever since. Others change in v4 itself — --cancel-grace-period and --signal-grace-period-seconds are replaced by --cancel-signal-timeout and --cancel-cleanup-timeout. The upgrade guide lists every one.
• The legacy Docker integration is removed. v4 removes the integration configured through BUILDKITE_DOCKER and BUILDKITE_DOCKER_COMPOSE_CONTAINER. If you use it, switch to the docker or docker-compose plugins, which are unaffected.

The upgrade guide has the complete list and the migration steps for each change.

What to do before 1 September

• Review the upgrade guide to check whether any change affects your agent configuration, hooks, plugins, or observability setup.
• Test v4. Install the beta and run representative jobs. The upgrade guide includes instructions for common installation methods.
• Decide how you'll receive v4. Agent installations that follow latest or stable will move to v4 on 1 September. If you pin the agent version, update the pin when you're ready.
• Need more time? Stay on v3 by switching to the oldstable release channel or setting the agent version to 3, depending on your installation method. v3 will continue to receive critical security and reliability fixes, and we'll announce an end-of-support date with plenty of notice.

Hosted Agents

If you use Hosted Agents, there is nothing to configure. Buildkite will manage the move to v4 over the weeks leading up to 1 September, and the agent version your Hosted Agents use will update automatically. We'll monitor the rollout closely and work directly with anyone who needs more time or hits a compatibility issue.

Questions or need a hand? Reach out to support@buildkite.com or your account team.

Debugging a failed build shouldn't start with a scavenger hunt. The Buildkite MCP Server now includes get_build_failure_summary, a new tool designed around Anthropic's best practices for writing tools for agents. It turns a common multi-step workflow into one purpose-built call, taking an agent from “the build failed” to a likely cause.

The tool brings together the build state, failed and broken jobs, useful log tails, error and warning annotations, and failed Test Engine executions. It even catches jobs that are still running but have already signaled that they'll fail. The result is deliberately bounded, giving the agent useful context without burying the conversation in logs.

When a failure needs a closer look, the agent can still follow up with the existing job, log, annotation, and test tools.

This is available in both the open-source MCP server and the Remote MCP server.

The Elastic CI Stack for AWS now supports Ubuntu 24.04 LTS as an alternative to Amazon Linux 2023 for Linux agents. Ubuntu images are available for both x86-64 and ARM64 instances.

Ubuntu can be selected with either supported deployment method:

• For CloudFormation, set the LinuxDistribution parameter to ubuntu2404 in Elastic CI Stack for AWS v6.70.0.
• For Terraform, set linux_distribution = "ubuntu2404" when using Terraform module v0.11.0.

Amazon Linux 2023 remains the default, so existing configurations continue to use it unless Ubuntu is explicitly selected.

The Buildkite MCP Server can now discover and load on-demand usage guides, and log searching is more efficient and reliable.

Discoverable skill guides for agents

Agents can now discover and load usage guides on demand, instead of relying solely on tool descriptions or the server's static instructions. Guides can be searched by keyword and loaded in full when needed, making it easier for an agent to find the right guidance for the task at hand. The first available guide covers debugging build failures, previously available only as a passive resource, and now discoverable and loadable directly by an agent.

More efficient, reliable log search

Searching build logs is now more token-efficient, cutting response size by roughly 40% in testing by trimming repeated, redundant data from each result. We've also fixed a few edge cases so log searches behave more predictably, including where a search starts from and how row numbers are reported.

These improvements apply to both the open-source MCP server and the Remote MCP server.

Teams using GitLab.com or GitLab Self-Managed can now build merge requests and test proposed merges against their target branches with merged-results builds. Merge request builds test the source branch, while merged-results builds test the proposed merge, helping catch integration failures before merging.

Set up merge request builds

In your pipeline's GitLab settings:

1. Enable Build merge requests to build a merge request when it opens or receives new commits. If Build branches is also enabled, Buildkite avoids creating a duplicate branch build for an active merge request.
2. Optionally, enable Build merged results commit to test the proposed merge with its target branch instead of testing the source branch alone.
3. If your target branch changes frequently, enable Rebuild when target branch changes to keep merged-results builds up to date.

To control build volume when a target branch changes, Buildkite rebuilds up to the 20 most recently created affected merge requests.

These settings are also available through the REST API. See the GitLab documentation to learn more.

The Buildkite MCP Server now returns leaner job data by default and surfaces build annotations without extra round trips.

Smaller, more targeted job listings

list_jobs now returns compact, actionable summaries by default, reducing token usage while keeping job IDs and failure diagnostics. Use detail_level: "detailed" for execution metadata, or detail_level: "full" for the previous, more verbose response. Summaries also include soft-failure, signal, step, and retry context when available.

list_jobs also gains step_key and group_key filters, so agents can fetch every job for a parallel step, or every job in a group, without paging through a build's full job list.

Annotation summaries on builds

get_build now includes lightweight annotation summaries, each with its ID, context, style, scope, associated job, and priority, without the potentially large annotation body. Up to 100 summaries are returned, with annotations_truncated set when there are more. Use list_annotations to fetch full annotation bodies or continue exploring.

These improvements apply to both the open-source MCP server and the Remote MCP server.

Test execution results uploaded to Test Engine are immutable, and so cannot be updated after initial upload. To allow metadata to be attached to execution records after upload we have introduced mutable tags.

Additional tags can be attached to executions in bulk via the execution tags API endpoint. Mutable tags must use the reserved mut. prefix, and can be used in filters in the same way as immutable tags.

The REST API's List Jobs endpoint now accepts optional step_key and group_key filters.

Use step_key to return the jobs for a specific step, including every job in a parallel step. Use group_key to return all jobs in a step group. These filters avoid fetching and filtering a build's full job list when you only need jobs from one step or group.

The Remote MCP Server's OAuth flow has had several reliability and security improvements, changing how MCP client sessions stay authenticated.

Sessions refresh instead of expiring

Previously, a session was tied to a fixed 7-day refresh token, after which an MCP client had to go through the full interactive authorization flow again. Sessions are now refreshed continuously in the background: access tokens are short-lived (1 hour) and refresh tokens roll forward for up to 30 days of continued use, so an actively used agent no longer hits a hard weekly re-authentication wall.

Clients that use Dynamic Client Registration (DCR) get a similar improvement to their client registration. Previously, a registration could expire on a fixed schedule even while a client was in active use. Now, each successful token refresh extends the registration's lifetime, so long-running agents no longer lose access mid-session.

Transient upstream errors (rate limits, timeouts) no longer force a re-authentication either. The server now retries these automatically and only asks a client to re-authenticate when the underlying Buildkite session has genuinely expired.

New: confirm the redirect before returning to your MCP client

Authorizing an MCP client now shows a confirmation page with the client's name and redirect URL before completing the flow. You can confirm or deny the request from there.

Note: if your MCP client automates the authorization redirect, it should expect this extra confirmation step in the flow.

New: rate limiting on the token endpoint

The OAuth token endpoint now rate limits requests per client and grant type to protect against excessive or misbehaving retries. Clients that exceed the limit receive an HTTP 429 response and should back off and retry shortly after.

For more information, see the Remote MCP Server documentation.

Organization administrators can now see when a user authorizes an OAuth application to access their organization in the Buildkite audit log.

Each OAUTH_APPLICATION_AUTHORIZED event identifies the user, OAuth application, organization, granted scopes, request IP address, and User Agent. This provides a clearer record of when tools using OAuth—such as the Buildkite CLI and Remote MCP Server—are granted access.

Refreshes of an existing OAuth authorization do not create duplicate authorization events.

Retried jobs now show their previous attempts directly in the build list. Click the job, or expand it with the chevron, to inspect earlier attempts without leaving the page.

We've heard from customers that it should be easier to see when jobs have been retried and quickly inspect what happened across attempts. This update makes it faster to compare retries, understand what changed, and debug flaky or failed jobs.

If you're not interested in previous job attempts, you can hide them by toggling off the Show past retry attempts display option.

Available now on the new Build page. If you haven't tried it yet, you can opt in from the Build page. Send feedback to support@buildkite.com.

The Buildkite MCP Server has new tools for working with jobs, smarter artifact and log handling, and more scalable pagination for large builds and test runs.

Dedicated tools for inspecting jobs

Agents can now look up job details directly with list_jobs and get_job, filter by state (for example, only failed jobs), page through large builds efficiently, and pull full agent details only when needed. This makes it much faster for an agent to zero in on "why did this build fail?" without wading through unrelated build metadata. list_agents and get_agent also now include os_id, arch, and queue by default, so agents can see where a job is running without requesting the full, token-heavy agent response.

Note: get_build and list_builds no longer return job data, and no longer accept detail_level, job_state, or include_agent — use list_jobs or get_job to fetch jobs instead. Your MCP client picks up the updated tool definitions automatically, so there's nothing to reconfigure.

Safer, more efficient artifact downloads

Artifacts now come back the smart way: small text files (under 64 KiB) are returned inline immediately, while larger or binary files get a temporary, ready-to-use download link, no API token juggling required. Listing artifacts across a build is also much lighter now, roughly halving the amount of data returned on large builds, so agents spend less time and budget just listing files before getting to the one they actually need.

Note: list_artifacts_for_build and list_artifacts_for_job no longer include download_url, url, dirname, glob_path, or original_path on each item. Use get_artifact with the id and job_id from a list result to fetch an artifact's content or download URL.

More scalable test result retrieval

Fetching failed test executions from large test runs now paginates properly against the Buildkite API, instead of pulling everything back at once. This means agents can work through large test suites reliably rather than hitting limits on big runs.

More reliable log handling

Under-the-hood improvements to log parsing make searching and tailing build logs more robust against messy or unusually large log output, with fewer dropped or malformed lines when an agent is digging through logs to debug a build.

Better resilience to rate limits

The server now handles Buildkite API rate limiting more gracefully, automatically retrying with proper backoff instead of failing outright. Agents should see fewer transient errors during heavy usage.

These improvements apply to both the open-source MCP server and the Remote MCP server.

The outbound IP address ranges used by Hosted Agents are being updated. The updated ranges are now visible in your Cluster's Networking page, but will not take effect until August 2, 2026. The new, larger ranges provide additional IP address space to support increasing workloads.

If you maintain IP allowlists or firewall rules, please review and update them before August 2 to ensure uninterrupted access.

Have questions or need help? Reach out to us at support@buildkite.com

Buildkite's REST API now exposes more of the job and runner context that agents, CLI tools, and MCP clients need to tell infrastructure failures apart from code failures.

The REST job object now includes signal and signal_reason. When signal_reason is present, it explains why the Buildkite Agent terminated the job, such as agent_stop or process_run_error. That gives automated debugging loops a clearer signal that the runner or process failed, rather than the code under test.

REST agent objects now include runner environment and lifecycle fields: os_id, arch, queue, connected_at, disconnected_at, lost_at, and stopped_at. These fields are also available through the agent embedded in a job response, so tools investigating a failed job can stay on the same API path and still answer questions like "did this runner disappear just before the job failed?"

Together, these additions make it easier for the Buildkite CLI, MCP clients, and other agentic CI workflows to diagnose failed jobs without falling back to the Buildkite UI. An agent can see whether a job was terminated by runner infrastructure, whether the runner was lost or stopped around the time of failure, and avoid blindly retrying builds that are unlikely to pass without infrastructure intervention.

These are additive read-only fields, with no migrations or GraphQL schema changes.