The OpenTelemetry Notification Service now has a Request Log panel that is available in preview. This gives you more visibility into your trace exports and helps you diagnose connectivity, authentication, and endpoint configuration issues.

The panel shows the last 20 outbound trace export requests. Each row shows the request UUID and exported span name (for example, buildkite.job or buildkite.step). When the endpoint responds, the row also shows the HTTP status code and request duration. Expanding a row shows:

• Request: Headers and body. Custom header values, such as API keys and bearer tokens, are redacted. The OTLP protobuf body is decoded into readable JSON, with trace and span IDs shown in hexadecimal.
• Response: Headers and body, or an error message when no HTTP response is received. Protobuf responses, including any partialSuccess rejections and error messages, are decoded into readable JSON.

This feature is in preview. Contact support@buildkite.com to have it enabled for your organization.

For more information, see the OpenTelemetry Request Log docs.

On 1 September 2026 (AEST), Buildkite Agent v4 becomes the stable release. From that date, installations and upgrades that follow the latest or stable release channels will use v4.

We expect most customers to be unaffected. v4 retires functionality that has been deprecated for years and makes current, supported behaviour the default, giving the agent a cleaner baseline for what we build next. New capabilities have continued shipping in v3, so for most setups the move to v4 changes nothing about how you use the agent.

You can test v4 in beta today to confirm your setup is ready before the switch. The v3 → v4 upgrade guide covers every change, with instructions for testing, planning your upgrade, or staying on v3 if you need more time.

What might affect you

• Secrets in pipeline uploads now fail the upload. Detected secrets cause pipeline upload to fail by default. Use --allow-secrets to opt out while you update your pipeline.
• post-checkout, post-command, and pre-exit hooks now run in reverse order. In v3, agent, repository, and plugin hooks run in the same order for every phase. In v4, these three run in reverse: the last configured plugin runs first, followed by earlier plugins, repository hooks, then agent hooks. This lets cleanup unwind setup in the opposite order, but may affect hooks that depend on the current sequence. The legacy-post-hook-order experiment temporarily restores the v3 order.
• Agent configuration variables propagate into child environments. propagate-agent-config-vars is now the default, so configuration such as BUILDKITE_GIT_* and BUILDKITE_SHELL passes into containers and other child processes. These variables appear in BUILDKITE_ENV_FILE as bare names rather than KEY=value entries — update any custom hooks or tools that parse every line as an assignment.
• OpenTracing is removed in favour of OpenTelemetry. v4 removes the agent's built-in Datadog APM tracing backend and its direct DogStatsD metrics integration. To keep sending traces to Datadog, enable OpenTelemetry with --opentelemetry-tracing (or BUILDKITE_OPENTELEMETRY_TRACING) and configure the Datadog Agent or another OpenTelemetry Collector to receive OTLP traces. For metrics, enable --opentelemetry-metrics (or BUILDKITE_OPENTELEMETRY_METRICS) and send them to the same collector, or scrape the agent's Prometheus endpoint with Datadog's OpenMetrics integration.
• Some configuration options are renamed or removed. Several have been deprecated for years: --meta-data was replaced by --tags in 2017 and has worked as an alias ever since. Others change in v4 itself — --cancel-grace-period and --signal-grace-period-seconds are replaced by --cancel-signal-timeout and --cancel-cleanup-timeout. The upgrade guide lists every one.
• The legacy Docker integration is removed. v4 removes the integration configured through BUILDKITE_DOCKER and BUILDKITE_DOCKER_COMPOSE_CONTAINER. If you use it, switch to the docker or docker-compose plugins, which are unaffected.

The upgrade guide has the complete list and the migration steps for each change.

What to do before 1 September

• Review the upgrade guide to check whether any change affects your agent configuration, hooks, plugins, or observability setup.
• Test v4. Install the beta and run representative jobs. The upgrade guide includes instructions for common installation methods.
• Decide how you'll receive v4. Agent installations that follow latest or stable will move to v4 on 1 September. If you pin the agent version, update the pin when you're ready.
• Need more time? Stay on v3 by switching to the oldstable release channel or setting the agent version to 3, depending on your installation method. v3 will continue to receive critical security and reliability fixes, and we'll announce an end-of-support date with plenty of notice.

Hosted Agents

If you use Hosted Agents, there is nothing to configure. Buildkite will manage the move to v4 over the weeks leading up to 1 September, and the agent version your Hosted Agents use will update automatically. We'll monitor the rollout closely and work directly with anyone who needs more time or hits a compatibility issue.

Questions or need a hand? Reach out to support@buildkite.com or your account team.

Debugging a failed build shouldn't start with a scavenger hunt. The Buildkite MCP Server now includes get_build_failure_summary, a new tool designed around Anthropic's best practices for writing tools for agents. It turns a common multi-step workflow into one purpose-built call, taking an agent from “the build failed” to a likely cause.

The tool brings together the build state, failed and broken jobs, useful log tails, error and warning annotations, and failed Test Engine executions. It even catches jobs that are still running but have already signaled that they'll fail. The result is deliberately bounded, giving the agent useful context without burying the conversation in logs.

When a failure needs a closer look, the agent can still follow up with the existing job, log, annotation, and test tools.

This is available in both the open-source MCP server and the Remote MCP server.

The Elastic CI Stack for AWS now supports Ubuntu 24.04 LTS as an alternative to Amazon Linux 2023 for Linux agents. Ubuntu images are available for both x86-64 and ARM64 instances.

Ubuntu can be selected with either supported deployment method:

• For CloudFormation, set the LinuxDistribution parameter to ubuntu2404 in Elastic CI Stack for AWS v6.70.0.
• For Terraform, set linux_distribution = "ubuntu2404" when using Terraform module v0.11.0.

Amazon Linux 2023 remains the default, so existing configurations continue to use it unless Ubuntu is explicitly selected.

The Buildkite MCP Server can now discover and load on-demand usage guides, and log searching is more efficient and reliable.

Discoverable skill guides for agents

Agents can now discover and load usage guides on demand, instead of relying solely on tool descriptions or the server's static instructions. Guides can be searched by keyword and loaded in full when needed, making it easier for an agent to find the right guidance for the task at hand. The first available guide covers debugging build failures, previously available only as a passive resource, and now discoverable and loadable directly by an agent.

More efficient, reliable log search

Searching build logs is now more token-efficient, cutting response size by roughly 40% in testing by trimming repeated, redundant data from each result. We've also fixed a few edge cases so log searches behave more predictably, including where a search starts from and how row numbers are reported.

These improvements apply to both the open-source MCP server and the Remote MCP server.

Teams using GitLab.com or GitLab Self-Managed can now build merge requests and test proposed merges against their target branches with merged-results builds. Merge request builds test the source branch, while merged-results builds test the proposed merge, helping catch integration failures before merging.

Set up merge request builds

In your pipeline's GitLab settings:

1. Enable Build merge requests to build a merge request when it opens or receives new commits. If Build branches is also enabled, Buildkite avoids creating a duplicate branch build for an active merge request.
2. Optionally, enable Build merged results commit to test the proposed merge with its target branch instead of testing the source branch alone.
3. If your target branch changes frequently, enable Rebuild when target branch changes to keep merged-results builds up to date.

To control build volume when a target branch changes, Buildkite rebuilds up to the 20 most recently created affected merge requests.

These settings are also available through the REST API. See the GitLab documentation to learn more.

The Buildkite MCP Server now returns leaner job data by default and surfaces build annotations without extra round trips.

Smaller, more targeted job listings

list_jobs now returns compact, actionable summaries by default, reducing token usage while keeping job IDs and failure diagnostics. Use detail_level: "detailed" for execution metadata, or detail_level: "full" for the previous, more verbose response. Summaries also include soft-failure, signal, step, and retry context when available.

list_jobs also gains step_key and group_key filters, so agents can fetch every job for a parallel step, or every job in a group, without paging through a build's full job list.

Annotation summaries on builds

get_build now includes lightweight annotation summaries, each with its ID, context, style, scope, associated job, and priority, without the potentially large annotation body. Up to 100 summaries are returned, with annotations_truncated set when there are more. Use list_annotations to fetch full annotation bodies or continue exploring.

These improvements apply to both the open-source MCP server and the Remote MCP server.

Test execution results uploaded to Test Engine are immutable, and so cannot be updated after initial upload. To allow metadata to be attached to execution records after upload we have introduced mutable tags.

Additional tags can be attached to executions in bulk via the execution tags API endpoint. Mutable tags must use the reserved mut. prefix, and can be used in filters in the same way as immutable tags.

The REST API's List Jobs endpoint now accepts optional step_key and group_key filters.

Use step_key to return the jobs for a specific step, including every job in a parallel step. Use group_key to return all jobs in a step group. These filters avoid fetching and filtering a build's full job list when you only need jobs from one step or group.

The Remote MCP Server's OAuth flow has had several reliability and security improvements, changing how MCP client sessions stay authenticated.

Sessions refresh instead of expiring

Previously, a session was tied to a fixed 7-day refresh token, after which an MCP client had to go through the full interactive authorization flow again. Sessions are now refreshed continuously in the background: access tokens are short-lived (1 hour) and refresh tokens roll forward for up to 30 days of continued use, so an actively used agent no longer hits a hard weekly re-authentication wall.

Clients that use Dynamic Client Registration (DCR) get a similar improvement to their client registration. Previously, a registration could expire on a fixed schedule even while a client was in active use. Now, each successful token refresh extends the registration's lifetime, so long-running agents no longer lose access mid-session.

Transient upstream errors (rate limits, timeouts) no longer force a re-authentication either. The server now retries these automatically and only asks a client to re-authenticate when the underlying Buildkite session has genuinely expired.

New: confirm the redirect before returning to your MCP client

Authorizing an MCP client now shows a confirmation page with the client's name and redirect URL before completing the flow. You can confirm or deny the request from there.

Note: if your MCP client automates the authorization redirect, it should expect this extra confirmation step in the flow.

New: rate limiting on the token endpoint

The OAuth token endpoint now rate limits requests per client and grant type to protect against excessive or misbehaving retries. Clients that exceed the limit receive an HTTP 429 response and should back off and retry shortly after.

For more information, see the Remote MCP Server documentation.

Organization administrators can now see when a user authorizes an OAuth application to access their organization in the Buildkite audit log.

Each OAUTH_APPLICATION_AUTHORIZED event identifies the user, OAuth application, organization, granted scopes, request IP address, and User Agent. This provides a clearer record of when tools using OAuth—such as the Buildkite CLI and Remote MCP Server—are granted access.

Refreshes of an existing OAuth authorization do not create duplicate authorization events.

Retried jobs now show their previous attempts directly in the build list. Click the job, or expand it with the chevron, to inspect earlier attempts without leaving the page.

We've heard from customers that it should be easier to see when jobs have been retried and quickly inspect what happened across attempts. This update makes it faster to compare retries, understand what changed, and debug flaky or failed jobs.

If you're not interested in previous job attempts, you can hide them by toggling off the Show past retry attempts display option.

Available now on the new Build page. If you haven't tried it yet, you can opt in from the Build page. Send feedback to support@buildkite.com.

The Buildkite MCP Server has new tools for working with jobs, smarter artifact and log handling, and more scalable pagination for large builds and test runs.

Dedicated tools for inspecting jobs

Agents can now look up job details directly with list_jobs and get_job, filter by state (for example, only failed jobs), page through large builds efficiently, and pull full agent details only when needed. This makes it much faster for an agent to zero in on "why did this build fail?" without wading through unrelated build metadata. list_agents and get_agent also now include os_id, arch, and queue by default, so agents can see where a job is running without requesting the full, token-heavy agent response.

Note: get_build and list_builds no longer return job data, and no longer accept detail_level, job_state, or include_agent — use list_jobs or get_job to fetch jobs instead. Your MCP client picks up the updated tool definitions automatically, so there's nothing to reconfigure.

Safer, more efficient artifact downloads

Artifacts now come back the smart way: small text files (under 64 KiB) are returned inline immediately, while larger or binary files get a temporary, ready-to-use download link, no API token juggling required. Listing artifacts across a build is also much lighter now, roughly halving the amount of data returned on large builds, so agents spend less time and budget just listing files before getting to the one they actually need.

Note: list_artifacts_for_build and list_artifacts_for_job no longer include download_url, url, dirname, glob_path, or original_path on each item. Use get_artifact with the id and job_id from a list result to fetch an artifact's content or download URL.

More scalable test result retrieval

Fetching failed test executions from large test runs now paginates properly against the Buildkite API, instead of pulling everything back at once. This means agents can work through large test suites reliably rather than hitting limits on big runs.

More reliable log handling

Under-the-hood improvements to log parsing make searching and tailing build logs more robust against messy or unusually large log output, with fewer dropped or malformed lines when an agent is digging through logs to debug a build.

Better resilience to rate limits

The server now handles Buildkite API rate limiting more gracefully, automatically retrying with proper backoff instead of failing outright. Agents should see fewer transient errors during heavy usage.

These improvements apply to both the open-source MCP server and the Remote MCP server.

The outbound IP address ranges used by Hosted Agents are being updated. The updated ranges are now visible in your Cluster's Networking page, but will not take effect until August 2, 2026. The new, larger ranges provide additional IP address space to support increasing workloads.

If you maintain IP allowlists or firewall rules, please review and update them before August 2 to ensure uninterrupted access.

Have questions or need help? Reach out to us at support@buildkite.com

Buildkite's REST API now exposes more of the job and runner context that agents, CLI tools, and MCP clients need to tell infrastructure failures apart from code failures.

The REST job object now includes signal and signal_reason. When signal_reason is present, it explains why the Buildkite Agent terminated the job, such as agent_stop or process_run_error. That gives automated debugging loops a clearer signal that the runner or process failed, rather than the code under test.

REST agent objects now include runner environment and lifecycle fields: os_id, arch, queue, connected_at, disconnected_at, lost_at, and stopped_at. These fields are also available through the agent embedded in a job response, so tools investigating a failed job can stay on the same API path and still answer questions like "did this runner disappear just before the job failed?"

Together, these additions make it easier for the Buildkite CLI, MCP clients, and other agentic CI workflows to diagnose failed jobs without falling back to the Buildkite UI. An agent can see whether a job was terminated by runner infrastructure, whether the runner was lost or stopped around the time of failure, and avoid blindly retrying builds that are unlikely to pass without infrastructure intervention.

These are additive read-only fields, with no migrations or GraphQL schema changes.

Read-only API access tokens no longer return a pipeline's webhook_url in REST and GraphQL responses.

Previously, a read_only token created by someone with pipeline write access could still retrieve the webhook_url. Because that URL can be used to trigger builds, a token intended to be read-only could effectively grant write access.

Buildkite now checks both the token's scope and its creator's permissions, applying whichever is more restrictive.

This is a breaking change if you rely on reading a pipeline's webhook_url using a read-only token. Update that automation to use a token with pipeline write scope instead. If you are using the GraphQL API, your token will need to have "Full access" GraphQL scope.

For more on token scopes, see the managing API tokens documentation.

We've made a few improvements to the way log groups render on the new build page. For a running job, the log group that's currently running stays open and on screen, and we've added live durations and sticky headers to make it easier to follow along.

• See the duration of the running log group: The running log group now shows a ticking duration and a bar that grows in real time, instead of a spinner you stare at until the group finishes.
• Follow the output as it streams: The group that's currently running expands automatically so its output stays visible. When the job moves on, it folds back to how the step defined it and the next group opens.
• Never lose your place: Scroll deep into a long expanded group and its header sticks just below the toolbar, then hands off to the next group as you pass it.

We've also shipped some fixes along the way to log search highlighting, and the log scrollbar now matches your active theme.

These improvements are available on the new build page. Give it a try if you haven't by opting in on the build page and tell us what you think at support@buildkite.com.

There's a new Metrics tab on every job that runs on Buildkite hosted agents, giving you a live view of how that job used its machine. No dashboards to wire up, no agent configuration — open a job and the charts are right there.

• Measure CPU load: See how hard the machine worked over the life of the job, so you can spot steps that are starved for CPU or right-size an oversized instance.
• Watch memory utilization: Track memory usage across the job to catch the spikes that lead to out-of-memory failures before they bite.
• Keep an eye on storage: Follow disk utilization over time to find jobs that are filling up the instance.
• Spot IO wait (Linux): On Linux instances, an extra chart surfaces time spent waiting on IO, making it easy to tell a disk-bound job apart from a CPU-bound one.
• Updates live as the job runs: The charts stream in while the job is still running and settle into a complete picture once it finishes, so you don't have to wait for the build to end to start investigating.

Use it to debug slow or failing jobs, right-size your hosted instances, and understand where your build minutes are actually going. Available now on jobs running on hosted agents — open any job and select the Metrics tab.

You can now fetch jobs separately from builds in the REST API, making large matrix or high-parallelism builds easier to query.

List builds without embedded jobs:

GET /v2/organizations/:org/pipelines/:pipeline/builds?exclude_jobs=true

Then fetch only the jobs you need:

# List jobs for one build, with filtering and pagination
GET /v2/organizations/:org/pipelines/:pipeline/builds/:build/jobs?state[]=failed&include_retried_jobs=false&per_page=100

# Fetch one job from a build
GET /v2/organizations/:org/pipelines/:pipeline/builds/:build/jobs/:uuid

# Fetch one job directly by UUID
GET /v2/organizations/:org/jobs/:uuid

Use exclude_jobs=true by default when listing builds unless you need embedded jobs, then use the jobs endpoints for targeted access.

See the REST API jobs documentation for endpoint details and examples.

Checkout

• Skip checkout on any agent by setting BUILDKITE_SKIP_CHECKOUT=true — no plugin or workaround required. Previously this meant clearing BUILDKITE_REPO, using a checkout plugin, or (in the Agent Stack for Kubernetes) setting checkout: skip: true.
• Skip fetching commits you already have with --git-skip-fetch-existing-commits, which skips the git fetch step during checkout when the target commit already exists locally. This saves time and network I/O on every build where the commit is already available, which is especially useful for agents using git mirrors.
• Bound how long a checkout attempt can run with --git-checkout-timeout (or BUILDKITE_GIT_CHECKOUT_TIMEOUT), specified in seconds. A hung git operation (clone, fetch, checkout, or submodule update) would previously wedge a job indefinitely; now an attempt that exceeds the timeout is cancelled and handed to the agent's existing checkout retry loop instead of blocking forever. The default is 0, which keeps the current behavior of waiting indefinitely.
• Control submodule cloning per pipeline or step by setting BUILDKITE_GIT_SUBMODULES at the pipeline or step level (or via a hook), without needing to reconfigure your agents.

Artifacts

• Faster downloads from your own S3 buckets. The agent splits each download into parallel ranged requests (a multipart download) instead of fetching the object as a single stream, which significantly speeds up downloads of large artifacts. This is enabled by default; you can opt out with --no-s3-multipart-download (or BUILDKITE_NO_S3_MULTIPART_DOWNLOAD).
• Tunable upload concurrency. Uploads have always run in parallel, but the number of concurrent uploads was fixed to the agent's CPU count, which can be a bottleneck for jobs that upload many small files, where wall time is dominated by per-request latency rather than CPU work. The default is unchanged, so existing agents behave exactly as before, but you can now raise (or lower) the limit with --concurrency on buildkite-agent artifact upload, the BUILDKITE_ARTIFACT_UPLOAD_CONCURRENCY environment variable, or fleet-wide via agent start --artifact-upload-concurrency.

More agent improvements

• More control over if_changed during pipeline upload. The --fetch-diff-base flag (or BUILDKITE_FETCH_DIFF_BASE) ensures the agent fetches the configured diff base before computing the file diff, so the local repository has an up-to-date copy of the base branch. The --changed-files-path flag (or BUILDKITE_CHANGED_FILES_PATH) lets you supply a pre-computed list of changed files instead of relying on git commands to detect changes.
• Hooks can detect job timeouts. The agent sets BUILDKITE_JOB_TIMED_OUT=true in the post-command hook environment when a job hits its timeout, so your hooks can distinguish a timeout from other cancellation reasons and react accordingly (for example, to send a specific notification or clean up differently).
• Fail fast on missing cloud tags with --fail-on-missing-tags (or BUILDKITE_AGENT_FAIL_ON_MISSING_TAGS), which exits the agent on startup if an enabled cloud tag source (EC2, ECS, or GCP) fails to return tags. Without this, an agent that fails to read its instance tags — for example due to cloud API rate limiting — would still connect to Buildkite but be misrouted or unusable for builds that rely on tag-based targeting. Failing fast makes the problem obvious and easy to diagnose.

These improvements rolled out across several agent releases. Update to Buildkite Agent v3.127.2 or later to take advantage of all of them. See the agent release notes for the complete list of changes.