For teams that have a large number of connected agents, we’ve added a new filter to the Agents page so you can quickly find which ones are busy working on jobs 🕵️‍♀️🕵️‍♀️🕵️‍♀️

We hope this makes it easier to find and interact with agents which are running jobs in your organization.

If you have any feedback we'd love to hear from you in our community Slack channel, or drop us an email to support@buildkite.com.

We know how frustrating finding and fixing flaky tests can be. And how costly your test suite performance degradations can become over time. We’ve been working to solve these problems for good, with our newest product — Buildkite Test Analytics.

As of today, Test Analytics is out of beta and available to everyone 🎉

• Automatically identify flaky tests, see what is causing the most disruption for your team, and get a head-start on fixing them for good.
• Get deep visibility and tracing for your test suite, integrated with your programming language and test framework.
• Set targets and identify problems as soon as they occur with configurable speed and reliability monitors.

Test Analytics is included in all Buildkite plans and is free to get started. You can even use it with any CI system, including GitHub Actions, CircleCI, and Jenkins ⚡️

Read more on the website, in our documentation — and if you're super excited, retweet!

Buildkite Agent v3.36.1 and the AWS Elastic Stack v5.9.0 are now available! 🎉

This agent version ships with experimental support for tracing CI runs through OpenTelemetry, as well as improvements to logging, and an experimental file locking system that should unlock more reliably when the agent hasn't shut down cleanly.

This agent release has been added to the v5.9.0 release of the elastic stack, which also:

• Adds ability to fetch EC2 instance tags via Instance Metadata
• Updates the Linux Kernel on elastic stack instances from 4.14 to 5.10
• Adds an option to enable EC2 Detailed Instance Monitoring

For full list of additions, changes, and fixes, see the buildkite-agent changelog and the elastic-ci-stack-for-aws changelog on GitHub.

We're changing the $BUILDKITE_PULL_REQUEST_REPO environment variable value supplied for GitHub and GitHub Enterprise repositories from the unauthenticated git protocol to https 🔒

GitHub announced some time ago that they are removing the unauthenticated git protocol. This change has been in effect since 15th March 2022. Now we're modifying how we generate this environment variable to match their change.

$BUILDKITE_PULL_REQUEST_REPO is not used by the Buildkite Agent to clone your repositories. The value is only provided as a reference, and is particularly useful for pull requests from repository forks. Some customers use this value to ensure that pull requests from forks come from trusted sources, for example.

We recommend reviewing your agent hooks and making sure any security rules that utilise this value are adjusted to be agnostic to the protocol used, and are at least able to handle https.

From Monday, 20th June 2022, all new builds will use a https:// protocol URL for $BUILDKITE_PULL_REQUEST_REPO. If you need a little more time, or would like this change to take effect earlier for your organization, please reach out via support@buildkite.com.

The 5.8.0 version of the AWS elastic stack is now available. 🚀

This release added:

• Ability to customise docker address pools to use more, slightly smaller networks rather than a few big ones
• Support for additional ARM/Graviton instance types: c7g, g5g, lm4gn, lm4gen, and x2gd
• SecretsBucketRegion parameter and updated s3secrets-hooks
• Docs on updating the different components #957 (@keithduncan)

It also fixed:

• Overwrite /usr/bin/buildkite-agent symlink if it already exists

For full list of additions, changes, and fixes, see the elastic-ci-stack-for-aws changelog on GitHub.

The 3.35.0 version of the buildkite-agent is now available. 🚀

This release has added:

• An option to skip updating the mirror when using git mirrors. Useful when git is mounted from an external volume
• The more secure SHA256 hashing algorithm alongside SHA1 when working with artifacts
• Additional security when creating directories, making them only accessible by current user and group

For full list of additions, changes, and fixes, see the buildkite-agent changelog on GitHub.

As announced in 2019, Schedules no longer need a user 🏃🏼‍♂️💨

Schedules created before then and not manually migrated have now had their build ownership user removed. Builds created from those schedules will no longer have a creator, which may affect trigger step permission, build.creator conditionals, and $BUILDKITE_BUILD_CREATOR environment variable checks.

Schedules created since the 2019 announcement are unaffected, as they never had a build ownership user.

We've added Build Matrix so you can create multiple jobs from a single, multi-variant step definition. 🧮

Build Matrices can also include multiple dimensions, different combinations with the adjustments key, and matrix jobs can be grouped together to clean up build pages. ⚡️

Read more about it in the blog, documentation, or on Twitter.

The 3.34.0 version of the buildkite-agent is now available. 🚀

This release has added:

• a new combination flag: spawn-with-priority
• locked down file permissions on Windows
• increased security by rejecting pipeline uploads containing redacted vars

For full list of additions, changes, and fixes, see the buildkite-agent changelog on GitHub.

We've added a new group pipeline step type so you can group steps together, set dependencies between them, reduce YAML repetition, and clean up your build pages 🏗✨

You can read more about it in the blog post, documentation, or on Twitter.

API Access Tokens can be restricted to allow access only from specific Allowed IP Addresses. Those restrictions have been honoured by the REST API, but not by the GraphQL API — until now. We've made sure these restrictions are also applied to GraphQL requests.

Check out the API Access Token documentation and configure your tokens on the API Access Tokens page.

Last week a serious vulnerability, CVE-2021-44228, was disclosed in the Java-based logging package Log4j. We’ve ensured that Buildkite internal systems, and our open source projects, are not vulnerable to this exploit.

We've performed an audit on our internal software and infrastructure, and we have no instances of Log4j in use directly or via dependencies, and therefore are not vulnerable to this exploit. Additionally we've reviewed our open source projects (including the Buildkite Agent and the Elastic CI Stack for AWS) and have verified they also don't have any use of Log4j and are not vulnerable to CVE-2021-44228.

We use a number of services from AWS and other cloud vendors, and are actively monitoring them to validate that they are not vulnerable and take any necessary mitigation.

If you haven't already, we also recommend updating any use of Log4j within your own build tooling.

If you have any further questions please contact support@buildkite.com.

The 5.7.2 version of the AWS elastic stack is now available. 🚀

This release includes:

• Upgrade Docker for Linux (20.10.9) and Windows (20.10.7)
• Upgrade docker-compose for Linux (1.29.2) and Windows (1.29.2)

It also fixes:

• BuildkiteAgentTokenParameterStorePath support for AWS Secrets Manager SSM references

For full list of additions, changes, and fixes, see the elastic-ci-stack-for-aws changelog on GitHub.

For teams using Datadog, we've recently made it easier to send information about your Buildkite pipelines to Datadog’s Continuous Integration Visibility. This is a simple integration that any organization using both Datadog and Buildkite can enable to get insights into their pipeline’s performance over time. 📈

For more details on the integration check out the documentation 📚

For teams that run Buildkite Pipelines and build Xcode based software projects for macOS, iOS, iPadOS, tvOS, and watchOS apps, you can now run your Buildkite Builds on AWS EC2 Mac instances using a CloudFormation template. 🎉

This experimental template creates an Auto Scaling group, launch template, and host resource group to launch a pool of EC2 Mac instances that run the Buildkite Agent.

📣 Big shoutout to Buildkite customer Oliver Koo for his early input into this feature 🙏

📚 For details on how to prepare and deploy this template to your AWS Account, checkout the Auto Scaling EC2 Mac documentation, or jump straight into Elastic CI Stack for EC2 Mac.

Identify, track, and fix problematic tests with ✨Test Analytics✨, now in private beta.

🔍 Identify flaky tests: See which tests are most disruptive with automatic flaky test identification and analysis over time.

👀 Monitor speed & reliability: Get alerts when slow tests are introduced.

📈 Deeply analyze performance: First-class framework integrations provide in-depth performance analysis, helping you find external dependency bottlenecks and more.

🏎 View real-time results: Watch your test suite run in real-time and get immediate performance insights.

Learn more and join our waitlist at https://buildkite.com/test-analytics.

In November 2018, we posted a changelog deprecating BUILDBOX_* environment variables from generating for new jobs.

From today, we no longer send BUILDBOX_* environment variables.

You can see our environment variable documentation for a complete list of current job environment variables.

The 3.33.3 version of the buildkite-agent and the 5.7.0 version of the AWS elastic stack are now available. 🚀

The 3.33.3 Agent release has added:

• Support for unset environment variables in Job Lifecycle Hooks

The 5.7.0 Elastic Stack release has added:

• Support for storing builds, git-mirrors, and Docker data on NVMe Instance Storage
• Retried login for ECR and generic Docker registries
• Experimental CloudFormation service role, listing the IAM Actions required to create, update, and delete the template
• A README feature matrix for Linux and Windows
• qemu and binfmt hooks for cross-architecture Docker image builds
• Support for AWS SSM sessions

For full list of additions, changes, and fixes, see the buildkite-agent changelog and the elastic-ci-stack-for-aws changelog on GitHub.

For teams running a lot of jobs who want more control over what order their jobs run in, we've introduced the Job Priority attribute to specify its priority within its queue. 🙋🏻‍♀️

It's a new attribute defined on a step in the pipeline configuration that changes how the dispatcher assigns work to agents. The higher the value you set for priority, the sooner a job will be prioritised. 🔝

Our recently released Eager Concurrency feature can also be used in conjunction with Job Priority.

Enterprise teams using Okta or Azure AD for SSO with their Buildkite organization can now optionally enable SCIM to automatically deprovision users ⚡️

Okta and Azure AD SSO with SAML is still available to all users.

A step-by-step guide to enabling SCIM support is available in our Single Sign-On with Okta and Azure AD documentation 📚