We've just shipped new features that'll help uplevel your build and test workflows with Buildkite, including some key announcements:

• Kubernetes agent stack
• OpenID Connect (OIDC) support
• flaky test tracker for automatically detecting flaky tests

Learn more about what's new in our March 2023 release ✨

We've updated the Build UI to be more intuitive with the following improvements:

• The issues tab is called the failures tab, making it easier for you to view failed jobs
• The organization name and avatar are now seen on each Build page, to make it easier view the organization you're editing settings for
• Block steps within groups are now easier to identify in the UI with a thickened purple border
• Pipelines can be bookmarked instead of starred
• Build summary now has a rebuild option, rather than edit steps and new build options
• You can now navigate to all builds by clicking on the builds icon
• Spacing and font fixes

It's also easier to make your pipelines stand out with:

• Pipeline avatars in place of build stats
• Configuration of pipeline color and emoji through pipeline settings

The latest agent release includes the job-api experiment, which enables an HTTP API within the agent, allowing jobs to inspect and mutate their environment without using bash. This is a big step towards supporting hooks and plugins in other languages.

Other experimental features include:

• allowing spawning agents with a descending priority, rather than the default ascending priority
• artifacts can now be uploaded to S3 compatible endpoints such as MinIO.

Other updates to this release include:

• the AWS Elastic Stack has been updated with a newer version of buildx
• the managed secrets buckets created by the stack now have public ACLs explicitly blocked

For a full list of additions, changes, fixes, and more details, see the buildkite-agent changelog and the elastic-ci-stack-for-aws changelog on GitHub.

The 3.44.0 version of the buildkite-agent and the 5.17.0 version of the AWS elastic stack are now available.

Major updates to the agent include:

• New experiment for running jobs under Kubernetes
• Docker images based on Ubuntu 22.04
• The ability to add claims to OIDC token requests

This agent release has been added to the 5.17.0 release of the elastic stack, as well as support for c7gn, m7g, and r7g instance type classes, and updates to Docker, Docker Compose, buildx, git, and the Linux kernel.

For a full list of additions, changes, and fixes, see the buildkite-agent changelog and the elastic-ci-stack-for-aws changelog on GitHub.

Update: This change has now been completed.

Over the past two years, GitHub, GitLab, and other Git services have updated their default branch names from "master" to "main" – you can read more about the motivation for the change in this statement from the Software Freedom Conservancy.

In line with this change, we will be updating the default branch for newly created pipelines from "master" to "main" from April 1st, 2023. This will not impact existing pipelines.

You can configure a given pipeline's default branch through the user interface, as well as the REST API and GraphQL.

If you would like to retain "master" as the default branch of new pipelines, you may set an organization-level default branch in Pipeline Settings, which will then be used for new pipelines:

If you've ever thought "I need more purple in my life" then we've got some exciting news for you... 💜

We've made some changes to make navigation as simple and obvious as possible across all parts of the product. We've introduced a more familiar navigation pattern to Test Analytics, updated the global nav to clearly identify where you are in the app, and introduced a new visual style to the sidebar. It's a coat of paint, but also a foundation for the features and improvements you can expect in the coming months.

We're focused on the next-generation of Buildkite—from small improvements to whole new products and experience–so you can consider this just a taste of what's to come.

Read the full post

We've updated the navigation in the documentation to help you find the content you need quickly. 🎉

These changes include:

• Having the on-page navigation highlight which section you're reading.
• Revamping the style of the global navigation bar to a single row.
• Focusing on up-to-date and high-value pages like security while deemphasizing deprecated content.
• Highlighting the Overview page and Getting started guide for Pipelines.
• Updating the typography to improve readability.

Try out the new navigation

Pipelines now considers your current branch when surfacing options for a new build. This removes the friction of a new build defaulting to main.

This has been a highly-requested feature from our customers, and we’re so excited to share that the change is now live.

The v3.43.1 version of the buildkite-agent and the v5.16.1 version of the AWS elastic stack are now available.

Agent v3.43.1 fixes a bug introduced in v3.43.0 where jobs running on agents in --acquire-job mode will fail after about ~4.5 minutes. Agents running in non-acquire mode are not affected, and most of our customers don't use --acquire-job mode.

AWS Elastic Stack v5.16.1 upgrades the agent to this new version. Most elastic stack installations will not be affected by the above agent bug, but we still recommend upgrading to the latest version.

For a full list of additions, changes, and fixes, see the buildkite-agent changelog and the elastic-ci-stack-for-aws changelog on GitHub.

The 3.43.0 version of the buildkite-agent and the 5.16.0 version of the AWS elastic stack are now available.

This agent release fixes a panic that was accidentally added in v3.42.0, adds a flag to allow empty results with doing an artifact search, and Docker images for linux/arm64. Also, start --acquire-job now retries while the job is locked.

This agent release has been added to the 5.16.0 release of the elastic stack, which also includes Git v2.39.1 to address recent security vulnerabilities in Git, and now pushes access logs to Cloudwatch for Linux instances.

Two weeks ago we released the 3.42.0 version of the agent and the 5.15.0 version of the AWS elastic stack. This agent has added an experimental inbuilt status page, and a new agent-startup hook. The 5.15.0 release of the elastic stack added a parameter to enable server side encryption on s3 buckets created in the stack.

For a full list of additions, changes, and fixes, see the buildkite-agent changelog and the elastic-ci-stack-for-aws changelog on GitHub.

Customers using SSO authentication can now easily configure the SSO session duration via the settings UI–enabling greater control over your security posture.

Once the specified duration elapses, the user session will expire requiring re-authentication to access Buildkite.

Learn more about how to set up your SSO session duration

We’ve released the Developer plan, which replaces the legacy Free plan. The Developer plan continues to be free-of-charge, with the following updates:

• Continue to use the Buildkite platform to build, test, and deploy software at scale, with your team of up to 3 users
• Retain your builds for up to 30 days

You can learn more about our plans on our pricing page.

The 3.41.0 version of the buildkite-agent and the 5.14.0 version of the AWS elastic stack are now available.

The agent has added the ability to set a name for tracing, and experimental support to request OIDC tokens from the Buildkite backend. This agent release has been added to the 5.14.0 release of the elastic stack, as well as a parameter to encrypt EBS volumes and, "Desired Capacity" has been added to the list of Cloudwatch metrics collected from the stack's Autoscaling Group.

For a full list of additions, changes, and fixes, see the buildkite-agent changelog and the elastic-ci-stack-for-aws changelog on GitHub.

The 3.40.0 version of the buildkite-agent and the 5.13.0 version of the AWS elastic stack are now available.

This release adds agent binaries for Windows/arm64, and fixes a few minor security issues. This agent release has been added to the release of the elastic stack.

For full list of additions, changes, and fixes, see the buildkite-agent changelog and the elastic-ci-stack-for-aws changelog on GitHub.

Last week a serious vulnerability (CVE-2022-3786, CVE-2022-3602), was disclosed in the OpenSSL library. We’ve ensured that Buildkite internal systems, and our open source projects, are not vulnerable to this exploit.

We've performed an audit on our internal software and infrastructure, and we have no instances of OpenSSL 3.0.x in use directly or via dependencies, and therefore are not vulnerable to this exploit.

Additionally we've reviewed our open source projects (including the Buildkite Agent and the Elastic CI Stack for AWS) and have verified they also don't have any use of OpenSSL 3.0.x and are not vulnerable to (CVE-2022-3786, CVE-2022-3602).

If you have any further questions please contact support@buildkite.com.

The 3.39.1 version of the buildkite-agent and the 5.11.2 version of the AWS Elastic Stack are now available.

The 3.39.1 version of the buildkite agent is a security update that addresses the security advisory, that was previously disclosed here. This advisory applied to agents running in the same environment as the recently-released bash 5.2.

The 5.11.2 version of the AWS Elastic Stack updates the buildkite-agent to v3.39.1 and also fixes an issue with the date command in the log collector.

For the full list of additions, changes, and fixes, see the buildkite-agent changelog and the elastic-ci-stack-for-aws changelog on GitHub.

UPDATE: Agent 3.39.1 and AWS Elastic Stack for Buildkite 5.11.2 have been released to resolve this issue. We recommend you update to these versions if you are running Bash 5.2.

Bash 5.2 and the Buildkite Agent have a compatibility issue. This issue may reveal the values of environment variables exported by hooks that contain multiple lines. We recommend avoiding updating Bash until the Agent has been updated as well.

This new version of Bash was released 16 days ago. It includes an update that changes how environment variables are exported. When variables contain multiple lines Bash now exports them using $'...' style quoting.

The Buildkite Agent allows using hooks to customise how jobs are run. These hooks are Bash scripts. Hooks can change environment variables, and those changes are propagated into later hooks and commands. This is done by exporting the variables from the Bash script and parsing that output. It doesn't yet understand this new style of quoting, so environment variables with newlines are currently parsing incorrectly, and so are being lost between hooks and commands.

The Agent will print the names of environment variables which are changed by hooks. This new style of quote is mis-parsed, however, and the value may be considered part of the name. This can cause an escaped version of the value to be printed within the job log by mistake.

We're working on a fix to the agent, and should have a release out shortly.

In the meantime, we recommend remaining on Bash 5.1 or lower. You can check your Bash version with bash --version.

If you are running Bash 5.2 already, we recommend auditing your environment variables to see if any secrets contain newlines. If so, we recommend revoking those secrets, and rotating them to new values once you have downgraded Bash, or upgraded the Agent.

Today we're rolling out some updates to Buildkite Pipelines that put developer productivity front and center by making failures more bold, and helping you find and fix failures faster:

Pipelines has grown many features over the years, with some of those additions making it harder to identify a failed build and figure out how to fix it. Check out the blog post to learn more about this first step, and our plans to deliver greater context within builds and across builds over time.

We'd love to hear what you think! Whether you reckon this is fresh as, or it’s missed the mark, drop into our Slack community, or send us an email: hello@buildkite.com 👋

Buildkite agent v3.38.0 and AWS Elastic Stack v5.11.0 are now available!

Agent v3.38.0 adds the ability to trace build jobs using OpenTelemetry. This lets you do all sorts of interesting performance tracking — which jobs are taking the longest, performance and error rate trends, and which job phases are taking up the most time.

Here's a screenshot of OpenTelemetry in action, as viewed from Datadog in waterfall view:

This agent release has been added to the v5.11.0 release of the AWS Elastic Stack, along with the ability to specify which tracing backend to use from the Elastic Stack definition, as well as the ability to specify an arbitrary set of environment variables to start the agent with.

For full list of additions, changes, and fixes, see the agent changelog and the elastic-ci-stack-for-aws changelog on GitHub.

Using the Create Pipeline REST API you can now specify the access level for each associated team 🔏

Previously, new pipelines could be created in teams, but only at the highest access level with the broadest permission. Now that access level can be varied to suit your pipeline and teams. This now matches the dashboard and GraphQL API.

Find out more in our docs about managing pipeline permissions with teams.

If you have any questions or feedback we'd love to hear from you in our community Slack channel, or drop us an email to support@buildkite.com.