Today we’re introducing the ability to configure multiple GitHub Enterprise Servers.

Some of our bigger teams with different requirements for source control have multiple GitHub Enterprise Server installations. We’ve only supported one per Buildkite organization — until now. If you’re on a compatible plan, you can now add as many different servers as you like from the repository providers page in your organization settings.

We love feedback! If you use this feature and love it, or it doesn’t quite do what you need, drop into our Slack community, or send us an email: hello@buildkite.com 👋

Last week a serious vulnerability (CVE-2022-3786, CVE-2022-3602), was disclosed in the OpenSSL library. We’ve ensured that Buildkite internal systems, and our open source projects, are not vulnerable to this exploit.

We've performed an audit on our internal software and infrastructure, and we have no instances of OpenSSL 3.0.x in use directly or via dependencies, and therefore are not vulnerable to this exploit.

Additionally we've reviewed our open source projects (including the Buildkite Agent and the Elastic CI Stack for AWS) and have verified they also don't have any use of OpenSSL 3.0.x and are not vulnerable to (CVE-2022-3786, CVE-2022-3602).

If you have any further questions please contact support@buildkite.com.

The 3.39.1 version of the buildkite-agent and the 5.11.2 version of the AWS Elastic Stack are now available.

The 3.39.1 version of the buildkite agent is a security update that addresses the security advisory, that was previously disclosed here. This advisory applied to agents running in the same environment as the recently-released bash 5.2.

The 5.11.2 version of the AWS Elastic Stack updates the buildkite-agent to v3.39.1 and also fixes an issue with the date command in the log collector.

For the full list of additions, changes, and fixes, see the buildkite-agent changelog and the elastic-ci-stack-for-aws changelog on GitHub.

UPDATE: Agent 3.39.1 and AWS Elastic Stack for Buildkite 5.11.2 have been released to resolve this issue. We recommend you update to these versions if you are running Bash 5.2.

Bash 5.2 and the Buildkite Agent have a compatibility issue. This issue may reveal the values of environment variables exported by hooks that contain multiple lines. We recommend avoiding updating Bash until the Agent has been updated as well.

This new version of Bash was released 16 days ago. It includes an update that changes how environment variables are exported. When variables contain multiple lines Bash now exports them using $'...' style quoting.

The Buildkite Agent allows using hooks to customise how jobs are run. These hooks are Bash scripts. Hooks can change environment variables, and those changes are propagated into later hooks and commands. This is done by exporting the variables from the Bash script and parsing that output. It doesn't yet understand this new style of quoting, so environment variables with newlines are currently parsing incorrectly, and so are being lost between hooks and commands.

The Agent will print the names of environment variables which are changed by hooks. This new style of quote is mis-parsed, however, and the value may be considered part of the name. This can cause an escaped version of the value to be printed within the job log by mistake.

We're working on a fix to the agent, and should have a release out shortly.

In the meantime, we recommend remaining on Bash 5.1 or lower. You can check your Bash version with bash --version.

If you are running Bash 5.2 already, we recommend auditing your environment variables to see if any secrets contain newlines. If so, we recommend revoking those secrets, and rotating them to new values once you have downgraded Bash, or upgraded the Agent.

Failed builds now jump straight to a new Issues tab on the build page 🗂

Builds with hundreds of jobs can hide what really matters. Instead of scrolling through the noise, Issues shows you only the failed annotations and jobs in a build. This helps you get your code shipping faster. And the rest of your jobs are only a tab click away.

Check out the blog post to learn more about this change.

As always, we'd love your feedback. Drop into our Slack community, or say hi: hello@buildkite.com 👋

Update, Sep 2023: We are no longer offering per pipeline build retention. For more information on retention per plan, see check out the docs.

Today, we're introducing build retention controls for pipelines so you can automatically remove old builds ⏰🧨

You can now configure pipelines to keep builds for a certain period of time. There's also an option to always keep a minimum number of the latest builds, so you don't lose context on pipelines that don't move quite so fast. These are available today — find any pipeline, then go to Settings, Builds, and look for the Build Retention section.

This change is currently optional, but we'll be announcing a broader rollout for build retention soon.

We'd love to hear from you if you find these controls useful, or if they don't quite fit your needs first. Drop into our Slack community, or send us an email: hello@buildkite.com 👋

Today we're rolling out some updates to Buildkite Pipelines that put developer productivity front and center by making failures more bold, and helping you find and fix failures faster:

Pipelines has grown many features over the years, with some of those additions making it harder to identify a failed build and figure out how to fix it. Check out the blog post to learn more about this first step, and our plans to deliver greater context within builds and across builds over time.

We'd love to hear what you think! Whether you reckon this is fresh as, or it’s missed the mark, drop into our Slack community, or send us an email: hello@buildkite.com 👋

We've released a new schema browser for our GraphQL API 🎉

You can now easily browse the entire structure of the API right there in the docs, without needing to log in first

Buildkite agent v3.38.0 and AWS Elastic Stack v5.11.0 are now available!

Agent v3.38.0 adds the ability to trace build jobs using OpenTelemetry. This lets you do all sorts of interesting performance tracking — which jobs are taking the longest, performance and error rate trends, and which job phases are taking up the most time.

Here's a screenshot of OpenTelemetry in action, as viewed from Datadog in waterfall view:

This agent release has been added to the v5.11.0 release of the AWS Elastic Stack, along with the ability to specify which tracing backend to use from the Elastic Stack definition, as well as the ability to specify an arbitrary set of environment variables to start the agent with.

For full list of additions, changes, and fixes, see the agent changelog and the elastic-ci-stack-for-aws changelog on GitHub.

You can now specify default timeouts for command steps ⏰💥

Timeouts can be specified on command steps already, but it's a pain to have to do it every time, and easy to forget. Setting default timeouts means you won't have command steps slip through the cracks and accidentally run forever, or consume too many job minutes.

Default and maximum timeouts can be specified at an organization and pipeline level. Defaults apply in order of specificity — the step's timeout will be used if supplied, otherwise it defaults to the pipeline's default if supplied, or finally to the organization. Maximums are always enforced, when supplied — the smallest value will be used.

These timeouts can be configured from your organization's Pipeline Settings page, or on a pipeline's Builds settings page.

We'd love to hear from you if you find these timeouts useful, or if they don't quite fit your needs first. Drop into our Slack community, or send us an email: hello@buildkite.com 👋

Today we're introducing a new "failing" build state for faster feedback 🏃‍♂️💨

We know it's frustrating when your build fails. But when it does, you want that information as soon as possible so you can get it green.

Now, as soon as a single job fails, the build can be marked as "failing". You'll see a GitHub Commit Status, or a Slack notification — whatever your team uses — as soon as we know your build is going to fail. And on the dashboard your build will show as failing. If you're using annotations or Test Analytics then your build can also tell you what to do to fix the problem while it's still finishing.

But fast feedback is only half of the equation. There's nothing worse than a build which takes forever just to tell you it's broken. If you're in a big team running lots of builds, running the rest of a failing build might be a waste of time. So we're also introducing "cancel on build failing."

1
2
3
4
steps:
  - command: bundle && bin/rails spec
    parallelism: 100
    cancel_on_build_failing: true

Adding the cancel_on_build_failing: true attribute to your command steps will cause those jobs to be canceled as soon as a build becomes failing. That way your agents can move on to the next piece of work. But we'll still send failing notifications before cancelation is complete.

We realise that some folks might need to update API clients or adjust pipeline and workflows, so this new feature is being introduced as opt-in for a smooth migration path. But you can turn on failing builds right now by going to your pipeline on the Buildkite dashboard, then going to the Settings page, Builds, and enabling Fail Fast:

You can also turn this on for your whole organization on your org's Pipeline Settings page.

The buttons will go away and the feature will be enabled for everybody on Monday, 5th September, 2022.

If this does exactly what you need, or you have a suggestion, we'd love to hear from you in our community Slack channel, or drop us an email to support@buildkite.com.

For customers triggering builds via GitHub, we've added a new environment variable to track the draft status of a pull request:

1
BUILDKITE_PULL_REQUEST_DRAFT="true"

If a pull request triggers a build while in a draft state, this variable will be present in all the build's command jobs. The variable will be absent otherwise.

This variable allows customizing your uploaded pipelines or running jobs at the agent level based on a pull request's draft status. For example, you might run a faster subset of your tests while a pull request is still in a draft state.

Check out our docs on environment variables for more information.

If you have any feedback, we'd love to hear from you in our community Slack channel, or drop us an email to support@buildkite.com.

Using the Create Pipeline REST API you can now specify the access level for each associated team 🔏

Previously, new pipelines could be created in teams, but only at the highest access level with the broadest permission. Now that access level can be varied to suit your pipeline and teams. This now matches the dashboard and GraphQL API.

Find out more in our docs about managing pipeline permissions with teams.

If you have any questions or feedback we'd love to hear from you in our community Slack channel, or drop us an email to support@buildkite.com.

Starting August 1st 2022, jobs which are not run within 30 days will automatically expire 🧹

In the past, it's been very easy to have lingering jobs in your Buildkite account which are never assigned an agent, and will never run. Not only does this create unnecessary noise and risk within your account, but it means that Buildkite’s job processing logic needs to handle years-old jobs.

With this change, we've introduced a new job state: expired. This is similar to the canceled state, and once a job is transitioned to this state, the build will fail.

This will be enabled for everyone on Monday, 1st August 2022, but you can opt in today at an organisation level, or a per-pipeline level, to start testing and verifying that it works with your own builds. Once enabled, jobs older than 30 days that haven't been run by an agent will be automatically transitioned to expired and their builds cancelled. This new state will also appear in the REST and GraphQL APIs.

To enable this today, see the "Job Expiry" section in your organization's pipelines settings page, or each pipeline's Pipeline Settings > Builds page:

If you have any questions or feedback we'd love to hear from you in our community Slack channel, or drop us an email to support@buildkite.com.

After releasing Test Analytics, we've been working on improving the navigation bar in the Docs to make it easier for you to find and read docs on both Pipelines and Test Analytics.

This change and other recent UI and UX improvements are already live in the docs.

Using %n in agent names is deprecated and will soon be removed 👋

The default name for a Buildkite Agent used to be agent-%n, resulting in agent-1, agent-2, etc. Pretty quickly, though, this gets tricky to keep unique, especially with a big database full of agent names. So to keep our database humming, we're removing support for %n.

We began this process with Buildkite Agent v3.27.0 released on 8th February, 2021, removing %n from the default agent name. Elastic CI Stack for AWS v5.2.0 was also updated to replace usage of %n with %spawn on the same day.

If you're using Elastic CI Stack for AWS, please make sure you're running v5.2.0 or newer. The version you're using should be visible on your agents page:

)

If you're running your agents another way, please review your agent configuration and confirm you're not using %n.

If you are using %n, there are a few alternatives:

• If you're running a single agent on each host, use %hostname.
• If you're running multiple agents, use the spawn option and %hostname-%spawn.
• If neither of the previous options work, use %random which will turn into a few random characters.

Check out our docs for a complete reference: https://buildkite.com/docs/agent/v3/configuration#name

We'll begin brownouts of %n on Monday, 8th August 2022. Finally, we'll remove support for %n on Monday, 5th September 2022. Agents which still use %n in their name after this date will fail to connect and accept work.

If you need some more time, or have a problem completing this migration, please drop us an email to support@buildkite.com.

For teams that have a large number of connected agents, we’ve added a new filter to the Agents page so you can quickly find which ones are busy working on jobs 🕵️‍♀️🕵️‍♀️🕵️‍♀️

We hope this makes it easier to find and interact with agents which are running jobs in your organization.

If you have any feedback we'd love to hear from you in our community Slack channel, or drop us an email to support@buildkite.com.

We know how frustrating finding and fixing flaky tests can be. And how costly your test suite performance degradations can become over time. We’ve been working to solve these problems for good, with our newest product — Buildkite Test Analytics.

As of today, Test Analytics is out of beta and available to everyone 🎉

• Automatically identify flaky tests, see what is causing the most disruption for your team, and get a head-start on fixing them for good.
• Get deep visibility and tracing for your test suite, integrated with your programming language and test framework.
• Set targets and identify problems as soon as they occur with configurable speed and reliability monitors.

Test Analytics is included in all Buildkite plans and is free to get started. You can even use it with any CI system, including GitHub Actions, CircleCI, and Jenkins ⚡️

Read more on the website, in our documentation — and if you're super excited, retweet!

Buildkite Agent v3.36.1 and the AWS Elastic Stack v5.9.0 are now available! 🎉

This agent version ships with experimental support for tracing CI runs through OpenTelemetry, as well as improvements to logging, and an experimental file locking system that should unlock more reliably when the agent hasn't shut down cleanly.

This agent release has been added to the v5.9.0 release of the elastic stack, which also:

• Adds ability to fetch EC2 instance tags via Instance Metadata
• Updates the Linux Kernel on elastic stack instances from 4.14 to 5.10
• Adds an option to enable EC2 Detailed Instance Monitoring

For full list of additions, changes, and fixes, see the buildkite-agent changelog and the elastic-ci-stack-for-aws changelog on GitHub.

We're changing the $BUILDKITE_PULL_REQUEST_REPO environment variable value supplied for GitHub and GitHub Enterprise repositories from the unauthenticated git protocol to https 🔒

GitHub announced some time ago that they are removing the unauthenticated git protocol. This change has been in effect since 15th March 2022. Now we're modifying how we generate this environment variable to match their change.

$BUILDKITE_PULL_REQUEST_REPO is not used by the Buildkite Agent to clone your repositories. The value is only provided as a reference, and is particularly useful for pull requests from repository forks. Some customers use this value to ensure that pull requests from forks come from trusted sources, for example.

We recommend reviewing your agent hooks and making sure any security rules that utilise this value are adjusted to be agnostic to the protocol used, and are at least able to handle https.

From Monday, 20th June 2022, all new builds will use a https:// protocol URL for $BUILDKITE_PULL_REQUEST_REPO. If you need a little more time, or would like this change to take effect earlier for your organization, please reach out via support@buildkite.com.