UPDATE: Agent 3.39.1 and AWS Elastic Stack for Buildkite 5.11.2 have been released to resolve this issue. We recommend you update to these versions if you are running Bash 5.2.

Bash 5.2 and the Buildkite Agent have a compatibility issue. This issue may reveal the values of environment variables exported by hooks that contain multiple lines. We recommend avoiding updating Bash until the Agent has been updated as well.

This new version of Bash was released 16 days ago. It includes an update that changes how environment variables are exported. When variables contain multiple lines Bash now exports them using $'...' style quoting.

The Buildkite Agent allows using hooks to customise how jobs are run. These hooks are Bash scripts. Hooks can change environment variables, and those changes are propagated into later hooks and commands. This is done by exporting the variables from the Bash script and parsing that output. It doesn't yet understand this new style of quoting, so environment variables with newlines are currently parsing incorrectly, and so are being lost between hooks and commands.

The Agent will print the names of environment variables which are changed by hooks. This new style of quote is mis-parsed, however, and the value may be considered part of the name. This can cause an escaped version of the value to be printed within the job log by mistake.

We're working on a fix to the agent, and should have a release out shortly.

In the meantime, we recommend remaining on Bash 5.1 or lower. You can check your Bash version with bash --version.

If you are running Bash 5.2 already, we recommend auditing your environment variables to see if any secrets contain newlines. If so, we recommend revoking those secrets, and rotating them to new values once you have downgraded Bash, or upgraded the Agent.

Today we're rolling out some updates to Buildkite Pipelines that put developer productivity front and center by making failures more bold, and helping you find and fix failures faster:

Pipelines has grown many features over the years, with some of those additions making it harder to identify a failed build and figure out how to fix it. Check out the blog post to learn more about this first step, and our plans to deliver greater context within builds and across builds over time.

We'd love to hear what you think! Whether you reckon this is fresh as, or it’s missed the mark, drop into our Slack community, or send us an email: hello@buildkite.com 👋

Buildkite agent v3.38.0 and AWS Elastic Stack v5.11.0 are now available!

Agent v3.38.0 adds the ability to trace build jobs using OpenTelemetry. This lets you do all sorts of interesting performance tracking — which jobs are taking the longest, performance and error rate trends, and which job phases are taking up the most time.

Here's a screenshot of OpenTelemetry in action, as viewed from Datadog in waterfall view:

This agent release has been added to the v5.11.0 release of the AWS Elastic Stack, along with the ability to specify which tracing backend to use from the Elastic Stack definition, as well as the ability to specify an arbitrary set of environment variables to start the agent with.

For full list of additions, changes, and fixes, see the agent changelog and the elastic-ci-stack-for-aws changelog on GitHub.

Using the Create Pipeline REST API you can now specify the access level for each associated team 🔏

Previously, new pipelines could be created in teams, but only at the highest access level with the broadest permission. Now that access level can be varied to suit your pipeline and teams. This now matches the dashboard and GraphQL API.

Find out more in our docs about managing pipeline permissions with teams.

If you have any questions or feedback we'd love to hear from you in our community Slack channel, or drop us an email to support@buildkite.com.

Starting August 1st 2022, jobs which are not run within 30 days will automatically expire 🧹

In the past, it's been very easy to have lingering jobs in your Buildkite account which are never assigned an agent, and will never run. Not only does this create unnecessary noise and risk within your account, but it means that Buildkite’s job processing logic needs to handle years-old jobs.

With this change, we've introduced a new job state: expired. This is similar to the canceled state, and once a job is transitioned to this state, the build will fail.

This will be enabled for everyone on Monday, 1st August 2022, but you can opt in today at an organisation level, or a per-pipeline level, to start testing and verifying that it works with your own builds. Once enabled, jobs older than 30 days that haven't been run by an agent will be automatically transitioned to expired and their builds cancelled. This new state will also appear in the REST and GraphQL APIs.

To enable this today, see the "Job Expiry" section in your organization's pipelines settings page, or each pipeline's Pipeline Settings > Builds page:

If you have any questions or feedback we'd love to hear from you in our community Slack channel, or drop us an email to support@buildkite.com.

After releasing Test Analytics, we've been working on improving the navigation bar in the Docs to make it easier for you to find and read docs on both Pipelines and Test Analytics.

This change and other recent UI and UX improvements are already live in the docs.

For teams that have a large number of connected agents, we’ve added a new filter to the Agents page so you can quickly find which ones are busy working on jobs 🕵️‍♀️🕵️‍♀️🕵️‍♀️

We hope this makes it easier to find and interact with agents which are running jobs in your organization.

If you have any feedback we'd love to hear from you in our community Slack channel, or drop us an email to support@buildkite.com.

Buildkite Agent v3.36.1 and the AWS Elastic Stack v5.9.0 are now available! 🎉

This agent version ships with experimental support for tracing CI runs through OpenTelemetry, as well as improvements to logging, and an experimental file locking system that should unlock more reliably when the agent hasn't shut down cleanly.

This agent release has been added to the v5.9.0 release of the elastic stack, which also:

• Adds ability to fetch EC2 instance tags via Instance Metadata
• Updates the Linux Kernel on elastic stack instances from 4.14 to 5.10
• Adds an option to enable EC2 Detailed Instance Monitoring

For full list of additions, changes, and fixes, see the buildkite-agent changelog and the elastic-ci-stack-for-aws changelog on GitHub.

We're changing the $BUILDKITE_PULL_REQUEST_REPO environment variable value supplied for GitHub and GitHub Enterprise repositories from the unauthenticated git protocol to https 🔒

GitHub announced some time ago that they are removing the unauthenticated git protocol. This change has been in effect since 15th March 2022. Now we're modifying how we generate this environment variable to match their change.

$BUILDKITE_PULL_REQUEST_REPO is not used by the Buildkite Agent to clone your repositories. The value is only provided as a reference, and is particularly useful for pull requests from repository forks. Some customers use this value to ensure that pull requests from forks come from trusted sources, for example.

We recommend reviewing your agent hooks and making sure any security rules that utilise this value are adjusted to be agnostic to the protocol used, and are at least able to handle https.

From Monday, 20th June 2022, all new builds will use a https:// protocol URL for $BUILDKITE_PULL_REQUEST_REPO. If you need a little more time, or would like this change to take effect earlier for your organization, please reach out via support@buildkite.com.

The 5.8.0 version of the AWS elastic stack is now available. 🚀

This release added:

• Ability to customise docker address pools to use more, slightly smaller networks rather than a few big ones
• Support for additional ARM/Graviton instance types: c7g, g5g, lm4gn, lm4gen, and x2gd
• SecretsBucketRegion parameter and updated s3secrets-hooks
• Docs on updating the different components #957 (@keithduncan)

It also fixed:

• Overwrite /usr/bin/buildkite-agent symlink if it already exists

For full list of additions, changes, and fixes, see the elastic-ci-stack-for-aws changelog on GitHub.

The 3.35.0 version of the buildkite-agent is now available. 🚀

This release has added:

• An option to skip updating the mirror when using git mirrors. Useful when git is mounted from an external volume
• The more secure SHA256 hashing algorithm alongside SHA1 when working with artifacts
• Additional security when creating directories, making them only accessible by current user and group

For full list of additions, changes, and fixes, see the buildkite-agent changelog on GitHub.

As announced in 2019, Schedules no longer need a user 🏃🏼‍♂️💨

Schedules created before then and not manually migrated have now had their build ownership user removed. Builds created from those schedules will no longer have a creator, which may affect trigger step permission, build.creator conditionals, and $BUILDKITE_BUILD_CREATOR environment variable checks.

Schedules created since the 2019 announcement are unaffected, as they never had a build ownership user.

The 3.34.0 version of the buildkite-agent is now available. 🚀

This release has added:

• a new combination flag: spawn-with-priority
• locked down file permissions on Windows
• increased security by rejecting pipeline uploads containing redacted vars

For full list of additions, changes, and fixes, see the buildkite-agent changelog on GitHub.

API Access Tokens can be restricted to allow access only from specific Allowed IP Addresses. Those restrictions have been honoured by the REST API, but not by the GraphQL API — until now. We've made sure these restrictions are also applied to GraphQL requests.

Check out the API Access Token documentation and configure your tokens on the API Access Tokens page.

Last week a serious vulnerability, CVE-2021-44228, was disclosed in the Java-based logging package Log4j. We’ve ensured that Buildkite internal systems, and our open source projects, are not vulnerable to this exploit.

We've performed an audit on our internal software and infrastructure, and we have no instances of Log4j in use directly or via dependencies, and therefore are not vulnerable to this exploit. Additionally we've reviewed our open source projects (including the Buildkite Agent and the Elastic CI Stack for AWS) and have verified they also don't have any use of Log4j and are not vulnerable to CVE-2021-44228.

We use a number of services from AWS and other cloud vendors, and are actively monitoring them to validate that they are not vulnerable and take any necessary mitigation.

If you haven't already, we also recommend updating any use of Log4j within your own build tooling.

If you have any further questions please contact support@buildkite.com.

The 5.7.2 version of the AWS elastic stack is now available. 🚀

This release includes:

• Upgrade Docker for Linux (20.10.9) and Windows (20.10.7)
• Upgrade docker-compose for Linux (1.29.2) and Windows (1.29.2)

It also fixes:

• BuildkiteAgentTokenParameterStorePath support for AWS Secrets Manager SSM references

For full list of additions, changes, and fixes, see the elastic-ci-stack-for-aws changelog on GitHub.

The 3.33.3 version of the buildkite-agent and the 5.7.0 version of the AWS elastic stack are now available. 🚀

The 3.33.3 Agent release has added:

• Support for unset environment variables in Job Lifecycle Hooks

The 5.7.0 Elastic Stack release has added:

• Support for storing builds, git-mirrors, and Docker data on NVMe Instance Storage
• Retried login for ECR and generic Docker registries
• Experimental CloudFormation service role, listing the IAM Actions required to create, update, and delete the template
• A README feature matrix for Linux and Windows
• qemu and binfmt hooks for cross-architecture Docker image builds
• Support for AWS SSM sessions

For full list of additions, changes, and fixes, see the buildkite-agent changelog and the elastic-ci-stack-for-aws changelog on GitHub.

The 3.32.3 version of the buildkite-agent and the 5.6.0 version of the AWS elastic stack are now available.

The 3.32.3 Agent release has added:

• Support for cross-region artifact buckets
• Improved error logging around AWS Credentials
• Logging to the artifact upload command to say where artifacts are being sent

The 5.6.0 Elastic Stack release has added:

• Cross-region secrets bucket support to git-credentials-s3-secrets
• AssumeRole support in the ECR Login plug-in

For full list of additions, changes, and fixes, see the buildkite-agent changelog and the elastic-ci-stack-for-aws changelog on GitHub.

From 1 October 2021, build artifacts hosted by Buildkite will be retained for six months from time of upload, after which they will be deleted. Artifacts uploaded before 1 April 2021 will also be deleted at this time.

Previously, build artifacts were retained indefinitely, which means we're currently storing over 7PB of data 🤯📈

Custom-hosted build artifacts are not affected by this change, and remain available to any customer who wants more control over their retention.

As always, you can reach out to us with any questions about this change.

We've released v5.5.0 of the Elastic Stack CI for AWS ☁️

Included in this release:

• Template validation rules for the Buildkite Agent token
• Secret redaction in build logs
• Support for the pre-bootstrap Buildkite Agent lifecycle hook

You can read the full release notes on the v5.5.0 release on GitHub.