Managing Log Output
Buildkite uses our open-source terminal Golang library to provide you with the best possible terminal rendering experience for your build logs, including ANSI terminal emulation to ensure spinners, progress bars, colors and emojis are rendered beautifully.
On this page:
You can group and collapse your build output by echoing
--- [group name] in your build output.
echo "--- A section of the build"
If you want to have the group open by default, use
+++ instead of
echo "+++ A section of the build"
If you'd like to open the previously defined group, use
^^^ +++. This is useful if a command within a group fails, and you'd like to have the group already open when you view the log.
echo "--- Bundling" bundle if [[ $? -ne 0 ]]; then echo "^^^ +++" echo "Bundler failed, oh no!!" fi
You can even include colors and emojis!
echo -e "--- Running \033[33mspecs\033[0m :cow::bell:"
Log output limits
If your build output exceeds 1MB then we'll only show the last 1MB of it in the rendered terminal output on your build page. In addition, your log file must not exceed 100MB else it may fail to upload.
If your log exceeds 1MB then we highly recommend reconfiguring your build tools to filter out unnecessary lines. Sometimes this isn't always possible, so you can use the below techniques to store and filter your log.
Storing the original log
One method for storing the original log is the Unix
tee command. It allows you to store the output stream of a command to a file and passing it straight through unchanged to the next command.
When this script is run it will store the original output of
your_build_command to the file
To store this file alongside your build, add the
artifact_paths attribute to the command step running your script:
When your build is finished the agent will upload
build.log as a build artifact, which will be downloadable from the "Artifacts" tab on your build page.
Filtering with grep
Grep is a Unix tool to help you filter lines of text that match a pattern. For example, the following script only sends Buildkite the matching lines as your log output, whilst storing the original log for artifact uploading.
#!/bin/bash set -euo pipefail your_build_command | tee build.log | grep 'some pattern'
Truncating with tail
Tail is a Unix tool that returns the last portion of a file. This is useful if your log output is exceeding our hard limit of 100MB. For example, the following script only sends Buildkite the last 90MB as your output log, whilst storing the original log for artifact uploading.
#!/bin/bash set -euo pipefail your_build_command | tee build.log | tail -c90000000
Improving Xcode logs with xcpretty
#!/bin/bash set -euo pipefail xcodebuild <build options> | tee -a build.log | xcpretty -c
Make sure to set the
-o pipefail option in your buildscript as above, otherwise the build failure status might not be passed through correctly.
Encryption and security
Buildkite has zero access to your source code in the pipelines and only receives and stores the log output of the builds and build artifacts in encrypted form.
Logs are AES-encrypted, and the build artifacts are encrypted in transit and at rest using AWS encryption (KMS or S3 SSE). As a result, the keys cannot be extracted on the Buildkite's side, and the AWS solutions provide mitigations against zero-day attacks and other security issues. Beyond this, the control over security measures within your infrastructure is up to you.
If you choose to host your build artifacts yourself, they end up in your private AWS bucket.
If you are a Buildkite Enterprise user, you can also set up a private AWS S3 build log archive location and store the logs in your private bucket.
To further tighten the security in a Buildkite organization, you can use the API Access Audit to track the actions of the users who have API Access Tokens that can access your organization's data via the REST and GraphQL API.
Redacted environment variables
The Buildkite agent can redact the values of environment variables whose names match common patterns for passwords, and other secure information, before the build log is uploaded to Buildkite.
The default environment variable name patterns are
*_TOKEN. You can replace the default patterns by setting redacted-vars on your agent.
For example, if you have environment variable
MY_SECRET="topsecret"and you run a command that outputs
This is topsecret info, the log output will actually be
This is [REDACTED] info.
Setting environment variables
Note that if you
pipeline.yml it is not redacted, but doing that is not recommended.