Creating dynamic pipelines and build annotations using Bazel

Before you start

To complete this tutorial, you'll need:

• A Buildkite account. If you don't have one already, create a 30-day free trial account.

  When you create a new organization as part of sign-up, you'll be guided through a flow to create and run a starter pipeline. Complete that before continuing.

• To enable the YAML steps editor in Buildkite:

  • Select Settings > YAML Migration to open the YAML migration settings.
  • Select Use YAML Steps for New Pipelines, then confirm the action in the modal.

• Git. This tutorial uses GitHub, but Buildkite can work with any version control system.

• To have made your own copy or fork of the bazel-buildkite-example repository within your own GitHub account.

Set up an agent

Buildkite Pipelines requires an agent running Bazel to build this pipeline. You can set up your own self-hosted agent to do this. However, you can get up and running more rapidly by creating a Buildkite hosted agent for macOS, instead.

Buildkite agents connect to Buildkite through a cluster, which provides a mechanism to organize your pipelines and agents together, such that the pipelines associated with a given cluster can only be built by the agents (defined within queues) in the same cluster.

By default, new Buildkite organizations have one cluster, named Default cluster, with a single queue, named with the key default. A cluster maintainer or Buildkite organization administrator can customize the cluster's name.

You need at least one agent configured within its own queue and cluster to run builds.

Create a Buildkite hosted agent for macOS

Unlike Linux hosted agents, which would require you to install Bazel or Bazelisk on the agent (for example, using an agent image), and implement other configurations to ensure that Bazel runs successfully on the agent (for example, ensuring Bazel runs as a non-root user), macOS hosted agents already come pre-installed with Bazelisk and ready to run Bazel.

You can create the first Buildkite hosted agent for macOS within a Buildkite organization for a two-week free trial, after which a usage cost (based on the agent's capacity) is charged per minute.

To create your macOS hosted agent:

1. Navigate to the cluster you want to run your pipeline in. To do this, select Agents in the global navigation to access the Clusters page.
2. Select the cluster (for example, Default cluster) to which the hosted agent will be added.

3. Follow the Create a Buildkite hosted queue > Using the Buildkite interface instructions to begin creating your hosted agent within its own queue.

   As part of this process:

   • Give this queue an intuitive key and description, for example, macos and Buildkite macOS hosted queue, respectively.
   • In the Select your agent infrastructure section, select Hosted.
   • Select macOS as the Machine type and Small for the Capacity.

4. Make your pipelines use your new macOS hosted agent by default, by ensuring its queue is the default queue. This should be indicated by (default) after the queue's key on the cluster's Queues page. If this is not the case and another queue is marked (default):

   1. On the cluster's Queues page, select the queue with the hosted agent you just created.
   2. On the queue's Overview page, select the Settings tab to open this page.
   3. In the Queue Management section, select Set as Default Queue.

Your Buildkite macOS hosted agent, as the new default queue, is now ready to use.

Set up a self-hosted agent

Setting up a self-hosted agent for this tutorial requires you to first install a Buildkite Agent in a self-hosted environment, and then install Bazel to the same environment.

Before installing and running a self-hosted agent, ensure you have:

• a cluster (for example, Default cluster) you can connect this agent to,
• a queue (for example, with the key macos) to which the agent will be associated with, and

• the value of an agent token (for example, Initial agent token), which you can configure for the agent.

  Be aware that since hosted agents are managed by Buildkite, there is no need to create agent tokens for these types of agents.

To install and run an agent in your own self-hosted infrastructure (including your own computer):

1. Decide where you want to run the agent.

   Most engineers start by running an agent on their local machine while playing around with pipeline definitions before setting up a long-term solution.

2. Follow the instructions for where you want to install the agent.

   To install locally, see:

   • macOS
   • Windows
   • Linux
   • Docker

   Or see all installation options.

   Ensure you configure the agent token, which connects the agent to your Buildkite account.

3. To confirm that your agent is running, and configured correctly with your credentials, go to Agents. You should see a list of all agents linked to the account and their status.

Last, to install Bazel, follow the relevant instructions to install Bazelisk (recommended) or the relevant Bazel package to the same operating system environment that your Buildkite agent was installed to.

Create a pipeline

Next, you'll create a new pipeline that builds an example Python project with Bazel, which in turn creates additional dynamically-generated steps in JSON format that Buildkite runs to build and test a hello-world library.

To create this pipeline:

1. Add a new pipeline in your Buildkite organization, select your GitHub account from the Any account dropdown, and specify your copy or fork of the 'bazel-buildkite-example' repository for the Git Repository value.

2. On the New Pipeline page, select the cluster associated with the agent you had set up with Bazel.

3. If necessary, provide a Name for your new pipeline.

4. Select the Cluster of the agent you had previously set up.

5. If your Buildkite organization already has the teams feature enabled, choose the Team who will have access to this pipeline.

6. Leave all other fields with their pre-filled default values, and select Create Pipeline. This associates the example repository with your new pipeline, and adds a step to upload the full pipeline definition from the repository.

Build the pipeline

Now that your pipeline has been set up and created in Buildkite Pipelines, it is ready to start being built, and we can start making commits to different areas of this project to see how these affect your dynamic pipeline builds.

Step 1: Create the first build

1. On the next page after creating your pipeline, which shows its name, select New Build. In the resulting dialog, create a build using the pre-filled details.

   1. In the Message field, enter a short description for the build. For example, My first build.
   2. Select Create Build.

2. Once the build has completed, visit your pipeline's build summary page, and verify that only the initial Compute the pipeline with Python step has been run.

Step 2: Make changes to both an app and library file

1. Edit one of the files within both the ./app and ./library directories, and commit and push this change to its main branch, with an appropriate message (for example, A change to both an app and a library file).

2. On your pipeline's build summary page, and notice that both the dynamically generated Build and test //library/... and Build and test //app/... Bazel package build steps have also been run.

3. Note also the Bazel Results build annotation on this pipeline build's results, which are generated from Bazel builds using the Bazel BEP Annotate Buildkite Plugin. This plugin is defined in the example Python project's utils.py file, which in turn, is used by the pipeline.py file.

Step 3: Make changes to only an app file

1. Edit one of the files within the ./app directory only, and commit and push this change to its main branch, with an appropriate message (for example, A change to only an app file).

2. On your pipeline's build summary page again, and notice that only the dynamically generated Build and test //app/... Bazel package build step is built.

Step 4: Make changes to only a library file

1. Edit one of the files within the ./library directory only, and commit and push this change to its main branch, with an appropriate message (for example, A change to only a library file).

2. On your pipeline's build summary page, notice that both the dynamically generated Build and test //library/... and Build and test //app/... Bazel package build steps have been built.

Why? According to each Bazel package's respective BUILD.bazel files in this project, //app has a dependency on //library. Therefore, if any change is made to a file in ./library, then ./app needs to be re-built to determine if the changes in ./library also affect those in ./app.

Next steps

That's it! You've successfully configured a Buildkite agent, built a Buildkite pipeline with an example Python program that:

• Builds pipeline steps dynamically.
• Uses Bazel to define Bazel package dependencies, and runs Bazel queries to determine which Bazel packages need to be built (based on their dependencies).
• Generates pipeline build annotations using the using the Bazel BEP Annotate Buildkite Plugin. 🎉

Learn more about dynamic pipelines from the Dynamic pipelines page.