Getting started with Pipelines

Before you start

To complete this tutorial, you'll need:

• A Buildkite account. If you don't have one already, create a 30-day free trial account.

  If you're signing up to Buildkite and need to create a new Buildkite organization, you'll initially be guided through a flow to create a starter pipeline.

  As part of this flow, upon reaching the Create your first pipeline page, select Create starter pipeline, and on the next Choose your agent setup page, select Skip agent setup, since you'll be doing these steps as part of this tutorial.

• To enable the YAML steps editor in Buildkite. If you haven't already done this:

  1. Select Settings > YAML Migration to open the Pipeline YAML Migration page.
  2. Select Use YAML Steps for New Pipelines, then confirm the action in the dialog.

• Git. This tutorial uses GitHub, but Buildkite can work with any version control system by adapting the relevant steps below to those for your particular Git-based setup.

Understand the architecture

Before creating a pipeline, take a moment to understand Buildkite's architectures and the advantages they provide. Buildkite provides both a hosted (as a managed solution) and self-hosted architecture for its build environments.

You can learn more about the differences between these architectures in Buildkite Pipelines architecture.

If you're already familiar with Buildkite Pipelines' architectures, continue on, bearing in mind that the remainder of this tutorial assumes that you already understand the fundamental differences between them.

Set up an agent

The program that executes work is called an agent in Buildkite. An agent is a small, reliable, and cross-platform build runner that connects your infrastructure to Buildkite. The agent polls Buildkite for work, runs jobs, and reports results. You can install agents on local machines, cloud servers, or other remote machines, as well as part of Buildkite hosted agents, which provides the quickest method to get up and running with Pipelines.

Buildkite agents connect to Buildkite through a cluster. Clusters provide a mechanism to organize your pipelines and agents together, such that the pipelines associated with a given cluster can only be built by the agents (defined within queues) in the same cluster.

By default, new Buildkite organizations have one cluster, named Default cluster with a single queue, named with the key default. A cluster maintainer or Buildkite organization administrator can customize the cluster's name.

You need at least one agent configured within its own queue and cluster to run builds.

Create a Buildkite hosted agent

You can create the first Buildkite hosted agent within a Buildkite organization for a two-week free trial, after which a usage cost (based on the agent's capacity) is charged per minute.

Before creating your Buildkite hosted agent, ensure you have a cluster (for example, Default cluster) you can connect this agent to.

To create a hosted agent:

1. Navigate to the cluster you want to run your pipeline in. To do this, select Agents in the global navigation to access the Clusters page.
2. Select the cluster (for example, Default cluster) to which the hosted agent will be added.

3. Follow the Create a Buildkite hosted queue > Using the Buildkite interface instructions to begin creating your hosted agent within its own queue.

   As part of this process:

   • In the Select your agent infrastructure section, select Hosted.
   • Follow the relevant sub-steps for configuring your hosted agent.

4. Make your pipelines use this hosted agent by default, by ensuring its queue is the default queue. This should be indicated by (default) after the queue's key on the cluster's Queues page. If this is not the case and another queue is marked (default):

   1. On the cluster's Queues page, select the queue with the hosted agent you just created.
   2. On the queue's Overview page, select the Settings tab to open this page.
   3. In the Queue Management section, select Set as Default Queue.

Your Buildkite hosted agent, as the new default queue, is now ready to use. You can now skip to the next step on creating a pipeline.

Install and run a self-hosted agent

Before installing and running a self-hosted agent, ensure you have:

• a cluster (for example, Default cluster) you can connect this agent to,
• a queue (for example, with the key default) to which the agent will be associated with, and

• the value of an agent token (for example, Initial agent token), which you can configure for the agent.

  Be aware that since hosted agents are managed by Buildkite, there is no need to create agent tokens for these types of agents.

To install and run an agent in your own self-hosted infrastructure (including your own computer):

1. Decide where you want to run the agent.

   Most engineers start by running an agent on their local machine while playing around with pipeline definitions before setting up a long-term solution.

2. Follow the instructions for where you want to install the agent.

   To install locally, see:

   • macOS
   • Windows
   • Linux
   • Docker

   Or see all installation options.

   Ensure you configure the agent token, which connects the agent to your Buildkite account.

To confirm that your agent is running, and configured correctly with your credentials, go to Agents. You should see a list of all agents linked to the account and their status.

Create a pipeline

Pipelines are how Buildkite represents a CI/CD workflow. You define each pipeline with a series of steps to run. When you trigger a pipeline, you create a build, and steps are dispatched as jobs to run on agents. Jobs are independent of each other and can run on different agents.

To create a pipeline:

1. Select Add to Buildkite for the appropriate example based on where your agent is running.

   • For Bash:
     
     (from the Bash example GitHub repository)
   • For PowerShell:
     
     (from the PowerShell example GitHub repository)

   Note: Both result in the same behavior—the pipeline definition is uploaded from the repository (.buildkite/pipeline.yml), then a script runs that prints output to the logs.

2. On the New Pipeline page, select the Cluster of the agent you had previously set up.

3. If the Team field appears further down the page, this means that your Buildkite organization already has the teams feature enabled. Therefore, choose the Team who will have access to this pipeline.

4. Leave all fields with their pre-filled default values and select Create Pipeline. This associates the example repository with your new pipeline, and adds a step to upload the full pipeline definition from the repository.

5. On the next page showing your pipeline name, select New Build. In the resulting dialog, create a build using the pre-filled details.

   1. In the Message field, enter a short description for the build. For example, My first build.
   2. Select Create Build.

The page for the build then opens and begins running:

Check the output

After triggering the build, you can view the output as it runs and the full results when complete. The output for each step shows in the job list.

Expand the row in the job list to view the output for a step. For example, selecting Example Script shows the following:

In the output, you'll see:

• A pre-command hook ran and printed some text in the logs.
• The agent checked out the repository.
• The agent accessed different environment variables shown in the job environment.
• The script ran and printed text to the logs and uploaded an image as an artifact of the build.

Beyond the log, select one of the other tabs to see the artifacts, a timeline breakdown, and the environment variables.

Next steps

That's it! You've installed an agent, run a build, and checked the output. 🎉

Now try creating your own pipeline.