Migrating from CircleCI? An open source tool to get you started

We’ve recently noticed more people migrating from CircleCI to Buildkite Pipelines, but some struggle translating their workflows. So, we built a tool to make it easier.

The Buildkite Migration tool is a small open source program that translates CircleCI workflow definitions into Buildkite pipeline definitions. In this post, we’ll cover what Buildkite Pipelines is, how to use the migration tool to translate CircleCI workflows, and what’s coming next.

What is Buildkite Pipelines?

Like CircleCI, Buildkite Pipelines is a CI/CD tool for automating workflows. It's known for pioneering the hybrid approach to CI/CD, where you separate the control plane from the compute (similar to CircleCI self-hosted runners):

• Buildkite dashboard: A software-as-a-service (SaaS) control plane for visualizing and managing CI/CD pipelines. This coordinates work and displays results with easy-to-follow logs for each pipeline step.
• Agents: Small, reliable, and cross-platform build runners. These are either hosted by you (on-premise or in the cloud) or by Buildkite (see hosted agents). They execute the work they receive from the Buildkite dashboard.

Decoupling the control plane and agents gives you:

• Control to scale your compute.
• A secure boundary around your private data.
• Flexibility to configure the build environment.

Build agents are open source and configurable so you can inspect the code, and optimize them for your application and build environment.

We believe this focus on security and putting control in the hands of our users is the reason more people are migrating to Buildkite. Companies are reconsidering their CI/CD requirements in the wake of security breaches and concerns over data privacy, and placing more importance on trust. Trust is a fundamental cornerstone in software development, and when that trust wavers, people often start looking for alternative paths forward.

Developers now demand solutions that not only provide powerful automation but also prioritize the security and privacy of their code and data. This is where Buildkite's hybrid CI/CD solution shines. Buildkite never needs to access your source code, secrets, or internal systems. You control the security perimeter with hosted or hybrid options and can adapt the system to fit your needs.

Similarities and differences

CircleCI and Buildkite Pipelines share some of the same foundations:

• YAML configuration: CircleCI and Buildkite rely on YAML to define their configuration files. This shared language for configuration simplifies the transition process, as you can often reuse or adapt your existing YAML configurations for Buildkite.
• Job and step definitions: In both platforms, you define individual jobs or steps within your workflows (pipelines in Buildkite). These jobs or steps outline specific tasks or actions, such as building, testing, or deploying your code. This common structure allows you to create similar workflows in Buildkite as you would in CircleCI.
• Parallelism and concurrency: CircleCI and Buildkite both support parallelism and concurrency, enabling you to run multiple jobs or steps simultaneously, which can significantly speed up your build and deployment processes. You can specify parallelism and concurrency settings in the YAML configuration of both platforms.
• Environment variables: Both platforms provide ways to manage environment variables within your workflow. You can securely store and access sensitive data and configuration details consistently across CircleCI and Buildkite.

However, there are some key differences—starting with terminology:

CircleCI workflows are composed of one or more jobs that are self-contained units of execution with one or more steps that run commands. In comparison, pipelines in Buildkite are composed of one or more steps that might contain commands or other types of work. And where CircleCI extends functionality with orbs, Buildkite uses plugins.

Once you’re using Buildkite, you’ll also notice:

• ⚡️ Unlimited concurrency.
• 🤖 Unlimited agents.
• ✨ Greater flexibility with configuration options, including lifecycle hooks.
• 👾 The ability to dynamically generate and update pipeline definitions at runtime, with logic written in your language of choice.
• 🤑 Simple and transparent pricing—pay us for a plan, manage your compute costs with hosting companies directly. No credit system.
• 🤝 Industry-leading support.
• 🔐 The option to adopt a zero trust security posture.
• 👀 Rich annotations to highlight key information from build logs.

How the migration tool works

CircleCI YAML in, Buildkite YAML out. The migration tool translates CircleCI features into the Buildkite equivalent for you.

Workflows, jobs, and commands

Workflows become groups of steps, with each step containing a command.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
version: 2.1

commands: # a reusable command with parameters
  greeting:
    parameters:
      to:
        default: "world"
        type: string
    steps:
      - run: echo "Hello <<parameters.to>>"
    jobs:
      my-job:
        docker:
          - image: cimg/base:stable
        steps:
          - greeting:
            to: "My-Name"
    workflows:
      my-workflow:
        jobs:
          - my-job

Conditionals

The filters, when, and unless conditionals become if clauses. Rather than the complexity of parameters and a full language for logical operators, Buildkite uses simple logic options inline. When there are conditionals inside jobs themselves that run on the same step, they are translated as Bash conditionals.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
version: 2.1

jobs:  
  test:    
    steps:      
      - run: echo 'Hello'      
      - when:          
        condition:            
          equal: [ main, << pipeline.git.branch >> ]          
        steps:            
          - run: echo 'Running in main'
workflows:  
  my-workflow:    
    when:      
      and:        
        - not:            
          matches:              
            pattern: "^main$"              
            value: << pipeline.git.branch >>        
        - or:            
          - equal: [ canary, << pipeline.git.tag >> ]            
          - << pipeline.parameters.deploy-canary >>    
    jobs:      
      - test

Parameters

Parameters are replaced with their values wherever they’re used.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
version: 2.1
commands:  
  print:    
    parameters:      
      message:        
        type: string        
        default: test    
    steps:      
      - run: echo << parameters.message >>
      
jobs:  
  daily-message:    
    parameters:      
      message:        
        type: string        
        default: testjob    
    steps:      
      - print:          
        message: Printing << parameters.message >>

workflows:  
  my-workflow:    
    jobs:      
      - daily-message:          
        message: testworkflow

Orbs

Orbs are equivalent to Buildkite plugins, and the translation is not currently supported. You’ll see a comment asking you to replace the orb with a plugin—either use one from the plugin ecosystem, or write your own.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
version: 2.1

orbs:  
  node: circleci/node@x.y

jobs:  
  install-node-example:    
    docker:      
      - image: cimg/base:stable    
      steps:      
        - checkout      
        - node/install:          
          install-yarn: true          
          node-version: '16.13'      
        - run: node --version

workflows:  
  test_my_app:    
  jobs:      
    - install-node-example

Executors

There's no direct translation into Buildkite. Executors become agent tags, which let you target agents with a particular configuration.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
version: 2.1

jobs:  
  orb-executor:    
    executor:      
      name: win/default # executor type      
      size: medium # can be medium, large, xlarge, 2xlarge    
      
    steps:      
    # Commands are run in a Windows virtual machine environment      
      - run: Hello, Orb Executor   
 
  self-runner:    
    machine: true    
    resource_class: win/default    
    steps:      
      - run: Hello, self runner  
      
  docker:    
    docker:      
      - image: circleci/python:3.6.2    
    steps:      
      - run: Hello from docker  
      
    multi-docker:    
      docker:      
        - image: circleci/python:3.6.2      
        - image: circleci/python:3.9-stretch-browsers    
      steps:      
        - run: Hello from multi-image docker  
        
    OSX:    
      macos:      
        xcode: 14.2.0    
      steps:      
        - run: Hello from OS X
        
  workflows:  
    test:    
    jobs:      
      - orb-executor
      - self-runner      
      - docker      
      - multi-docker      
      - OSX

Migrating a CircleCI workflow to Buildkite

Let’s see the tool in action using the following CircleCI workflow definition:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
version: 2.1

# Define the jobs we want to run for this project
jobs:  
  build:    
    docker:      
      - image: cimg/base:2023.03    
    steps:      
      - checkout      
      - run: echo "this is the build job"  
  test:    
    docker:      
      - image: cimg/base:2023.03    
    steps:      
      - checkout      
      - run: echo "this is the test job"

# Orchestrate our job run sequence
workflows:  
  build_and_test:    
    jobs:      
      - build      
      - test

To follow along yourself, you’ll need:

• Docker installed and running
• Your favorite command line tool—we’re using iTerm2.
• Git

To translate a CircleCI workflow to Buildkite:

1. On the command line, clone the migration tool by running:

   1
   git clone https://github.com/buildkite/migration

2. Navigate to the cloned directory, and from inside the directory at the root, start the app by running:

   1
   docker-compose up webui

   This continues running while you use the app.

3. Open the web UI in your browser at http://localhost:9292.

   You'll see a simple interface with a panel to paste in your CircleCI workflow on the left, and a panel for the resulting Buildkite pipeline definition on the right.

4. Copy the CircleCI workflow definition from above and paste it into the left hand panel.

5. Select Buildkite-ify to translate the workflow definition.

   You'll see the pipeline definition appear in the right hand panel, ready for use in Buildkite.

Please note that while the translator tool aims to create a configuration that closely mirrors your CircleCI setup, there may be minor differences between the two platforms. Inspect the translated configuration and make any necessary adjustments to suit Buildkite's specific requirements. Check out Defining steps and surrounding documentation to learn how to customize your pipeline definition.

When you’re happy with your pipeline definition, you can start using it in Buildkite. Sign up for free and learn how to run your first agent. If you need help, check out the Getting started guide or reach out to support for help.

For more information about the migration tool and alternative ways to run it, check the README.

Next steps

The migration tool already covers many typical use cases, but it's still under active development. We're working on:

• Translating orbs to Buildkite plugins.
• Investigating ways to make it easier to consume the tool through an API, CLI, or web app.
• Fixing bugs and generally improving the translation process.

The tool is open source (github.com/buildkite/migration), so you can help us make it work better for you. If you have suggestions, feedback, or run into problems, open issues to let us know or contribute to the code.

Buildkite Pipelines is a CI/CD tool designed for developer happiness. Easily follow and decipher logs, get observability into key build metrics, and tune for enterprise-grade speed, scale, and security. Every new signup gets a free 30-day trial to test out the key features. See Buildkite Pipelines to learn more.