Migrate from CircleCI

Understand the differences

Most CircleCI concepts translate to Buildkite Pipelines directly, but there are still differences that need to be understood to ensure a comfortable transition from one platform to another.

System architecture

CircleCI is a fully hosted CI/CD platform that runs jobs on CircleCI-managed or self-hosted runners.

Buildkite Pipelines uses a hybrid model:

• A SaaS platform (the Buildkite dashboard) for visualization and pipeline management.
• Buildkite agents for executing jobs — through Buildkite hosted agents or through self-hosted agents in your own infrastructure as the Buildkite agent is open source and can run on local machines, cloud servers, or containers.

See Buildkite Pipelines architecture for more details.

Security

The hybrid architecture of Buildkite Pipelines provides a unique approach to security. Buildkite Pipelines takes care of the security of the SaaS platform, including user authentication, pipeline management, and the web interface. The Buildkite agents, which run on your infrastructure, allow you to maintain control over the environment, security, and other build-related resources.

While Buildkite Pipelines provides its own secrets management capabilities, you can also configure Buildkite Pipelines so that it doesn't store your secrets. Buildkite Pipelines does not have or need access to your source code. Only the agents you host within your infrastructure would need access to clone your repositories, and your secrets that provide this access can also be managed through secrets management tools hosted within your infrastructure.

Learn more about Security and Secrets in Buildkite Pipelines.

Pipeline configuration concepts

In CircleCI, the core description of work is a workflow defined in .circleci/config.yml, containing jobs with multiple steps.

In Buildkite Pipelines, a pipeline is the core description of work, typically defined in a pipeline.yml file (usually in .buildkite/).

A Buildkite pipeline contains different types of steps for different tasks:

• Command step: Runs one or more shell commands on one or more agents.
• Wait step: Pauses a build until all previous jobs have completed.
• Block step: Pauses a build until unblocked.
• Input step: Collects information from a user.
• Trigger step: Creates a build on another pipeline.
• Group step: Displays a group of sub-steps as one parent step.

While CircleCI traditionally maps each project one-to-one to a repository, Buildkite pipelines are fully decoupled from repositories. You can create multiple pipelines per repository, trigger pipelines across repositories, or run pipelines independently of any repository.

Triggering a Buildkite pipeline creates a build, and any command steps are dispatched as jobs to run on agents. A common practice is to define a pipeline with a single step that uploads the pipeline.yml file in the code repository. The pipeline.yml contains the full pipeline definition and can be generated dynamically.

Plugin system

CircleCI uses orbs, which are reusable packages that bundle jobs, commands, and executors together. Buildkite Pipelines uses Buildkite plugins, which are referenced directly in pipeline definitions. Unlike orbs, Buildkite plugins focus on modifying agent behavior at the step level. They are shell-based, run on individual agents, and are pipeline- or step-specific with independent versioning. Plugin failures are isolated to individual builds, and compatibility issues are rare.

Provision agent infrastructure

Buildkite agents run your builds, tests, and deployments. They can run as Buildkite hosted agents where the infrastructure is provided for you, or on your own infrastructure (self-hosted), similar to self-hosted runners in CircleCI.

For self-hosted agents, consider:

• Infrastructure type: On-premises, cloud (AWS, GCP), or container platforms (Docker, Kubernetes).
• Resource usage: Evaluate CPU, memory, and disk requirements based on your current CircleCI resource class usage.
• Platform dependencies: Ensure agents have required tools and libraries. Unlike CircleCI, where Docker images provide pre-configured environments, Buildkite agents require explicit tool installation or pre-built agent images.
• Network: Agents poll the Buildkite agent API over HTTPS so no incoming firewall access is needed.
• Scaling: Scale agents independently based on concurrent job requirements.
• Build isolation: Use agent tags and clusters to target specific agents.

For Buildkite hosted agents, see the Getting started guide. For self-hosted agents, see the Installation guides for your infrastructure type.

Pipeline translation fundamentals

Before translating your CircleCI configuration, note the key differences in pipeline syntax, step execution, artifact handling, Docker configuration, and agent targeting between CircleCI and Buildkite Pipelines.

Files and syntax

This table outlines the fundamental differences in pipeline files and their syntax between CircleCI and Buildkite Pipelines.

The YAML-based pipeline syntax of Buildkite Pipelines is simpler. Where CircleCI relies on parameters to conditionally include or exclude jobs and workflows, Buildkite Pipelines uses dynamic pipelines to generate the entire pipeline definition at build time using scripts written in any language. This approach provides more flexibility without the complexity of parameter declarations and conditional logic scattered throughout your configuration.

Step execution

Both CircleCI and Buildkite Pipelines run steps in parallel by default. In CircleCI, steps within a job run sequentially, but jobs within a workflow run in parallel unless you specify requires. In Buildkite Pipelines, all steps run in parallel unless you add explicit ordering.

Each Buildkite step is fully isolated from other steps, similar to how CircleCI jobs are isolated from each other. Steps can run on different agents, with no shared filesystem or state between them.

To make a Buildkite pipeline run its steps in a specific order, use the depends_on attribute or a wait step.

# Buildkite Pipelines: Explicit sequencing with depends_on

steps:
  - label: "Lint"
    key: lint
    command: npm run lint

  - label: "Test"
    key: test
    command: npm test

  - label: "Build"
    depends_on: [lint, test]
    command: npm run build

Workspace and artifact handling

In CircleCI, persist_to_workspace and attach_workspace share files between jobs. In Buildkite Pipelines, each step runs in a fresh workspace on potentially different agents. Use buildkite-agent artifact upload and buildkite-agent artifact download to share artifacts:

# Buildkite Pipelines
steps:
  - label: "Build"
    key: "build"
    command:
      - npm run build
      - buildkite-agent artifact upload "dist/**/*"

  - label: "Deploy"
    depends_on: "build"
    command:
      - buildkite-agent artifact download "dist/**/*" .
      - npm run deploy

Other options for sharing state between steps:

• Reinstall per step: Simple for fast-installing dependencies like npm ci.
• Meta-data: Use meta-data to exchange lightweight key-value pairs between steps at runtime without file-based sharing.
• Cache plugin: Similar to CircleCI's save_cache/restore_cache, use the Buildkite cache plugin for larger dependencies using cloud storage (S3, GCS). The plugin's manifest attribute works like CircleCI's {{ checksum }} to generate a cache key from a file.
• External storage: Custom solutions for complex state management.

Docker images and executors

CircleCI executors define the execution environment (Docker image, resource class, environment variables). Buildkite Pipelines separates these concerns:

For example, in Buildkite Pipelines:

# Buildkite Pipelines
steps:
  - label: "Test"
    env:
      NODE_ENV: test
    plugins:
      - docker#v5.13.0:
          image: node:18
    command: npm test

For Docker-based builds, use the Docker plugin or the Docker Compose plugin to run commands inside containers.

plugins:
  - docker#v5.13.0:
      image: cimg/node:18.17
      shell: ["/bin/bash", "-l", "-e", "-c"]
      propagate-uid-gid: true

Agent targeting

CircleCI uses resource_class and executors to control where jobs run:

# CircleCI
jobs:
  build:
    docker:
      - image: cimg/node:20.0
    resource_class: large
    steps:
      - checkout
      - run: make build

  deploy:
    machine:
      image: ubuntu-2204:current
    resource_class: medium
    steps:
      - checkout
      - run: make deploy

Buildkite Pipelines uses a pull-based model where agents poll queues for work using the agents attribute. Map CircleCI resource classes to queues with agents sized to match your workload requirements. This model provides better security (no incoming connections), easier scaling with ephemeral agents, and more resilient networking:

# Buildkite Pipelines
steps:
  - label: "Build"
    command: "make build"
    agents:
      queue: "large"
  - label: "Deploy"
    command: "make deploy"
    agents:
      queue: "production"

You can also use custom agent tags beyond queue to target agents by capability, for example agents: { os: "linux", arch: "arm64" }. For Windows or macOS jobs, route to platform-specific queues using agents: { queue: "windows" } or agents: { queue: "macos" }.

Translate an example CircleCI configuration

This section guides you through the process of translating a CircleCI configuration example (which builds a Node.js app) into a Buildkite pipeline. If you want to see the finished result first, skip to the complete pipeline or the refactored version with YAML aliases.

Step 1: Understand the source configuration

The following CircleCI configuration shows an example:

version: 2.1

orbs:
  node: circleci/node@5.2

executors:
  node-executor:
    docker:
      - image: cimg/node:20.0

jobs:
  lint:
    executor: node-executor
    steps:
      - checkout
      - node/install-packages
      - run: npm run lint

  test:
    executor: node-executor
    steps:
      - checkout
      - node/install-packages
      - run: npm test
      - store_test_results:
          path: test-results
      - store_artifacts:
          path: coverage

  build:
    executor: node-executor
    steps:
      - checkout
      - node/install-packages
      - run: npm run build
      - store_artifacts:
          path: dist

workflows:
  ci:
    jobs:
      - lint
      - test
      - build:
          requires:
            - lint
            - test

This workflow lints, tests, and builds a Node.js application, with the build job depending on lint and test completing first.

Step 2: Create a basic Buildkite pipeline structure

Create a .buildkite/pipeline.yml file in your repository. Start with a basic structure that maps each CircleCI job to a Buildkite Pipelines step:

steps:
  - label: ":eslint: Lint"
    key: lint
    command:
      - echo "Lint step placeholder"

  - label: ":test_tube: Test"
    key: test
    command:
      - echo "Test step placeholder"

  - label: ":wrench: Build"
    key: build
    command:
      - echo "Build step placeholder"

Notice the immediate differences in this pipeline syntax from CircleCI:

• No version: declaration needed.
• No orbs:, executors:, or jobs: blocks.
• No checkout step as Buildkite agents check out code automatically.
• Emoji support in labels without plugins.
• Key assignment for dependency references.

Step 3: Configure the step dependencies

The build step should run only after lint and test complete successfully. Configure explicit dependencies on the build step, which prevents it from running if either the lint or test steps fail:

  - label: ":wrench: Build"
    key: build
    depends_on: [lint, test]
    command:
      - echo "Build step placeholder"

Without this depends_on attribute, all three steps would run simultaneously, due to the parallel-by-default behavior of Buildkite Pipelines.

Step 4: Add the actual commands

Replace the placeholder commands with real commands. The node/install-packages orb command becomes npm ci:

  - label: ":eslint: Lint"
    key: lint
    commands:
      - npm ci
      - npm run lint

Step 5: Add the Docker plugin for container builds

Replace the CircleCI executor with the Docker plugin to run commands inside a container:

  - label: ":eslint: Lint"
    key: lint
    plugins:
      - docker#v5.13.0:
          image: "node:20"
    commands:
      - npm ci
      - npm run lint

Step 6: Implement artifact collection

Add artifact collection using the artifact_paths attribute. This replaces CircleCI's store_artifacts:

  - label: ":test_tube: Test"
    key: test
    plugins:
      - docker#v5.13.0:
          image: "node:20"
    commands:
      - npm ci
      - npm test
    artifact_paths:
      - coverage/**/*

Step 7: Review the complete pipeline

The complete example CircleCI pipeline translated to a Buildkite pipeline:

steps:
  - label: ":eslint: Lint"
    key: lint
    plugins:
      - docker#v5.13.0:
          image: "node:20"
    commands:
      - npm ci
      - npm run lint

  - label: ":test_tube: Test"
    key: test
    plugins:
      - docker#v5.13.0:
          image: "node:20"
    commands:
      - npm ci
      - npm test
    artifact_paths:
      - coverage/**/*

  - label: ":wrench: Build"
    depends_on: [lint, test]
    plugins:
      - docker#v5.13.0:
          image: "node:20"
    commands:
      - npm ci
      - npm run build
    artifact_paths:
      - dist/**/*

Step 8: Refactor with YAML aliases

Eliminate the duplication using YAML aliases:

common:
  docker: &docker
    docker#v5.13.0:
      image: "node:20"

steps:
  - label: ":eslint: Lint"
    key: lint
    plugins:
      - *docker
    commands:
      - npm ci
      - npm run lint

  - label: ":test_tube: Test"
    key: test
    plugins:
      - *docker
    commands:
      - npm ci
      - npm test
    artifact_paths:
      - coverage/**/*

  - label: ":wrench: Build"
    depends_on: [lint, test]
    plugins:
      - *docker
    commands:
      - npm ci
      - npm run build
    artifact_paths:
      - dist/**/*

By anchoring the plugin map rather than the entire plugins array, individual steps can override or extend their plugin list when needed. The final result is shorter than the original CircleCI configuration, with no duplication and a cleaner, more readable structure.

Translating common patterns

This section covers translation patterns for CircleCI features not covered in the example walkthrough.

Environment variables

CircleCI job-level environment maps to the env attribute in Buildkite Pipelines. For more information, see environment variables. For pipeline-wide variables, use a top-level env attribute.

You can also define environment variables at the agent level using agent hooks, making them available to all pipelines running on those agents. If you use the Elastic CI Stack for AWS, you can scope agent-level variables to specific pipelines.

# Buildkite Pipelines
env:
  NODE_ENV: production

steps:
  - label: "Build"
    command: npm run build

Contexts and secrets

CircleCI contexts are named collections of environment variables attached to jobs at the workflow level. Translate based on content type:

• For secrets: Use Buildkite cluster secrets with the secrets: attribute in pipeline YAML, or an external secrets manager.
• For non-secret variables: Use the env: attribute directly. Use YAML anchors to share variables across steps.

Approval jobs

CircleCI's type: approval jobs create manual gates in a workflow. The equivalent in Buildkite Pipelines is a block step:

# Buildkite Pipelines
steps:
  - label: "Build"
    key: "build"
    command: npm run build

  - block: "🚀 Deploy to production?"
    key: "hold"
    depends_on: "build"

  - label: "Deploy"
    depends_on: "hold"
    command: npm run deploy

Matrix builds

CircleCI matrix jobs translate to the native build matrix support in Buildkite Pipelines:

# Buildkite Pipelines
steps:
  - label: "Test (Node {{matrix.node_version}})"
    plugins:
      - docker#v5.13.0:
          image: node:{{matrix.node_version}}
    command: npm test
    matrix:
      setup:
        node_version:
          - "18"
          - "20"
          - "22"

Parallelism

CircleCI's parallelism key maps to the parallelism attribute in Buildkite Pipelines:

# Buildkite Pipelines
steps:
  - label: "Test"
    parallelism: 4
    command: npm test

Buildkite Pipelines parallelism creates multiple jobs with BUILDKITE_PARALLEL_JOB and BUILDKITE_PARALLEL_JOB_COUNT environment variables. For intelligent test distribution based on timing data (equivalent to circleci tests split --split-by=timings), use Test Engine.

Branch and tag filtering

Buildkite Pipelines offers two approaches to step-level filtering:

• branches: attribute: For simple patterns, with ! prefix for negation (for example, branches: "!dev !staging").
• if: conditionals: For complex patterns or regex matching (for example, if: build.branch !~ /^feature\/experimental-/). The branches: and if: attributes cannot be used together on the same step.

For tag-only builds, use if: build.tag =~ /^v/.

For pipeline-wide branch restrictions that prevent builds from being created entirely, configure this in Pipeline Settings under Branch Limiting, not in YAML.

Scheduled workflows

CircleCI supports scheduled pipelines configured through the CircleCI UI, API, or the legacy triggers: key in YAML. In Buildkite Pipelines, scheduled builds are configured in the Buildkite UI under your pipeline's Settings > Schedules.

Dynamic configuration

CircleCI's dynamic configuration pattern uses setup: true with the continuation orb to generate pipelines at runtime. Buildkite Pipelines handles this natively with buildkite-agent pipeline upload:

# Buildkite Pipelines
steps:
  - label: "<img class="emoji" title="pipeline" alt=":pipeline:" src="https://buildkiteassets.com/emojis/img-buildkite-64/pipeline.png" draggable="false" /> Generate pipeline"
    command: |
      ./generate-config.sh | buildkite-agent pipeline upload

For path-based dynamic configuration (similar to CircleCI's path-filtering orb), use conditionals, dynamic pipelines with change detection logic, or the declarative if_changed attribute. For monorepos, the Monorepo Diff plugin watches for changes across directories and triggers the appropriate pipelines automatically.

Reusable commands

CircleCI commands are reusable step sequences with parameters. For simple reuse in Buildkite Pipelines, use YAML anchors. For parameterized reuse, use dynamic pipelines where a step generates and uploads pipeline YAML at runtime using buildkite-agent pipeline upload.

Concept mapping reference

This table provides a mapping between CircleCI concepts and their Buildkite Pipelines equivalents:

Key differences and benefits of migrating to Buildkite Pipelines

Beyond the syntax differences shown in the example pipeline translation, migrating to Buildkite Pipelines provides several architectural advantages:

• Hybrid architecture with infrastructure control: Run builds on Buildkite hosted agents, self-hosted agents in your own environment, or a mix of both. Source code and secrets stay within your infrastructure, and agents connect outbound over HTTPS with no incoming firewall access required.
• No concurrency limits: Scale by adding agents without platform-imposed concurrency caps. Buildkite Pipelines supports 100,000+ concurrent agents with turnkey autoscaling through the Elastic CI Stack for AWS or Agent Stack for Kubernetes.
• Dynamic pipelines: Instead of relying on parameters and static conditional logic, dynamic pipelines generate and modify steps at runtime using any language. This is especially valuable for monorepos and complex build graphs.
• Predictable pricing: No credit-based billing or per-user charges based on commit activity. See pricing for details.

For a detailed comparison, see Advantages of migrating from CircleCI.

Be aware of common pipeline-translation mistakes, which might include:

• Forgetting about fresh workspaces (leading to missing dependencies).
• Using cimg/* Docker images without login shell and UID/GID workarounds.
• Over-parallelizing interdependent steps.
• Assuming tools from orbs are available (when you need explicit installation).

Next steps

Explore these resources to enhance your migrated pipelines:

• Defining your pipeline steps
• Buildkite agent overview
• Plugins directory
• Dynamic pipelines and the Buildkite SDK
• Buildkite agent hooks
• Using conditions
• Annotations
• Security, Secrets, and permissions
• Integrations
• Test Engine for test insights

You can try the Buildkite pipeline converter to see how your existing CircleCI configuration might look converted to Buildkite Pipelines.

With a basic understanding of the differences between Buildkite Pipelines and CircleCI, if you haven't already done so, run through the Getting started with Pipelines guide to get yourself set up to run pipelines in Buildkite Pipelines, and create your own pipeline.

For migration assistance, contact support@buildkite.com.