Sparse Checkout Buildkite Plugin

[!NOTE] Sparse checkout is now supported natively. Buildkite agent v3.136.0 and later support sparse checkout through the checkout.sparse attribute, and the native checkout also retries failed fetches. For most pipelines we recommend it over this plugin.

For non-cone patterns (the equivalent of this plugin’s no_cone), also set BUILDKITE_GIT_SPARSE_CHECKOUT_MODE: no-cone in the step’s env (requires Git 2.35 or later). See --git-sparse-checkout-mode.

Keep using this plugin if you need clean_checkout, cleanup_sparse_state or post_checkout.unshallow, or if your agents are older than v3.136.0. See Migrating from the sparse checkout plugin.

A Buildkite plugin for creating a sparse checkout of a repository.

This is useful for pipeline upload steps that dont need to access files outside the .buildkite directory. If your repository is large, this plugin will speed up your pipelines by only pulling the files relevant to the step.

Configuration

These are all the options available to configure this plugin’s behaviour.

Required

paths (list of string)

Paths accepted by git sparse-checkout set.

Optional

no_cone (‘true’ or ‘false’)

Whether to pass --no-cone to git sparse-checkout so that the paths are considered to be a list of patterns.

skip_ssh_keyscan (‘true’ or ‘false’)

Whether to skip ssh-keyscan step. This will skip adding each ssh public key into the known-hosts file. Only use if ssh keys are already setup.

clean_checkout (true, false, or only-upon-failure)

Controls how the checkout hook cleans the repository before sparse-checkout.

• false (default): Runs git clean -ffxdq before fetch. If sparse-checkout or git checkout fails, the job fails.
• true: Runs aggressive cleanup before fetch (remove lock files, git reset --hard HEAD, git clean -ffxdq). Use when interrupted jobs often leave a corrupted tree. Note that for some workloads, this aggressive cleanup could negate the expected performance benefits of using a sparse checkout.
• only-upon-failure: Runs git clean -ffxdq before fetch (same as false). If configuring sparse-checkout or checking out the commit fails (for example, local changes that would be overwritten), the hook runs the same aggressive cleanup as true once and retries sparse-checkout and checkout.

What aggressive cleanup fixes:

• Stale git lock files (from interrupted operations)
• Corrupted git index
• Uncommitted changes in tracked files
• Untracked and ignored files

⚠️ Warning: true and the retry path of only-upon-failure destroy ALL local changes and remove ALL untracked files. git clean -ffxdq with -x also removes ignored files (such as credentials, local configuration, or cache files). Only use aggressive cleanup when you’re certain no important local data needs to be preserved.

Use true for pipeline upload jobs that don’t need to preserve local changes. Use only-upon-failure when you want a fast path on healthy agents but automatic recovery from dirty checkouts left by cancelled jobs.

cleanup_sparse_state (‘true’ or ‘false’)

Tear down sparse-checkout state after the job finishes, so that subsequent jobs on the same agent that do not use sparse checkout are not affected.

When git sparse-checkout runs, it writes .git/config.worktree and sets extensions.worktreeConfig, core.sparseCheckout, and core.sparseCheckoutCone in git config. On agents with persistent build directories, this state persists across jobs — causing subsequent non-sparse jobs to silently inherit the sparse paths and fail to find files outside them.

In most setups you do not need this option. The plugin’s hooks/environment already isolates sparse checkouts into a -sparse-suffixed build directory, so non-sparse jobs on the same agent land elsewhere and never see sparse state. Enable cleanup_sparse_state only when your agent configuration overrides BUILDKITE_BUILD_CHECKOUT_PATH after the plugin’s env hook runs, causing sparse and non-sparse checkouts to share the same directory.

Cleanup runs at pre-exit after the job’s command finishes, so the next job on the same directory starts clean. This runs regardless of job success or failure. Cleanup runs at pre-exit rather than post-checkout so that sparse state stays active for the duration of the job command.

A second pass runs at pre-checkout as a safety net for cases where a previous job’s pre-exit never fired (for example, agent crashes or SIGKILL).

The cleanup removes .git/config.worktree and unsets extensions.worktreeConfig, core.sparseCheckout, and core.sparseCheckoutCone. The working tree files are left intact. This deliberately avoids git sparse-checkout disable, which re-materialises the full working tree (expensive on large monorepos).

verbose (‘true’ or ‘false’)

Enable verbose logging with bash execution tracing (set -x). This shows each command being executed and can help debug issues with ssh-keyscan, git operations, or other checkout problems. When enabled, you’ll see detailed output including command arguments and any error messages from underlying tools.

merge_ref_retry_attempts (integer)

How many times to try fetching the GitHub pull-request merge ref (refs/pull/<n>/merge) when using merge-ref checkout (see BUILDKITE_PULL_REQUEST_USING_MERGE_REFSPEC below). Defaults to 3. Set to 0 to skip merge-ref fetch attempts and fall back immediately. Must be a whole number of at least 0. Values above 10 are capped at 10 with a warning, and anything else fails the job.

GitHub computes the merge ref some time after a push, and never creates one for a pull request with merge conflicts. While the merge ref is missing, the plugin waits 2 seconds before the second attempt and 5 seconds before each later one, so the default waits about 7 seconds before it falls back. Any other error, such as Permission denied (publickey), is retried with the same backoff as fetch_attempts. If the last attempt fails with such an error, the job fails instead of falling back.

fetch_attempts (integer)

How many times to try the initial git clone and the commit (or branch, when BUILDKITE_COMMIT=HEAD) fetch before failing the job. The merge-ref fetch uses merge_ref_retry_attempts instead. Defaults to 1, which fails on the first error. Set it to 6 to match the agent’s built-in checkout (buildkite/agent#3822), which covers short network or GitHub outages such as Permission denied (publickey). Any failure is retried. Must be a whole number of at least 1. Values above 10 are capped at 10 with a warning, and anything else fails the job.

The wait doubles after each failed attempt, starting at 1 second, plus up to 1 second of jitter. The total wait before the job fails is about 2^(attempts - 1) - 1 seconds, so it grows fast:

Attempts	Waits between attempts	Total wait
2	1s	about 1s
3	1s, 2s	about 3s
6	1s, 2s, 4s, 8s, 16s	about 31s
8	1s … 64s	about 2 minutes
10	1s … 256s	about 8.5 minutes

merge_ref_retry_attempts uses these waits for errors other than a missing merge ref, and the same maximum.

post_checkout (object)

Options that run after the sparse checkout completes, in the post-checkout hook.

unshallow (‘true’ or ‘false’)

Convert the shallow clone into a full-depth clone by running git fetch --unshallow origin after checkout. This is useful when your build requires full git history (for example, changelog generation, git log, or git blame). If the repository is already unshallow, this step is skipped.

Environment Variables

BUILDKITE_PULL_REQUEST_USING_MERGE_REFSPEC

When BUILDKITE_PULL_REQUEST_USING_MERGE_REFSPEC=true, the plugin fetches the GitHub merge ref and retries any failure up to merge_ref_retry_attempts times (default 3). See merge_ref_retry_attempts above for the waits. If the last attempt fails because the merge ref is still missing, it falls back to the normal non-merge-ref target (BUILDKITE_BRANCH when BUILDKITE_COMMIT=HEAD, otherwise BUILDKITE_COMMIT). Any other error fails the job.

Example

Below is an example of using sparse-checkout plugin.

steps:
  - label: "Pipeline upload"
    command: "buildkite-agent pipeline upload"
    plugins:
      - sparse-checkout#v1.10.0:
          paths:
            - .buildkite

Unshallowing the repository

If your build step requires full git history, enable the unshallow option under post_checkout:

steps:
  - label: "Build with full history"
    command: "make changelog"
    plugins:
      - sparse-checkout#v1.10.0:
          paths:
            - src
            - .buildkite
          post_checkout:
            unshallow: true

Handling corrupted repository states

If your jobs are frequently cancelled during the git clone phase, you may encounter failures due to uncommitted changes left in the repository. Use the clean_checkout option to handle this:

steps:
  - label: "Pipeline upload with clean checkout"
    command: "buildkite-agent pipeline upload"
    plugins:
      - sparse-checkout#v1.10.0:
          paths:
            - .buildkite
          clean_checkout: true

To avoid aggressive cleanup on every build but recover when a cancelled job leaves local changes that block checkout:

steps:
  - label: "Pipeline upload"
    command: "buildkite-agent pipeline upload"
    plugins:
      - sparse-checkout#v1.10.0:
          paths:
            - .buildkite
          clean_checkout: only-upon-failure

Cleaning up sparse-checkout state when the default path isolation is overridden

If your agent configuration causes sparse and non-sparse jobs to share the same checkout directory (overriding the plugin’s -sparse path isolation), sparse-checkout state can leak between them. Enable cleanup_sparse_state to clean this up at pre-exit and pre-checkout:

steps:
  - label: "Sparse build"
    command: "make build"
    plugins:
      - sparse-checkout#v1.10.0:
          paths:
            - src
          cleanup_sparse_state: true

The plugin will clean up stale sparse config in pre-exit (protecting the next job on the same directory from our own sparse state) and again in pre-checkout (protecting the current run from a previous interrupted job where pre-exit never fired).

Testing

docker run --rm -ti -v "${PWD}":/plugin buildkite/plugin-tester:latest

Contributing

1. Fork the repo
2. Make the changes
3. Run the tests
4. Commit and push your changes
5. Send a pull request

📜 License

The package is available as open source under the terms of the MIT License.