Git mirrors

When to use Git mirrors

Git mirrors optimize performance and reduce network bandwidth, helping you minimize the time it takes to re-clone large repositories. They're particularly useful in self-hosted build infrastructure where multiple agents are running.

Key benefits of Git mirrors:

• Speed up cloning by sharing objects across checkouts instead of fetching everything repeatedly.
• Reduce disk usage by storing common objects once in the mirror.
• Achieve faster builds with less time spent on Git operations.
• Maintain a complete copy of the repository on your infrastructure as a local backup.
• Handle large repositories more efficiently, especially those with many files and extensive history.

How Git mirrors work

Git mirrors leverage two core Git features:

• git clone --mirror creates a complete copy of the remote repository, including all branches and references. This differs from git clone --bare by capturing everything from the remote. This is crucial when the agent doesn't "know" which branch it needs to build ahead of time.

• git clone --reference creates checkouts that borrow objects from the mirror instead of fetching them from the remote. This saves both network bandwidth and disk space. One caveat is that checkouts depend on the mirror remaining healthy and available.

Setting up Git mirrors

Configure Git mirroring using the --git-mirrors-path flag on agents. This flag sets the central location (the Git mirror directory) where the agent stores mirrors.

Use these agent configuration options:

• git-clone-mirror-flags
• git-fetch-base-branch
• git-mirrors-lfs-cache
• git-mirrors-lock-timeout
• git-mirrors-path
• git-mirrors-skip-update
• git-skip-fetch-existing-commits

On the Elastic CI Stack for AWS, enable Git mirrors by setting the BuildkiteAgentEnableGitMirrors CloudFormation parameter to true. To avoid the initial full clone on freshly launched instances, you can also pre-populate mirrors from S3 archives at instance boot — see Git mirror seeding.

Caching Git LFS objects in mirrors

By default, a Git mirror caches only Git objects. When Git LFS is enabled for a checkout, each checkout downloads its own LFS objects from the LFS server, even when it shares a mirror with other jobs on the same host.

Set the git-mirrors-lfs-cache agent configuration option to true to also cache LFS objects in the mirror. While the agent holds the mirror's update lock, it fetches the job's LFS objects into the mirror. Each checkout then reuses those objects from the mirror instead of downloading them from the LFS server again. If an object is missing from the mirror or the prefetch fails, the checkout falls back to downloading it from the LFS server as normal.

This option is off by default, because whether it helps depends on the host:

• The mirror grows with every LFS object it fetches, and nothing prunes it automatically. On a persistent or shared mirror volume, this can use a significant amount of disk space over time.
• The LFS prefetch runs while the mirror's update lock is held, so other jobs that need the same mirror wait longer. A slow LFS download can cause the lock wait to exceed git-mirrors-lock-timeout.
• Mirrors are often stored on network or shared volumes, where reading a cached object isn't always faster than downloading it from the LFS server.

git-mirrors-lfs-cache requires git-mirrors-path to be set, and like the other git-mirrors-* options, it can only be set as agent configuration—not from pipeline environment variables, hooks, or plugins.

Git submodules

Git submodules allow one repository to reference another, enabling code in the first repository to use contents from the second. When you enable mirrors, the agent also mirrors submodules. You can create and update submodule mirrors the same way as regular mirrors.

Submodule mirrors require special handling on the agent. The agent must update the submodule configuration in the main repository using git submodule update --reference <submodule_mirror>. This must be done for each submodule, which means parsing the submodule configuration and then iterating over the submodules to update them.

Common issues with Git mirrors

This section covers common issues with Git mirrors and how to solve or prevent them.

Parallelism issues

When multiple agents fetch from the same mirror simultaneously, conflicts may occur. The Buildkite agent implements a locking system that prevents multiple agents from updating the same mirror simultaneously. This file-based lock works across multiple machines, even when the mirror directory is a network file share.

Checkout corruption

A mirror repository and a reference clone (checkout) work together. The checkout borrows most of its objects from the mirror instead of storing them locally. When git fetch updates the mirror, it automatically triggers maintenance that cleans up objects Git considers unnecessary. By default, git fetch runs git maintenance run --auto. However, some of those objects may actually be needed by the checkout. If this happens, you'll see checkout errors and need to delete the checkout directory and re-clone (which is quick with mirrors enabled).

Updating mirrors

The command git remote update is commonly used to refresh mirrors because it updates all references. However, git fetch origin <branch> is the preferred approach for most CI/CD use cases because it fetches only the objects a particular job requires. The Buildkite agent now uses git fetch origin <branch> instead of git remote update for this reason. Remember that git remote update also runs auto maintenance that may cause the checkout corruption mentioned above.