Buildah container builds

Key difference from Docker

Buildah does not use the Docker daemon, which means images built with Buildah are managed separately from Docker images. When using the Docker Buildkite plugin to run Buildah, the images built by Buildah won't be visible to Docker commands running outside the Buildah container. As a result:

• docker images won't show Buildah-built images
• Images must be pushed to a registry to be shared between Buildah and Docker environments
• Buildah stores its images in its own storage backend

Using Buildah with Elastic CI Stack for AWS

To use Buildah with the Elastic CI Stack for AWS, you will need to run Buildah inside a container using the Docker Buildkite plugin.

Building and pushing to Buildkite Package Registries

The following example shows how to build a container image and push it to Buildkite Package Registries using OIDC authentication:

env:
  PACKAGE_REGISTRY_NAME: "my-docker-registry"
  BUILDAH_ISOLATION: "chroot"

steps:
  - label: "🐳 Build and Push with Buildah"
    plugins:
      - docker#v5.13.0:
          image: "quay.io/buildah/stable:latest"
          privileged: true
          userns: "host"
          mount-buildkite-agent: true
    command: |
      buildah bud \
        --format docker \
        --file Dockerfile \
        --tag packages.buildkite.com/${BUILDKITE_ORGANIZATION_SLUG}/${PACKAGE_REGISTRY_NAME}/myapp:${BUILDKITE_BUILD_NUMBER} \
        .

      # Verify the image was built
      buildah images

      # Authenticate using OIDC and push to registry
      buildkite-agent oidc request-token \
        --audience "https://packages.buildkite.com/${BUILDKITE_ORGANIZATION_SLUG}/${PACKAGE_REGISTRY_NAME}" \
        --lifetime 300 | \
      buildah login \
        --authfile ./bk-oidc-auth.json \
        "packages.buildkite.com/${BUILDKITE_ORGANIZATION_SLUG}/${PACKAGE_REGISTRY_NAME}" \
        --username buildkite \
        --password-stdin

      buildah push \
        --authfile ./bk-oidc-auth.json \
        packages.buildkite.com/${BUILDKITE_ORGANIZATION_SLUG}/${PACKAGE_REGISTRY_NAME}/myapp:${BUILDKITE_BUILD_NUMBER}

Configuration breakdown

This section explains the components of the configuration necessary for using Buildah with Buildkite Pipelines.

Environment variables

• PACKAGE_REGISTRY_NAME: the name of your Buildkite Package Registry.
• BUILDAH_ISOLATION: "chroot": sets the isolation mode for Buildah. The chroot mode provides good isolation without requiring additional privileges.

Docker plugin configuration

• image: "quay.io/buildah/stable:latest": uses the official Buildah container image.
• privileged: true: grants extended privileges to the container. Is required for Buildah to create and manage container images.
• userns: "host": uses the host's user namespace and is necessary for Buildah to function correctly in this configuration.
• mount-buildkite-agent: true: mounts the Buildkite agent binary into the container, enabling the use of buildkite-agent oidc request-token.

Buildah commands

• buildah bud: builds an image from a Dockerfile.
  • --format docker: produces a Docker-compatible image format.
  • --file Dockerfile: specifies the path to the Dockerfile.
  • --tag: tags the resulting image.
• buildah images: lists the built images (useful for verification).
• buildah login: authenticates with a container registry.
  • --authfile: specifies where to store authentication credentials.
  • --username and --password-stdin: provide credentials for authentication.
• buildah push: pushes the image to a registry.
  • --authfile: uses the authentication file created during login.

Understanding the components

This section covers the key components and configuration options for running Buildah with the Elastic CI Stack for AWS.

• Container images: the official Buildah image that runs in privileged mode is quay.io/buildah/stable:latest.
• Security contexts: the configuration shown uses privileged mode where the container runs as root with privileged: true, bypassing most security controls.
• Storage driver: Buildah uses the following container storage backends:
  • overlay: fast and efficient, used by default. Modern Buildah images support overlay without requiring /dev/fuse or additional configuration.
  • vfs: fallback option that works in all environments but slower, especially with bigger images. Can be specified with --storage-driver vfs if overlay encounters issues.
• Storage paths: when running in the container as root (privileged), Buildah uses the system location /var/lib/containers.
• Build isolation: the recommended isolation mode for Buildah container environments is BUILDAH_ISOLATION=chroot. It provides good isolation without requiring additional privileges.

Customizing the build

This section contains instructions for customizing your Buildah builds.

Using build arguments

You can pass build arguments to your Dockerfile:

command: |
  buildah bud \
    --format docker \
    --build-arg VERSION=${BUILDKITE_BUILD_NUMBER} \
    --build-arg COMMIT=${BUILDKITE_COMMIT} \
    --file Dockerfile \
    --tag packages.buildkite.com/${BUILDKITE_ORGANIZATION_SLUG}/${PACKAGE_REGISTRY_NAME}/myapp:${BUILDKITE_BUILD_NUMBER} \
    .

Targeting specific build stages

For multi-stage Dockerfiles, you can target a specific stage:

command: |
  buildah bud \
    --format docker \
    --target production \
    --file Dockerfile \
    --tag packages.buildkite.com/${BUILDKITE_ORGANIZATION_SLUG}/${PACKAGE_REGISTRY_NAME}/myapp:${BUILDKITE_BUILD_NUMBER} \
    .

Using alternative storage driver

If you encounter issues with the default overlay driver, you can use vfs as a fallback:

command: |
  buildah bud \
    --storage-driver vfs \
    --format docker \
    --file Dockerfile \
    --tag packages.buildkite.com/${BUILDKITE_ORGANIZATION_SLUG}/${PACKAGE_REGISTRY_NAME}/myapp:${BUILDKITE_BUILD_NUMBER} \
    .

Troubleshooting

This section describes common issues for Buildah and the ways of solving these issues.

Permission denied errors

• Ensure privileged: true is configured in the Docker plugin.
• Verify the userns: "host" setting is present.
• Confirm the BUILDAH_ISOLATION environment variable is set to "chroot".

Storage driver errors

• The default overlay driver should work in privileged mode.
• If overlay fails, try --storage-driver vfs as a fallback (this is a slower but more compatible approach).
• Check that the storage volume has sufficient space.

Registry authentication failures

• Ensure the mount-buildkite-agent: true setting is configured so buildkite-agent oidc request-token is available.
• Verify that the OIDC token audience matches your Package Registry URL exactly.
• Check that the authentication file is being passed correctly to the buildah push command.

Image not found after build

Remember that Buildah images are separate from Docker images. If you need to use the image in subsequent steps:

• Push the image to a registry and pull it in later steps.
• Use the same Buildah container for all operations on that image.