Prometheus metrics

How to enable Prometheus monitoring

The Agent Stack for Kubernetes controller can expose Prometheus metrics for monitoring and observability. To enable Prometheus monitoring, complete these two steps:

1. Enable metrics port exposure in the Helm chart.
2. Create a PodMonitor resource for scraping.

Enabling metrics port exposure

Configure the prometheus-port option in your Helm deployment to expose the metrics endpoint. You can use either the command-line or the value file approach.

Command-line approach

Use the following command to expose the metrics endpoint:

helm upgrade --install agent-stack-k8s oci://ghcr.io/buildkite/helm/agent-stack-k8s \
    --namespace buildkite \
    --create-namespace \
    --set agentToken=<buildkite-cluster-agent-token> \
    --set config.prometheus-port=8080

Values file approach

Set the following configuration in your values file:

# values.yml
agentToken: "<buildkite-cluster-agent-token>"
config:
  prometheus-port: 8080
  tags:
    - queue=kubernetes

And run the following command:

helm upgrade --install agent-stack-k8s oci://ghcr.io/buildkite/helm/agent-stack-k8s \
    --namespace buildkite \
    --create-namespace \
    --values values.yml

This exposes metrics on port 8080 at the /metrics endpoint within the controller pod.

Creating a PodMonitor for scraping

If you're using Prometheus Operator, create a PodMonitor resource to automatically scrape metrics from the controller:

# buildkite-podmonitor.yml
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: buildkite-agent-stack
  namespace: buildkite
  labels:
    app: buildkite-agent-stack
spec:
  selector:
    matchLabels:
      app: agent-stack-k8s  # Replace with your actual Helm release name followed by "-agent-stack-k8s"
  podMetricsEndpoints:
  - port: metrics
    path: /metrics
    interval: 30s

Apply the PodMonitor:

kubectl apply -f buildkite-podmonitor.yml

Verification

Verify that monitoring is working correctly:

# Check that the metrics port is exposed
kubectl get pods -n buildkite -o wide
kubectl port-forward -n buildkite deployment/agent-stack-k8s 8080:8080

# In another terminal, test metrics endpoint
curl http://localhost:8080/metrics

# Verify PodMonitor is created and discovered
kubectl get podmonitor -n buildkite

Notes on using the metrics

Most metrics below are counter metrics, designed to be used in conjunction with the rate and a time window. These are named ending in _total.

PromQL examples:

• rate(buildkite_scheduler_job_create_success_total[10m]) - jobs successfully created per second over a 10 minute window.
• rate(buildkite_scheduler_job_create_errors_total[10m]) - errors per second of failures to create jobs over a 10-minute window.

Some metrics are gauges, which can be useful for diagnosing particular issues.

A few metrics are native histograms, which requires the Prometheus feature flag to be enabled (--enable-feature=native-histograms). These are mostly latency histograms named ending in _seconds, and again work well with rate:

• rate(buildkite_job_end_to_end_seconds[10m]) - histogram of time in seconds that jobs spent between being returned from a query to Buildkite and being created in Kubernetes, over a 10 minute window.
• rate(buildkite_monitor_job_query_seconds[10m]) - histogram of time spent querying Buildkite for jobs that can be scheduled, over a 10 minute window.

Some counters don't exist until their first increment

Counter metrics that have labels (for example, a reason or delete_reason column in the tables below) are created per label value the first time that value actually occurs, not at startup. A freshly installed controller, or one that simply hasn't hit a particular error yet, won't report buildkite_scheduler_job_create_errors_total at all—not even as 0—until a job creation error with some reason has actually happened. Counters without labels, such as buildkite_scheduler_job_create_success_total, are registered immediately and do start at 0.

This matters for service-level indicators (SLIs) and alerts built on rate(): querying a labeled counter that has never fired returns no data, which can be mistaken for "zero errors" rather than "this metric doesn't exist yet." For an aggregated query, sum(rate(buildkite_scheduler_job_create_errors_total[5m])) or vector(0) reliably resolves to 0 instead of no data, since the or vector(0) fallback only applies when the left-hand side is empty. See Existential issues with metrics for more detail, including patterns for preserving specific label combinations rather than aggregating them away.

Labels and their meanings

completion_watcher

Cleans up sidecar containers that the Buildkite agent doesn't manage directly, once the agent container has terminated. Without this, those containers would otherwise keep the pod running indefinitely after the job has already finished.

deduper

Prevents the same Buildkite job from being scheduled more than once, by tracking which jobs are currently in flight.

job_watcher

Watches Kubernetes Jobs (rather than Pods directly) for two problems: a Job that never had a pod created for it, and a Job whose pod was created but failed before the Buildkite agent running in it could acquire the Buildkite job. For problems with a pod that has already been created, see pod_watcher below.

limiter

Limits how many jobs are in flight at once, queuing jobs over the configured limit until a slot is available.

monitor

Polls Buildkite for jobs to run, filters them by agent tags, and passes matching jobs into the rest of the scheduling pipeline (reserver, then limiter, then deduper, then scheduler).

pod_watcher

Watches pods that have already been created, and handles failures that happen after scheduling: an image that won't pull, a pod stuck in Pending for too long, or a Buildkite job that's canceled while its pod is still pending. For a job that never got a pod at all, see job_watcher above.

scheduler

Creates the Kubernetes Job for each Buildkite job passed to it by the deduper. The scheduler is the final step in the scheduling pipeline.

Other