Controller configuration
This page covers the available commands for:
agent-stack-k8s [flags]agent-stack-k8s [command]
All references to "controller" on this page refer to the Agent Stack for Kubernetes controller.
Available commands
| Command | Description |
|---|---|
completion |
Generate the autocompletion script for the specified shell |
help |
Help about any command |
lint |
A tool for linting Buildkite pipelines |
version |
Prints the version |
Use agent-stack-k8s [command] --help for more information about a command.
Flags
| Flag and value type if applicable | Description |
|---|---|
|
string
|
The name of the Buildkite agent token secret. Default:buildkite-agent-token
|
|
string
|
The Buildkite API token with GraphQL scopes. |
|
string
|
The UUID of the Buildkite cluster. The agent token must be for the Buildkite cluster. |
|
string
|
The config file path. |
|
|
Debug logs. |
|
string
|
Sets a default PullPolicy for image-check init containers, used if an image pull policy is not set for the corresponding container in a podSpec or podSpecPatch. |
|
string
|
Configures a default image pull policy for containers that do not specify a pull policy and non-init containers created by the stack itself. Default:IfNotPresent
|
|
duration
|
The duration after starting a Kubernetes job that the controller will wait before considering failing the job due to a missing pod (for example, when the podSpec specifies a missing service account). Default:30s
|
|
string
|
The Buildkite GraphQL endpoint URL. |
|
integer
|
Sets the amount of results returned by GraphQL queries when retrieving jobs to be scheduled. Default:100
|
|
|
Displays help for the agent-stack-k8s. |
|
string
|
The image to use for the Buildkite agent. Default:ghcr.io/buildkite/agent:3.91.0
|
|
duration
|
Duration after starting a pod that the controller will wait before considering cancelling a job due to ImagePullBackOff (e.g., when the podSpec specifies container images that cannot be pulled). Default:30s
|
|
duration
|
Controls the interval between job state queries while a pod is still Pending. Default:5s
|
|
integer
|
Controls the interval between job state queries while a pod is still Pending. Default:5
|
|
duration
|
The time to retain Kubernetes jobs after completion. Default:10m0s
|
|
integer
|
The maximum number of seconds a Kubernetes job is allowed to run before terminating all pods and failing. Default:21600
|
|
integer
|
The burst value of the K8s client rate limiter. Default:20
|
|
integer
|
The QPS value of the K8s client rate limiter. Default:10
|
|
integer
|
The maximum jobs in flight, where a value of 0 means no maximum. Default:25
|
|
string
|
The Kubernetes namespace to create resources in. Default:default
|
|
string
|
The Buildkite organization name to watch. |
|
integer
|
Sets the maximum depth of pagination when retrieving Buildkite jobs to be scheduled. Increasing this value will increase the number of requests made to the Buildkite GraphQL API and number of jobs to be scheduled on the Kubernetes cluster. Default:1
|
|
duration
|
The time to wait between polling for new jobs (minimum 1s
|
|
string
|
The bind address to expose the pprof profiler (for example, |
|
|
Causes the controller to prohibit the Kubernetes plugin specified within jobs (pipeline YAML). Enabling this causes jobs with a Kubernetes plugin to fail, preventing the pipeline YAML from having any influence over the podSpec. |
|
uint16
|
The bind port to expose Prometheus /metrics. Specifying 0 disables this feature. |
|
duration
|
Duration after querying jobs in Buildkite that the data is considered valid Default:10s
|
|
strings
|
A comma-separated list of agent tags. The "queue" tag must be unique (for example, "queue=kubernetes,os=linux") Default:[queue=kubernetes]
|
|
bool
|
Allow the controller to pause processing the jobs when the queue is paused on Buildkite. false
|
Kubernetes node selection
The Buildkite Agent Stack for Kubernetes controller can be deployed to particular Kubernetes Nodes, using the Kubernetes PodSpec nodeSelector field.
The nodeSelector field can be defined in the controller's configuration:
# values.yml
...
nodeSelector:
teamowner: "services"
config:
...
Additional environment variables for the controller container
If the Buildkite Agent Stack for Kubernetes controller container requires extra environment variables in order to correctly operate inside your Kubernetes cluster, they can be added to your values YAML file and applied during a deployment with Helm.
The controllerEnv field can be used to define extra Kubernetes EnvVar environment variables that will apply to the Buildkite Agent Stack for Kubernetes controller container:
# values.yml
...
controllerEnv:
- name: KUBERNETES_SERVICE_HOST
value: "10.10.10.10"
- name: KUBERNETES_SERVICE_PORT
value: "8443"
config:
...
Custom annotations for the controller
If you need to add custom annotations to the Agent Stack for Kubernetes controller pod, these annotations can be defined in your values YAML file and applied during a deployment with Helm. Note that the controller pod will also have the annotations checksum/config and checksum/secrets to track changes to the configuration and secrets.
The annotations field can be used to define custom annotations that will be applied to the Buildkite Agent Stack for Kubernetes controller pod:
# values.yml
...
annotations:
kubernetes.io/description: "Agent Stack K8s Controller"
prometheus.io/scrape: "true"
prometheus.io/port: "8080"
config:
...
Cleaning up old Buildkite Pipelines jobs
If you are using Kubernetes v1.23 and earlier, you may sometimes find that old jobs are still present in your Kubernetes cluster and are not getting automatically cleaned up. This may consume unnecessary space and potentially cause other disruptions with deployments.
If you notice old Buildkite Pipelines jobs still present in your Kubernetes cluster, you can use the clean-up-job.yaml script (with usage instructions provided at the top of this file) located in Agent Stack for Kubernetes repository to clean up your old Buildkite jobs.