Controller configuration
This page covers the available commands for:
agent-stack-k8s [flags]agent-stack-k8s [command]
All references to "controller" on this page refer to the Agent Stack for Kubernetes controller.
Available commands
| Command | Description |
|---|---|
completion |
Generate the autocompletion script for the specified shell |
help |
Help about any command |
lint |
A tool for linting Buildkite pipelines |
version |
Prints the version |
Use agent-stack-k8s [command] --help for more information about a command.
Flags
| Flag and value type if applicable | Description |
|---|---|
|
string
|
The name of the Buildkite agent token secret. Default:buildkite-agent-token
|
|
|
Permits podSpecPatch to modify the |
|
string
|
The UUID of the Buildkite cluster. The agent token must be for the Buildkite cluster. |
|
string
|
The config file path. |
|
|
Debug logs. |
|
string
|
Sets a default PullPolicy for image-check init containers, used if an image pull policy is not set for the corresponding container in a podSpec or podSpecPatch. |
|
string
|
Configures a default image pull policy for containers that do not specify a pull policy and non-init containers created by the stack itself. |
|
integer
|
The maximum number of seconds a pod will run after being told to terminate, if not otherwise set by a podSpec. Default:60
|
|
duration
|
The duration after starting a Kubernetes job that the controller will wait before considering failing the job due to a missing pod (for example, when the podSpec specifies a missing service account). Default:30s
|
|
bool
|
Allow the controller to pause processing the jobs when the queue is paused on Buildkite. false
|
|
|
Displays help for the agent-stack-k8s. |
|
duration
|
Timeout for HTTP requests to the Buildkite agent API. Default:60s
|
|
string
|
The container image to use for the Buildkite agent. Defaults to a version of |
|
string
|
Configures the CPU resource limit for all 200m
|
|
string
|
Configures the memory resource limit for all 128Mi
|
|
duration
|
Duration after starting a pod that the controller will wait before considering cancelling a job due to ImagePullBackOff (e.g., when the podSpec specifies container images that cannot be pulled). Default:30s
|
|
integer
|
The maximum number of seconds a Kubernetes job is allowed to run before terminating all pods and failing. Default:21600
|
|
duration
|
Controls the interval between job state queries while a pod is still Pending. Default:5s
|
|
integer
|
The number of concurrent goroutines for converting Buildkite jobs into Kubernetes jobs. Default:25
|
|
string
|
The prefix to use when creating Kubernetes job names. Default:buildkite-
|
|
duration
|
The time to retain Kubernetes jobs after completion. Default:10m0s
|
|
integer
|
The burst value of the K8s client rate limiter. Default:20
|
|
integer
|
The QPS value of the K8s client rate limiter. Default:10
|
|
string
|
Sets the log format. One of logfmt
|
|
|
Logs full HTTP request and response payloads. Only active when log level is debug. This may log sensitive information including tokens and secrets. |
|
string
|
Sets the log level. One of info
|
|
integer
|
The maximum jobs in flight, where a value of 0 means no maximum. Default:25
|
|
string
|
The Kubernetes namespace to create resources in. Default:default
|
|
|
Disables colored log output (ANSI escape codes). Colors are disabled automatically when the output is not a terminal. |
|
string
|
The Buildkite organization name to watch. |
|
integer
|
Sets the maximum number of pages when retrieving Buildkite jobs to be scheduled. Increasing this value will increase the number of requests made to the Buildkite API and number of jobs to be scheduled on the Kubernetes cluster. Default:2
|
|
integer
|
Sets the maximum number of jobs per page when retrieving Buildkite jobs to be scheduled. Default:1000
|
|
duration
|
The time to wait between polling for new jobs (minimum 1s
|
|
string
|
The bind address to expose the pprof profiler (for example, |
|
|
Causes the controller to prohibit the Kubernetes plugin specified within jobs (pipeline YAML). Enabling this causes jobs with a Kubernetes plugin to fail, preventing the pipeline YAML from having any influence over the podSpec. |
|
uint16
|
The bind port to expose Prometheus /metrics. Specifying 0 disables this feature. |
|
duration
|
Controls the interval between pagination cursor resets. Increasing this value will increase the number of jobs to be scheduled but also delay picking up any jobs that were missed from the start of the query. Default:10s
|
|
string
|
The Buildkite queue to poll for jobs. If set, overrides the queue tag. |
|
|
Disables and skips all |
|
strings
|
A comma-separated list of agent tags. The "queue" tag must be unique (for example, "queue=kubernetes,os=linux"). Default:[queue=kubernetes]
|
|
integer
|
Sets the maximum number of jobs the controller will hold in the work queue. Default:1000000
|
Kubernetes node selection
The Buildkite Agent Stack for Kubernetes controller can be deployed to particular Kubernetes Nodes, using the Kubernetes PodSpec nodeSelector field.
The nodeSelector field can be defined in the controller's configuration:
# values.yml
...
nodeSelector:
teamowner: "services"
config:
...
Additional environment variables for the controller container
If the Buildkite Agent Stack for Kubernetes controller container requires extra environment variables in order to correctly operate inside your Kubernetes cluster, they can be added to your values YAML file and applied during a deployment with Helm.
The controllerEnv field can be used to define extra Kubernetes EnvVar environment variables that will apply to the Buildkite Agent Stack for Kubernetes controller container:
# values.yml
...
controllerEnv:
- name: KUBERNETES_SERVICE_HOST
value: "10.10.10.10"
- name: KUBERNETES_SERVICE_PORT
value: "8443"
config:
...
Custom annotations for the controller
If you need to add custom annotations to the Agent Stack for Kubernetes controller pod, these annotations can be defined in your values YAML file and applied during a deployment with Helm. Note that the controller pod will also have the annotations checksum/config and checksum/secrets to track changes to the configuration and secrets.
The annotations field can be used to define custom annotations that will be applied to the Buildkite Agent Stack for Kubernetes controller pod:
# values.yml
...
annotations:
kubernetes.io/description: "Agent Stack K8s Controller"
prometheus.io/scrape: "true"
prometheus.io/port: "8080"
config:
...
Cleaning up old Buildkite Pipelines jobs
If you are using Kubernetes v1.23 and earlier, you may sometimes find that old jobs are still present in your Kubernetes cluster and are not getting automatically cleaned up. This may consume unnecessary space and potentially cause other disruptions with deployments.
If you notice old Buildkite Pipelines jobs still present in your Kubernetes cluster, you can use the clean-up-job.yaml script (with usage instructions provided at the top of this file) located in Agent Stack for Kubernetes repository to clean up your old Buildkite jobs.