Workload identity federation

How it works

An administrator creates a workload identity before the workload connects. Each token exchange then works as follows:

1. The workload requests an OIDC token from its identity provider, using the audience configured on the workload identity.
2. The identity provider returns a signed OIDC token that contains claims about the workload, such as its Buildkite organization, pipeline, and branch.
3. The workload sends the workload identity's client ID and the OIDC token to the Buildkite token exchange endpoint, https://buildkite.com/oauth/token.
4. Buildkite checks the token against the workload identity, as described in OIDC token requirements, and returns a short-lived API access token. The access token acts as the workload identity, not as a person.
5. The workload uses the access token to call the Buildkite REST API.
6. Buildkite returns the API response if the token's pipeline access and scopes allow the request.

Create a workload identity

You need to be a Buildkite organization administrator to create a workload identity. You can also create and manage workload identities with the workload identities REST API. For the settings to use on a specific platform, follow Working with Buildkite agents or Working with AWS.

To create a workload identity:

1. Navigate to your organization's Workload Identities settings, and select New Workload Identity.
2. In Details, enter a Name and Description.

3. In Trust Policy, describe the OIDC tokens that Buildkite trusts:

   • Provider: Select the Buildkite Agents, AWS, GitHub Actions, or Custom OIDC configuration.
   • OIDC issuer: The token's iss claim.
   • OIDC audience: The token's aud claim. Keep the default, which is unique to your Buildkite organization.
   • OIDC subject: The token's sub claim. Accepts * wildcards, but keep it as specific as possible.
   • Claims (optional): Claims that must have exact values, to lock the workload identity to tokens from one owner. Learn more in Lock tokens to an owner with claims.
   • Claim conditions (optional): An expression for any other check on the token's claims. Learn more in Write claim conditions.
   • JWKS override (optional, under Advanced settings): If the issuer doesn't support OIDC discovery, add a JSON Web Key Set (JWKS) URL or inline JWKS JSON.

   Note: A trust policy that's too broad is a serious security risk, because it lets workloads that you didn't intend mint API access tokens for your Buildkite organization. Make the OIDC subject as restrictive as possible, and use claims and claim conditions to narrow it further.

4. In Resource Access, restrict which pipelines tokens can reach. By default, tokens can't reach any pipelines. Learn more in Choose pipeline access.

5. Expand Grantable Scopes, and select the maximum REST API scopes that a token can have. A token exchange request can ask for any subset of these scopes. Resource Access doesn't limit the read_organizations and write_organizations scopes, which let a token read your Buildkite organization and manage its members.

6. Expand Default Scopes, and select the scopes that a token gets when the request doesn't specify any. Default scopes must be a subset of the grantable scopes.

7. In Security, set the following:

   • Maximum token lifetime (seconds): From 30 to 3,600 seconds (one hour), which is the default. A token can expire sooner. Learn more in Token lifetime.
   • Allowed IPs for token exchange requests (optional, recommended): Space-separated CIDR ranges that can exchange tokens.
   • Allowed IPs for minted token usage (optional, recommended): Space-separated CIDR ranges that can use minted tokens.

8. Select Create Workload Identity.

The workload identity's page shows the Client ID, Token endpoint, and OIDC audience that the workload needs to exchange an OIDC token. These values aren't secrets, so you can store them in CI variables or commit them to your repository.

A workload identity trusts one OIDC issuer and one audience. Create one for each platform that your workloads run on.

Lock tokens to an owner with claims

Claims lock a workload identity to OIDC tokens from one owner, such as your Buildkite organization. The field holds a JSON object. Each key names a top-level claim in the token, and each value is the exact value that the claim must have:

{
  "organization_id": "00000000-0000-4000-8000-000000000001"
}

Buildkite agents in every Buildkite organization share the same issuer. A workload identity for Buildkite agents must therefore have an organization_id or organization_slug claim.

Buildkite compares claims as follows:

• JSON types: Values must match exactly, including their JSON type, so the string "42" doesn't match the number 42.
• Missing claims: The token doesn't match.
• Nested and array claims: Keys only name top-level claims, and a claim whose value is an array or an object never matches. Use claim conditions for these claims.

Claims can hold up to 20 claims, and can't include iss, aud, sub, exp, iat, nbf, or jti, which Buildkite checks separately.

Write claim conditions

Claim conditions hold a Common Expression Language (CEL) expression for any check that claims can't make, such as a list of allowed values, a branch prefix, or alternatives across several claims. The expression reads the token's claims from the claims variable. Buildkite only mints an API access token when the expression returns true.

For example, this expression on a Buildkite agent's OIDC token accepts builds of release branches, except release/legacy, from two pipelines:

claims.pipeline_id in ["00000000-0000-4000-8000-000000000002",
                       "00000000-0000-4000-8000-000000000003"]
  && claims.build_branch.startsWith("release/")
  && claims.build_branch != "release/legacy"

The following expressions show common checks:

When the expression reads a claim that the token doesn't have, it fails, and Buildkite refuses the token. Use has() to check for an optional claim first. Comparisons check JSON types, as they do for claims.

Buildkite won't save an expression that:

• Is longer than 4,096 bytes
• Reads any variable other than claims
• Uses a macro that loops over a list, such as all(), exists(), map(), or filter()
• Returns something other than a boolean
• Returns true for a token with no claims, such as true or !has(claims.environment)

Choose pipeline access

The Pipelines setting in Resource Access limits which pipelines a workload identity's tokens can reach:

• None: No pipelines. A new workload identity starts with this option.
• Selected: Up to 20 pipelines that you choose.
• All: Every pipeline in your Buildkite organization, including pipelines created later.

For other pipelines, the REST API returns 404 Not Found. A token keeps the pipeline access that the workload identity had when the token was minted. To remove access from existing tokens, disable the workload identity, which revokes them.

Exchange an OIDC token

A workload gets an OIDC token, exchanges it for an API access token, and then calls the REST API:

# 1. Get an OIDC token, for example from a Buildkite agent.
OIDC_TOKEN="$(buildkite-agent oidc request-token \
  --audience "$BUILDKITE_AUDIENCE")"

# 2. Exchange it for a short-lived Buildkite API access token.
BUILDKITE_API_TOKEN="$(curl -sSf https://buildkite.com/oauth/token \
  -d "grant_type=client_credentials" \
  -d "client_id=$BUILDKITE_CLIENT_ID" \
  -d "client_assertion_type=urn:ietf:params:oauth:client-assertion-type:jwt-bearer" \
  -d "client_assertion=$OIDC_TOKEN" | jq -r .access_token)"

# 3. Call the REST API with the access token.
curl -sSf -H "Authorization: Bearer $BUILDKITE_API_TOKEN" \
  https://api.buildkite.com/v2/organizations/acme-inc/builds

Set BUILDKITE_CLIENT_ID and BUILDKITE_AUDIENCE to the workload identity's Client ID and OIDC audience. How the workload gets its OIDC token depends on the platform. Learn more in Working with Buildkite agents and Working with AWS.

Request parameters

Response

A successful response returns an API access token with the workload identity's default scopes. The expires_in value is the token's lifetime in seconds:

{
  "access_token": "bkwi_...",
  "token_type": "Bearer",
  "expires_in": 299,
  "scope": "read_builds"
}

The token only works for your Buildkite organization, and only with the REST API.

Token lifetime

An API access token never outlives the OIDC token that it was exchanged for. The access token expires at the earliest of:

• The OIDC token's expiry, from its exp claim
• The exchange time plus the workload identity's Maximum token lifetime
• The exchange time plus the request's expires_in value, if set

To get a shorter-lived token, add an expires_in parameter to the request, in seconds.

Exchange an OIDC token soon after the workload requests it, because the access token only lasts as long as the OIDC token has left.

Cache the access token and reuse it until it's close to expiry, instead of exchanging a token for every API request. To keep calling the API after that, such as in a long-running job, request a new OIDC token and exchange it.

Get a new OIDC token for each exchange

A workload identity accepts each OIDC token only once. Until the OIDC token expires, Buildkite refuses to exchange it again, and returns an invalid_client error. Someone who copies an OIDC token after the workload exchanged it can't use it to mint API access tokens.

Request a new OIDC token for every exchange, including retries. For example, a retry after a network error fails if Buildkite accepted the first request.

Buildkite identifies OIDC tokens by their jti (JWT ID) claim. Buildkite agent, GitHub Actions, and AWS STS tokens have a unique jti claim. A workload can exchange an OIDC token without a jti claim more than once, until it expires.

OIDC token requirements

Buildkite only mints a token when all of the following are true:

• The workload identity is enabled.
• The request comes from an IP address that the workload identity's Allowed IPs for token exchange requests and the organization's API IP allowlist allow, if set.
• The OIDC token's iss claim exactly matches the OIDC issuer.
• The OIDC token's aud claim equals the OIDC audience, or is an array that contains it.
• The OIDC token's sub claim matches the OIDC subject.
• The OIDC token is signed with RS256, RS384, RS512, PS256, PS384, PS512, ES256, ES384, or ES512, and the signature matches a key from the issuer or the JWKS override.
• The OIDC token hasn't expired, and its iat and nbf claims are no more than 60 seconds in the future.
• The OIDC token's lifetime (its exp claim minus its iat claim) is one hour or less.
• The OIDC token's claims match Claims, and the Claim conditions expression returns true.
• If the OIDC token has a jti claim, the claim is a non-empty string that the workload identity hasn't already accepted.

Buildkite agent, GitHub Actions, and AWS STS tokens last five minutes by default, and CircleCI tokens last one hour. GitLab CI/CD tokens last as long as the job timeout, and fail this check when the timeout is longer than one hour. Request OIDC tokens that last no longer than the workload needs.

By default, Buildkite finds the issuer's signing keys with OIDC discovery, from the jwks_uri in the issuer's discovery document. If you set the JWKS override to an HTTPS URL, Buildkite fetches the keys from that URL instead. Buildkite caches fetched keys for up to one hour, so providers can rotate keys without changes to the workload identity. You can also add the JWKS inline as JSON. Inline keys never refresh, so update them before the provider signs tokens with a new key.

Troubleshooting

The token endpoint returns RFC 6749 §5.2 error responses with an error code and, for some errors, an error_description:

{
  "error": "invalid_scope",
  "error_description": "Requested scopes exceed grantable scopes"
}

Common errors

Find why a token exchange failed

Buildkite returns an invalid_client error for every verification failure. When the OIDC token is expired, reused, or malformed, or its claims don't match the workload identity, the error_description says so. Buildkite doesn't describe other failures, such as a wrong client ID, issuer, audience, or signature, a disabled workload identity, or a request from an IP address that isn't allowed. Failed exchanges don't appear in the audit log.

To find the cause of a failure without a description, decode the OIDC token in the workload. Decoding doesn't need a key:

# The header, which holds the signing algorithm (alg) and key ID (kid)
echo "$OIDC_TOKEN" | jq -R 'split(".")[0] | gsub("-"; "+") | gsub("_"; "/") | @base64d | fromjson'

# The payload, which holds the claims
echo "$OIDC_TOKEN" | jq -R 'split(".")[1] | gsub("-"; "+") | gsub("_"; "/") | @base64d | fromjson'

Print only the decoded header and payload, never the token itself, because someone else could exchange it before it expires.

Compare the result with the OIDC token requirements. Common causes are:

• The iss claim and the OIDC issuer differ by a trailing /.
• The OIDC subject doesn't match the whole sub claim. Use * for the parts that change between runs, such as the commit and step key in a Buildkite agent token.
• A claim has the wrong JSON type, such as "123456789" for a claim of 123456789.
• A claim condition reads a claim that the token doesn't have. Check for the claim with has() first.
• The request uses the wrong client ID, the workload identity is disabled, or the request comes from an IP address that isn't allowed.
• The workload sent an OIDC token that the workload identity had already accepted, for example in a retry. Learn more in Get a new OIDC token for each exchange.

Fix API requests that return 404

A REST API request returns 404 Not Found for an existing pipeline, build, or job when the token can't reach that pipeline. Add the pipeline to the workload identity's Resource Access setting, then exchange a new OIDC token.