NewCI/CD in 2023. Check out the December Release for usage metrics, platform improvements, and a sneak peek at upcoming features.

Signed Git commits with Sigstore, Gitsign and OIDC


Humans regularly sign git commits as proof of authorship, and security-sensitive workflows can use those signatures to gate on trusted users only. What about commits created by automated workflows? We’re seeing an increase in customers creating git commits via automation in Buildkite Pipelines, but typically those commits are not signed.

This article demonstrates how to use Sigstore and Gitsign with Buildkite OpenID Connect (OIDC) to sign commits created as part of your automation flows, making it possible to prove which Buildkite pipeline created a commit.

Why sign Git commits?

Cryptographically signed Git commits validate the identity of the signer—irrespective of their network or location. Verifying these signatures significantly reduces the risk of unauthorized code changes and strengthens the security of the software delivery lifecycle.

One common place we see git commits generated in Buildkite pipelines is scheduled builds that automatically update repositories at regular intervals. By signing these Git commits, developers and downstream pipelines can reliably trace the commit directly back to the source pipeline.

Another common scenario is GitOps workflows that build new artifacts, such as Docker images, and deploy them through a Git commit to a separate configuration repository. Signing these commits firmly establishes where and when automation triggered a change for deployment.

Your toolkit: OIDC, Sigstore and Gitsign

OpenID Connect (OIDC)

Example OIDC token

OIDC token

An OIDC token is a signed JSON Web Token (JWT) provided by Buildkite containing information about the pipeline and job, including the pipeline and organization slugs, plus job specifics like the branch, the commit SHA, the job ID and the agent ID. They’re normally passed around base64 encoded, but here is an example without the encoding and signature:

{ "iss": "https://agent.buildkite.com", "sub": "organization:acme-inc:pipeline:my-app:ref:refs/heads/main:commit:9", "aud": "https://buildkite.com/acme-inc", "iat": 1669014898, "nbf": 1669014898, "exp": 1669014898, "organization_slug": "acme-inc", "pipeline_slug": "my-app", "build_number": 1, "build_branch": "main", "build_commit": "9f3182061f1e2cca4702c368cbc039b7dc9d4485", "step_key": "build", "job_id": "0814990a-477b-4fa8-9968-49074483cee", "agent_id": "0814990a-4782-42b5-afc1-16715b10b8ff" }

Sigstore

Sigstore is a platform with multiple projects that help secure software supply chains. Two foundational projects that are relevant here are Fulcio and Rekor.

Fulcio is a free code signing Certificate Authority, built to make short-lived x509 certificates. The certificates contain an OIDC identity: either an email address or a URL to a product like Buildkite.

Rekor is an immutable transparency log that (amongst other things) can verify when an x509 certificate was issued and when code signature was made.

Both Fulcio and Rekor can be deployed privately, but public good shared instances are available on the Internet and used by default.

Gitsign

Gitsign is a program that uses Fulcio to sign Git commits with an OIDC identity via a x509 certificate, and Rekor to help verify the commits later. It looks for the SIGSTORE_ID_TOKEN environment variable. If found, the OIDC token in the value is used as the identity to fetch a short lifetime x509 certificate from Sigstore and sign the commit.

Starting from Gitsign version 0.6.0, developers can now sign commits using a Buildkite OIDC token.

How to sign commits in Buildkite

To start signing commits, you will need to:

  • Install Gitsign v0.6.0 or better in your agent environment.
  • Include a command script that:
    • Sets the value of the SIGSTORE_ID_TOKEN key to a Buildkite OIDC token.
    • Configures Git to sign all commits and tags.
    • Signs commits using Gitsign.
    • Tells Gitsign to expect an x509 certificate.
#!/bin/sh set -e echo "+++ fetching OIDC token from Buildkite" # set the SIGSTORE_ID_TOKEN as Buildkite OIDC token SIGSTORE_ID_TOKEN="$(buildkite-agent oidc request-token --audience sigstore)" echo "+++ creating signed Git commit" # setup the working directory git checkout git config --local commit.gpgsign true # Sign all commits git config --local tag.gpgsign true # Sign all tags git config --local gpg.x509.program gitsign # Use gitsign for signing git config --local gpg.format x509 # gitsign expects x509 args # create a new commit on a branch and push up to GitHub BRANCH_NAME="branch-`date +%s`" git checkout -b ${BRANCH_NAME} echo `date --iso-8601=seconds` > the-date.txt git add the-date.txt git commit the-date.txt -m "update" echo "+++ pushing branch to github" git push origin -u ${BRANCH_NAME}

Verifying a Git commit was created by a Buildkite build

Many Source Control Systems will detect the commit signature, however they may not trust it as Sigstore generated certificates are not signed by common Certificate Authorities. For example, GitHub will display the commits as signed but unverified:

Screenshot of GitHub commit UI showing the certificate credentials of the git commit

Commits are signed with x509 certificates, but will still show as unverified.

To verify the commit manually, use Gitsign directly and provide the Buildkite Pipeline that you expect to have created the signature:

  • Replace https://buildkite.com/acme-inc/my-app with your pipeline URL.
  • Note that the OIDC issuer is the Buildkite agent.
$ gitsign verify \ --certificate-identity=https://buildkite.com/acme-inc/my-app \ --certificate-oidc-issuer=https://agent.buildkite.com HEAD

If the signature is valid, you'll see that Gitsign verifies it:

gitsign: Signature made using certificate ID 0x8cd9e267dcc37c109a5c1f1811ad1608c692465c | CN=sigstore-intermediate,O=sigstore.dev gitsign: Good signature from [https://buildkite.com/acme-inc/my-app](https://agent.buildkite.com) Validated Git signature: true Validated Rekor entry: true Validated Certificate claims: true

If the signature is invalid, an error is returned:

Error: none of the expected identities matched what was in the certificate, got subjects [joe@example.com] with issuer https://accounts.google.com

And if the signature is missing, an error is returned:

Error: unsupported signature type

In addition to running this verification manually, you may find it useful to verify the commits in deploy pipelines to ensure only changes from a trusted source can reach production.

For debugging, you can print an x509 certificate with this command:

git cat-file commit HEAD | sed -n '/BEGIN/, /END/p' | sed 's/^ //g' | sed 's/gpgsig //g' | sed 's/SIGNED MESSAGE/PKCS7/g' | openssl pkcs7 -print -print_certs -text

Example certificate output:

... Certificate: Data: Version: 3 (0x2) Serial Number: 0b:1e:1a:e3:c8:1d:39:12:64:18:64:aa:0a:ab:85:ce:52:e1:aa:2f Signature Algorithm: ecdsa-with-SHA384 Issuer: O=sigstore.dev, CN=sigstore-intermediate Validity Not Before: May 27 12:17:56 2023 GMT Not After : May 27 12:27:56 2023 GMT Subject: Subject Public Key Info: Public Key Algorithm: id-ecPublicKey Public-Key: (256 bit) pub: 04:60:90:72:34:1a:49:35:e6:26:84:80:8f:0d:02:59:38:b3:d6:e0:bd:2d:a6:72:73:5b:ea:34:ba:45:4a:e6:0a:c2:e0:cf:04:d3:07:fd:50:05:0e:2a:a9:a1:17:2e:c6:ad:b1:a4:30:63:af:80:1c:4d:2a:67:ef:59:3d:30:12 ASN1 OID: prime256v1 NIST CURVE: P-256 X509v3 extensions: X509v3 Key Usage: critical Digital Signature X509v3 Extended Key Usage: Code Signing X509v3 Subject Key Identifier: 57:18:0D:CA:B1:2B:AE:2B:65:86:78:90:55:98:B2:80:2D:00:28:3A X509v3 Authority Key Identifier: DF:D3:E9:CF:56:24:11:96:F9:A8:D8:E9:28:55:A2:C6:2E:18:64:3F X509v3 Subject Alternative Name: critical URI:https://buildkite.com/acme-inc/my-app 1.3.6.1.4.1.57264.1.1: https://agent.buildkite.com 1.3.6.1.4.1.57264.1.8:..https://agent.buildkite.com CT Precertificate SCTs: Signed Certificate Timestamp: Version : v1 (0x0) Log ID : DD:3D:30:6A:C6:C7:11:32:63:19:1E:1C:99:67:37:02:A2:4A:5E:B8:DE:3C:AD:FF:87:8A:72:80:2F:29:EE:8E Timestamp : May 27 12:17:56.918 2023 GMT Extensions: none Signature : ecdsa-with-SHA256 30:45:02:20:18:E5:5E:D5:33:D3:45:84:48:92:AE:30:5B:C4:CA:6B:6B:A6:FF:58:5A:AC:E0:98:C9:5C:47:EC:78:20:CD:62:02:21:00:8A:A9:C5:D0:8E:F8:04:3F:C3:D1:86:0C:89:90:7D:0D:25:3F:39:E0:35:6A:1C:1D:E9:51:93:04:15:FA:D2:D5 Signature Algorithm: ecdsa-with-SHA384 Signature Value: 30:65:02:30:72:e9:30:f1:fe:79:0b:cd:82:93:fc:54:57:35:55:2e:0b:18:fe:59:9a:78:16:7f:b4:2f:a1:a4:43:2d:d2:f4:6d:03:3d:35:c9:bb:a4:57:c8:58:fc:5f:d0:86:ff:b3:02:31:00:bc:86:6d:6b:57:b8:08:11:7b:ef:df:02:a1:5f:11:28:71:a1:fc:6c:16:56:71:78:a1:b2:e5:1b:66:8d:e8:ca:73:08:64:ae:6a

Cryptographically signing automatically generated Git commits significantly increases the security and traceability of your software supply chain.

Coupling short-lived X.509 certificates with ephemeral OIDC tokens ensures the integrity and authenticity of these changes, no matter their origin.

By adopting these practices, your organization strengthens its security posture and establishes a more robust software development lifecycle.

Further reading