Amazon EventBridge

The Amazon EventBridge notification service in Buildkite lets you stream events in real-time from your Buildkite account to your AWS account.

Events

Once you’ve configured an Amazon EventBridge notification service in Buildkite, the following events are published to the partner event bus:

Detail Type Description
Build Created A build has been created
Build Started A build has started
Build Finished A build has finished
Job Scheduled A job has been scheduled
Job Started A command step job has started running on an agent
Job Finished A job has finished
Job Activated A block step job has been unblocked via the web or API
Agent Connected An agent has connected
Agent Lost An agent has been marked as lost
Agent Disconnected An agent has disconnected
Agent Stopping An agent is stopping
Agent Stopped An agent has stopped

Configuring

In your Buildkite Organization’s Notification Settings, add an Amazon EventBridge notification service:

Screenshot of Add Buildkite Amazon EventBridge Button

Once you’ve entered your AWS region and AWS Account ID, a Partner Event Source will be created in your AWS account matching the "Partner Event Source Name" shown on the settings page:

Screenshot of Buildkite Amazon EventBridge Notification Settings

You can then start consuming the events in your AWS account. The links to "Partner Event Sources Console" and "Event Rules" take you to the relevant pages in your AWS Console.

Filtering

When creating your EventBridge rule you can specify an "Event Pattern" filter to limit which events will be processed. You can use this to respond only to certain events based on the type, or any attribute from within the event payload.

For example, to only process Build Finished events you’d configure your rule with the following event pattern:

Screenshot of configuring an EventBridge Event Pattern filter

You can use any event property in your custom event pattern. For example, the following event pattern allows only “Build Started" and "Build Finished" events containing a particular pipeline slug:

{
  "detail-type": [ "Build Started", "Build Finished" ],
  "detail": {
    "pipeline": {
      "slug": "some-pipeline"
    }
  }
}

See the Example Event Payloads for full list of properties, and the AWS EventBridge Event Patterns documentation for full details on the pattern syntax.

Logging

To debug your EventBridge events you can configure a rule to route the event bus directly to AWS CloudWatch Logs:

Screenshot of configuring an EventBridge Rule to send to CloudWatch Logs

You can then use CloudWatch Logs Insights to query and inspect the live events from your event bus, by choosing the event log group configured above:

Screenshot of CloudWatch Logs Insights

Lambda Example: Track agent wait times via CloudWatch Metrics

You can use the following AWS Lambda and Job Started event to publish a CloudWatch metric which tracks how long jobs are waiting for agents to become available:

const AWS = require("aws-sdk");
const cloudWatch = new AWS.CloudWatch();

exports.handler = (event, context, callback) => {
  const waitTime =
    new Date(event.detail.job.started_at) -
    new Date(event.detail.job.runnable_at);

  console.log(`Job started after waiting ${waitTime} seconds`);

  cloudWatch.putMetricData(
    {
      Namespace: "Buildkite",
      MetricData: [
        {
          MetricName: "Job Agent Wait Time",
          Timestamp: new Date(),
          StorageResolution: 1,
          Unit: "Seconds",
          Value: waitTime,
          Dimensions: [
            {
              Name: "Pipeline",
              Value: event.detail.pipeline.slug
            }
          ]
        }
      ]
    },
    (err, data) => {
      if (err) console.log(err, err.stack);
      callback(null, "Finished");
    }
  );
};

Example Event Payloads

Build Created

{
  "detail-type": "Build Created",
  "detail": {
    "version": 1,
    "build": {
      "uuid": "8fcaa7b9-e175-4110-9f48-f79949806a31",
      "graphql_id": "QnVpbGQtLS04ZmNhYTdiOS1lMTc1LTQxMTAtOWY0OC1mNzk5NDk4MDZhMzE=",
      "number": 123456,
      "commit": "5a741616cdf07dc87c5adafe784321eeeb639e33",
      "message": "Merge pull request #456 from my-org/chore/update-deps",
      "branch": "master",
      "state": "scheduled",
      "started_at": null,
      "finished_at": null,
      "source": "webhook"
    },
    "pipeline": {
      "uuid": "88d73553-5533-4f56-9c16-fb38d7817d8f",
      "graphql_id": "UGlwZWxpbmUtLS04OGQ3MzU1My01NTMzLTRmNTYtOWMxNi1mYjM4ZDc4MTdkOGY=",
      "slug": "my-pipeline",
      "repo": "git@somewhere.com:project.git"
    },
    "organization": {
      "uuid": "a98961b7-adc1-41aa-8726-cfb2c46e42e0",
      "graphql_id": "T3JnYW5pemF0aW9uLS0tYTk4OTYxYjctYWRjMS00MWFhLTg3MjYtY2ZiMmM0NmU0MmUw",
      "slug": "my-org"
    }
  }
}

Build Started

{
  "detail-type": "Build Started",
  "detail": {
    "version": 1,
    "build": {
      "uuid": "8fcaa7b9-e175-4110-9f48-f79949806a31",
      "graphql_id": "QnVpbGQtLS04ZmNhYTdiOS1lMTc1LTQxMTAtOWY0OC1mNzk5NDk4MDZhMzE=",
      "number": 123456,
      "commit": "5a741616cdf07dc87c5adafe784321eeeb639e33",
      "message": "Merge pull request #456 from my-org/chore/update-deps",
      "branch": "master",
      "state": "started",
      "started_at": "2019-08-11 06:01:16 UTC",
      "finished_at": null,
      "source": "webhook"
    },
    "pipeline": {
      "uuid": "88d73553-5533-4f56-9c16-fb38d7817d8f",
      "graphql_id": "UGlwZWxpbmUtLS04OGQ3MzU1My01NTMzLTRmNTYtOWMxNi1mYjM4ZDc4MTdkOGY=",
      "slug": "my-pipeline",
      "repo": "git@somewhere.com:project.git"
    },
    "organization": {
      "uuid": "a98961b7-adc1-41aa-8726-cfb2c46e42e0",
      "graphql_id": "T3JnYW5pemF0aW9uLS0tYTk4OTYxYjctYWRjMS00MWFhLTg3MjYtY2ZiMmM0NmU0MmUw",
      "slug": "my-org"
    }
  }
}

Build Finished

{
  "detail-type": "Build Finished",
  "detail": {
    "version": 1,
    "build": {
      "uuid": "8fcaa7b9-e175-4110-9f48-f79949806a31",
      "graphql_id": "QnVpbGQtLS04ZmNhYTdiOS1lMTc1LTQxMTAtOWY0OC1mNzk5NDk4MDZhMzE=",
      "number": 123456,
      "commit": "5a741616cdf07dc87c5adafe784321eeeb639e33",
      "message": "Merge pull request #456 from my-org/chore/update-deps",
      "branch": "master",
      "state": "passed",
      "started_at": "2019-08-11 06:01:16 UTC",
      "finished_at": "2019-08-11 06:01:35 UTC",
      "source": "webhook"
    },
    "pipeline": {
      "uuid": "88d73553-5533-4f56-9c16-fb38d7817d8f",
      "graphql_id": "UGlwZWxpbmUtLS04OGQ3MzU1My01NTMzLTRmNTYtOWMxNi1mYjM4ZDc4MTdkOGY=",
      "slug": "my-pipeline",
      "repo": "git@somewhere.com:project.git"
    },
    "organization": {
      "uuid": "a98961b7-adc1-41aa-8726-cfb2c46e42e0",
      "graphql_id": "T3JnYW5pemF0aW9uLS0tYTk4OTYxYjctYWRjMS00MWFhLTg3MjYtY2ZiMmM0NmU0MmUw",
      "slug": "my-org"
    }
  }
}

Job Scheduled

{
  "detail-type": "Job Scheduled",
  "detail": {
    "version": 1,
    "job": {
      "uuid": "9e6c3f19-4fdb-4e8e-b925-28cd7504e17f",
      "graphql_id": "Sm9iLS0tOWU2YzNmMTktNGZkYi00ZThlLWI5MjUtMjhjZDc1MDRlMTdm",
      "type": "script",
      "label": ":nodejs: Test",
      "step_key": "node_test",
      "command": "yarn test",
      "agent_query_rules": [
        "queue=default"
      ],
      "exit_status": null,
      "passed": false,
      "soft_failed": false,
      "state": "assigned",
      "runnable_at": "2019-08-11 06:01:14 UTC",
      "started_at": null,
      "finished_at": null,
      "unblocked_by": null
    },
    "build": {
      "uuid": "8fcaa7b9-e175-4110-9f48-f79949806a31",
      "graphql_id": "QnVpbGQtLS04ZmNhYTdiOS1lMTc1LTQxMTAtOWY0OC1mNzk5NDk4MDZhMzE=",
      "number": 123456,
      "commit": "5a741616cdf07dc87c5adafe784321eeeb639e33",
      "message": "Merge pull request #456 from my-org/chore/update-deps",
      "branch": "master",
      "state": "started",
      "source": "webhook"
    },
    "pipeline": {
      "uuid": "88d73553-5533-4f56-9c16-fb38d7817d8f",
      "graphql_id": "UGlwZWxpbmUtLS04OGQ3MzU1My01NTMzLTRmNTYtOWMxNi1mYjM4ZDc4MTdkOGY=",
      "slug": "my-pipeline",
      "repo": "git@somewhere.com:project.git"
    },
    "organization": {
      "uuid": "a98961b7-adc1-41aa-8726-cfb2c46e42e0",
      "graphql_id": "T3JnYW5pemF0aW9uLS0tYTk4OTYxYjctYWRjMS00MWFhLTg3MjYtY2ZiMmM0NmU0MmUw",
      "slug": "my-org"
    }
  }
}

Job Started

{
  "detail-type": "Job Started",
  "detail": {
    "version": 1,
    "job": {
      "uuid": "9e6c3f19-4fdb-4e8e-b925-28cd7504e17f",
      "graphql_id": "Sm9iLS0tOWU2YzNmMTktNGZkYi00ZThlLWI5MjUtMjhjZDc1MDRlMTdm",
      "type": "script",
      "label": ":nodejs: Test",
      "step_key": "node_test",
      "command": "yarn test",
      "agent_query_rules": [
        "queue=default"
      ],
      "exit_status": null,
      "passed": false,
      "soft_failed": false,
      "state": "started",
      "runnable_at": "2019-08-11 06:01:14 UTC",
      "started_at": "2019-08-11 06:01:16 UTC",
      "finished_at": null,
      "unblocked_by": null
    },
    "build": {
      "uuid": "8fcaa7b9-e175-4110-9f48-f79949806a31",
      "graphql_id": "QnVpbGQtLS04ZmNhYTdiOS1lMTc1LTQxMTAtOWY0OC1mNzk5NDk4MDZhMzE=",
      "number": 123456,
      "commit": "5a741616cdf07dc87c5adafe784321eeeb639e33",
      "message": "Merge pull request #456 from my-org/chore/update-deps",
      "branch": "master",
      "state": "started",
      "source": "webhook"
    },
    "pipeline": {
      "uuid": "88d73553-5533-4f56-9c16-fb38d7817d8f",
      "graphql_id": "UGlwZWxpbmUtLS04OGQ3MzU1My01NTMzLTRmNTYtOWMxNi1mYjM4ZDc4MTdkOGY=",
      "slug": "my-pipeline",
      "repo": "git@somewhere.com:project.git"
    },
    "organization": {
      "uuid": "a98961b7-adc1-41aa-8726-cfb2c46e42e0",
      "graphql_id": "T3JnYW5pemF0aW9uLS0tYTk4OTYxYjctYWRjMS00MWFhLTg3MjYtY2ZiMmM0NmU0MmUw",
      "slug": "my-org"
    }
  }
}

Job Finished

{
  "detail-type": "Job Finished",
  "detail": {
    "version": 1,
    "job": {
      "uuid": "9e6c3f19-4fdb-4e8e-b925-28cd7504e17f",
      "graphql_id": "Sm9iLS0tOWU2YzNmMTktNGZkYi00ZThlLWI5MjUtMjhjZDc1MDRlMTdm",
      "type": "script",
      "label": ":nodejs: Test",
      "step_key": "node_test",
      "command": "yarn test",
      "agent_query_rules": [
        "queue=default"
      ],
      "exit_status": 0,
      "passed": true,
      "soft_failed": false,
      "state": "finished",
      "runnable_at": "2019-08-11 06:01:14 UTC",
      "started_at": "2019-08-11 06:01:16 UTC",
      "finished_at": "2019-08-11 06:01:35 UTC",
      "unblocked_by": null
    },
    "build": {
      "uuid": "8fcaa7b9-e175-4110-9f48-f79949806a31",
      "graphql_id": "QnVpbGQtLS04ZmNhYTdiOS1lMTc1LTQxMTAtOWY0OC1mNzk5NDk4MDZhMzE=",
      "number": 123456,
      "commit": "5a741616cdf07dc87c5adafe784321eeeb639e33",
      "message": "Merge pull request #456 from my-org/chore/update-deps",
      "branch": "master",
      "state": "started",
      "source": "webhook"
    },
    "pipeline": {
      "uuid": "88d73553-5533-4f56-9c16-fb38d7817d8f",
      "graphql_id": "UGlwZWxpbmUtLS04OGQ3MzU1My01NTMzLTRmNTYtOWMxNi1mYjM4ZDc4MTdkOGY=",
      "slug": "my-pipeline",
      "repo": "git@somewhere.com:project.git"
    },
    "organization": {
      "uuid": "a98961b7-adc1-41aa-8726-cfb2c46e42e0",
      "graphql_id": "T3JnYW5pemF0aW9uLS0tYTk4OTYxYjctYWRjMS00MWFhLTg3MjYtY2ZiMmM0NmU0MmUw",
      "slug": "my-org"
    }
  }
}

Job Activated

{
  "detail-type": "Job Activated",
  "detail": {
    "version": 1,
    "job": {
      "uuid": "9e6c3f19-4fdb-4e8e-b925-28cd7504e17f",
      "graphql_id": "Sm9iLS0tOWU2YzNmMTktNGZkYi00ZThlLWI5MjUtMjhjZDc1MDRlMTdm",
      "type": "manual",
      "label": ":rocket: Deploy",
      "step_key": "manual_deploy",
      "command": null,
      "agent_query_rules": [],
      "exit_status": null,
      "passed": false,
      "soft_failed": false,
      "state": "finished",
      "runnable_at": null,
      "started_at": null,
      "finished_at": null,
      "unblocked_by": {
        "uuid": "c07c69c6-11d2-4375-9148-9d0338b0a836",
        "graphql_id": "VXNlci0tLWMwN2M2OWM2LTExZDItNDM3NS05MTQ4LTlkMDMzOGIwYTgzNg==",
        "name": "bell"
      }
    },
    "build": {
      "uuid": "8fcaa7b9-e175-4110-9f48-f79949806a31",
      "graphql_id": "QnVpbGQtLS04ZmNhYTdiOS1lMTc1LTQxMTAtOWY0OC1mNzk5NDk4MDZhMzE=",
      "number": 123456,
      "commit": "5a741616cdf07dc87c5adafe784321eeeb639e33",
      "message": "Merge pull request #456 from my-org/chore/update-deps",
      "branch": "master",
      "state": "started",
      "source": "webhook"
    },
    "pipeline": {
      "uuid": "88d73553-5533-4f56-9c16-fb38d7817d8f",
      "graphql_id": "UGlwZWxpbmUtLS04OGQ3MzU1My01NTMzLTRmNTYtOWMxNi1mYjM4ZDc4MTdkOGY=",
      "slug": "my-pipeline",
      "repo": "git@somewhere.com:project.git"
    },
    "organization": {
      "uuid": "a98961b7-adc1-41aa-8726-cfb2c46e42e0",
      "graphql_id": "T3JnYW5pemF0aW9uLS0tYTk4OTYxYjctYWRjMS00MWFhLTg3MjYtY2ZiMmM0NmU0MmUw",
      "slug": "my-org"
    }
  }
}

Agent Connected

{
  "detail-type": "Agent Connected",
  "detail": {
    "version": 1,
    "agent": {
      "uuid": "288139c5-728d-4c22-88e3-5a926b6c4a51",
      "graphql_id": "QWdlbnQtLS0yODgxMzljNS03MjhkLTRjMjItODhlMy01YTkyNmI2YzRhNTE=",
      "connection_state": "connected",
      "name": "my-agent-1",
      "version": "3.13.2",
      "ip_address": "3.80.193.183",
      "hostname": "ip-10-0-2-73.ec2.internal",
      "pid": "18534",
      "priority": 0,
      "meta_data": [
        "aws:instance-id=i-0ce2c738afbfc6c83"
      ],
      "connected_at": "2019-08-10 09:44:40 UTC",
      "disconnected_at": null,
      "lost_at": null
    },
    "organization": {
      "uuid": "a98961b7-adc1-41aa-8726-cfb2c46e42e0",
      "graphql_id": "T3JnYW5pemF0aW9uLS0tYTk4OTYxYjctYWRjMS00MWFhLTg3MjYtY2ZiMmM0NmU0MmUw",
      "slug": "my-org"
    },
    "token": {
      "uuid": "df75860c-94f9-4275-91cb-3986590f45b5",
      "created_at": "2019-08-10 07:44:40 UTC",
      "description": "Default agent token"
    }
  }
}

Agent Disconnected

{
  "detail-type": "Agent Disconnected",
  "detail": {
    "version": 1,
    "agent": {
      "uuid": "288139c5-728d-4c22-88e3-5a926b6c4a51",
      "graphql_id": "QWdlbnQtLS0yODgxMzljNS03MjhkLTRjMjItODhlMy01YTkyNmI2YzRhNTE=",
      "connection_state": "disconnected",
      "name": "my-agent-1",
      "version": "3.13.2",
      "ip_address": "3.80.193.183",
      "hostname": "ip-10-0-2-73.ec2.internal",
      "pid": "18534",
      "priority": 0,
      "meta_data": [
        "aws:instance-id=i-0ce2c738afbfc6c83"
      ],
      "connected_at": "2019-08-10 09:44:40 UTC",
      "disconnected_at": "2019-08-10 09:54:40 UTC",
      "lost_at": null
    },
    "organization": {
      "uuid": "a98961b7-adc1-41aa-8726-cfb2c46e42e0",
      "graphql_id": "T3JnYW5pemF0aW9uLS0tYTk4OTYxYjctYWRjMS00MWFhLTg3MjYtY2ZiMmM0NmU0MmUw",
      "slug": "my-org"
    },
    "token": {
      "uuid": "df75860c-94f9-4275-91cb-3986590f45b5",
      "created_at": "2019-08-10 07:44:40 UTC",
      "description": "Default agent token"
    }
  }
}

Agent Lost

{
  "detail-type": "Agent Lost",
  "detail": {
    "version": 1,
    "agent": {
      "uuid": "288139c5-728d-4c22-88e3-5a926b6c4a51",
      "graphql_id": "QWdlbnQtLS0yODgxMzljNS03MjhkLTRjMjItODhlMy01YTkyNmI2YzRhNTE=",
      "connection_state": "lost",
      "name": "my-agent-1",
      "version": "3.13.2",
      "ip_address": "3.80.193.183",
      "hostname": "ip-10-0-2-73.ec2.internal",
      "pid": "18534",
      "priority": 0,
      "meta_data": [
        "aws:instance-id=i-0ce2c738afbfc6c83"
      ],
      "connected_at": "2019-08-10 09:44:40 UTC",
      "disconnected_at": "2019-08-10 09:54:40 UTC",
      "lost_at": "2019-08-10 09:54:40 UTC"
    },
    "organization": {
      "uuid": "a98961b7-adc1-41aa-8726-cfb2c46e42e0",
      "graphql_id": "T3JnYW5pemF0aW9uLS0tYTk4OTYxYjctYWRjMS00MWFhLTg3MjYtY2ZiMmM0NmU0MmUw",
      "slug": "my-org"
    },
    "token": {
      "uuid": "df75860c-94f9-4275-91cb-3986590f45b5",
      "created_at": "2019-08-10 07:44:40 UTC",
      "description": "Default agent token"
    }
  }
}

Agent Stopping

{
  "detail-type": "Agent Stopping",
  "detail": {
    "version": 1,
    "agent": {
      "uuid": "288139c5-728d-4c22-88e3-5a926b6c4a51",
      "graphql_id": "QWdlbnQtLS0yODgxMzljNS03MjhkLTRjMjItODhlMy01YTkyNmI2YzRhNTE=",
      "connection_state": "stopping",
      "name": "my-agent-1",
      "version": "3.13.2",
      "ip_address": "3.80.193.183",
      "hostname": "ip-10-0-2-73.ec2.internal",
      "pid": "18534",
      "priority": 0,
      "meta_data": [
        "aws:instance-id=i-0ce2c738afbfc6c83"
      ],
      "connected_at": "2019-08-10 09:44:40 UTC",
      "disconnected_at": null,
      "lost_at": null
    },
    "organization": {
      "uuid": "a98961b7-adc1-41aa-8726-cfb2c46e42e0",
      "graphql_id": "T3JnYW5pemF0aW9uLS0tYTk4OTYxYjctYWRjMS00MWFhLTg3MjYtY2ZiMmM0NmU0MmUw",
      "slug": "my-org"
    },
    "token": {
      "uuid": "df75860c-94f9-4275-91cb-3986590f45b5",
      "created_at": "2019-08-10 07:44:40 UTC",
      "description": "Default agent token"
    }
  }
}

Agent Stopped

{
  "detail-type": "Agent Stopped",
  "detail": {
    "version": 1,
    "agent": {
      "uuid": "288139c5-728d-4c22-88e3-5a926b6c4a51",
      "graphql_id": "QWdlbnQtLS0yODgxMzljNS03MjhkLTRjMjItODhlMy01YTkyNmI2YzRhNTE=",
      "connection_state": "stopped",
      "name": "my-agent-1",
      "version": "3.13.2",
      "ip_address": "3.80.193.183",
      "hostname": "ip-10-0-2-73.ec2.internal",
      "pid": "18534",
      "priority": 0,
      "meta_data": [
        "aws:instance-id=i-0ce2c738afbfc6c83"
      ],
      "connected_at": "2019-08-10 09:44:40 UTC",
      "disconnected_at": "2019-08-10 09:54:40 UTC",
      "lost_at": null
    },
    "organization": {
      "uuid": "a98961b7-adc1-41aa-8726-cfb2c46e42e0",
      "graphql_id": "T3JnYW5pemF0aW9uLS0tYTk4OTYxYjctYWRjMS00MWFhLTg3MjYtY2ZiMmM0NmU0MmUw",
      "slug": "my-org"
    },
    "token": {
      "uuid": "df75860c-94f9-4275-91cb-3986590f45b5",
      "created_at": "2019-08-10 07:44:40 UTC",
      "description": "Default agent token"
    }
  }
}