Elastic CI Stack for AWS

The Elastic CI Stack for AWS gives you a private, autoscaling Buildkite Agent cluster in your own AWS account. This guide will lead you through getting started with the stack :cloudformation:🎉

Creating an AWS EC2 Key Pair

The first thing you’ll need is an EC2 Key Pair to allow you to SSH into EC2 instances created by the Elastic CI stack. If you already have a Key Pair that you'd like to use, you can head straight to Launching the Stack.

To create an EC2 Key Pair, sign into the AWS Console, and go to the EC2 service:

AWS EC2

Then go to “Key Pairs” under “Network & Security”:

AWS EC2 Key Pair

Click the “Create Key Pair” button in the top left hand corner and give it a name.

AWS EC2 Create Key Pair

Launching the Stack

To launch the stack, you’ll first need to create a Buildkite API Access Token for the stack to use. The stack includes a scheduled Lamda function which connects to the Buildkite API to autoscale EC2 instances.

Go to the “Agents” page on Buildkite and select the “AWS” tab:

Buildkite AWS Agents

Click the “Create an API Access Token” button:

Buildkite AWS Agents Setup Tokens

The form is pre-filled with everything you’ll need:

Buildkite API Access Token form

Click “Create New API Access Token” and you'll be taken back to the AWS tab on the agents page with the token ready to use.

Now you’re ready to launch the stack! Click “Launch Stack” :red_button:

Launch Buildkite Elastic CI Stack
AWS Select Template Screen

After clicking “Next” you’ll configure the stack using the details from Buildkite:

AWS Parameters

By default the stack uses a job queue of default but you can specify any other queue name you’d like. See the Buildkite Agent Job Queue docs for more info.

Once you're ready, click the “I acknowledge that this template might cause AWS CloudFormation to create IAM resources.” checkbox then click “Create”:

AWS Create Stack Button

After creating the stack, you'll be taken to the CloudFormation console. Click the “Refresh” button in the top right hand corner of the screen until you see the stack being created.

Waiting for the AWS Stack to be created

You've now got a working Elastic CI Stack ready to run builds! 🎉

Running your first build

We’ve created a sample bash-parallel-example sample pipeline for you to test with your new autoscaling stack. Click the “Add to Buildkite” button below (or on the GitHub Readme):

Add Bash Example to Buildkite

Your pipeline is now ready for it’s first build. Click "New Build" in the top right and choose a build message (perhaps a little party :parrot:?):

Triggering Buildkite Build

Once your build is created head back to AWS EC2 Autoscaling console to monitor the Elastic CI stack creating new EC2 instances:

AWS EC2
AWS EC2 Auto Scaling Group Menu

Select the “buildkite-AgentAutoScaleGroup-xxx” group and then the the “Instances” tab. You'll see instances starting up to run your new build and after a few minutes they'll transition from Pending to InService:

AWS Auto Scaling Group Launching

Once the instances are ready you'll see them appear on your Buildkite Agents page:

Buildkite Connected Agents

And then your build will start running on your new agents:

Running your first build

Congratulations on running your first Elastic CI Stack build on Buildkite! 👏🎉

Further documentation

You can read more about the Elastic CI Stack over on the GitHub repository:

https://github.com/buildkite/elastic-ci-stack-for-aws