Launching and Running Elastic CI Stack for AWS

The Elastic CI Stack for AWS gives you a private, autoscaling Buildkite Agent cluster in your own AWS account. This guide leads you through getting started with the stack :cloudformation:🎉

Before you start

Most Elastic CI Stack features are supported on both Linux and Windows. The following AMIs are available in all the supported regions:

  • Amazon Linux 2 (64-bit x86)
  • Amazon Linux 2 (64-bit ARM, Graviton)
  • Windows Server 2019 (64-bit x86)

See this overview page for extensive information on compatibility, configuration, security, and other topics that will allow you to effectively use Elastic CI Stack for AWS.

Required and recommended skills

The Elastic CI Stack for AWS does not require familiarity with the underlying AWS services to deploy it. However, to run builds, some familiarity with the following AWS services is required:

  • AWS CloudFormation
  • Amazon EC2 (to select an EC2 InstanceType stack parameter appropriate for your workload)
  • Amazon S3 (to copy your git clone secret for cloning and building private repositories)

Elastic CI Stack provides defaults and pre-configurations suited for most use cases without the need for additional customization. Still, you'll benefit from familiarity with VPCs, availability zones, subnets, and security groups for custom instance networking.

For post-deployment diagnostic purposes, deeper familiarity with EC2 is recommended to be able to access the instances launched to execute Buildkite jobs over SSH or AWS Systems Manager Sessions.

Billable services

The Elastic CI Stack template deploys several billable Amazon services that do not require upfront payment and operate on a pay-as-you-go principle, with the bill proportional to usage.

Service name Purpose Required
EC2 Deployment of instances ☑️
EBS Root disk storage of EC2 instances ☑️
Lambda Scaling of Auto Scaling group and modifying Auto Scaling group's properties ☑️
Systems Manager Parameter Store Storing the Buildkite agent token ☑️
CloudWatch Logs Logs for instances and Lambda scaler ☑️
CloudWatch Metrics Metrics recorded by Lambda scaler ☑️
S3 Charging based on storage and transfers in/and out of the secrets bucket (on by default) ❌

Buildkite services are billed according to your plan.

Launching the stack

Go to the Agents page on Buildkite and select the AWS tab:

Buildkite AWS Agents

Click Launch Stack :red_button:

Launch Buildkite Elastic CI Stack
AWS Select Template Screen

After clicking Next, configure the stack using your Buildkite agent token:

AWS Parameters

If you don't know your agent token, there is a Reveal Agent Token button available on the right-hand side of the Agents page:

Reveal Agent Token

By default the stack uses a job queue of default, but you can specify any other queue name you like. See the Buildkite Agent Job Queue docs for more info.

Review the parameters, see Elastic CI Stack for AWS Parameters for more details.

Once you're ready, check these three checkboxes:

  • I acknowledge that AWS CloudFormation might create IAM resources.
  • I acknowledge that AWS CloudFormation might create IAM resources with custom names.
  • I acknowledge that AWS CloudFormation might require the following capability: CAPABILITY_AUTO_EXPAND

Then click Create stack:

AWS Create Stack Button

After creating the stack, Buildkite takes you to the CloudFormation console. Click the Refresh icon in the top right hand corner of the screen until you see the stack status is CREATE_COMPLETE.

AWS Elastic CI Stack Create Complete

You now have a working Elastic CI Stack ready to run builds! 🎉

Running your first build

We've created a sample bash-parallel-example sample pipeline for you to test with your new autoscaling stack. Click the Add to Buildkite button below (or on the GitHub Readme):

Add Bash Example to Buildkite

Click Create Pipeline. Depending on your organization's settings, the next step will vary slightly:

  • If your organization uses the web-based steps editor (default), your pipeline is now ready for its first build. You can skip to the next step.
  • If your organization has been upgraded to the YAML steps editor, you should see a Choose a Starting Point wizard. Select Pipeline Upload from the list:
    Upload Pipeline from Version Control

Click New Build in the top right and choose a build message (perhaps a little party :partyparrot:?):

Triggering Buildkite Build

Once your build is created, head back to AWS EC2 Auto Scaling Groups to watch the Elastic CI Stack creating new EC2 instances:

AWS EC2 Auto Scaling Group Menu

Select the buildkite-AgentAutoScaleGroup-xxxxxxxxxxxx group and then the Instances tab. You'll see instances starting up to run your new build and after a few minutes they'll transition from Pending to InService:

AWS Auto Scaling Group Launching

Once the instances are ready they will appear on your Buildkite Agents page:

Buildkite Connected Agents

And then your build will start running on your new agents:

Your First Build

Congratulations on running your first Elastic CI Stack build on Buildkite! 👏🎉

Further documentation

To gain a better understanding of how Elastic CI Stack works and how to use it most effectively and securely, check out the following resources: