Those responsible for managing infrastructure and associated user accounts and permissions across a large team understand the importance of having a central place to manage these things. They’ll also understand that the ease with which they’re able to manage these things has a direct correlation with potential security risks and scalability. Eliminating manual tasks is a core principle of SRE, especially when it reduces cost, and automation improves how quickly we can adapt to change; in technologies, environments, and integrations with third-party tools, and reduces the chance of human error.
Infrastructure as Code exists as a way to improve efficiency, remain nimble and adaptable to change. It can also be the key to fostering DevOps practices in engineering teams – allowing developers to better understand and be involved in defining configuration as part of their software development processes.
Defining configuration as code:
- Provides consistency and clearly documents conventions
- Reduces the chance of introducing errors
- Allows infrastructure changes to be tested just as application code
- Offers all the benefits that using a source control manager provides:
- Centralised read/write and team permissions
- Rollbacks and versioning
- A robust commit and review process on PRs
- Surfaces configuration details for others to learn from and contribute to.
It’s no surprise that our users also want to manage their CI/CD resources with Infrastructure as Code.
Why Terraform?
HashiCorp Terraform is one of the most popular and extensible open-source tools for defining and provisioning repeatable cloud infrastructure through code, and has an active and vibrant community of users. Terraform allows users to configure infrastructure and services using HashiCorp Configuration Language (HCL), a declarative language that is user-friendly, and super powerful. It offers the ability to design modular abstraction layers, to reference output variables from one module for use in different modules. And it gives teams the ability to create focused, reusable Terraform configurations grouped by function or service as opposed to one large file with potentially hundreds or thousands of lines of code.
We wanted tools that worked the way we worked: using a code repository, collaborating on changes with pull requests, testing our changes, and run in our terminal. Terraform fits this bill very nicely. And what we can run in our terminal we can also run using Buildkite, with bonus points: the logs can be recorded and shared, we can see who is making changes and when, and we can tightly control concurrency, all with security in mind.
Samuel CochranProduct Manager, Pipelines, Buildkite
Introducing the Buildkite Terraform provider
“The less you have to think about as a software engineer the better. We try to make magic like that happen everywhere as much as possible.” And that’s why we use the Terraform provider…It helps us reinforce best practices and standards. And by defining pipelines via Terraform, it helps the CI team make sure that everybody's using pipelines defined in the repository, and away from the Buildkite UI in terms of making steps and pipelines – which we think is the best pattern because it's next to their code and keeps everything consistent between teams. We tell them: Don't think about that; just use this.”
Chloe Hutchison, CI Team Lead & Principal SRE, Elastic
Many of our users rely on Terraform to manage their infrastructure and over the years have built and used their own custom Buildkite providers. We now have an official Buildkite provider in the Terraform Registry that is built and fully supported by us. Our users are now better able to manage their Buildkite CI/CD resources as infrastructure as code.
The official Buildkite provider allows users to configure and manage:
- Agent tokens
- Pipelines
- Pipeline schedules
- Teams
Chloe says the CI team at Elastic:
“doesn't like to use 'click-ops' at all if we can avoid it…we had things that were provisioned by 'click-ops' before there were good Terraform providers available and it's a nightmare to manage; changes get lost, people get confused, and nothing is consistent.”
Elastic’s CI team uses the Buildkite Terraform provider via the Cloud Development Kit for Terraform (CDKTF). CDKTF allows users to use familiar programming languages like TypeScript, Go, .NET, Python, and Java to define and provision infrastructure, giving users access to the Terraform ecosystem while also leveraging their existing toolchains.
Chloe describes the benefits of CDKTF as being, “a programming layer on top of Terraform which allows us to use multiple providers, and do a bit of fancy code around it…I like doing it that way because it means that if there are any limitations with one of the providers that we're using, we can just code around them.”
“Our teams are using it to set up repos, CI pipelines, team memberships, and to specify Access Control Lists (ACLs). We have a data structure that teams can use to say “Give me a whole new project”. It goes and creates the GitHub repository, the Vault secrets (to be used in CI), Buildkite pipelines, and does a bunch of stuff in the background so that teams get a secure environment for each Buildkite agent they run. It can even set up scheduled builds”, says Chloe.
Check out Jarryd Tillbrook's demo to learn how to use Terraform to configure remote state, include official providers, create a Buildkite pipeline, trigger builds automatically once a commit is pushed to GitHub, and configure an Elastic Stack of autoscaling Buildkite agents in AWS.
Additional resources
For more information on using Terraform with Buildkite, check out these resources:
- Terraform Provider Docs
- Managing CI/CD Resources with Terraform, Jarryd Tillbrook at HashiTalks AU/NZ, 2022
- Managing CI/CD Resources with Terraform talk demo repository
- Official Buildkite Provider repository
- Terraform Techniques with Buildkite, Samuel Cochran, 2016
