Rewriting the Buildbox Agent

TLDR: There is a new version of the agent written in Golang. Learn how to upgrade from the Ruby agent and install it from scratch at https://github.com/buildbox/agent

When I was building Buildbox, one of the biggest problems I faced was "How do I run builds on CI servers that I don't control?".

My first idea was to use SSH, however, that would have required the customer giving Buildbox SSH access to their servers - which sounded like a bad idea. Also, I wasn't sure of the security and scalability implications of Buildbox having potentially hundreds of SSH connections open to servers all over the world. I scratched that idea pretty quick.

The next idea I had was to write a small program that the customers would run on their CI server, which would co-ordinate the builds for them. This was the approach I ended up going with. I wrote it in Ruby, because that's what I know best and what I'm most comfortable working with.

I got a prototype out pretty quickly using childprocess and faraday, and eventually shipped it as a Ruby gem.

The first few months of Buildbox went by, and this setup worked pretty well. But there were a few problems...

Installation

Installing a Ruby gem is easy, if you already have Ruby installed. If you didn't have Ruby installed, and you hadn't installed it before, then you're in for a world of confusion. Sure, you can just apt-get install ruby, but what version of Ruby will it be? Will it be compatible with the childprocess gem I was using? Is it going to leak memory? What about JRuby? Will that work? Not knowing what version of Ruby it will run under, made things tricky. And I didn't want to have to require a particular version.

Memory

Ruby seems to enjoy eating RAM. The Buildbox agent is a long running proccess that you shouldn't have to restart. When booting up the agent for the first time, without it having run any builds, would hover around 70MB of RAM, slowly getting bigger and bigger. I had reports of some agents using up to 3.5GB of RAM. Short of asking the customer to use a version of Ruby that had better GC management, this was a problem that was pretty much unsolveable.

Security

When setting up a project on Buildbox, the customer has to upload their entire build script to the app. The agent would then download that script, run it on their CI server, and report the results back. This approach was simple, easy to build, and easy to document. But there are security implications with this approach that I wanted to fix.

The Solution

I decided to rewrite the agent. Take the lessons learned from the Ruby agent, and write something that would address the above problems.

After hearing good things about Golang, I decided to give a try. After a few frustrating late nights...

It's 4:00 AM, and this is me with Golang now: http://t.co/JupoPsOSYf

— Keith Pitt (@keithpitt) March 14, 2014

Golang has beat me tonight.

— Keith Pitt (@keithpitt) January 29, 2014

I was able to get something working. Over the past few weeks, I've been tweaking and getting it ready, and it's now ready to release to the world. I was able to also address all the above problems, and I'm really happy with the results. You can get the source code to the new agent on GitHub: https://github.com/buildbox/agent

Installation

Golang lets me ship self contained binaries that have no dependencies. This is great, and means I can bundle up all the installation steps in a script that you can run on your CI server. No more Ruby required!

1
bash -c "`curl -sL https://raw.githubusercontent.com/buildbox/agent/master/install.sh`"

I've also written a Manual Installation Guide if you want to install it yourself.

Memory

The new agent uses around 1.5MB to 3.3MB of RAM. This is amazing. I want to leave as much RAM on the server as possible for your builds to use. It's also super quick.

Security

Instead of uploading the build script to Buildbox, you instead have the script stored in your project repository and tell Buildbox the path to script. When the agent needs to run a build, it will run a bootstrap.sh file (example here) that will clone your repo, checkout the right commit, and run your build script.

And that's it! A big shout out to Mark Wolfe for helping out with the agent <3 <3

If you have any issues, ping me on Twitter @keithpitt, submit a GitHub issue, or send me an email at: keith@buildkite.com