When I was building Buildbox, one of the biggest problems I faced was "How do I run builds on CI servers that I don't control?".
My first idea was to use SSH, however, that would have required the customer giving Buildbox SSH access to their servers - which sounded like a bad idea. Also, I wasn't sure of the security and scalability implications of Buildbox having potentially hundreds of SSH connections open to servers all over the world. I scratched that idea pretty quick.
The next idea I had was to write a small program that the customers would run on their CI server, which would co-ordinate the builds for them. This was the approach I ended up going with. I wrote it in Ruby, because that's what I know best and what I'm most comfortable working with.
The first few months of Buildbox went by, and this setup worked pretty well. But there were a few problems...
Installing a Ruby gem is easy, if you already have Ruby installed. If you
didn't have Ruby installed, and you hadn't installed it before, then you're in
for a world of confusion. Sure, you can just
apt-get install ruby, but what
version of Ruby will it be? Will it be compatible with the childprocess gem I
was using? Is it going to leak memory? What about JRuby? Will that work? Not
knowing what version of Ruby it will run under, made things tricky. And I
didn't want to have to require a particular version.
Ruby seems to enjoy eating RAM. The Buildbox agent is a long running proccess
that you shouldn't have to restart. When booting up the agent for the first
time, without it having run any builds, would hover around
70MB of RAM,
slowly getting bigger and bigger. I had reports of some agents using up to
3.5GB of RAM. Short of asking the customer to use a version of Ruby that had
better GC management, this was a problem that was pretty much unsolveable.
When setting up a project on Buildbox, the customer has to upload their entire build script to the app. The agent would then download that script, run it on their CI server, and report the results back. This approach was simple, easy to build, and easy to document. But there are security implications with this approach that I wanted to fix.
I decided to rewrite the agent. Take the lessons learned from the Ruby agent, and write something that would address the above problems.
After hearing good things about Golang, I decided to give a try. After a few frustrating late nights...
It's 4:00 AM, and this is me with Golang now: http://t.co/JupoPsOSYf— Keith Pitt (@keithpitt) March 14, 2014
Golang has beat me tonight.— Keith Pitt (@keithpitt) January 29, 2014
I was able to get something working. Over the past few weeks, I've been tweaking and getting it ready, and it's now ready to release to the world. I was able to also address all the above problems, and I'm really happy with the results. You can get the source code to the new agent on GitHub: https://github.com/buildbox/agent
Golang lets me ship self contained binaries that have no dependencies. This is great, and means I can bundle up all the installation steps in a script that you can run on your CI server. No more Ruby required!
bash -c "`curl -sL https://raw.githubusercontent.com/buildbox/agent/master/install.sh`"
I've also written a Manual Installation Guide if you want to install it yourself.
The new agent uses around
3.3MB of RAM. This is amazing. I want to
leave as much RAM on the server as possible for your builds to use. It's also
Instead of uploading the build script to Buildbox, you instead have the script stored
in your project repository and tell Buildbox the path to script. When the agent
needs to run a build, it will run a
bootstrap.sh file (example here)
that will clone your repo, checkout the right commit, and run your build script.
And that's it! A big shout out to Mark Wolfe for helping out with the agent <3 <3