Online CI providers are the new black

Hosted continuous integration services, what do they do and why do you need one?

Imagine the possibility of having your tests running each time you push changes to your git(hub) repository without having to:

• set up a test environment;
• mess up your configuration with multiple setups;
• consume local CPU & I/O;
• hesitate (is my change worth running the tests?).

We already have continuous integration servers for years, but they are a pain in the ass. We don’t want to have to set them up, we don’t want to update them and we certainly don’t want to get a call each time the server is slow or crashed. One year ago, their was no solution beside having an active and pro-efficient IT guy :-).

Today, a large panel of choices has become available.

Section intitulée travisci-the-community-leaderTravisCI – the community leader

February 2011 – https://travis-ci.org/

You must already have seen them a lot, especially if you are a Symfony contributor. Travis is everywhere in the Open-Source world. The code source is on Github, and you should be able to run the whole service on your servers (oh, but don’t).

It works with a simple .travis.yml file at the root of your project, telling Travis how to setup and run your tests. This file can be validated via a command-line tool. 

Travis supports Ruby, JavaScript (Node.js), Java, Scala, PHP, Haskell and Erlang, and they have a public API if you want to build your own tools on top of it.

Travis was originally only used for Open-Source projects as they were no private builds capabilities. But since a few months and their community fund-raising, some people (including me) can beta-test the private version of Travis, called TravisPro, which allows companies to run automated tests on private projects.

They have the best community around. You can go to IRC, ask questions, find solutions and see them implemented right away by some awesome dude. It is driven by a small German company and runs on top of Rails 3, Ember.js, AMQP and Chef.

You can do a lot of customization via the .travis.yml file, and some people use it to compile and run tests on Android applications too.

While the situation about private / paying plans is not clear by now (how much and for what), it’s the greatest service for Open-Source.

Section intitulée circleci-the-american-start-upCircleCI – the American start-up

September 2011 – https://circleci.com/

CircleCI is the main competitor of TravisPro, as they provide Continuous Integration on the cloud for companies starting at 19$/month for a single repository. The build interface of CircleCI is really clear and reading through console logs is a bliss:

CircleCI automatically detects many settings for Ruby, Python, Node.js, Java and Clojure. As for PHP, you’ll have to build your own configuration file (circle.yml) but in the end everythings works as expected.

What is really impressive with this service is the speed of a build. For a Symfony2 project, with a composer install call with 20+ dependencies, Doctrine2 setup and fixtures loading, the PHPUnit test suite runs in about 1m30 only. Travis spends 3m50 for the same work.

CircleCI exposes an API, but at the moment you can’t select PHP version for your tests, neither add a non-github repository.

Support is great and it’s really easy to talk directly with the CEO, they are a young but mature start-up and I advise you to test them up!

Section intitulée tddium-to-watch-closelyTddium – to watch closely

November 2011 – https://www.tddium.com/

It’s currently a Ruby and JRuby only service, but they provide really great tools and plan to support Python, PHP and Node.js soon. There is already a private beta for Python and PHP is coming next.  With their command line tool, you can add any Git repository to the build system, self-hosted or Github like. They automatically detect a lot of settings for running your test suite but of course everything can be configured via a tddium.yml file.

What’s different about them is you pay for what you use: CPU time. The more you need, the more expensive it get. So when you have a 10 worker hours / month plan (15$) and you exceed the quota, there are extra charges (0.75$/hour, which is cheap). Be sure to have flexible billing on this, you do not want your tests shutted down for a money issue… Hopefully, I have heard from them that they are going to introduce a flat-rate high performance plan, that will eliminate the need for overages. 

Section intitulée coconut-ci-the-french-start-upCoconut CI – the French start-up

July 2012 – https://coconut-ci.com/

Built on top of Symfony 1.4 and Jenkins-php-api, they are not thigh to Github like the others ones. They use Mozilla Persona for the registration / login process, and has you are not giving away a GitHub permission, you will be forced to add manually an SSH key if you want to use Github. The counterpart is that you are free to use any git server you want.

They support atoum, PHPUnit and plan to add Behat and Selenium as soon as possible. 

When creating a new job, you can select some presets for atoum, Composer and PHPUnit (more soon), and you can choose a PHP version between 5.3 and 5.4 too. Unfortunately, this service is very young and it does not support any RDBM currently. 

Good points are CoconutCI already provides a REST Api and addresses the main pitfalls of TravisCI: private builds and not github-only projects. 

Section intitulée buildhive-a-jenkins-in-the-cloudBuildhive – a Jenkins in the cloud

May 2012 – https://buildhive.cloudbees.com/

Based heavily on Jenkins (even for the ugly design), this service is linked to your Github account – but it only builds public repositories. Cloudbees is a Java service but Buildhive (which is one of their product) is also running Rails, PHP, Python, Ant and Maven.  I got some Tomcat 500 errors when configuring my project, I advise you to avoid this service for now… But as it’s free, give it a try!

Section intitulée continuous-run-your-tests-on-your-awsContinuous – run your tests on your AWS

May 2011 – https://continuous.io/

Continuous is a small, self-funded and young start-up, and unlike every other services, they run builds on your Amazon Web Services (AWS) account. Each build startup a new micro AWS instance, which is shutted down right after the tests and cost something like 2¢ per build.

This allows a maximal flexibility on the test instance: you are root and can install anything you want or need.

Like other services, you setup your test suite in a file called .continuousrc (or in the Project admin page). They are built to run Python and Django, with code coverage and JSLint too. There is no PHP support, you will have to install it yourself, and will therefore cause a higher building time.

Section intitulée there-is-more-of-themThere is more of them

I will not write about Semaphoreapp for Ruby, CISimple for mobile application, Shiningpanda for Python and Java, Hosted-Ci for iOs and Mac, Bamboo for Jira users…

Listing and comparing all the available services would take ages. There is no perfect solution for everybody, and as usual with technology, you will have to test them yourself and make a choice based on your needs.

The criterias not to be forgotten are:

• Automatic deployment: you may want to deploy each successful build to a QA server
• CI tools: is Jenkins a requirement? Will you be able to add plugins you need¹? What about plugging your own tools to the build system?
• Supported technologies: maybe you are running Python today, but what if you switch to Ruby tomorrow?
• Cost: you may want to pay only for what you use at a start, and pay a fixed monthly fee later.

What’s really missing on all those services is quality tools. When setting up a Jenkins internally, you can add CodeSniffer, PHPMD, JSLint… There are a lot of tools we want to plug into a continuous integration system, to extract quality metrics like cyclomatic complexity, number of errors in a commit (and who to blame for it)… At the moment, all the services I saw are just about running tests.

Beside this default, there are two major factors that block the adoption of these services:

• companies may not want or can have their products source-code running outside their own servers or countries. Github addressed this issue with Github Entreprise.
• there is still a lot of evangelization work to do about continuous integration and test, a lot of companies still consomme a huge amount of time with QC team on thing that should be automatised.

As of today you have no excuse to not have a CI server running your tests, can it be a self-hosted Jenkins or a Circle account.

Section intitulée update-february-2012Update – February 2012

Coconut CI disappear, and a new player pop-out in October 2012: https://drone.io/. This is already a great service, check them out!