At my new place of employment (okay, so it’s been a few months already — time flies!) we recently decided to put a continuous integration server in place. We had started with cruise control because a company we partnered with already had it setup and in place. Cruise control has never been the most user-friendly and easy to use server and so we set out to find another one.
Before you think this is a review of a bunch of integration servers, I’ll just say it’s not and get that out of the way. We didn’t really have the time do a thorough review of every server out there. We just needed an integration server that was easy to setup and run. Luckily we found it when we tried out Bamboo from the folks over at Atlassian. We are already using Jira and Confluence in house so it also made sense to just get most of our communication tools from one vendor. It was extremely easy to get going and to integrate with our Subversion repository. Its unit test run integration seems to work extremely well, also. We know we aren’t using all of its functionality but as we have time and new requirements pop-up I’m sure it will get more extensive use (abuse) in our shop.
The thing that struck me, however, was just how many vendors there are that have continuous integration server products. The ones that we found were Parabuild, Pulse, Bamboo, Anthil, Cruise Control, Team City, and Gauntlet (from Borland). I know there are some others such as Continuum and Luntbuild but I didn’t really even have them on my list.
I guess I look at this and see two things. One, the market is extremely saturated. Some of these products are probably not going to see a lot of adoption. These generally aren’t expensive products and so you have to sell a lot of them to justify even having a few full time developers and QA engineers on the product. The second is that this shows just how much adoption there is for continuous integration and testing in Java development shops nowadays. I think this is a great thing because it means more and more developers will get exposure to the concept of unit testing and continuous integration. I still see too few developers that have exposure and experience with this concept.
I think next I’ll come back to more discussions about hiring woes and some recommendations for developers that are out looking for a new job.
Predictable Chaos 
Dude, you have no idea what you are talking about. The market isn’t saturated. It even hasn’t arrived yet.
Regards,
Slava Imeshev
Why do you think it’s not saturated? I just don’t see the market supporting as many integration server products that I see out there. The developer tools market has an amazing ability to consolidate down to a couple of products in the long run. And the long run usually doesn’t even take that long. Maybe because I’m not in your industry I don’t know what I’m talking about but when I see that many products that do similar things it makes me thing that some are just not going to survive.
The current market is saturated. But Slava is right in one aspect, there is a new market coming. Things are changing very fast in the CI server market. I recently wrote an article about where it is heading and what we at JetBrains are doing about it. See here.
That’s a really interesting article Rob. My friend Phil was telling me after he read my post yesterday that they were using TeamCity and like it. I think the delayed commit feature was one he really liked.
We are pretty much doing what you mention in your post. Developers here are supposed to synch to subversion, build and test locally, commit and then keep going. They get notified if their commit breaks the build or not. For us, a failed unit test is a broken build. All unit tests must pass to be considered a successful build. We are only a team of 4 or so people right now, though, so realistically we don’t have some of the issues that larger teams will run into. Our code base is only about 6 months old so builds still only take a few minutes even with a pretty comprehensive test suite.
I think one area that a CI server could go into is improving the code coverage analysis tools. It would be great if they kept historical data and could show trending over time (i.e. the overall percentage of the code base that’s covered) and be able to show that trending within a package hierarchy such as functional area. Code coverage is good but long term trending to see if you’re falling behind or getting better is just one more metric that can be useful to have.
I think this almost necessitates another post. 🙂
Mike – good post, glad you’re enjoying Bamboo!
Are you using Bamboo’s Clover plugin to get trending data on your code coverage?
Also, if you’re not already I’d recommend using IM as the fastest mechanism for build breakage feedback. Bamboo works with any Jabber server, including Wildfire and GTalk so there’s plenty of quick ways to get notified. It’s largely what we use internally for quick developer feedback.
m
Hey Mike, good name, btw. Right now we’re just using Emma in our build scripts to get coverage data but I have been looking at getting a license for Clover. I just haven’t had enough time to get to it yet.
I will definitely check out the IM notification mechanism, though. That would be a great way to get quick feedback besides e-mail which can sometimes go unnoticed right away.
Mike,
You might want to check this link for the defintion of the market saturation: http://en.wikipedia.org/wiki/Market_saturation
In any case, competion is good for you, the consumer.
By the way, have you had a chance to look at our Parabuild? This is a mature, stable, feature-reach Continous Integration and build management sever. Parabuild was the first to offer full-blown remote builds.
Hope this helps,
Regards,
Slava Imeshev
Hi Slava,
Well, based on that particular definition the market isn’t saturated but I still think there are too many products to survive/thrive in the marketplace. It doesn’t really matter how big the market is…the market tends to pick a few favorites and go with those. A couple of those that I listed will thrive while the others will either stagnate or be discontinued. It’s impossible to say which ones they will be because it’s not always the best product that actually wins. I mean, looking at it from our situation, how can we justify the time to look at 7 products when we’re a small team? Even medium sized development groups will most likely look at just a couple of those and make a decision.
I completely agree that competition is good for the consumer. You get more ideas out of more products and then the other products will adopt the ideas that work best.
And unfortunately I didn’t really get a chance to look at Parabuild. We knew Cruise Control wasn’t enough and we identified Bamboo and Team City initially since we already use Jira, Confluence and IDEA and then were going to look beyond those if neither worked for us. Parabuild was on the second list at that stage. Bamboo worked really well right “out of the box” and so we went with it. In a perfect world we would have had more time to really do a comprehensive review of tools but we’re in a rush and with only a couple of developers we just couldn’t justify a long review cycle.
Hi Mike!
I have been using the professional version of Luntbuild: Quickbuild and the eatures are really very interesting and flexibility is the key word in quickbuild, though there wasnt any plugin available for quickbuild with confluence but the reports could be easily extracted througha sql query from jira db and pushed into confluence in the desired format using shell scripts and this support for shell scritps is an advantage which though available inmost build management systems but cant be used to the fullest like in quickbuild.
Another interesting feature was of inheritance which though i’m evaluating Bamboo, I cant use it as adeptly as in quickbuild.