Code Coverage Tools
While unit testing is a good start and it’s admirable to try and insure all your unit tests actually pass, it’s somewhat useless if you don’t know how much or what part of your code base doesn’t benefit from unit testing. Even the most committed developers of unit testing or TDD will miss covering some portion of their code. How do you try and prevent this from happening? Enter code coverage tools. The best known is probably Clover from Cenqua but for those of us working on our own home grown projects or that just like to use open source tools there is Emma.
Works Great!
I’ve been using it for MyThingo and it’s been very helpful. I’ve uncovered a few bugs that I didn’t know existed and hadn’t been uncovered by my unit testing up until now. Emma works by instrumenting your code and then you execute your unit tests using the instrumented class files. At runtime you set variables that tell Emma were to keep it’s coverage database and then after your tests execute you generate the report from that data. It seems to work very well and has a pretty comprehensive report of coverage including views of all your code with color coding to show which lines have and have not been executed during your tests.
Shortcomings
I have to admit that I miss the spit and polish that Clover seems to have. For example, Clover tells you how many times a line has been executed. While not critical it is a nice to have and I would expect this data is available in clover but just not exposed. If I get some time I may look into the report generating code and see what’s available. The integration of Clover in to Eclipse is also a very nice to have feature but again, isn’t critical.
Conclusion
Code coverage is one of those things that isn’t talked about much in unit testing circles. Okay, it is at times but not many developers I have worked with over the years really pay much attention to it. It is a critical part of your code testing infrastructure, however. You can’t just accept the fact that your unit tests pass. You must know how much of your code base is actually tested and then make sure that the parts that aren’t tested aren’t necessarily critical. It’s something I’ll harp on but in addition you have to make it brain dead simple. It has to be part of your build and not require any special attention by developers. It has to be part of your nightly build so reports are kept up to date as well. And finally, someone actually has to pay attention to the results and encourag developers to increase their coverage scores. If you do all this I think you’ll definitely see an improvement in the quality of the code that is delivered to QA and your users. And just think, better code going in to QA tends to mean shorter QA cycles which means faster delivery.
Predictable Chaos 
I always wondered why 100% (or 90% or any number) test coverage is more important than the quality of tests. I mean, a lazy developer could make some dummy tests, they will pass and it will show perfect coverage on Emma, and everyone will be happy. But in the end is useless. I would prefer not to stick to a predetermined coverage, but to put more emphasis in the quality of tests, and be sure not waste time on trivial code that doesn’t need to be tested, like setters and getters.
I agree that coverage report is important to learn about missing tests, but to be fanatical about 100% coverage is something I’ll never understand.
I agree with you on the 100% coverage ideal. Most people seem to agree that an 80% target is pretty good target. I look at each class after I finish a round of unit tests and determine if the uncovered code truly needs more coverage before I decide my next step. I also have to look at the tests themselves and see if I have covered adequate success, failure and exceptional execution cases.
-Mike
Coverage measurement is just another tool, and a coverage percentage is really not meaningful in isolation. Used for good (as opposed to evil) coverage reporting can be very useful in helping you focus your limited testing resources. It can also make writing tests less frustrating by giving you feedback whilst actively writing tests. The percentage itself can be useful if it is tracked over time. Whether it is 7% or 97% what matters is that as you add new code it doesn’t go down too much.
[disclaimer: I work for cenqua who make clover] Clover has a couple of unique features that help. You can exclude code that you don’t want to measure coverage on with a regex (thereby making the % a little more meaningful). It can produce a historical report that charts coverage over time. It has IDE plugins.
Cheers,
Pete.
Hi Pete,
I agree that any metric can be abused. At some point you really just have to trust people to use the tests to help themselves vs. abusing the system. Tracking coverage changes over time is a great metric.
I agree anyone can write set of tests that will give 90 percent coverage but will not test the product well. Still I think if you aim to write ‘good’ tests the better coverage you get the better your product is tested. I use netbeans and we even developed a special plugin that visualizes unit tests code coverage. We found it really helpful for our projects as we could see what exactly lines are not covered. The plugin is available open-source at http://unitcover.dev.java.net
If you’re looking for an Eclipse integration of EMMA you might check out my recent project at http://www.mountainminds.com/products/eclemma. It’s a free Java code coverage tool based on eclipse.
100% coverage is nice because it’s a red/green indicator that indicates that the build is not correct.
This does not mean that all your files need to be covered–I’m currently trying to figure out how to exclude classes that are nothing more than property-bags, but having the go/no-go is pretty nice.