Code Coverage Tools
While unit testing is a good start and it’s admirable to try and insure all your unit tests actually pass, it’s somewhat useless if you don’t know how much or what part of your code base doesn’t benefit from unit testing. Even the most committed developers of unit testing or TDD will miss covering some portion of their code. How do you try and prevent this from happening? Enter code coverage tools. The best known is probably Clover from Cenqua but for those of us working on our own home grown projects or that just like to use open source tools there is Emma.
I’ve been using it for MyThingo and it’s been very helpful. I’ve uncovered a few bugs that I didn’t know existed and hadn’t been uncovered by my unit testing up until now. Emma works by instrumenting your code and then you execute your unit tests using the instrumented class files. At runtime you set variables that tell Emma were to keep it’s coverage database and then after your tests execute you generate the report from that data. It seems to work very well and has a pretty comprehensive report of coverage including views of all your code with color coding to show which lines have and have not been executed during your tests.
I have to admit that I miss the spit and polish that Clover seems to have. For example, Clover tells you how many times a line has been executed. While not critical it is a nice to have and I would expect this data is available in clover but just not exposed. If I get some time I may look into the report generating code and see what’s available. The integration of Clover in to Eclipse is also a very nice to have feature but again, isn’t critical.
Code coverage is one of those things that isn’t talked about much in unit testing circles. Okay, it is at times but not many developers I have worked with over the years really pay much attention to it. It is a critical part of your code testing infrastructure, however. You can’t just accept the fact that your unit tests pass. You must know how much of your code base is actually tested and then make sure that the parts that aren’t tested aren’t necessarily critical. It’s something I’ll harp on but in addition you have to make it brain dead simple. It has to be part of your build and not require any special attention by developers. It has to be part of your nightly build so reports are kept up to date as well. And finally, someone actually has to pay attention to the results and encourag developers to increase their coverage scores. If you do all this I think you’ll definitely see an improvement in the quality of the code that is delivered to QA and your users. And just think, better code going in to QA tends to mean shorter QA cycles which means faster delivery.