Code Coverage Tools

While unit testing is a good start and it’s admirable to try and insure all your unit tests actually pass, it’s somewhat useless if you don’t know how much or what part of your code base doesn’t benefit from unit testing. Even the most committed developers of unit testing or TDD will miss covering some portion of their code. How do you try and prevent this from happening? Enter code coverage tools. The best known is probably Clover from Cenqua but for those of us working on our own home grown projects or that just like to use open source tools there is Emma.

Works Great!

I’ve been using it for MyThingo and it’s been very helpful. I’ve uncovered a few bugs that I didn’t know existed and hadn’t been uncovered by my unit testing up until now. Emma works by instrumenting your code and then you execute your unit tests using the instrumented class files. At runtime you set variables that tell Emma were to keep it’s coverage database and then after your tests execute you generate the report from that data. It seems to work very well and has a pretty comprehensive report of coverage including views of all your code with color coding to show which lines have and have not been executed during your tests.

Shortcomings

I have to admit that I miss the spit and polish that Clover seems to have. For example, Clover tells you how many times a line has been executed. While not critical it is a nice to have and I would expect this data is available in clover but just not exposed. If I get some time I may look into the report generating code and see what’s available. The integration of Clover in to Eclipse is also a very nice to have feature but again, isn’t critical.

Conclusion

Code coverage is one of those things that isn’t talked about much in unit testing circles. Okay, it is at times but not many developers I have worked with over the years really pay much attention to it. It is a critical part of your code testing infrastructure, however. You can’t just accept the fact that your unit tests pass. You must know how much of your code base is actually tested and then make sure that the parts that aren’t tested aren’t necessarily critical. It’s something I’ll harp on but in addition you have to make it brain dead simple. It has to be part of your build and not require any special attention by developers. It has to be part of your nightly build so reports are kept up to date as well. And finally, someone actually has to pay attention to the results and encourag developers to increase their coverage scores. If you do all this I think you’ll definitely see an improvement in the quality of the code that is delivered to QA and your users. And just think, better code going in to QA tends to mean shorter QA cycles which means faster delivery.

Persistence Testing

One of the things that is always a pain when unit testing is when you need to test functionality that utilizes persistence. The unit tests that are easy are the ones that just test the functionality in your POJOs that doesn’t require persistence to test. Then there are the tests you want to write to make sure that your persistence configuration is correct. For example, you want to make sure that you’ve got your associations in Hibernate configured properly or that changes to data model don’t break things when you go to save. I’ve made mistakes before where I added a property that was required but some of my code didn’t set that property. It’d be nice to know that by having a unit test tell me that saving the object fails in certain situations.

The difficult part in this is getting a database configured. No one really wants to have one or more databases that are just for testing. It’s a pain to maintain and is going to be slow. When you unit test most times you are going to want to start with a clean slate and then populate with the data that is necessary for your tests. All of this overhead for a test? Not Ideal. What are some options?

Enter Hypersonic

Hypersonic is a Java database that can be run either on disk or in-memory. The in-memory is what we’re most interested in for testing because it doesn’t require a location on disk to run, is extremely fast since it doesn’t require disk access and is compact.

What I do is have a separate hibernate.cfg.xml file for unit testing that is identical in most regards to my production hibernate.cfg.xml except for the fact that it uses a straight JDBC connection vs. a DataSource reference for its connection source. Additionally, I tell Hibernate to create the database schema upon startup. Here is an example of some of the entries I set:

     <property name="hibernate.connection.url">
          jdbc:hsqldb:mem:testdb
     </property>
     <property name="hibernate.connection.driver_class>
          org.hsqldb.jdbcDriver
     </property>
     <property name="hibernate.dialect>
          org.hibernate.dialect.HSQLDialect
     </property>
     <property name="hibernate.connection.provider.class>
          org.hibernate.connection.DriverManagerConnectionProvider
     </property>
     <property name="hibernate.hbm2ddl.auto>
          create-drop
     </property>

The key here is the JDBC url, jdbc:hsqldb:mem:testdb, which tells Hypersonic to run in-memory vs. on disk and then the setting for hibernate.hbm2ddl which is create-drop and tells Hibernate to perform automatic schema generation.

I also use Spring as my primary application container/framework and so what I also do is have an applicationContext.env.xml for my normal application database environment and then a testApplicationContext.env.xml for testing. The main difference is the normal one configures the Hibernate SessionFactory with the production Hibernate configuration file and the test environment file configures it with the test Hibernate configuration. When I go to run unit tests that need both Spring and Hibernate I will use the test application context in place of the normal one. It’s pretty easy to do since I have a base class used for testing that will build out the application context during startup. It does this by just listing all of the Spring configuration files during startup and passing them in to the appropriate ApplicationContext sub-class.

Give this approach a try if you’ve been struggling with unit testing that involves database operations. It has saved me a lot of time and headache since I started using it. You can send me an e-mail if you have any questions about this.

Next up, code coverage with Emma.

A while back I started working on a project in the evenings that turned in to MyThingo. I decided from the get go to make sure I built exhaustive unit tests for the codebase. At the beginning I was using JUnit but I had read about TestNG a few times so I decided to give it a try. It was an amazingly fast transition. If you want to give it a try, just follow the documentation that comes with it.

After I started using it the little things that it does better are what kept me using it. Here are a few of the things that I use a lot:

Annotation Based Configuration

Rather than having to use a naming convention or extending a particular TestCase I can just mark a method as testable. I don’t need to have all my methods start with “test”. That gets annoying. Here’s an example of how it looks:

@Test
public void verifyAddition {
    assert 4 == Number.add(2,2) : "2 + 2 should have added up to 4";
}

Obviously the unit test above is not real but it shows how you mark a test method and that you don’t have to name it testVerifyAddition. The class this method is in is just a standard POJO as well.

Another annotation that comes in really handy is the ExpectedExceptions annotation. This tells TestNG that the method should fail unless it receives an exception of the specified type. Here’s the difference between the normal way of testing for a particular exception in JUnit and one in TestNG. First, the JUnit way:

public void testForNullArgumentException() {     
    try {         
        new Foo.doMethod(null);
        fail("I should have received a NullArgumentException");     
    } catch(NullArgumentException e) {         
        return;     
    } 
}

Now the way it’s done in TestNG:

@Test
@ExpectedExceptions(NullArgumentException.class)
public void nullArgInMethodX {
    new Foo().doMethod(null);
}

A little simpler, heh? None of what I’ve shown is impossible in JUnit (obviously) but overall it just makes testing easier. And the key to getting folks to unit test is to make it as painless as possible.

More Flexible Execution Profiles
One of the areas of JUnit that can drive a person crazy is that the class is dumped and loaded for every test. That means you have to setup and tear down after every test method. The idea is that you start with a clean slate for every test. But in the real world that’s not always ideal nor is it performant. Sometimes I just want to initialize items for the entire test class and other things for every method. TestNG lets you do this.

The way you do this is with the @Configuration annotation. This annotation can be added to methods and then its options allow to specify how the annotation affects the test class. For example, the beforeTestMethod parameter tells TestNG that the method should be run before each test method in the class. With beforeTestClass the method should be executed once at the beginning of the test run for this class. There are other options you can find in the TestNG documentation but these are the ones I use the most.

The last item I thought I’d go over is the ability to tell TestNG that a method X depends on the successful execution of other method(s) and if the dependent method(s) fails then there is no use in executing the test method in question. This can cut down on additional test execution times.

One of the issues I’ve had in getting other developers to unit test is that it’s usually just a pain to do tests beyond simple unit testing. You have to face facts that some developers just don’t care about it as much as you do. The key is making it as painless as possible and to make it relatively easy to mimic the environment that the code is going to run in. I’ve covered some of the areas where TestNG helps accomplish the simple/easier/faster part and in a future post I’ll cover some of the other areas such as setting up your Spring environment in conjunction with Hibernate using Hypersonic and auto-created database schemas using in-memory database. Another topic will be the use of easy mock for testing features that would normally require a container such as Tomcat. If you’re interested in those, leave me a comment or shoot me an e-mail and I’ll let you know when those are up and ready.