Unit Tests is a procedure every developer gets to know during her/his education. Additionally, it is one of the most discussed topics in computer science. It’s not about whether Unit Tests are useful, but rather how thoroughly the code should be tested or how high the priority of these tests is during development. But how do you set the standard for good and thorough unit tests? Which ingredients make delicious unit tests possible?

Code Coverage

Yes, of course. First thing that comes to mind is the code coverage of the Unit Tests. Not only the number of lines of code covered. But also the branches and different test classes which are based e.g. on different input parameters.

There are many tools that help to get a good indication of code coverage. These can be installed by plugins in the own development environment ( e.g. EclEmma for Eclipse ) or are even already brought along ( e.g. the Code Coverage Runner in IntelliJ IDEA ). External tools like SonarCube can also measure code coverage and provide different additional statistics.

So, is code coverage a tangible value for your own test quality? Unfortunately, only partially. The code coverage of a test merely states whether a test reaches and executes every part of the code including branches. However, it is important to recognise that your own tests may not cover all error cases or even have tested all test classes although a high-test coverage has already been achieved. So how can these missing tests be discovered?

Mutation Testing Frameworks

The mutation of tests is a great approach to identify possible missing or weak tests. To recognise the quality of the tests, these frameworks manipulate the source code on byte code level and exchange e.g. comparison operators like > with <=. The following code can be used as an example. Of course, it can be replaced with every other code sample or different code complexity.

1 public boolean check ( final int value )
2 {
3 if( value < 128.0 )
4 {
5 return true ;
6 }
7
8 else {
9 return false ;
10 }
11 }

The mutation framework changes the code like this:

1 public boolean check ( final int value )
2 {
3 if( value >= 128.0 ) // Changed here
4 {
5 return true ;
6 }
7
8 else
9 {
10 return false ;
11 }
12 }

After that there must be at least one existing test case which detects this manipulation and fails. If not, the tests should either be supplemented by a test case or be reworked.

But like everywhere in computer science there is also a condition which the programmer himself has to fulfil. To obtain strong mutation tests, both the input parameters and the output parameters must be checked. In our example the parameter ‘value’ and also the return value must be checked in the unit test. Otherwise, only so-called weak mutation tests with less significance are obtained.

The advantage is an automated possibility to assess the quality of tests and to detect possible missing cases very quickly. In addition, no code or test adaptations are required and it can be integrated into any project rather quickly.

A well-known and intuitive framework is for example PITest. It is usable with maven, ant, gradle or simple command line tools.

Develop a recipe

Code coverage tools and mutation frameworks are a very powerful solution for good Unit Tests. The interaction is very feasible, because both are automatically present in the project and need little overhead for integration. Of course, there are other ingredients for an optimal Unit Test recipe. However, these two already allow a very good result and a significant increase in quality of the project code and tests.

Bon appétit.

Have fun. Enjoy coding.
Your INNO coding team.