Unit-Testing: The Martinizer

I. Introduction & Philosophy

The “Martinizer” is a little tool I’ve created to help improve unit-test coverage on large Java projects.  The name is a tip-of-the-hat to “Uncle Bob” Martin of TDD fame, with a little word-play on “One hour Martinizing”, a dry-cleaning service.  (DRY, as in Don’t Repeat Yourself — get it?)

The tool itself is pretty simple, but a bit of history may help clarify its intended purpose and use.  I do a lot of work on a large Java project that has less-than-ideal unit-test coverage.  As part of that work, I teach, encourage, and sometimes nag other developers to improve the quantity and quality of their tests.

A key benefit of unit-tests is that they act as a “safety net” when making changes to existing code.  If I know that the code I’m working on is well covered by tests, then I am more able to make changes, and refactor code as necessary, knowing that the tests will alert me if I stray.  (“Well covered” means more than strict coverage; it also means that the tests really check for the proper results of the code, not just did the code execute.)

On the other hand, when I’m working with code that is only partly covered, the safety net has holes in it, and my confidence in my ability to make changes (without breaking functionality) is lessened.  And in a large project, with a history of less-than-ideal test coverage, it can feel like it’s an uphill climb to add tests, even for small changes to existing code.

Over the years, I’ve tried several approaches to encourage developers to improve the safety net, including:

• Graphing code coverage and displaying it publicly.
• Sorting classes by frequency of change and lack of coverage.  (I created a script to this end, called “Vulture”.)
• Monitoring commits for overall decreases in coverage, and kicking out automated emails to the committers when it happens.  (Code-named the “Nagamatic”.)

but none of these particularly hit the “sweet spot” where a minimum of effort returns a maximum benefit.  Working from a suggestion by my colleague Jens Wessling, I tried a different approach.  What we’re especially interested in is:

• code that has changed
• that isn’t covered by tests

So why not just report on that?  That is the purpose of “The Martinizer”.   It tells you which lines you added or changed, that aren’t covered by tests.  That’s it.

II. What’s It Do?

In brief. the Martinizer compares two different git commits of a Java project.  It finds the new or changed lines of code in the newer commit, that isn’t fully covered by tests, and shows them to you.  The output looks like:

com.mycompany.access DatabaseListManager:
3849 subjectSortLocale = value;
3850 }

com.mycompany.pages.maint SearchToolDisplay:
44 if (history != null && ! resultsId.equals("-1")) {
91 if (displayCount < 0)
92 return "AUTHENTICATION ERROR";

com.mycompany.pages.maint SearchToolPOST:
102 String embeddedLoginToken = request.getParameter("token");
103 if (! authenticateFromToken(embeddedLoginToken)) {
104 return linkSource.createPageRenderLinkWithContext("maint/SearchToolDisplay", "-1", -1);

The example shows the output from a single run of the Martinizer.  It reported on three classes, and shows the line numbers, and line text, of lines that were added or changed, but were not covered by tests.

The immediate benefit of this kind of report is to keep things from getting worse.  In a large, perhaps legacy, project, no individual developer can reasonably be tasked with adding full coverage.  But they can be held responsible for “holding the line”, for making sure new work is always covered.

But there’s a more subtle purpose at work as well.  When a system like the Martinizer is in place, it can help reinforce everyone’s discipline and motivation.  Knowing that everyone is being held responsible, that no-one can back-slide, supports each individual’s desire to move forward, to add more and better tests.

There are a variety of ways the Martinizer can be used on a day-to-day basis.  The specifics will depend on your particular circumstance.  Approaches include:

1. Generate reports at regular intervals.  This is my personal favorite for the “large project” I mentioned earlier.  Once an hour, a cron job on a Linux server does a git pull on the trunk of that project, and runs the martinizer, comparing the current commit with the commit from the previous hour.  The report is emailed to a central person (me), who then forwards it on to the likely suspects.    This does have a bit of a “Big Brother” feel, which has both pros and cons.  But it could also be easily extended to automatically forward the reports on to everyone named in the commits in the last hour.
2. Generate a report on-demand.  Allow individual developers to compare (say) their active branch with trunk (or trunk as it was when they branched).  This is the most pro-active approach, and we also use this in my workplace.
3. Generate report on a CI build.  Perhaps the most obvious option is to automatically generate (and send) the report on every continuous-integration (e.g. Jenkins) build.

Each of these approaches can be used at different times, and for different purposes:

1. When you’re ready to merge your working branch into the trunk.
2. When you’ve been making changes in several places, and want a little help remembering where you need to add coverage.  (Assuming you didn’t TDD in the first place… or even if you did but just want to be certain.)
3. When you’re code-reviewing someone else’s changes.
4. When you’re overseeing a larger group, and want to help ensure the social contract and safety net for everyone.  (Which requires a mixture of tact and firmness that can be tricky, but also very rewarding.)

III. How it works

The Martinizer is not a full-fledged application.  It is a collection of a few simple tools that can be combined in several ways, as per the above.  It assumes the use of, and depends heavily on, ‘git’ and on Jacoco.  The tools are:

1. ChangedCodeScanner.  A small Java project (jar) that simply takes the output from ‘git diff’ and reformats it in a way that the martinizer script(s) can read.
2. martinizer.  This is the core shell script that does all the work.  It calls ‘git diff’, runs the output through ChangedCodeScanner, maps the changed lines against the report from Jacoco, and writes the final report to stdout.
3. runMartinizer.  This is a script for approach II.1, above.  It is normally run from cron, assumes the project is checked out in a specific location on the server, takes care of remembering the commit from the previous run, and emails the report to someone.

The entire project is available for download from github.com/wcroth55/TheMartinizer.  It is released as “open source” (Artistic License 2.0); the details are in the LICENSE file in the project.

IV. Notes, thoughts, and disclaimers about the Martinizer output

1. Not everything is meaningful.  If you reformatted a line, or if you renamed a variable in a method, the Martinizer may tell you about it.  If it looks like something that really deserves test coverage, you may want to add a test.  But you may not be “on the hook” just because you reformatted existing code.
2. Not everything is important.  In the example above, SearchToolPost is a temporary utility class that is only being used by developers, and will most likely go away.  I might not care about unit-tests for it.
3. “Not covered” means not 100% covered.  In the example above, line 44 of SearchToolDIsplay has three possible execution paths.  If only two of the paths are tested, the Martinizer considers it “not covered”.  (Emma would mark it as yellow.)  Your change may only affect a path that is tested.  Use common sense, and don’t go overboard.
4. Some things are dumb.  E.g. line 3850 in DatabaseListManager.  Do we really need to worry about coverage of a closing right-brace?
5. It’s because of the legacy (code).  If you’re TDD-ing all your code, if you write unit-tests for everything, you don’t need the Martinizer.  But if you’re working with legacy code, or you need help ‘encouraging’ people to keep writing tests, this is for you!
6. Filter for surprises.  On the positive side, the whole point of the Martinizer is to help you find things that are surprising.  If you see a line in a Martinizer report and go “whoa, that’s not tested?!”… then it may be telling you something important.
7. Other people may be watching.  When you’re writing or changing code, remember that your code reviewer might use the Martinizer.  We’re not going all “Big Brother” here, but it may help you think twice about whether your code needs a unit-test.  (Think of it instead as a tiny, crude, automated, after-the-fact form of pair programming.)

V. Comments and Suggestions

I welcome questions and suggestions, and if you use the Martinizer, I’d love to hear about your experiences.