You want to refactor a class and the tests you were counting on are not there. Or they are there, but they aim somewhere else and none of them would notice if you got this particular piece wrong. The code works, presumably, since it has been in production for years. You just have no way to prove that it still works once you have touched it.

Characterization tests are for this situation. You write tests that describe what the code does right now, and then you refactor with those tests watching your back.

What a characterization test is

You call the code, you look at what comes back, and you write an assertion that matches. If you cannot predict the output, guess, run the test, and let the failure message tell you the real value. Then put that value in the assertion.

That last step sounds like cheating, and it would be if the point was to verify that the code is correct. It isn't. The point is to record what the code does, so that any later change to it shows up as a failing test.

Which means that if the current behaviour is wrong, you pin the bug. On purpose. A method that returns null on an empty list where it should have thrown gets a test asserting null. That feels bad, and it is still the right move. You are about to move the code around and you need to know whether the behaviour survived the move. Fixing the bug is a separate step, taken later, with tests already in place to tell you exactly what you changed.

Why this is not Test-Driven Development

Both are red-green loops, so they sometimes get lumped together. However, they solve opposite problems.

In Test-Driven Development, TDD, the test comes first and the production code does not exist yet. The test is a design tool. I know of devs who would like TDD to mean Test-Driven Design. Writing a test forces you to decide what the thing is called and what it hands back before you have committed to an implementation. It is an expression of intent.

A characterization test comes second. The code already exists, the design decisions were made by someone who may not be around anymore, and you get no design feedback from the exercise at all. The test says nothing about whether the code is any good. It only tells you when you have changed it. That is a much smaller thing than TDD gives you, and it is the only thing available once the code is already written.

Approval testing does the tedious part

Guessing a value and then copying the real one out of the failure message is mechanical enough that there is a tool for it. ApprovalTests, by Llewellyn Falco and Lars Eckart, is built around that step. You hand Approvals.verify() whatever the code produced. The first run writes what it got to a received file and you read it. If it describes the behaviour you have, you approve it, the file becomes the assertion, and every later run compares against it and fails on any difference.

I have used ApprovalTests properly once, at a workshop years ago, and it never became part of how I work. So take this as a pointer rather than a report from daily practice. I am including it because the idea is right, not because I can tell you how it holds up after a year in a real code base.

For characterization work it beats hand-written assertions in one specific way. It scales to output you would never sit down and type out, like a generated report or a whole object graph. There is also a combination feature that runs every permutation of the inputs you give it and puts the lot in a single approved file, which covers a tangled method a great deal faster than writing the cases one at a time.

An approved file pins everything in the output, so the obvious worry is that a timestamp or a generated id will break a test that has nothing to do with your refactoring. That is handled. There are scrubbers for it, and you hand them to the verification as an option: DateScrubber for dates and times, GuidScrubber for generated ids, a RegExScrubber for whatever else varies, and Scrubbers.scrubAll to combine several of them. What is left is deciding which parts of the output are behaviour and which are noise. That is the same decision you make when you write an assertion by hand.

Aim at the seam, not at every call

The common mistake is writing too many of them. Pin every internal method call and you have cemented the current structure rather than the current behaviour. The refactoring you wanted to do is then blocked by your own tests. Every method you extract breaks something, and you end up rewriting the safety net at the same time as the code it was supposed to protect. At that point you do not have a safety net, you have two things changing at once.

Aim at the seam you are going to work behind instead. Find the outermost point where the behaviour is still observable and where you have no intention of changing the interface. A public entry point or an API boundary. Whatever is going to look the same from the outside when you are done. Everything inside that boundary is then free to move.

Aiming a test at something that is not an endpoint can feel wrong, but systems have behaviour at every level, not only where a user can see it. I made that argument at length in a series back in 2012, building the same model behind a JSF page, a Wicket page, a Swing GUI and two web services. The behaviour did not change from one to the next, only the way it was verified.

When I moved LogEze from Spark to Javalin, the characterization tests went at the shared super class that every controller ran through. The controllers themselves were already covered, but those tests reached the super class from the outside and none of them aimed at it. That class was the one thing I was about to rebuild underneath, so it needed tests describing what it did before the rebuild started. I have written about that migration in more detail.

The tests I did not write

LogEze has good test coverage. It also has holes, and I know roughly where they are. Before I had a coding assistant I would add tests when I had to, but never as many as the code actually needed. Writing them by hand is boring work, and I am as good at talking myself out of boring work as anyone else.

So the refactoring would get postponed. Not decided against, just postponed, because the honest version of the task was a couple of hours of writing tests for code I did not even intend to keep, followed by twenty minutes of the work I actually wanted to do. Those two hours never fit into the day.

These days I point the assistant at the code and tell it to characterize the mess before I touch it. I make sure there are no changes in the production code first, so the tests describe unchanged behaviour rather than something I have already half broken. Then I read what came out, which is the part I cannot delegate, and only after that do I start changing things. When I break something, I hear about it immediately instead of hearing about it from a user.

The work did not become more interesting, it just became cheap enough that I actually do it. Technical debt survives because the cost of each small cleanup adds up to more than anyone can justify. Lower the cost of the safety net and some of that backlog starts moving again.

When not to bother

So should every piece of untested code get a characterization test before you touch it? No, there are a few cases where it is the wrong tool or simply not needed.

If the code is new, write the test first. You get design feedback out of it, which is worth a great deal more than a snapshot of behaviour that does not exist yet.

If you already know the behaviour is wrong and fixing it is the job, there is no reason to pin something you will delete an hour later. Write the test you want instead. And before you start on any of this, look at what you already have. A test that aims at the seam you care about is a safety net whatever reason it was written for. Read it rather than trusting a coverage report, because the number will not tell you whether the test asserts anything at all.

Conclusion

Characterization tests are the cheapest way I know to make untested code safe to change. They describe what the code does, not what it should do. That is a limitation and it is also the point, since they go red the moment you change something by accident. Put them at the seam, do the refactoring, and then go back and fix the bugs you pinned on the way in.

Resources