Filed under: TDD, Test automation, — Tags: JUnit, Java — Thomas Sundberg — 2016-03-25
A test must be independent. That is, a test should never depend on the result of another test.
If you run two tests in a specific order, you should expect the same result as if you run them in the reverse order.
This is not always the case. It happens that I see tests written that changes the state of an application and other tests that rely on this new state. These tests are coupled and they have to be executed in the correct order. The second test will fail if the first test has not been executed.
I had such an issue the other day, a test that created a few files was impossible to run two times in a row. The reason was that the state was not reset before each execution. A proper setup of the expected state before executing the test was not done.
I have also seen cases where a test suit works on the developers machine, but not on the continuous integration server. The reason? The executions where done in a different order on the continuous integration server and there where coupled tests.
It is a problem because it doesn't allow you to work on a small part of your system in isolation. Working on an isolated part is good because it is smaller than the whole and therefore easier to understand. Easier to understand means easier to get working.
The state of an application can often be reset if we test on a newly started application. Restarting the application before each test execution is one option. There is a draw back, starting the application before each execution can be very slow. Some applications takes many seconds to start. Our feedback cycle suffers. Changes saved in a database, and therefore a new state for the application, will still be a problem.
It is important that a test don't rely on any state it isn't responsible for. All tests must take control over their world and prepare its expected state. This has a nice side effect. If a test fails, you are able to see the state it left the world in when it failed. You are able to do a post-mortem and trouble shoot in a repeatable manner. You can always re-run the failing test and get the state back.
One way to make sure we don't allow any coupling between tests would be to execute them in a random order. This makes it hard to write tests that must be executed in a specific order. They would randomly fail and that is a good sign that there are couplings between tests that must be removed.
This is implemented in Mavens Surefire plugin. You are able to define that the test should be executed in random order. I wrote about this a while ago.
What I really would like to see is that random is the default for tools like JUnit. The JUnit developers claim that the order is more or less random from JDK 7. More or less is not good enough. Really making sure that the execution order is random would be a great default. JUnit has annotations for executing tests in order. I can see situations where it can be used, but currently I can't see any situation where it can be used in a good way. The only reason I see to execute tests in a specific order is to handle dependencies between them. And dependencies between tests is bad.
If you can't run a test in isolation, don't run the entire test suit. Instead, do whatever it takes to allow the test you want to run isolated. You have a problem and ignoring it will not make it smaller. The opposite is more likely, it will grow.
I would like to thank Malin Ekholm for proof reading.
RSS