Wednesday, January 9, 2008
Test your Unit Tests?
When I was working on a project in the past I stopped the webserver and database before running the unit tests. To my surprise I found 40+ unit tests started failing. As you would guess the tests did not use the proper test doubles (mocks/stubs..).
The next interesting finding was some tests started failing when they were run repeatedly (the same test multiple times in a single build).
Both of the above problems are not typical in a fresh development project with people having the right experience with testing/TDD. However these problems could emerge when your team has
- less experience in testing or
- you are working on a legacy codebase which is not testable.
With the legacy codebase, refactoring the codebase is an option to make it testable. However you need tests to refactor with confidence. Its a cyclic dependency problem. So the idea should be to add tests as early as possible. Such tests may be often added at a higher level and may not be proper unit tests. Such tests should be refactored later as the refactoring of the legacy codebase progresses. Due to various reasons some of such tests remain unattended and remain in the codebase. Most of the times the reason why they continue to be in the system is because we do not know that they needs to be addressed.
So what can we do about it?
1.Automate your build scripts to shutdown all the integrating systems, as a part of your build script, before you run the unit tests. Stop the webserver, database server, disable network connections etc..
2.Automate the build script to repeat the tests (the same test multiple times in a single build) occassionally (may be with the nightly build). We can use a decorated TestRunner for this.
So we have added two parameters of the unit test quality to our build
the 'unitness' and 'repeatability'. ¶ 4:48 AM
The next interesting finding was some tests started failing when they were run repeatedly (the same test multiple times in a single build).
Both of the above problems are not typical in a fresh development project with people having the right experience with testing/TDD. However these problems could emerge when your team has
- less experience in testing or
- you are working on a legacy codebase which is not testable.
With the legacy codebase, refactoring the codebase is an option to make it testable. However you need tests to refactor with confidence. Its a cyclic dependency problem. So the idea should be to add tests as early as possible. Such tests may be often added at a higher level and may not be proper unit tests. Such tests should be refactored later as the refactoring of the legacy codebase progresses. Due to various reasons some of such tests remain unattended and remain in the codebase. Most of the times the reason why they continue to be in the system is because we do not know that they needs to be addressed.
So what can we do about it?
1.Automate your build scripts to shutdown all the integrating systems, as a part of your build script, before you run the unit tests. Stop the webserver, database server, disable network connections etc..
2.Automate the build script to repeat the tests (the same test multiple times in a single build) occassionally (may be with the nightly build). We can use a decorated TestRunner for this.
So we have added two parameters of the unit test quality to our build
the 'unitness' and 'repeatability'. ¶ 4:48 AM
Comments:
<< Home
One thing I want to clear up:
Unit tests shouldn't depend on outside systems. However you should still have tests that integrate with outside systems, right? I am not clear if you are advising people not to do that.
There is nothing wrong with a codebase that has tests dependant on outside systems. Just don't call those tests unit tests. ActiveRecord is a good example of a codebase that has tests dependant on outside systems for a good reason (to verify it works against different databases).
The other thing I wanted to mention:
Automating the build script to repeat the tests is actually a well known practice called continuous integration. Check out Martin Fowler's article here: http://martinfowler.com/articles/continuousIntegration.html If you already knew that then I am missing out on what you mean and I beg further explanation of your idea.
Unit tests shouldn't depend on outside systems. However you should still have tests that integrate with outside systems, right? I am not clear if you are advising people not to do that.
There is nothing wrong with a codebase that has tests dependant on outside systems. Just don't call those tests unit tests. ActiveRecord is a good example of a codebase that has tests dependant on outside systems for a good reason (to verify it works against different databases).
The other thing I wanted to mention:
Automating the build script to repeat the tests is actually a well known practice called continuous integration. Check out Martin Fowler's article here: http://martinfowler.com/articles/continuousIntegration.html If you already knew that then I am missing out on what you mean and I beg further explanation of your idea.
I am not against integration tests. I am talking more about the tests which are meant to be unit tests and still talks to outside systems(please see in the post how we ended up having such tests working on a legacy project).
About repeating the tests, what I meant is in a single build run the same test (say FunnyTest) multiple times, so as to make sure that the each test run is independent of other.
Please see my trivial example. If I do not have a proper SetUp/TearDown the test will fail on the second run. Hope that clarifies.
MyFunnyTest {
testForFun(){
appendSomethingToFile("something");
assert("something", readFromFile())
}
}
Post a Comment
About repeating the tests, what I meant is in a single build run the same test (say FunnyTest) multiple times, so as to make sure that the each test run is independent of other.
Please see my trivial example. If I do not have a proper SetUp/TearDown the test will fail on the second run. Hope that clarifies.
MyFunnyTest {
testForFun(){
appendSomethingToFile("something");
assert("something", readFromFile())
}
}
Subscribe to Post Comments [Atom]
<< Home
