Noeud:Specify the Testing Goals, Noeud « Next »:, Noeud « Up »:Designing a Test Suite


Specify the Testing Goals

A test suite is a project in and of itself.
As much as possible, you should define the objectives to be reached by the test suite, and you even may isolate different aspects that should be covered by different test suites, or even different test tools.

Knowing who will run the test suite is crucial: a maintainer runs additional tests and only needs raw information such as a core dump, while when run by a user it should ease the writing of a precise bug report.

Keep in mind that message exchanges are extremely costly in the bug tracking process: the user is likely to be lax about answering simple questions. Without any answer, you will know a bug is sneaking, imperceptible to you, in the program.

You might come up with different test suites, some of them exercising the code with time consuming internal checks activated, or simply changing something in the environment. For instance you might want to run your programs with special tools which considerably improve the severity of the testing conditions that the users do not necessarily have: debugging memory management libraries, bound checking compilers, etc.

Running all these tests can be extremely time consuming1, and can discourage users from running them. While this should never be a reason not to test a feature, you should nonetheless keep this factor in mind, and maybe design a consistency test suite which is to be run by maintainers only, and a "public" test suite, which would be run by virtually all the users.

Notes de bas de page

  1. The GCC test suite takes hours to run on most architectures. While performed in a quarter of an hour on my machine, the Autoconf test suite has been reported to take up to 9 hours by some users. Using Dmalloc at its maximum checking possibility slows down a program by several orders of magnitude, making it hardly usable.