Noeud:Develop the Interface, Noeud « Next »:, Noeud « Previous »:Specify the Testing Goals, Noeud « Up »:Designing a Test Suite


Develop the Interface

A test suite is a tool.
Because programs are sensitive to the environment1, playing with a failure is often needed, either to get more details using some specific feature, or after a variable was changed, and simply after the program is --hopefully-- fixed. All these scenarios may happen on the user's side, therefore you should make it easy for them: have a clean interface, and some user documentation. Bug reporters are well-meaning, but have often little time and varying skills: let your interface be simple enough for you to ask "please run this simple command". Conversely, some users are ready to spend some time tracking the origin of a failure: everything must be done to play easily with variation of the test scenario.

Because unfortunately not all the failures are detected by the test suite, you might also be interested in equipping the programs with verbosity/tracing options. Many programs provide a --verbose option which produces information about critical internal variables, the various actions that are performed etc. It often helps locating when the bug was triggered.

No one would ever work with a Boolean compiler: while compilation succeeded is all we need, compilation failed is definitely not enough information for a programmer to track an error. Test suites are very similar: we all expect test suite succeeded, but test suite failed, or even test suite failed: test 51 failed, are not enough. Your testing scheme should provide accurate information on the tests that were performed, what was expected, and what was obtained.

If the test suite may be run by users, then you should pay even more attention to providing as much information as needed to understand a failure at distance. For instance, have the test suite wrap all the pertinent information in a log file, and ask for this file.

Notes de bas de page

  1. Like for plants and fishes, have your sensitive programs listen to classical music instead of hard core techno. Do as I do: use earphones.