Simple Uses of Autotest

Just like configure.ac, the very first thing a test suite needs is an identity: the name and version of its hosting package, AT_PACKAGE_STRING, an address where failures should be reported, AT_PACKAGE_BUGREPORT, and optionally, the test suite's own name, given as argument to the macro AT_INIT. When a test suite is embedded in a package, its identity is automatically provided by configure, see Autotesting GNU M4; for the time being we will simply m4_define them.

Invoking AT_INIT is mandatory, as is AC_INIT in the Autoconf world:

AT_INIT ([name])

Macro

Initialize Autotest. Giving a name to the test suite is encouraged if your package includes several test suites.

Then, of course, it needs tests. A simple test will suffice for our purpose, for instance checking that m4 supports the most common options required by the GNU Coding Standards: --version and --help.

     # Process with autom4te to create an -*- Autotest -*- test suite.
     
     m4_define([AT_PACKAGE_STRING],    [GNU Programming 2])
     m4_define([AT_PACKAGE_BUGREPORT], [gnuprog2-devel@sourceforge.org])
     
     AT_INIT([Standard Options: 1])
     
     AT_SETUP([Standard Options])
     AT_CHECK([m4 --version])
     AT_CHECK([m4 --help])
     AT_CLEANUP
     

     Example 12.6: std-opt1.at -- An Autotest Source
     

This test suite is composed of a single test group, named "Standard Options". This test group is composed of two steps: checking the reaction of m4 when given --version and when given --help.

Test groups are enclosed between AT_SETUP/AT_CLEANUP pairs:

AT_SETUP (title)

Macro

Begin a test group named title. This title is really the identifier of the test group, used in quiet and verbose outputs. It should be short, but descriptive.

AT_CLEANUP

Macro

End a test group.

To prevent a test group from corrupting another one (via trailing files, modified environment variables and so on), test groups are run by distinct sub-shells in distinct subdirectories. As a direct consequence, test groups cannot share files or variables. To enforce this clean separation between test groups, Autotest ignores anything that is not in a test group. As a consequence, you can run the whole test suite or just some selected test groups in any order without fearing unexpected side effects due to the testing framework itself.

Our unique test group is composed of two steps: AT_CHECK([m4 --version]) stands for "run m4 --version and expect a success".

To "compile" this Autotest source file into a Bourne shell-script, run autom4te:

     $ autom4te -l autotest testsuite.at -o testsuite
     

and then run it:

     ## -------------------------------------------------- ##
     ## GNU Programming 2 test suite: Standard Options: 1. ##
     ## -------------------------------------------------- ##
       1: std-opt1.at:8     FAILED near `std-opt1.at:9'
     ## ----------------------------------------------- ##
     ## ERROR: Suite unsuccessful, 1 of 1 tests failed. ##
     ## ----------------------------------------------- ##
     
     You may investigate any problem if you feel able to do so, in which
     case the test suite provides a good starting point.
     ...
     

Ugh! Something went wrong in our surprisingly simple test suite! The test suite is then rerun verbosely, creating a detailed log file, std-opt1.log, and suggesting sending it to the maintainers.

     ...
     Now, failed tests will be executed again, verbosely, and logged
     in the file std-opt1.log.
     ## -------------------------------------------------- ##
     ## GNU Programming 2 test suite: Standard Options: 1. ##
     ## -------------------------------------------------- ##
     1. std-opt1.at:8: testing Standard Options...
     std-opt1.at:9: m4 --version
     --- /dev/null   Sat Apr 14 10:11:43 2001
     +++ at-stdout   Tue Oct  2 21:33:14 2001
     @ -0,0 +1,6 @
     +GNU m4 1.4q
     +Written by Rene' Seindal and Gary V. Vaughan.
     +
     +Copyright 1989-1994, 1999, 2000 Free Software Foundation, Inc.
     +This is free software; see the source for copying conditions.  There is NO
     +warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
     1. std-opt1.at:8: FAILED near `std-opt1.at:9'
     ## ------------------------ ##
     ## std-opt1.log is created. ##
     ## ------------------------ ##
     
     Please send `std-opt1.log' to <gnuprog2-devel@sourceforge.org>,
     along with all information you think might help.
     

     Example 12.7: std-opt1 Run
     

Reading the verbose output or std-opt1.log is frightening at first, but with some practice you will soon it find rather easy since it is based on common tools such as diff. But let's first spot the failed test group and the guilty test:

     1. std-opt1.at:8: testing Standard Options...
     std-opt1.at:9: m4 --version
     --- /dev/null   Sat Apr 14 10:11:43 2001
     +++ at-stdout   Tue Oct  2 21:33:14 2001
     @ -0,0 +1,6 @
     +GNU m4 1.4q
     +Written by Rene' Seindal and Gary V. Vaughan.
     +
     +Copyright 1989-1994, 1999, 2000 Free Software Foundation, Inc.
     +This is free software; see the source for copying conditions.  There is NO
     +warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
     1. std-opt1.at:8: FAILED near `std-opt1.at:9'
     

Each test group is marked at its beginning and its end (1. std-opt1.at:8). Then each test is presented (std-opt1.at:9), and, if it failed, some information on the nature of the failure is reported. There are at most three aspects which are checked: the exit status, the standard output, and the standard error output. Here, the unified diff header reports the standard output is not what was expected:

     --- /dev/null   Sat Apr 14 10:11:43 2001
     +++ at-stdout   Tue Oct  2 21:33:14 2001
     @ -0,0 +1,6 @
     +GNU m4 1.4q
     +Written by Rene' Seindal and Gary V. Vaughan.
     +
     +Copyright 1989-1994, 1999, 2000 Free Software Foundation, Inc.
     +This is free software; see the source for copying conditions.  There is NO
     +warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
     

In this report, lines starting with a dash correspond to what was expected, and lines starting with a plus to what was observed. Here, what was observed is

     GNU m4 1.4q
     Written by Rene' Seindal and Gary V. Vaughan.
     
     Copyright 1989-1994, 1999, 2000 Free Software Foundation, Inc.
     This is free software; see the source for copying conditions.  There is NO
     warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
     

while the expected content was empty. Indeed our first test, AT_CHECK([m4 --version]), makes no provision for some output. It reads as "run m4 --version, expect a success (exit status should be 0), no standard output, nor standard error output".

Note that no error was reported for the failure of the second test, exercising --help. This is typical of test groups: it suffices that a single test fails for the rest of the test group to be ignored. The size of test groups is sometimes a matter of taste, but in general a single test group matches a single use scenario. Making long test groups is attractive (less to type etc.) and harmless, but testing independent features should always be done in distinct test groups.

In the present case, --version and --help support are definitely two different features: they will be part of different test group. But since they are somewhat related, we will simply keep them close, under the "Standard Options" banner for instance.

We have to fix our tests:

AT_CHECK (commands, [status = 0], [stdout], [stderr])

Macro

Execute a test by performing given shell commands. These commands should normally exit with status, while producing expected stdout and stderr contents.

But what shall we actually expect as standard output? Reproducing the exact result of m4 --version means your test suite will fail for any other version of GNU M4, similarly with --help. The easiest is to ignore the standard output, by passing ignore to AT_CHECK:

     AT_CHECK([m4 --version], [], [ignore])
     

This is what I would have done if I was not writing an Autotest tutorial... Here, we will make sure that something is output; and actually, to be even more specific, that both options' output contain m4. It would certainly be a bug if they didn't...

Autotest does not provide direct support for such content checking¹. Its model is strictly based on equality, therefore we have to find a means to transform our partial control into a strict equality. There are at least two means to achieve this goal, each having its pros and cons.

The first one that comes to one's mind simply consists in using grep, or better yet (more for religious issues than for technically sound reasons), fgrep:

     AT_CHECK([m4 --version | fgrep m4], [], [ignore])
     

This solution is definitely the simplest, and that's actually its only interest... Imagine the test fails some day: what information will its failure bring?

Almost nothing.

You will not know if m4 failed, since in portable Bourne shell programming there is no means to determine if an upstream command failed in a pipe: the exit status of the pipe is that of the last command². You will not know either if m4 --version output a big fat nothing, or nothing that satisfied fgrep, since the only output is that of fgrep. And last but not least, in the latter case, you won't know what was output.

The second solution circumvents these issues simply by decomposing the pipe: first we run m4 --version and save its output, and then we check it:

     AT_CHECK([m4 --version >m4-version])
     AT_CHECK([fgrep m4 m4-version], [], [ignore])
     

Alas, this innocent first test will sooner or later cause you a heart attack: I guarantee someone will report problems, most likely an Ultrix user... You just fell into one of the obscure bugs that affect some hosts: they do not support multiple file descriptor redirections. Autotest definitely needs to save the standard output to check its contents, therefore it does redirect both the standard output and the standard error output, hence you must not redirect them again. Equally unfortunate, we know of no way to warn the Autotest user she wrote such an non portable command, we can only ask her to keep it in mind (and anyway, if she doesn't, a fellow Ultrix user will remind her). Note too, that this is why you must use ignore, and not attempt to redirect the standard output (or error) to /dev/null.

To address this issue, Autotest may be asked to save the standard output in a file for later examination: use stdout to save it into the file stdout:

     AT_CHECK([m4 --version], [], [stdout])
     AT_CHECK([fgrep m4 stdout], [], [ignore])
     

The case of the --help is exactly similar, so much indeed that writing a macro would factor a lot of code.