Checking Module Support

A major aspect of GNU M4 is its wonderful handling of modules. As a matter of fact, the executable m4 is nothing but an empty shell which sole ability is almost reduced to handling --help and --version (which we already exhaustively tortured). Such a major feature must be exercised, and in fact, any conscientious maintainer will take a sadist pleasure at writing the most perverted possible tests. Gary is one such person, and we will follow his tracks, checking that modules can be loaded and unloaded.

Our victim will be the gnu module, which contains both builtins, such as builtin, and macros, such as __gnu__. We will check that they are undefined at start up when --traditional is specified, defined when gnu is loaded, and undefined again when the module is unloaded, and defined when loaded again:

     # Process with autom4te to create an -*- Autotest -*- test suite.
     # modules.at -- Testing GNU M4 module support.
     
     AT_INIT([Modules support])
     
     AT_SETUP([Modules loading and unloading])
     
     AT_DATA([[input.m4]],
     [[define(`status',
     ``$1': ifdef(`$1', `defined', `non defined')')
     status(`builtin'), status(`__gnu__')
     load(`gnu')status(`builtin'), status(`__gnu__')
     unload(`gnu')status(`builtin'), status(`__gnu__')
     load(`gnu')status(`builtin'), status(`__gnu__')
     ]])
     
     AT_CHECK([[m4 --traditional --load-module=load input.m4]], [],
     [[
     builtin: non defined, __gnu__: non defined
     builtin: defined, __gnu__: defined
     builtin: non defined, __gnu__: non defined
     builtin: defined, __gnu__: defined
     ]])
     
     AT_CLEANUP
     

     Example 12.16: modules.at -- An Autotest Source Checking M4
     Modules Support
     

which indeed runs as expected: 100% of one test passes. Note however that this test is actually quite weak, with some more effort it would have been better to check that the functionalities of builtin and __gnu__ are still working.

Now, again, we can run the test suite on our working copy of m4, by a simple ./modules AUTOTEST_PATH=$HOME/src/m4/src. But observe that if you replace m4 with strace m4 or something equivalent, and ask for the standard error to be ignore'd, then you get something similar to:

     $ ./modules -v AUTOTEST_PATH=$HOME/src/m4/src | grep gnu
     open("/home/akim/src/m4/modules/.libs/gnu.so.0", O_RDONLY) = 3
     open("/usr/local/libexec/m4/gnu.la", O_RDONLY) = 4
     read(4, "# gnu.la - a libtool library fil"..., 4096) = 708
     open("/usr/local/libexec/m4/gnu.la", O_RDONLY) = 4
     read(4, "# gnu.la - a libtool library fil"..., 4096) = 708
     

Although /home/akim/src/m4/modules/.libs/gnu.so.0 is reassuring, since it demonstrates that Libtool took care of loading the non installed version of the gnu module, the /usr/local/libexec/m4/gnu.la part is still a bit frightening: what if, after all, we were mixing installed modules with a non installed m4? As a matter of fact, this problem is extremely frequent since today many executables use auxiliary files. For instance the Autoconf collection heavily depends on a configuration file named autom4te.cfg and on many M4 files, Bison and Flex need to find "skeleton" files¹, Automake needs scripts like install-sh and missing, Makefile components *.am etc. When testing the working copy of these tools the risk of mixing installed and non installed bits is high, and will of course result in insignificant results, whether the test suite passes or not.

The most common answer is having the test suite pass some combinations of options and environment variables to make sure the tools load non installed files. In the current case, it means replacing the previous AT_CHECK invocation with something like

     AT_CHECK([[m4 -G -M $HOME/src/m4/modules -m load input.m4]], ...
     

Now we have the converse problem: the test suite will always involve non installed modules, even when exercising an installed m4.

The easy answer to this dilemma is: "forget about testing an installed program, after all it should have been tested before being installed", in other words "forget about testing any other copy of the program than the one in the same build tree as this test suite".

It is worth mentioning the case of programs invoking other programs in the same package. Autoconf is a typical example: autom4te is slaved by autoconf, autoheader, autoscan and autoupdate, all of them being run by autoreconf! Yet this is an improvement over the previous situation where, for instance, autoheader ran autoconf, itself using autom4te. In order to enforce this relationship, all of them had hard coded heuristics like this:

     # Default AUTOCONF to the name under which `autoconf' is installed
     # when `./configure --program-transform-name' and similar is used.
     : ${AUTOCONF=@autoconf-name@}
     dir=`echo "$0" | sed -e 's,[^/]*$,,'`
     # We test "$dir/autoconf" in case we are in the build tree,
     # in which case the names are not transformed yet.
     for autoconf in "$AUTOCONF" \
                     "$dir/@autoconf-name@" \
                     "$dir/autoconf" \
                     "@bindir@/@autoconf-name@"; do
       test -f "$autoconf" && break
     done
     

     Example 12.17: Excerpt of autoheader Looking for autoconf
     

Let's list a few consequences:

• Because directories are hard coded, if you move something, the behavior is undefined.
• The innocent test -f "$autoconf" is hell! If the user specified some option --foo in AUTOCONF, then this snippet will look for the file autoconf --foo.
• If the user specified AUTOCONF=autoconf-2.13, then again it won't be honored since instead of letting the system look for it in the PATH, this snippet just checks if autoconf-2.13 is present in the current directory.
• User specifications are silently ignored! In neither of the previous cases the user will be warned of what happened.

The explosion of the European Launcher, see Ariane 501, caused by a $500 000 000 bug, was caused by a comparable kind of problem: the real application was polluted with code unrelated to a normal use. Today, the Autoconf programs simply include:

     : ${AUTOCONF=@autoconf-name@}
     

and that's all! Let PATH handle the rest: when being tested in the build tree, the wrappers are run and handle all the dark magic. Now the behavior, relying on standard Unix interface, is predictable, and will properly fail when it should.

It is a mistake to dedicate a test suite to the special layout of a package in the process of being built. We strongly discourage you from going to the dark side of the testing force.

So, we are back to our problem: how can we test both an installed program, and an non installed program while having each copy use its own files? Keep in mind that a test suite must be seen as a user, albeit eccentric and demanding: present the same interface in both cases to the test suite. Then the answer is obvious: provide a wrapper, a small shell script which takes care of running a non installed program, and let the PATH handle the rest.

This wrapper must give a perfect illusion, it must pass the Turing test: an observer shouldn't be able to tell the difference. Note that we also just solved the problem left in the previous section --lt-m4 vs. m4 in error messages--: this wrapper must hide this detail. Since this wrapper depends upon configuration options, it is configure which will instantiate from a template.

In the case of m4, this template, m4.in, is just:

     #! /bin/sh
     # @configure_input@
     # Wrapper around a non installed m4 to make it work as an installed one.
     
     "@top_buildpath@/src/m4" \
           --module-directory="@top_buildpath@/modules" \
           ${1+"$@"} 2>/tmp/m4-$$
     status=$?
     # Normalize stderr.
     sed 's,^[^:]*[lt-]*m4[.ex]*:,m4:,' /tmp/m4-$$ >&2
     rm /tmp/m4-$$
     
     exit $status
     

     Example 12.18: m4.in -- A Wrapper around a non installed m4
     

How does this script work? We pass the option --module-directory so that it uses the non installed modules instead of those possibly installed on the system; and we used sed to normalize the name of the executable in the error messages. This sed invocation deserves some explanations:

[^:]*

aims at removing the possible leading path. In particular, when configured with --disable-shared, @top_buildpath@/src/m4 is an genuine binary, which will display its full path.

[lt-]*

is a portable approximation of \(lt-\)\?, matching Libtool's prefix when not configured with --disable-shared. In this case, there is no real need to normalize a possible directory specification because the bin/m4 wrapper modifies the PATH to run the actual executable, in which case the name is indeed simply m4.

[.ex]*

takes care of the possible .exe extension on some poor hosts.

Approximating even further, for instance with s/^[^:]*:/m4:/, will sooner or later destroy other standard error output than m4's signature, the output of dumpdef for instance. One could also rely on a redefinition of PATH in which case the normalization can be simplified.

Note that these wrappers are also a good place where special magical tricks can be performed. For instance, as described in Other Uses of a Test Suite, the test suite can be a good place for profiling. Configure the package using ./configure CFLAGS='-pg' (and --disable-shared if, as is the case of GNU M4, the application is heavily composed of libraries), and add a few lines to save the profiling data file, gmon.out, for a later use (voir Profiling and Optimising Your Code):

     #! /bin/sh
     # @configure_input@
     # Wrapper around a non installed m4 to make it work as an installed one.
     
     "@top_buildpath@/src/m4" \
           --module-directory="@top_buildpath@/modules" \
           ${1+"$@"} 2>/tmp/m4-$$
     status=$?
     test -d gmon || mkdir gmon
     mv gmon.out gmon/$$
     # Normalize stderr.
     sed 's,^[^:]*[lt-]*m4[.ex]*:,m4:,' /tmp/m4-$$ >&2
     rm /tmp/m4-$$
     
     exit $status
     

     Example 12.19: m4.in -- A Profiling Wrapper
     

then run the test suite, then gprof -s ../src/m4 gmon/* && rm -rf gmon, and finally gprof ../src/m4 gmon.sum. Enjoy!

Well, there is not much to enjoy, because the GNU M4 test suite is really paying attention to testing independent features, and includes almost no torture tests. But applying the same trick on an M4 based package, such as Autoconf (i.e., installing the wrapper above as tests/m4 in Autoconf's tests directory) provides a excellent base for profile-guided improvements.