Noeud:Checking Module Support, Noeud « Next »:Testing Optional Features, Noeud « Previous »:Checking dnl and define, Noeud « Up »:Stand-alone Test Suite
A major aspect of GNU M4 is its wonderful handling of modules. As a
matter of fact, the executable m4
is nothing but an empty
shell which sole ability is almost reduced to handling --help
and --version
(which we already exhaustively tortured). Such a
major feature must be exercised, and in fact, any conscientious
maintainer will take a sadist pleasure at writing the most perverted
possible tests. Gary is one such person, and we will follow his tracks,
checking that modules can be loaded and unloaded.
Our victim will be the gnu
module, which contains both builtins,
such as builtin
, and macros, such as __gnu__
. We will
check that they are undefined at start up when --traditional
is
specified, defined when gnu
is loaded, and undefined again when
the module is unloaded, and defined when loaded again:
# Process with autom4te to create an -*- Autotest -*- test suite. # modules.at -- Testing GNU M4 module support. AT_INIT([Modules support]) AT_SETUP([Modules loading and unloading]) AT_DATA([[input.m4]], [[define(`status', ``$1': ifdef(`$1', `defined', `non defined')') status(`builtin'), status(`__gnu__') load(`gnu')status(`builtin'), status(`__gnu__') unload(`gnu')status(`builtin'), status(`__gnu__') load(`gnu')status(`builtin'), status(`__gnu__') ]]) AT_CHECK([[m4 --traditional --load-module=load input.m4]], [], [[ builtin: non defined, __gnu__: non defined builtin: defined, __gnu__: defined builtin: non defined, __gnu__: non defined builtin: defined, __gnu__: defined ]]) AT_CLEANUP
Example 12.16: modules.at
-- An Autotest Source Checking M4
Modules Support
Modules Support
which indeed runs as expected: 100% of one test passes. Note however
that this test is actually quite weak, with some more effort it would
have been better to check that the functionalities of builtin
and
__gnu__
are still working.
Now, again, we can run the test suite on our working copy of
m4
, by a simple ./modules
AUTOTEST_PATH=$HOME/src/m4/src
. But observe that if you replace
m4
with strace m4
or something equivalent, and ask for the
standard error to be ignore
'd, then you get something similar to:
$ ./modules -v AUTOTEST_PATH=$HOME/src/m4/src | grep gnu open("/home/akim/src/m4/modules/.libs/gnu.so.0", O_RDONLY) = 3 open("/usr/local/libexec/m4/gnu.la", O_RDONLY) = 4 read(4, "# gnu.la - a libtool library fil"..., 4096) = 708 open("/usr/local/libexec/m4/gnu.la", O_RDONLY) = 4 read(4, "# gnu.la - a libtool library fil"..., 4096) = 708
Although /home/akim/src/m4/modules/.libs/gnu.so.0
is reassuring,
since it demonstrates that Libtool took care of loading the non
installed version of the gnu
module, the
/usr/local/libexec/m4/gnu.la
part is still a bit frightening:
what if, after all, we were mixing installed modules with a non
installed m4
? As a matter of fact, this problem is extremely
frequent since today many executables use auxiliary files. For instance
the Autoconf collection heavily depends on a configuration file named
autom4te.cfg
and on many M4 files, Bison and Flex need to find
"skeleton" files1, Automake needs scripts like install-sh
and missing
,
Makefile components *.am
etc. When testing the working copy of
these tools the risk of mixing installed and non installed bits is high,
and will of course result in insignificant results, whether the test
suite passes or not.
The most common answer is having the test suite pass some combinations
of options and environment variables to make sure the tools load non
installed files. In the current case, it means replacing the previous
AT_CHECK
invocation with something like
AT_CHECK([[m4 -G -M $HOME/src/m4/modules -m load input.m4]], ...
Now we have the converse problem: the test suite will always involve non
installed modules, even when exercising an installed m4
.
The easy answer to this dilemma is: "forget about testing an installed program, after all it should have been tested before being installed", in other words "forget about testing any other copy of the program than the one in the same build tree as this test suite".
It is worth mentioning the case of programs invoking other programs in
the same package. Autoconf is a typical example: autom4te
is
slaved by autoconf
, autoheader
, autoscan
and autoupdate
, all of them being run by autoreconf
!
Yet this is an improvement over the previous situation where, for
instance, autoheader
ran autoconf
, itself using
autom4te
. In order to enforce this relationship, all of them
had hard coded heuristics like this:
# Default AUTOCONF to the name under which `autoconf' is installed # when `./configure --program-transform-name' and similar is used. : ${AUTOCONF=@autoconf-name@} dir=`echo "$0" | sed -e 's,[^/]*$,,'` # We test "$dir/autoconf" in case we are in the build tree, # in which case the names are not transformed yet. for autoconf in "$AUTOCONF" \ "$dir/@autoconf-name@" \ "$dir/autoconf" \ "@bindir@/@autoconf-name@"; do test -f "$autoconf" && break done
Example 12.17: Excerpt ofautoheader
Looking forautoconf
Let's list a few consequences:
test -f "$autoconf"
is hell! If the user specified
some option --foo
in AUTOCONF
, then this snippet will
look for the file autoconf --foo
.
AUTOCONF=autoconf-2.13
, then again it won't
be honored since instead of letting the system look for it in the
PATH
, this snippet just checks if autoconf-2.13
is
present in the current directory.
The explosion of the European Launcher, see Ariane 501, caused by a $500 000 000 bug, was caused by a comparable kind of problem: the real application was polluted with code unrelated to a normal use. Today, the Autoconf programs simply include:
: ${AUTOCONF=@autoconf-name@}
and that's all! Let PATH
handle the rest: when being
tested in the build tree, the wrappers are run and handle all the dark
magic. Now the behavior, relying on standard Unix interface, is
predictable, and will properly fail when it should.
It is a mistake to dedicate a test suite to the special layout of a package in the process of being built. We strongly discourage you from going to the dark side of the testing force.
So, we are back to our problem: how can we test both an installed
program, and an non installed program while having each copy use its own
files? Keep in mind that a test suite must be seen as a user,
albeit eccentric and demanding: present the same interface in both cases
to the test suite. Then the answer is obvious: provide a wrapper, a
small shell script which takes care of running a non installed program,
and let the PATH
handle the rest.
This wrapper must give a perfect illusion, it must pass the Turing test:
an observer shouldn't be able to tell the difference. Note that we also
just solved the problem left in the previous section --lt-m4
vs. m4
in error messages--: this wrapper must hide this detail.
Since this wrapper depends upon configuration options, it is
configure
which will instantiate from a template.
In the case of m4
, this template, m4.in
, is just:
#! /bin/sh # @configure_input@ # Wrapper around a non installed m4 to make it work as an installed one. "@top_buildpath@/src/m4" \ --module-directory="@top_buildpath@/modules" \ ${1+"$@"} 2>/tmp/m4-$$ status=$? # Normalize stderr. sed 's,^[^:]*[lt-]*m4[.ex]*:,m4:,' /tmp/m4-$$ >&2 rm /tmp/m4-$$ exit $status
Example 12.18:m4.in
-- A Wrapper around a non installedm4
How does this script work? We pass the option
--module-directory
so that it uses the non installed modules
instead of those possibly installed on the system; and we used
sed
to normalize the name of the executable in the error
messages. This sed invocation deserves some explanations:
[^:]*
--disable-shared
,
@top_buildpath@/src/m4
is an genuine binary, which will display
its full path.
[lt-]*
\(lt-\)\?
, matching Libtool's
prefix when not configured with --disable-shared
. In this
case, there is no real need to normalize a possible directory
specification because the bin/m4
wrapper modifies the PATH
to run the actual executable, in which case the name is indeed simply
m4
.
[.ex]*
.exe
extension on some poor hosts.
Approximating even further, for instance with s/^[^:]*:/m4:/
,
will sooner or later destroy other standard error output than
m4
's signature, the output of dumpdef
for instance.
One could also rely on a redefinition of PATH
in which case the
normalization can be simplified.
Note that these wrappers are also a good place where special magical
tricks can be performed. For instance, as described in Other Uses of a Test Suite, the test suite can be a good place for profiling.
Configure the package using ./configure CFLAGS='-pg'
(and
--disable-shared
if, as is the case of GNU M4, the
application is heavily composed of libraries), and add a few lines to
save the profiling data file, gmon.out
, for a later use
(voir Profiling and Optimising Your Code):
#! /bin/sh # @configure_input@ # Wrapper around a non installed m4 to make it work as an installed one. "@top_buildpath@/src/m4" \ --module-directory="@top_buildpath@/modules" \ ${1+"$@"} 2>/tmp/m4-$$ status=$? test -d gmon || mkdir gmon mv gmon.out gmon/$$ # Normalize stderr. sed 's,^[^:]*[lt-]*m4[.ex]*:,m4:,' /tmp/m4-$$ >&2 rm /tmp/m4-$$ exit $status
Example 12.19: m4.in
-- A Profiling Wrapper
then run the test suite, then gprof -s ../src/m4 gmon/* && rm -rf
gmon
, and finally gprof ../src/m4 gmon.sum
. Enjoy!
Well, there is not much to enjoy, because the GNU M4 test suite is
really paying attention to testing independent features, and includes
almost no torture tests. But applying the same trick on an M4 based
package, such as Autoconf (i.e., installing the wrapper above as
tests/m4
in Autoconf's tests
directory) provides a
excellent base for profile-guided improvements.
Bison and Flex both generate tables describing the grammar specificities, but the engine, the code which ``executes'' these tables is independent of the grammar. This code, instead of being hard coded in the executables, is stored in a file named the skeleton. Once Bison and Flex documented, ref to there. (FIXME: Didier says this footnote should be removed. Once Bison and Flex documented, ref to there..)