Noeud:The GNU M4 Test Suite, Noeud « Next »:Using Autotest with the GNU Build System, Noeud « Up »:Autotesting GNU M4
Based on the general advice presented in Ordering the Tests, the
first steps consisted in determining the sequence of kinds of tests to
be performed: (i) exercising macro and builtin definitions and uses,
(ii) builtins (builtins.at
, (iii) special options
(options.at
, (iv) complex hand crafted tests (others.at
),
(v) module support (modules.at
, and (vi) generated tests,
automatically extracted from the documentation (generated.m4
).
Because we repeatedly run m4
with some common options, we also
define two macros which will make our life easier: AT_CHECK_M4
which is a simple check, and AT_TEST_M4
, which is a whole test
group in itself. As advised in Look for Realism, for options such
as --warning
, we pass -d
, --debug
, to
constantly check the debugging output (which is actually produced only
when special macros such as traceon
are invoked). We also pass
the option -b
, --batch
, to make sure the test suite is
interruptible. This point deserves some slightly out off-topic detailed
explanations, typically a footnote, but that too lengthy to fit down
there...
Interactive programs, such as shells, often check whether their standard input is a TTY (basically meaning that the standard input is the user herself, not a file), and then neutralize CTRL-c, since it would result in exiting the shell. GNU M4 follows the same rule, and therefore, if you runm4
, typing CTRL-c will have no effect, while runningm4 <input.m4
keeps CTRL-c activated. But what happens when runningm4 input.m4
? The standard input is notinput.m4
, the latter is just an argument passed tom4
on its command line, but the standard input is not redirected, and therefore it remains the same as before (typically, users run./testsuite
from an interactive shell, hence the standard input oftestsuite
is a TTY, inherited bym4
, thereforem4
considers it is in interactive mode!).If for some reason
input.m4
makesm4
go into in infinite loop, then you are doomed, you will have to usekill
to terminate the process. It should be noted that other interactive programs are still sensible to CTRL-c: they stop the current operation and resume to the prompt. GNU M4 has no such feature, which makes it even worse. I strongly encourage using systematically--batch
.Finally, please note that sometimes, for some reason, a test might not behave as expected and may be expecting some input from the standard input. Then the test suite will appear to be stuck. If you experience this, if some user reports a never ending test group, suggest that they run
./testsuite </dev/null
. If this time the test group ends, ask her to run./testsuite -x
: the last command was the one expecting data from the standard input.
Example 12.23: Testing Interactive Programs
Putting all this together gives the following testsuite.at
. To
save trees, the license is not included below.
# Process with autom4te to create an -*- Autotest -*- test suite. # Test suite for GNU M4. # Copyright 2001 Free Software Foundation, Inc. # We need at least Autotest 2.52g, otherwise fail now. m4_version_prereq([2.52g]) # AT_CHECK_M4(ARGS, [EXIT-STATUS = 0], [STDOUT = `'], [STDERR = `']) # ------------------------------------------------------------------ m4_define([AT_CHECK_M4], [AT_CHECK([m4 -b -d $1], [$2], [$3], [$4]) ]) # AT_TEST_M4(TITLE, INPUT, [STDOUT = `'], [STDERR = `']) # ------------------------------------------------------ # Run m4 on INPUT, expecting a success. m4_define([AT_TEST_M4], [AT_SETUP([$1]) AT_DATA([[input.m4]], [$2]) AT_CHECK_M4([[input.m4]], 0, [$3], [$4]) AT_CLEANUP ]) # We use `dnl' in zillions of places... m4_pattern_allow([^dnl$]) # We exercise m4. AT_TESTED([m4]) ## ----------- ## ## The suite. ## ## ----------- ## AT_INIT # Macro definitions, uses, tracing etc. m4_include([macros.at]) # Torturing builtins. m4_include([builtins.at]) # Options. m4_include([options.at]) # Hand crafted tests. m4_include([others.at]) # Torturing the modules support. m4_include([modules.at]) # From the documention. m4_include([generated.at])
Example 12.24: GNU M4's testsuite.at
Most tests are straightforward and do not deserve special attention; to
see AT_CHECK_M4
and AT_TEST_M4
in action, see the GNU
M4 distribution. We will focus on excerpts of modules.at
and
generated.at
.
Originally, when the test suite was only a set of handwritten shell
scripts, a few of them were testing the loading and unloading of
modules, sometimes testing relative path to modules, sometimes absolute
paths, some were exercising the option --module-directory
,
others the environment variable M4MODPATH
, and others
LTDL_LIBRARY_PATH
. Looking at this set of shell scripts it was
barely possible to verify their coverage: were there cases which were
not tested? In addition, did all the tests have the same strength (the
inputs were sometimes different)? As advised in Maintain the Test Suite, these specific tests were generalized into a unique test group
macro, and it then became easy to be sure all the possibilities were
covered. Since in addition these tests depend on modtest
, a
module written specially for exercising the modules, and therefore which
is not to be installed, the macro is equipped with a preliminary test to
skip the test group when it missing. Please note that since these tests
aim at checking that modtest
can be found, using
AT_CHECK([m4 -m modtest.la || exit 77])
is taking the risk that
actual failures be considered as skipped tests.
## ---------------------------- ## ## Exercising the test module. ## ## ---------------------------- ## # AT_TEST_M4_MODTEST(TITLE, ENV-VARS, M4-OPTIONS) # ----------------------------------------------- # Skip if modtest is not present (we are not in the package). m4_define([AT_TEST_M4_MODTEST], [AT_SETUP([$1]) AT_KEYWORDS([module]) AT_CHECK([test -f $top_builddir/modules/modtest.la || exit 77]) AT_DATA([input.m4], [[load(`modtest') test Dumpdef: dumpdef(`test'). unload(`modtest') test Dumpdef: dumpdef(`test'). ]]) AT_CHECK([$2 m4 -m load -d input.m4 $3], 0, [[ Test module called. Dumpdef: . test Dumpdef: . ]], [[Test module loaded. test: <test> Test module unloaded. m4: input.m4: 6: Warning: dumpdef: undefined name: test ]]) AT_CLEANUP ]) AT_TEST_M4_MODTEST([--module-directory: absolute path], [], [-M $top_buildpath/modules]) AT_TEST_M4_MODTEST([--module-directory: relative path], [], [-M $top_builddir/modules]) AT_TEST_M4_MODTEST([M4MODPATH: absolute path], [M4MODPATH=$top_buildpath/modules], []) AT_TEST_M4_MODTEST([M4MODPATH: relative path], [M4MODPATH=$top_builddir/modules], []) AT_TEST_M4_MODTEST([LTDL_LIBRARY_PATH: absolute path], [LTDL_LIBRARY_PATH=$top_buildpath/modules], []) AT_TEST_M4_MODTEST([LTDL_LIBRARY_PATH: relative path], [LTDL_LIBRARY_PATH=$top_builddir/modules], [])
The last bit of testing we will pay attention to is the case of the
tests extracted from the documentation. The file generated.at
is
produced by a simple Awk program, generate.awk
, which we won't
detail here, see the GNU M4 distribution. The idea is simple:
convert the example from the Texinfo documentation into actual tests.
For instance, the following excerpt of the node "Dumpdef" of
m4.texinfo
(voir Displaying macro definitions):
@example define(`foo', `Hello world.') @result{} dumpdef(`foo') @error{}foo: `Hello world.' @result{} dumpdef(`define') @error{}define: <define> @result{} @end example
rendered as
define(`foo', `Hello world.') => dumpdef(`foo') error-->foo: `Hello world.' => dumpdef(`define') error-->define: <define> =>
is turned into:
## --------- ## ## Dumpdef. ## ## --------- ## AT_SETUP([[Dumpdef]]) AT_KEYWORDS([[documentation]]) # ../doc/m4.texinfo:1673 AT_DATA([[input.m4]], [[define(`foo', `Hello world.') dumpdef(`foo') dumpdef(`define') ]]) AT_CHECK_M4([[input.m4]], 0, [[ ]], [[foo: `Hello world.' define: <define> ]]) AT_CLEANUP