Noeud:The GNU M4 Test Suite, Noeud « Next »:, Noeud « Up »:Autotesting GNU M4



The GNU M4 Test Suite

Based on the general advice presented in Ordering the Tests, the first steps consisted in determining the sequence of kinds of tests to be performed: (i) exercising macro and builtin definitions and uses, (ii) builtins (builtins.at, (iii) special options (options.at, (iv) complex hand crafted tests (others.at), (v) module support (modules.at, and (vi) generated tests, automatically extracted from the documentation (generated.m4).

Because we repeatedly run m4 with some common options, we also define two macros which will make our life easier: AT_CHECK_M4 which is a simple check, and AT_TEST_M4, which is a whole test group in itself. As advised in Look for Realism, for options such as --warning, we pass -d, --debug, to constantly check the debugging output (which is actually produced only when special macros such as traceon are invoked). We also pass the option -b, --batch, to make sure the test suite is interruptible. This point deserves some slightly out off-topic detailed explanations, typically a footnote, but that too lengthy to fit down there...

Interactive programs, such as shells, often check whether their standard input is a TTY (basically meaning that the standard input is the user herself, not a file), and then neutralize CTRL-c, since it would result in exiting the shell. GNU M4 follows the same rule, and therefore, if you run m4, typing CTRL-c will have no effect, while running m4 <input.m4 keeps CTRL-c activated. But what happens when running m4 input.m4? The standard input is not input.m4, the latter is just an argument passed to m4 on its command line, but the standard input is not redirected, and therefore it remains the same as before (typically, users run ./testsuite from an interactive shell, hence the standard input of testsuite is a TTY, inherited by m4, therefore m4 considers it is in interactive mode!).

If for some reason input.m4 makes m4 go into in infinite loop, then you are doomed, you will have to use kill to terminate the process. It should be noted that other interactive programs are still sensible to CTRL-c: they stop the current operation and resume to the prompt. GNU M4 has no such feature, which makes it even worse. I strongly encourage using systematically --batch.

Finally, please note that sometimes, for some reason, a test might not behave as expected and may be expecting some input from the standard input. Then the test suite will appear to be stuck. If you experience this, if some user reports a never ending test group, suggest that they run ./testsuite </dev/null. If this time the test group ends, ask her to run ./testsuite -x: the last command was the one expecting data from the standard input.

     Example 12.23: Testing Interactive Programs
     

Putting all this together gives the following testsuite.at. To save trees, the license is not included below.

     # Process with autom4te to create an -*- Autotest -*- test suite.
     
     # Test suite for GNU M4.
     # Copyright 2001 Free Software Foundation, Inc.
     
     # We need at least Autotest 2.52g, otherwise fail now.
     m4_version_prereq([2.52g])
     
     # AT_CHECK_M4(ARGS, [EXIT-STATUS = 0], [STDOUT = `'], [STDERR = `'])
     # ------------------------------------------------------------------
     m4_define([AT_CHECK_M4],
     [AT_CHECK([m4 -b -d $1], [$2], [$3], [$4])
     ])
     
     # AT_TEST_M4(TITLE, INPUT, [STDOUT = `'], [STDERR = `'])
     # ------------------------------------------------------
     # Run m4 on INPUT, expecting a success.
     m4_define([AT_TEST_M4],
     [AT_SETUP([$1])
     AT_DATA([[input.m4]], [$2])
     AT_CHECK_M4([[input.m4]], 0, [$3], [$4])
     AT_CLEANUP
     ])
     
     # We use `dnl' in zillions of places...
     m4_pattern_allow([^dnl$])
     
     # We exercise m4.
     AT_TESTED([m4])
     
     ## ----------- ##
     ## The suite.  ##
     ## ----------- ##
     
     AT_INIT
     
     # Macro definitions, uses, tracing etc.
     m4_include([macros.at])
     
     # Torturing builtins.
     m4_include([builtins.at])
     
     # Options.
     m4_include([options.at])
     
     # Hand crafted tests.
     m4_include([others.at])
     
     # Torturing the modules support.
     m4_include([modules.at])
     
     # From the documention.
     m4_include([generated.at])
     
     Example 12.24: GNU M4's testsuite.at
     

Most tests are straightforward and do not deserve special attention; to see AT_CHECK_M4 and AT_TEST_M4 in action, see the GNU M4 distribution. We will focus on excerpts of modules.at and generated.at.

Originally, when the test suite was only a set of handwritten shell scripts, a few of them were testing the loading and unloading of modules, sometimes testing relative path to modules, sometimes absolute paths, some were exercising the option --module-directory, others the environment variable M4MODPATH, and others LTDL_LIBRARY_PATH. Looking at this set of shell scripts it was barely possible to verify their coverage: were there cases which were not tested? In addition, did all the tests have the same strength (the inputs were sometimes different)? As advised in Maintain the Test Suite, these specific tests were generalized into a unique test group macro, and it then became easy to be sure all the possibilities were covered. Since in addition these tests depend on modtest, a module written specially for exercising the modules, and therefore which is not to be installed, the macro is equipped with a preliminary test to skip the test group when it missing. Please note that since these tests aim at checking that modtest can be found, using AT_CHECK([m4 -m modtest.la || exit 77]) is taking the risk that actual failures be considered as skipped tests.

     ## ---------------------------- ##
     ## Exercising the test module.  ##
     ## ---------------------------- ##
     
     # AT_TEST_M4_MODTEST(TITLE, ENV-VARS, M4-OPTIONS)
     # -----------------------------------------------
     # Skip if modtest is not present (we are not in the package).
     m4_define([AT_TEST_M4_MODTEST],
     [AT_SETUP([$1])
     AT_KEYWORDS([module])
     
     AT_CHECK([test -f $top_builddir/modules/modtest.la || exit 77])
     
     AT_DATA([input.m4],
     [[load(`modtest')
     test
     Dumpdef: dumpdef(`test').
     unload(`modtest')
     test
     Dumpdef: dumpdef(`test').
     ]])
     
     AT_CHECK([$2 m4 -m load -d input.m4 $3], 0,
     [[
     Test module called.
     Dumpdef: .
     
     test
     Dumpdef: .
     ]],
     [[Test module loaded.
     test:   <test>
     Test module unloaded.
     m4: input.m4: 6: Warning: dumpdef: undefined name: test
     ]])
     
     AT_CLEANUP
     ])
     
     AT_TEST_M4_MODTEST([--module-directory: absolute path],
                        [], [-M $top_buildpath/modules])
     
     AT_TEST_M4_MODTEST([--module-directory: relative path],
                        [], [-M $top_builddir/modules])
     
     AT_TEST_M4_MODTEST([M4MODPATH: absolute path],
                        [M4MODPATH=$top_buildpath/modules], [])
     
     AT_TEST_M4_MODTEST([M4MODPATH: relative path],
                        [M4MODPATH=$top_builddir/modules], [])
     
     AT_TEST_M4_MODTEST([LTDL_LIBRARY_PATH: absolute path],
                        [LTDL_LIBRARY_PATH=$top_buildpath/modules], [])
     
     AT_TEST_M4_MODTEST([LTDL_LIBRARY_PATH: relative path],
                        [LTDL_LIBRARY_PATH=$top_builddir/modules], [])
     

The last bit of testing we will pay attention to is the case of the tests extracted from the documentation. The file generated.at is produced by a simple Awk program, generate.awk, which we won't detail here, see the GNU M4 distribution. The idea is simple: convert the example from the Texinfo documentation into actual tests. For instance, the following excerpt of the node "Dumpdef" of m4.texinfo (voir Displaying macro definitions):

     @example
     define(`foo', `Hello world.')
     @result{}
     dumpdef(`foo')
     @error{}foo:	`Hello world.'
     @result{}
     dumpdef(`define')
     @error{}define:	<define>
     @result{}
     @end example
     

rendered as

     define(`foo', `Hello world.')
     =>
     dumpdef(`foo')
     error-->foo:	`Hello world.'
     =>
     dumpdef(`define')
     error-->define:	<define>
     =>
     

is turned into:

     ## --------- ##
     ## Dumpdef.  ##
     ## --------- ##
     
     AT_SETUP([[Dumpdef]])
     AT_KEYWORDS([[documentation]])
     
     # ../doc/m4.texinfo:1673
     AT_DATA([[input.m4]],
     [[define(`foo', `Hello world.')
     dumpdef(`foo')
     dumpdef(`define')
     ]])
     
     AT_CHECK_M4([[input.m4]], 0,
     [[
     
     
     ]],
     [[foo:	`Hello world.'
     define:	<define>
     ]])
     
     AT_CLEANUP