Testing Optional Features

Some packages provide optional features, possibly depending upon configuration options. Therefore a test suite will exercise programs with different behaviors at runtime, or put more simply, it will test different programs sharing the same name. Quite by a mere chance, GNU M4 is one such program: depending upon the system and/or the user's will, support for extended precision arithmetics based on GMP¹ might be compiled.

Therefore we need optional tests. The special exit status value 77 tells AT_CHECK to disregard the rest of the test group, independent of the results (unless the test expects 77 as exit value, see (FIXME: Inside Test Groups.), AT_CHECK, for more details).

Autotest based test suites provide the user with a means to pass configuration options to the tests: atlocal. The file atlocal is automatically generated from its template, atlocal.in, by AC_CONFIG_TESTDIR (Autotest Test Suite (FIXME: Embedding an Autotest Test Suite.)), and is loaded (when present) by any Autotest test suite. In our case, we merely need to know if GMP support was compiled in, which configure knows via the value of the variable USE_GMP: either yes or no. Therefore, our template atlocal.in is:

     # -*- shell-script -*-
     # @configure_input@
     # Configurable variable values for M4 test suite.
     # Copyright 2000, 2001 Free Software Foundation, Inc.
     
     # Some tests cannot be performed with all the configurations.
     USE_GMP=@USE_GMP@
     

     Example 12.20: tests/atlocal.in -- User Test Variables
     

There remains to write the test itself, exercising the mpeval module:

     # Process with autom4te to create an -*- Autotest -*- test suite.
     # mpeval1.at -- Testing GNU M4 GMP support.
     
     AT_INIT([GMP support])
     
     AT_SETUP([GMP: Exponentiation])
     
     AT_CHECK([test "$USE_GMP" = yes || exit 77])
     
     AT_DATA([[input.m4]],
     [[mpeval(`2**100')
     ]])
     
     AT_CHECK([[m4 -m mpeval input.m4]], 0,
     [[1267650600228229401496703205376
     ]])
     
     AT_CLEANUP
     

     Example 12.21: mpeval1.at -- Exercising an Optional Feature Using
     atlocal
     

create mpeval1, and run it:

     ## ------------------------------------------------ ##
     ## GNU Programming 2E 0.0a test suite: GMP support. ##
     ## ------------------------------------------------ ##
       1: mpeval.at:6       ok (skipped near `mpeval.at:8')
     ## ---------------------------------------- ##
     ## All 1 tests were successful (1 skipped). ##
     ## ---------------------------------------- ##
     

Hm... Something went wrong.... I'm sure the m4 I just installed has GMP support...

If you spend some time analyzing the failure, you'll find a simple explanation: the example is presented as it if were part of GNU M4 distribution, while in fact it's part of a different package, gnu-prog2. The latter has no information with respect to the configuration options used for the GNU M4 which was installed!

The problem we face is well known to Autoconf gurus: differences between configure time data (not very different from compile time data), and execution time data. Some softwares make decision with regards to their behavior at runtime, e.g., some hosts can be big endian or little endian depending on runtime environment. In such a case, you should just follow the program's footsteps and adjust your tests to runtime conditions. As a matter of fact, still following the motto "the test suite is a user" (voir Look for Realism), you should depend as little as possible on configuration options: a user has no reason to know the decision made by her system administrator. In the case of GMP, it means first asking m4 whether it knows a module named mpeval, and then checking it:

     # Process with autom4te to create an -*- Autotest -*- test suite.
     # mpeval2.at -- Testing GNU M4 GMP support.
     
     AT_INIT([GMP support: Runtime Check])
     
     AT_SETUP([GMP: Exponentiation])
     
     AT_CHECK([m4 -m mpeval </dev/null || exit 77])
     
     AT_DATA([[input.m4]],
     [[mpeval(`2**100')
     ]])
     
     AT_CHECK([[m4 -m mpeval input.m4]], 0,
     [[1267650600228229401496703205376
     ]])
     
     AT_CLEANUP
     

     Example 12.22: mpeval2.at -- Exercising an Optional Feature at
     Runtime
     

This time, the mpeval module is properly exercised:

     ## --------------------------------------------------------------- ##
     ## GNU Programming 2E 0.0a test suite: GMP support: Runtime check. ##
     ## --------------------------------------------------------------- ##
       1: mpeval.at:6
     ## ---------------------------- ##
     ## All 1 tests were successful. ##
     ## ---------------------------- ##
     

Creating stand-alone test suites is still rare, and hackers writing test suites for other people's packages are even less common²: the most common use of Autotest is writing a portable test suite shipped with the tested package, within the GNU Build System, together with Automake and Autoconf. It turns out miraculously to be the topic of the next section, Autotesting GNU M4.