Running an Autotest Test Suite

An Autotest test suite is often named testsuite, but we advertise for more explicit names, such as test-m4, test-joepackage etc. This way, it can be installed or sent to other people.

To test a package, such as GNU M4, just run its test suite:

     $ cd m4-1.5/tests
     $ ./testsuite
     ## ---------------------- ##
     ## GNU m4 1.5 test suite. ##
     ## ---------------------- ##
     
     Macro definitions.
     
       1: macros.at:29      ok
       2: macros.at:71      ok
       3: macros.at:105     ok
     ...
     Options.
     
      21: options.at:26     ok
      22: options.at:52     ok
     
     Composite macros.
     ...
     Documentation examples.
     
      43: generated.at:14   ok
     ...
      78: generated.at:1404 ok
      79: generated.at:1438 ok
     ## ----------------------------- ##
     ## All 79 tests were successful. ##
     ## ----------------------------- ##
     

All the test groups (here, numbered from 1 to 79) are run. If some failed, a log file would have been created, full of details that might help the GNU M4 maintainer to understand the failure. For instance in the following example, I have manually changed the test group 44 for it to fail:

     $ ./testsuite 44
     ## ---------------------- ##
     ## GNU m4 1.5 test suite. ##
     ## ---------------------- ##
      44: generated.at:48   FAILED near `generated.at:60'
     ## ----------------------------------------------- ##
     ## ERROR: Suite unsuccessful, 1 of 1 tests failed. ##
     ## ----------------------------------------------- ##
     You may investigate any problem if you feel able to do so, in which
     case the test suite provides a good starting point.
     
     Now, failed tests will be executed again, verbosely, and logged
     in the file testsuite.log.
     ## ---------------------- ##
     ## GNU m4 1.5 test suite. ##
     ## ---------------------- ##
     44. generated.at:48: testing Define...
     generated.at:60: m4 -b -d input.m4
     --- -   Sat Jan 12 17:31:38 2002
     +++ at-stdout     Sat Jan 12 17:31:38 2002
     @ -1,3 +1,3 @
     
     -Hello world.
     +Hello universe.
     
     44. generated.at:48: FAILED near `generated.at:60'
     ## ------------------------- ##
     ## testsuite.log is created. ##
     ## ------------------------- ##
     
     Please send `testsuite.log' to <bug-m4@gnu.org>,
     along with all information you think might help.
     

Autotest test suites support the following arguments:

--list

-l

List all the tests (or only the selection), including their possible keywords.

To change environment variables, set of tests, or verbosity of the test suite:

variable=value

Set the environment variable to value. The variable AUTOTEST_PATH specifies the testing path to prepend to PATH.

number

number-number

number-

-number

Add the corresponding test groups to the selection.

--keywords=keywords

-k keywords

Add to the selection the test groups which title or keywords (arguments to AT_SETUP or AT_KEYWORDS, see (FIXME: Test Groups.)) match all the keywords of the comma separated list keywords.

Running ./testsuite -k autoupdate,FUNC will select all the tests tagged with autoupdate and FUNC (as in AC_CHECK_FUNC, AC_FUNC_FNMATCH etc.) while ./testsuite -k autoupdate -k FUNC runs all the tests tagged with autoupdate or FUNC.

--errexit

-e

If any test fails, immediately abort testing. It implies --debug.

--verbose

-v

Force more verbosity in the detailed output of what is being done. This is the default for debugging scripts.

--debug

-d

Do not remove the files after a test group was performed --but they are still removed before, therefore using this option is sane when running several test groups. Do not create debugging scripts. Do not log (in order to preserve supposedly existing full log file). This is the default for debugging scripts.

--trace

-x

Trigger shell tracing of the test groups.