Noeud:Writing Autotest Macros, Noeud « Next »:, Noeud « Previous »:Simple Uses of Autotest, Noeud « Up »:Stand-alone Test Suite



Writing Autotest Macros

Autotest provides a minimal set of macros, flexible enough to meet the needs, but too limited to perform strict tests on some executables. This is on purpose: there is no universal magical way to check an executable (or if you have one, please contact the Autotest maintainers as soon as possible). You will have to write your testing tools, i.e., since we are describing an M4 based programming environment, you will have to write your own test macros.

We will write a macro which to factor a whole test group exercising m4 on --version and --help, and as usual, the most difficult task will be finding a good name for it. I suggest AT_CHECK_M4_STD_OPTION (users willing to contribute better names are most welcome: send submissions to gnuprog2-devel@sourceforge.org, along with all information you think might help). The whole std-opt2.at now contains:

     # Process with autom4te to create an -*- Autotest -*- test suite.
     
     m4_define([AT_PACKAGE_STRING],    [GNU Programming 2])
     m4_define([AT_PACKAGE_BUGREPORT], [gnuprog2-devel@sourceforge.org])
     
     AT_INIT([Standard Options: 2])
     
     AT_BANNER([Standard Options.])
     
     # AT_CHECK_M4_STD_OPTION(OPTION)
     # ------------------------------
     # Check that `m4 OPTION' outputs something containing `m4'.
     m4_define([AT_CHECK_M4_STD_OPTION],
     [AT_SETUP([$1 support])
     AT_CHECK([m4 $1], [], [stdout])
     AT_CHECK([fgrep m4 stdout], [], [ignore])
     AT_CLEANUP])
     
     AT_CHECK_M4_STD_OPTION([--version])
     AT_CHECK_M4_STD_OPTION([--help])
     
     Example 12.8: std-opt2.at -- An Autotest Source
     

which gives:

     ## -------------------------------------------------- ##
     ## GNU Programming 2 test suite: Standard Options: 2. ##
     ## -------------------------------------------------- ##
     
     Standard Options.
     
       1: std-opt2.at:19    ok
       2: std-opt2.at:20    ok
     ## ---------------------------- ##
     ## All 2 tests were successful. ##
     ## ---------------------------- ##
     
     Example 12.9: std-opt2 Run
     

Tada!

Let's go a step further: after all, these two test groups are fairly generic: we could very well introduce a higher level macro to check whether some program supports --version and --help! We will exercise autoconf, gcc, litbool, and m4.

     # Process with autom4te to create an -*- Autotest -*- test suite.
     
     m4_define([AT_PACKAGE_STRING],    [GNU Programming 2])
     m4_define([AT_PACKAGE_BUGREPORT], [gnuprog2-devel@sourceforge.org])
     
     AT_INIT([Standard Options: 3])
     
     # _AT_CHECK_STD_OPTION(PROGRAM, OPTION)
     # -------------------------------------
     # Check that `PROGRAM OPTION' outputs something containing `PROGRAM'.
     m4_define([_AT_CHECK_STD_OPTION],
     [AT_SETUP([$1 $2 support])
     AT_CHECK([$1 $2], [], [stdout])
     AT_CHECK([fgrep $1 stdout], [], [ignore])
     AT_CLEANUP])
     
     # AT_CHECK_STD_OPTIONS(PROGRAM)
     # -----------------------------
     # Check that PROGRAM respects the GCS wrt --version, and --help.
     m4_define([AT_CHECK_STD_OPTIONS],
     [AT_BANNER([$1 Standard Options.])
     AT_TESTED([$1])
     _AT_CHECK_STD_OPTION([$1], [--version])
     _AT_CHECK_STD_OPTION([$1], [--help])
     ])
     
     AT_CHECK_STD_OPTIONS([autoconf])
     AT_CHECK_STD_OPTIONS([gcc])
     AT_CHECK_STD_OPTIONS([libtool])
     AT_CHECK_STD_OPTIONS([m4])
     
     Example 12.10: std-opt3.at -- An Autotest Source
     

which produces:

     ## -------------------------------------------------- ##
     ## GNU Programming 2 test suite: Standard Options: 3. ##
     ## -------------------------------------------------- ##
     
     autoconf Standard Options.
     
       1: std-opt3.at:27    ok
       2: std-opt3.at:27    ok
     
     gcc Standard Options.
     
       3: std-opt3.at:28    FAILED near `std-opt3.at:28'
       4: std-opt3.at:28    ok
     
     libtool Standard Options.
     
       5: std-opt3.at:29    ok
       6: std-opt3.at:29    ok
     
     m4 Standard Options.
     
       7: std-opt3.at:30    ok
       8: std-opt3.at:30    ok
     ## ----------------------------------------------- ##
     ## ERROR: Suite unsuccessful, 1 of 8 tests failed. ##
     ## ----------------------------------------------- ##
     
     You may investigate any problem if you feel able to do so, in which
     case the test suite provides a good starting point.
     
     Now, failed tests will be executed again, verbosely, and logged
     in the file std-opt3.log.
     ## -------------------------------------------------- ##
     ## GNU Programming 2 test suite: Standard Options: 3. ##
     ## -------------------------------------------------- ##
     3. std-opt3.at:28: testing gcc --version support...
     std-opt3.at:28: gcc --version
     stdout:
     2.95.2
     std-opt3.at:28: fgrep gcc stdout
     stdout:
     std-opt3.at:28: exit code was 1, expected 0
     3. std-opt3.at:28: FAILED near `std-opt3.at:28'
     ## ------------------------ ##
     ## std-opt3.log is created. ##
     ## ------------------------ ##
     
     Please send `std-opt3.log' to <gnuprog2-devel@sourceforge.org>,
     along with all information you think might help.
     
     Example 12.11: std-opt3 Run
     

Please note that we clearly achieved our goal thanks to the two step test: we know exactly why it failed with gcc, since the whole output is displayed in the logs, there is no need for additional interaction with the user, had the failure occurred on such a "remote" environment.