Using Autotest with the GNU Build System

As far as Autoconf is concerned, you only have to invoke AC_CONFIG_TESTDIR:

AC_CONFIG_TESTDIR (test-directory, [autotest-path = test-directory])

Macro

Ask for the creation of test-directory/atconfig, which contains Autotest private information related to the layout of the package tree. The test suite default path, AUTOTEST_PATH, is set to autotest-path. This colon-separated path should include the directories of the programs to exercise, relative to the top level of the package.

and create the wrapper around m4:

     AC_CONFIG_TESTDIR(tests)
     AC_CONFIG_FILES([tests/m4], [chmod +x tests/m4])
     AC_CONFIG_FILES([tests/Makefile tests/atlocal])
     

Given that the current version of Automake, 1.5, does not provide Autotest support, one merely uses the regular Makefile snippets in Makefile.am ((FIXME: Embedding an Autotest Test Suite.), and in particular example 12.5).

One interesting bit is the handling of the generated tests. Because the source files of the tests are expected to be found in the source tree, even though generated.at is generated (surprise!), we have to qualify its path:

     m4_texinfo = $(top_srcdir)/doc/m4.texinfo
     generate   = $(AWK) -f $(srcdir)/generate.awk
     $(srcdir)/generated.at: $(srcdir)/generate.awk $(m4_texinfo)
             $(generate) $(m4_texinfo) >$@t
             mv $@t $@
     

As already described in (FIXME: Embedding an Autotest Test Suite.), you hook the testsuite to Automake's check-local target:

     check-local: atconfig $(TESTSUITE)
             $(SHELL) $(srcdir)/$(TESTSUITE)
     

and finally, after so many pages to read, you can run happily make check, and stare happily at

     ## ----------------------------- ##
     ## All 76 tests were successful. ##
     ## ----------------------------- ##
     

One of the most important targets provided by Automake is distcheck, which basically checks that your packages behaves properly: it compiles cleanly when using a separate build directory, the test suite succeeds, installs the package in some temporary directory etc. This gives you the guarantee that all the files needed to compile and test your package are shipped. Run it, and see how Autotest boxes are so much more beautiful than Automake's...

     ...
     ## ----------------------------- ##
     ## All 76 tests were successful. ##
     ## ----------------------------- ##
     ...
     =======================================
     m4-1.5.tar.gz is ready for distribution
     =======================================
     

Unfortunately, although a plain distcheck is a very significant sign that your package behaves properly, it offers no guarantee that you did not forget to install some files! Don't laugh, I have already been trapped; it even happened that I produced incorrect paths in installed configuration files, while the test suite and distcheck were very happy because... they were not using the package as a user would do, they were bypassing configuration files etc. Lack of realism...

Fortunately Automake provides the installcheck target, which is run by distcheck, after the package was installed! Alleluia! Hook the test suite to installcheck, but setting the AUTOTEST_PATH so that the installed m4 be run:

     # Run the test suite on the *installed* tree.
     installcheck-local:
             $(SHELL) $(TESTSUITE) AUTOTEST_PATH=$(exec_prefix)/bin
     

This time, we really did all we could to test our package.