There are two kinds of tests in the Automake testsuite (both implemented
as shell scripts). The scripts with the '.sh' suffix are "simple"
tests, their outcome completely determined by their exit status. Those
- with the '.tap' suffix use the TAP protocol. If you want to run a test
- by hand, you can do so directly if it is a simple test:
+ with the '.tap' suffix use the TAP protocol.
- ./t/nogzip.sh
+ If you want to run a test by hand, you should be able to do so using the
+ 'runtest' script provided in the Automake distribution:
- (it will be verbose by default), while if it is a TAP test you can pass
- it to your preferred TAP runner, as in e.g.:
+ ./runtest t/nogzip.sh
+ ./runtest t/add-missing.tap
- prove --verbose --merge ./t/add-missing.tap
-
- The tests can also be run directly in a VPATH build, as with:
-
- /path/to/srcdir/t/nogzip.sh
- prove --verbose --merge /path/to/srcdir/t/add-missing.tap
+ This will run the test using the correct shell, and should also work in
+ VPATH builds. Note that, to run the TAP tests this way, you'll need to
+ have the prove(1) utility available in $PATH.
Supported shells
----------------
- By default, the tests are run by the $SHELL detected at configure
- time. They also take care to re-execute themselves with that shell,
- unless told not to. So, to run the tests with a different shell, say
- '/path/to/another/sh', the user must use:
-
- AM_TESTS_REEXEC=no /path/to/another/sh ./t/foo.sh
- AM_TESTS_REEXEC=no prove -v -e /path/to/another/sh ./t/bar.tap
-
- to run a test directly, and:
-
- make check LOG_COMPILER=/path/to/sh (GNU make)
- LOG_COMPILER=/path/to/sh make -e check (non-GNU make)
+ By default, the tests are run by a proper shell detected at configure
+ time. Here is how you can run the tests with a different shell, say
+ '/bin/my-sh':
- to run the test(s) through the makefile test driver.
+ # Running through the makefile test driver.
+ make check AM_TEST_RUNNER_SHELL=/bin/my-sh (GNU make)
+ AM_TEST_RUNNER_SHELL=/bin/my-sh make -e check (non-GNU make)
- The test scripts are written with portability in mind, so that they
- should run with any decent Bourne-compatible shell.
+ # Run a test directly from the command line.
+ AM_TEST_RUNNER_SHELL=/bin/my-sh ./runtest t/foo.sh
- However, some care must be used with Zsh, since, when not directly
- started in Bourne-compatibility mode, it has some incompatibilities
- in the handling of $0 which conflict with our usage. Our testsuite
- can automatically work around these incompatibilities when a version
- 4.3 or later of Zsh is used, but unfortunately not when an older
- version of Zsh is used. Thus, if you want to run a test script, say
- 'foo.sh', with Zsh 4.2, you *can't* simply do "zsh foo.sh", but
- you *must* resort to:
-
- AM_TESTS_REEXEC=no zsh -o no_function_argzero foo.sh
-
- Note that this problem does not occur if Zsh is executed through
- a symlink with a basename of 'sh', since in that case Zsh starts
- in Bourne compatibility mode. So you should be perfectly safe
- when /bin/sh is Zsh, even a it's version < 4.3.
+ The test scripts are written with portability in mind, and should run
+ with any decent POSIX shell. However, it is worth nothing that older
+ versions of Zsh (pre-4.3) exhibited several bugs and incompatibilities
+ with our uses, and are thus not supported for running Automake's test
+ scripts.
Reporting failures
tools which can be taken for granted because they're listed in the
GNU Coding Standards (for example, 'gzip').
-* Include ./defs in every test script (see existing tests for examples
- of how to do this).
+* Include 'test-init.sh' in every test script (see existing tests for
+ examples of how to do this).
* Use the 'skip_' function to skip tests, with a meaningful message if
possible. Where convenient, use the 'warn_' function to print generic
parallel one. For those tests that are *not* meant to work with the
parallel testsuite harness at all (these should be very very few),
set the shell variable 'am_serial_tests' to "yes" before including
- ./defs.
+ test-init.sh.
* Some tests in the Automake testsuite are auto-generated; those tests
might have custom extensions, but their basename (that is, with such
to be confused with auto-generated tests; for example, 'u-v-w.sh'
or 'option-w0.tap' are *not* valid name for hand-written tests.
-* ./defs brings in some commonly required files, and sets a skeleton
- configure.ac. If possible, append to this file. In some cases
- you'll have to overwrite it, but this should be the exception. Note
- that configure.ac registers Makefile.in but do not output anything by
+* test-init.sh brings in some commonly required files, and sets a skeleton
+ configure.ac. If possible, append to this file. In some cases you'll
+ have to overwrite it, but this should be the exception. Note that
+ configure.ac registers Makefile.in but do not output anything by
default. If you need ./configure to create Makefile, append AC_OUTPUT
- to configure.ac. In case you don't want ./defs to pre-populate your
- test directory (which is a rare occurrence), set the 'am_create_testdir'
- shell variable to "empty" before sourcing ./defs.
+ to configure.ac. In case you don't want your test directory to be
+ pre-populate by test-init.sh (this should be a rare occurrence), set
+ the 'am_create_testdir' shell variable to "empty" before sourcing
+ test-init.sh.
* By default, the testcases are run with the errexit shell flag on,
to make it easier to catch failures you might not have thought of.
changes the test by adding a failing command after the last command,
the test will spuriously fail because '$?' is nonzero at the end.
Note that this is relevant even if the errexit shell flag is on, in
- case the test contains commands like "grep ... Makefile.in && Exit 1"
+ case the test contains commands like "grep ... Makefile.in && exit 1"
(and there are indeed a lot of such tests).
* Use $ACLOCAL, $AUTOMAKE, $AUTOCONF, $AUTOUPDATE, $AUTOHEADER,
$PERL, $MAKE, $EGREP, and $FGREP, instead of the corresponding
commands.
+* When you want to redirect the output from a make invocation, use the
+ 'run_make' function rather than calling $MAKE directly. Not only is
+ this more idiomatic, but it also avoid possible spurious racy failures
+ when the make invocations in the testsuite are run in parallel mode
+ (as with "make check AM_TESTSUITE_MAKE='make -j4"').
+
+* Do not override Makefile variables using make arguments, as in e.g.:
+
+ $MAKE prefix=/opt install # BAD
+
+ This is not portable for recursive targets (with non-GNU make,
+ targets that call a sub-make may not pass "prefix=/opt" along).
+ Instead, use the 'run_make' function, which automatically uses
+ the AM_MAKEFLAGS to propagate the variable definitions along to
+ sub-make:
+
+ run_make prefix=/opt install # GOOD
+
* Use '$sleep' when you have to make sure that some file is newer
than another.
likely to fail, display its output even in the failure case, before
exiting.
-* Use 'Exit' rather than 'exit' to abort for leave early from a test
- case.
-
* Use '$PATH_SEPARATOR', not hard-coded ':', as the separator of
PATH's entries.
example, if you need to copy or grep an automake-provided script,
do not assume that they can be found in the '$top_srcdir/lib'
directory, but use '$am_scriptdir' instead. The complete list of
- such "$am_...dir" variables can be found in the 'defs-static.in'
+ such "$am_...dir" variables can be found in the 't/ax/test-defs.in'
file.
* When writing input for lex, include the following in the definitions
by a C++ compiler, for similar reasons (i.e., the isatty(3) function
from that same unistd.h header would be required otherwise).
-* Before commit: make sure the test is executable, add the tests to
- TESTS in Makefile.am, add it to XFAIL_TESTS in addition if needed,
- write a ChangeLog entry, send the diff to <automake-patches@gnu.org>.
+* Add any new test to handwritten_TESTS in 't/list-of-tests.mk', and
+ to XFAIL_TESTS in addition if needed (that is, if the test is expected
+ to fail).
* In test scripts, prefer using POSIX constructs over their old
Bourne-only equivalents:
- use $(...), not `...`, for command substitution;
- - use $((...), not `expr ...`, for arithmetic processing;
+ - use $((...)), not `expr ...`, for arithmetic processing;
- liberally use '!' to invert the exit status of a command, e.g.,
in idioms like "if ! CMD; then ...", instead of relying on clumsy
paraphrases like "if CMD; then :; else ...".
generated configure scripts do not ensure they will find a truly POSIX
shell (even though they will prefer and use it *if* it's found).
-* Do not test an Automake error with "$AUTOMAKE && Exit 1", or in three
+* Do not test an Automake error with "$AUTOMAKE && exit 1", or in three
years we'll discover that this test failed for some other bogus reason.
This happened many times. Better use something like
Note this doesn't prevent the test from failing for another reason,
but at least it makes sure the original error is still here.
-
-* Do not override Makefile variables using make arguments, as in e.g.:
-
- $MAKE prefix=/opt install
-
- This is not portable for recursive targets (targets that call a
- sub-make may not pass "prefix=/opt" along). Use the following
- instead:
-
- prefix=/opt $MAKE -e install
projects / platform / upstream / automake.git / blobdiff