1 The Automake test suite
15 By default, verbose output of a test 't/foo.sh' or 't/foo.tap' is retained
16 in the log file 't/foo.log'. Also, a summary log is created in the file
17 'test-suite.log' (in the top-level directory).
19 You can use '-jN' for faster completion (it even helps on a uniprocessor
20 system, due to unavoidable sleep delays, as noted below):
24 To rerun only failed tests:
28 To run only tests that are newer than their last results:
30 make -k check RECHECK_LOGS=
32 To run only selected tests:
34 make -k check TESTS="t/foo.sh t/bar.tap" (GNU make)
35 env TESTS="t/foo.sh t/bar.tap" make -e -k check (non-GNU make)
37 To run the tests in cross-compilation mode, you should first configure
38 the automake source tree to a cross-compilation setup. For example, to
39 run with a Linux-to-MinGW cross compiler, you will need something like
42 ./configure --host i586-mingw32msvc --build i686-pc-linux-gnu
44 To avoid possible spurious error, you really have to *explicitly* specify
45 '--build' in addition to '--host'; the 'lib/config.guess' script can help
46 determine the correct value to pass to '--build'.
47 Then you can just run the testsuite in the usual way, and the test cases
48 using a compiler should automatically use a cross-compilation setup.
56 XFAIL - expected failure
60 XPASS - unexpected success
63 SKIP - skipped tests (third party tools not available)
64 ERROR - some unexpected error condition
70 There are two kinds of tests in the Automake testsuite (both implemented
71 as shell scripts). The scripts with the '.sh' suffix are "simple"
72 tests, their outcome completely determined by their exit status. Those
73 with the '.tap' suffix use the TAP protocol. If you want to run a test
74 by hand, you can do so directly if it is a simple test:
78 (it will be verbose by default), while if it is a TAP test you can pass
79 it to your preferred TAP runner, as in e.g.:
81 prove --verbose --merge ./t/add-missing.tap
83 The tests can also be run directly in a VPATH build, as with:
85 /path/to/srcdir/t/nogzip.sh
86 prove --verbose --merge /path/to/srcdir/t/add-missing.tap
92 By default, the tests are run by the $SHELL detected at configure
93 time. They also take care to re-execute themselves with that shell,
94 unless told not to. So, to run the tests with a different shell, say
95 '/path/to/another/sh', the user must use:
97 AM_TESTS_REEXEC=no /path/to/another/sh ./t/foo.sh
98 AM_TESTS_REEXEC=no prove -v -e /path/to/another/sh ./t/bar.tap
100 to run a test directly, and:
102 make check LOG_COMPILER=/path/to/sh (GNU make)
103 LOG_COMPILER=/path/to/sh make -e check (non-GNU make)
105 to run the test(s) through the makefile test driver.
107 The test scripts are written with portability in mind, so that they
108 should run with any decent Bourne-compatible shell.
110 However, some care must be used with Zsh, since, when not directly
111 started in Bourne-compatibility mode, it has some incompatibilities
112 in the handling of $0 which conflict with our usage. Our testsuite
113 can automatically work around these incompatibilities when a version
114 4.3 or later of Zsh is used, but unfortunately not when an older
115 version of Zsh is used. Thus, if you want to run a test script, say
116 'foo.sh', with Zsh 4.2, you *can't* simply do "zsh foo.sh", but
117 you *must* resort to:
119 AM_TESTS_REEXEC=no zsh -o no_function_argzero foo.sh
121 Note that this problem does not occur if Zsh is executed through
122 a symlink with a basename of 'sh', since in that case Zsh starts
123 in Bourne compatibility mode. So you should be perfectly safe
124 when /bin/sh is Zsh, even a it's version < 4.3.
130 Send verbose output, i.e., the contents of test-suite.log, of failing
131 tests to <bug-automake@gnu.org>, along with the usual version numbers
132 (which Automake, which Autoconf, which operating system, which make
133 version, which shell, etc.)
140 * If you plan to fix a bug, write the test case first. This way you'll
141 make sure the test catches the bug, and that it succeeds once you have
144 * Add a copyright/license paragraph.
146 * Explain what the test does, i.e., which features it checks, which
147 invariants it verifies, or what bugs/issues it guard against.
149 * Cite the PR number (if any), and the original reporter (if any), so
150 we can find or ask for information if needed.
152 * If a test checks examples or idioms given in the documentation, make
153 sure the documentation reference them appropriately in comments, as
156 @c Keep in sync with autodist-config-headers.sh
161 * Use "required=..." for required tools. Do not explicitly require
162 tools which can be taken for granted because they're listed in the
163 GNU Coding Standards (for example, 'gzip').
165 * Include ./defs in every test script (see existing tests for examples
168 * Use the 'skip_' function to skip tests, with a meaningful message if
169 possible. Where convenient, use the 'warn_' function to print generic
170 warnings, the 'fail_' function for test failures, and the 'fatal_'
171 function for hard errors. In case a hard error is due to a failed
172 set-up of a test scenario, you can use the 'framework_fail_' function
175 * For those tests checking the Automake-provided test harnesses that
176 are expected to work also when the 'serial-tests' Automake option
177 is used (thus causing the serial testsuite harness to be used in the
178 generated Makefile), place a line containing "try-with-serial-tests"
179 somewhere in the file (usually in a comment).
180 That will ensure that the 'gen-testsuite-part' script generates a
181 sibling of that test which uses the serial harness instead of the
182 parallel one. For those tests that are *not* meant to work with the
183 parallel testsuite harness at all (these should be very very few),
184 set the shell variable 'am_serial_tests' to "yes" before including
187 * Some tests in the Automake testsuite are auto-generated; those tests
188 might have custom extensions, but their basename (that is, with such
189 extension stripped) is expected to end with "-w" string, optionally
190 followed by decimal digits. For example, the name of a valid
191 auto-generated test can be 'color-w.sh' or 'tap-signal-w09.tap'.
192 Please don't name hand-written tests in a way that could cause them
193 to be confused with auto-generated tests; for example, 'u-v-w.sh'
194 or 'option-w0.tap' are *not* valid name for hand-written tests.
196 * ./defs brings in some commonly required files, and sets a skeleton
197 configure.ac. If possible, append to this file. In some cases
198 you'll have to overwrite it, but this should be the exception. Note
199 that configure.ac registers Makefile.in but do not output anything by
200 default. If you need ./configure to create Makefile, append AC_OUTPUT
201 to configure.ac. In case you don't want ./defs to pre-populate your
202 test directory (which is a rare occurrence), set the 'am_create_testdir'
203 shell variable to "empty" before sourcing ./defs.
205 * By default, the testcases are run with the errexit shell flag on,
206 to make it easier to catch failures you might not have thought of.
207 If this is undesirable in some testcase, you can use "set +e" to
208 disable the errexit flag (but please do so only if you have a very
211 * End the test script with a ':' command. Otherwise, when somebody
212 changes the test by adding a failing command after the last command,
213 the test will spuriously fail because '$?' is nonzero at the end.
214 Note that this is relevant even if the errexit shell flag is on, in
215 case the test contains commands like "grep ... Makefile.in && Exit 1"
216 (and there are indeed a lot of such tests).
218 * Use $ACLOCAL, $AUTOMAKE, $AUTOCONF, $AUTOUPDATE, $AUTOHEADER,
219 $PERL, $MAKE, $EGREP, and $FGREP, instead of the corresponding
222 * Use '$sleep' when you have to make sure that some file is newer
225 * Use cat or grep or similar commands to display (part of) files that
226 may be interesting for debugging, so that when a user send a verbose
227 output we don't have to ask him for more details. Display stderr
228 output on the stderr file descriptor. If some redirected command is
229 likely to fail, display its output even in the failure case, before
232 * Use 'Exit' rather than 'exit' to abort for leave early from a test
235 * Use '$PATH_SEPARATOR', not hard-coded ':', as the separator of
238 * It's more important to make sure that a feature works, than make
239 sure that Automake's output looks correct. It might look correct
240 and still fail to work. In other words, prefer running 'make' over
241 grepping Makefile.in (or do both).
243 * If you run $ACLOCAL, $AUTOMAKE or $AUTOCONF several times in the
244 same test and change configure.ac by the meantime, do
246 rm -rf autom4te*.cache
248 before the following runs. On fast machines the new configure.ac
249 could otherwise have the same timestamp as the old autom4te.cache.
251 * Use filenames with two consecutive spaces when testing that some
252 code preserves filenames with spaces. This will catch errors like
253 `echo $filename | ...`.
255 * Make sure your test script can be used to faithfully check an
256 installed version of automake (as with "make installcheck"). For
257 example, if you need to copy or grep an automake-provided script,
258 do not assume that they can be found in the '$top_srcdir/lib'
259 directory, but use '$am_scriptdir' instead. The complete list of
260 such "$am_...dir" variables can be found in the 'defs-static.in'
263 * When writing input for lex, include the following in the definitions
267 #define YY_NO_UNISTD_H 1
270 to accommodate non-ANSI systems, since GNU flex generates code that
271 includes unistd.h otherwise. Also add:
273 int isatty (int fd) { return 0; }
275 to the definitions section if the generated code is to be compiled
276 by a C++ compiler, for similar reasons (i.e., the isatty(3) function
277 from that same unistd.h header would be required otherwise).
279 * Before commit: make sure the test is executable, add the tests to
280 TESTS in Makefile.am, add it to XFAIL_TESTS in addition if needed,
281 write a ChangeLog entry, send the diff to <automake-patches@gnu.org>.
283 * In test scripts, prefer using POSIX constructs over their old
284 Bourne-only equivalents:
286 - use $(...), not `...`, for command substitution;
287 - use $((...), not `expr ...`, for arithmetic processing;
288 - liberally use '!' to invert the exit status of a command, e.g.,
289 in idioms like "if ! CMD; then ...", instead of relying on clumsy
290 paraphrases like "if CMD; then :; else ...".
291 - prefer use of ${param%pattern} and ${param#pattern} parameter
292 expansions over processing by 'sed' or 'expr'.
294 * Note however that, when writing Makefile recipes or shell code in a
295 configure.ac, you should still use `...` instead, because the Autoconf
296 generated configure scripts do not ensure they will find a truly POSIX
297 shell (even though they will prefer and use it *if* it's found).
299 * Do not test an Automake error with "$AUTOMAKE && Exit 1", or in three
300 years we'll discover that this test failed for some other bogus reason.
301 This happened many times. Better use something like
304 grep 'expected diagnostic' stderr
306 Note this doesn't prevent the test from failing for another reason,
307 but at least it makes sure the original error is still here.
309 * Do not override Makefile variables using make arguments, as in e.g.:
311 $MAKE prefix=/opt install
313 This is not portable for recursive targets (targets that call a
314 sub-make may not pass "prefix=/opt" along). Use the following
317 prefix=/opt $MAKE -e install