Stefano Lattarini [Thu, 11 Aug 2011 09:28:03 +0000 (11:28 +0200)]
Merge branch 'master' into test-protocols
* master:
coverage: possible infinite recursion in the test harness
test defs: yet more environment cleanup
Stefano Lattarini [Wed, 10 Aug 2011 14:03:35 +0000 (16:03 +0200)]
coverage: possible infinite recursion in the test harness
Motivated by a regression in the 'test-protocols' branch.
* tests/parallel-tests-fork-bomb.test: New test, checking that
if $(TEST_SUITE_LOG) is in $(TEST_LOGS), we obtain a diagnosed
error rather than a make hang or a fork bomb.
* tests/Makefile.am (TESTS): Update.
Stefano Lattarini [Wed, 10 Aug 2011 08:44:56 +0000 (10:44 +0200)]
test defs: yet more environment cleanup
* tests/defs: Also unset the TEST_SUITE_LOG variable.
Stefano Lattarini [Tue, 9 Aug 2011 21:01:23 +0000 (23:01 +0200)]
test harness: recursive make invocations must use $(AM_MAKEFLAGS)
* lib/am/check.am ($(TEST_SUITE_LOG)): Pass $(AM_MAKEFLAGS)
to $(MAKE) when issuing a recursive make call.
Problem detected by the `sc_tests_make_without_am_makeflags'
maintainer check.
Stefano Lattarini [Tue, 9 Aug 2011 20:54:03 +0000 (22:54 +0200)]
Merge branch 'master' into test-protocols
* master:
tests: fix various blunders in 'suffix-chain.test'
Stefano Lattarini [Tue, 9 Aug 2011 20:44:55 +0000 (22:44 +0200)]
tests: fix various blunders in 'suffix-chain.test'
* tests/suffix-chain.test: Fix various blunders that were causing
this test to fail spuriously: append to `configure.in', not to
`configure.ac'; add a `;' character after the `return' statement
in an input file; use tabs, not spaces, to indent Makefile recipes;
and use more uncommon suffixes that don't trigger built-in recipes
with FreeBSD make.
Stefano Lattarini [Tue, 9 Aug 2011 16:04:14 +0000 (18:04 +0200)]
Merge branch 'master' into test-protocols
* master:
tests: don't let a known Solaris make bug poison too many tests
gitignore: more use of anchors
Stefano Lattarini [Tue, 9 Aug 2011 15:51:51 +0000 (17:51 +0200)]
tests: don't let a known Solaris make bug poison too many tests
See automake bug#7670 and bug#7824.
* tests/defs (make_can_chain_suffix_rules): New function, tell
whether the make implementation in use can chain suffix rules
automatically or not.
* tests/suffix3.test: Use it to avoid issuing calls to make that
are unportable to make implementations that are not smart enough
to chain suffix rules automatically.
* tests/suffix8.test: Use it to avoid issuing calls to make that
* tests/suffix10.test: Use it to avoid issuing calls to make that
* tests/suffix11.test: Use it to avoid issuing calls to make that
* tests/suffix-chain.test: New test, exposes the limitation that
we have papered over in the tests above.
Stefano Lattarini [Tue, 9 Aug 2011 14:43:08 +0000 (16:43 +0200)]
gitignore: more use of anchors
* .gitignore: Anchor files that are intended to be ignored only
if found in the same directory of the `.gitignore' file, not also
in its subdirectories.
* doc/.gitignore, doc/amhello/.gitignore, lib/Automake/.gitignore,
lib/Automake/tests/.gitignore, tests/.gitignore: Likewise. Also,
where needed, add new entries that were once implied by the
non-anchored entries in the upper-level `.gitignore' files.
Stefano Lattarini [Tue, 9 Aug 2011 13:27:30 +0000 (15:27 +0200)]
Merge branch 'master' into test-protocols
* master:
testsuite: avoid spurious failure of self checks on Solaris
Stefano Lattarini [Tue, 9 Aug 2011 13:24:03 +0000 (15:24 +0200)]
testsuite: avoid spurious failure of self checks on Solaris
This is just a quick workaround to prevent having too much noise
in the testsuite; more proper fixes have been committed into the
'testsuite-work' branch.
* tests/self-check-dir.test: Relax a test to avoid unimportant
failure on Solaris 10.
* tests/self-check-exit.test: Likewise.
Stefano Lattarini [Mon, 8 Aug 2011 12:42:42 +0000 (14:42 +0200)]
Merge branch 'master' into test-protocols
* master:
test defs: more environment cleanup
Stefano Lattarini [Mon, 8 Aug 2011 12:40:59 +0000 (14:40 +0200)]
Merge branch 'maint'
* maint:
test defs: more environment cleanup
Stefano Lattarini [Mon, 8 Aug 2011 12:39:16 +0000 (14:39 +0200)]
test defs: more environment cleanup
* tests/defs.in: Also unset variables AM_COLOR_TESTS and
AM_TESTS_ENVIRONMENT.
Stefano Lattarini [Mon, 8 Aug 2011 11:18:50 +0000 (13:18 +0200)]
test harness: more minor VPATH-rewrite issues
* tests/am/check.am (am__set_TESTS_bases, $(TEST_SUITE_LOGS),
am--redo-logs, recheck, recheck-html): Cosmetic fixlets to
minimize the risk of unwanted VPATH rewrites.
(check-TESTS): Likewise, and normalize trailing whitespace
since we are at it.
Bugs exposed by test cases `check6-p.test' and `check7-p.test'.
Stefano Lattarini [Mon, 8 Aug 2011 10:50:12 +0000 (12:50 +0200)]
test harness: "make -n check" works correctly with Solaris make
To avoid some recipes being executed with "make -n", we used the
trick of issuing recursive make calls not directly with $(MAKE),
but with a simple indirection instead:
...
am__stealth_MAKE = $(MAKE)
...
target:
... $(am__stealth_MAKE) ...
...
This hack worked with GNU make, but, unfortunately, not with
Solaris make. The present change removes the hack, and uses more
proper idioms to avoid unwanted recipe executions with "make -n".
Bug exposed by test case `parallel-tests-dry-run.test'.
* lib/am/check.am (am__stealth_MAKE): Remove, no more used.
(.log.trs): Break recipe in multiple shell invocations.
($(TEST_SUITE_LOG)): Likewise, and move some corner-case parts of
the recipe to ...
(am--redo-logs): ... this new internal helper .PHONY target.
Stefano Lattarini [Mon, 8 Aug 2011 08:40:28 +0000 (10:40 +0200)]
Merge branch 'master' into test-protocols
* master:
tests: fix spurious failure with Solaris make
Stefano Lattarini [Mon, 8 Aug 2011 08:29:25 +0000 (10:29 +0200)]
Merge branch 'maint'
* maint:
tests: fix spurious failure with Solaris make
Stefano Lattarini [Mon, 8 Aug 2011 08:25:00 +0000 (10:25 +0200)]
Merge branch 'am-distcheck-configure-flags' into maint
* am-distcheck-configure-flags:
tests: fix spurious failure with Solaris make
Stefano Lattarini [Mon, 8 Aug 2011 08:20:47 +0000 (10:20 +0200)]
tests: fix spurious failure with Solaris make
* tests/distcheck-configure-flags-am.test: Avoid using `+=' too
liberally with AM_DISTCHECK_CONFIGURE_FLAGS, since the line breaks
so introduced, in conjunction with single quotes, might confuse
Solaris make.
Stefano Lattarini [Mon, 8 Aug 2011 07:31:03 +0000 (09:31 +0200)]
test harness: fixlet in workaround for VPATH rewrite issue
Fixes a blunder in previous commit `v1.11-983-gfda3de5'.
* lib/am/check.am (am__set_TESTS_bases): Add forgotten end-of-line
anchor to sed expression. In comments, add reference to ...
* tests/test-trs-basic.test: ... this test, which has been adapted
and tightened (and tweaked).
Stefano Lattarini [Sun, 7 Aug 2011 23:39:12 +0000 (01:39 +0200)]
fix: regenerate Makefiles
Stefano Lattarini [Sun, 7 Aug 2011 23:23:42 +0000 (01:23 +0200)]
test harness: work around a VPATH rewrite issue
* lib/am/check.am (am__TEST_BASES): Removed, it's role taken
over by ...
(am__set_TESTS_bases): ... these new variable.
($(TEST_SUITE_LOG): Use it, to avoid VPATH rewrite issues.
* automake.in (handle_tests): Update the code for the cleanup
of the `.trs' file to use `$(TEST_LOGS)' instead of
`$(am__TEST_BASES)'.
* tests/test-harness-vpath-rewrite.test: New test, better
exposing the problem fixed by this change
* tests/Makefile.am (TESTS): Add it.
* tests/test-trs-basic.test: Update and extend.
Stefano Lattarini [Sun, 7 Aug 2011 22:16:56 +0000 (00:16 +0200)]
parallel-tests: work around Solaris XPG4 make segfault
Issue exposed by test `posixsubst-tests-p.test', and similar to
the problem solved by commit `v1.11-159-ge7aa360'.
* lib/am/check.am [%?PARALLEL_TESTS%] (check-TESTS): Trim trailing
whitespace from $list, to avoid triggering a nasty bug (potential
segfault) on Solaris XPG4 make and Heirloom make.
Stefano Lattarini [Sun, 7 Aug 2011 21:48:02 +0000 (23:48 +0200)]
testsuite: fix weird spurious failure with Solaris /bin/sh
Solaris /bin/sh, when killed with a SIGTERM or SIGINT signal, can
apparently end up exiting with exit status 208, instead of leaving
the correct wide exit status to the parent. See:
<http://dbaspot.com/shell/396118-bourne-shell-exit-code-term.html>
Work around this incompatibility.
* tap-signal.test: Look for the above Solaris bug.
(signal_caught): Adapt to handle it.
Stefano Lattarini [Sun, 7 Aug 2011 21:05:37 +0000 (23:05 +0200)]
testsuite: fix another spurious failure on Solaris make
* tests/parallel-tests-log-override-recheck.test: Filter make
output before grepping it, for make implementations that, like
Solaris' one, print the whole of the failed recipe on failure.
Stefano Lattarini [Sun, 7 Aug 2011 20:50:23 +0000 (22:50 +0200)]
testsuite: fix two spurious failures on Solaris make
* tests/tests-environment-and-log-compiler.test: Do not fail if
"make -n check" fails, as that is issued only for debugging, its
semantics being tested in other test scripts.
* tests/test-driver-strip-vpath.test: Likewise.
Stefano Lattarini [Sun, 7 Aug 2011 19:34:56 +0000 (21:34 +0200)]
tap driver: refactor and remove dead code
* lib/tap-driver (get_tap_line): Removed, ...
(all callers): ... updated to use $parser->next directly instead.
(peek_tap_line, unget_tap_line, @tap_lines): Remove, they're not
used anymore.
(TAP_PEEKING): Block deleted, all its subroutines and variables
having been removed.
Stefano Lattarini [Sun, 7 Aug 2011 18:26:34 +0000 (20:26 +0200)]
testsuite: fix spurious failure in a test on TAP support
* tests/tap-passthrough-exit.test: When the `--ignore-exit' flag
of the TAP driver is used, don't look for a message reporting the
non-zero exit statuses of tests in the log files; such message is
not expected to be there anymore. Related simplifications.
Stefano Lattarini [Sun, 7 Aug 2011 18:07:35 +0000 (20:07 +0200)]
tap: plan location is more liberal w.r.t. non-TAP lines
With this change, only lines that are TAP results will matter
w.r.t. the position of the TAP plan in the input; for example,
this input:
this is a non-TAP line
# and this a TAP diagnostic line
1..1
ok 1
was considered to be an error, diagnosed with a message "test
plan in middle of output"; as effect of the current change, such
input is now valid. This is more consistent with the behaviour
of the `prove' utility.
* lib/tap-driver ($lineno): Removed, no more needed.
($tap_stopped): New global variable.
(stringify_test_result): Return "ERROR" if a TAP result is found
when `$tap_stopped' is set to true.
(handle_tap_test): Diagnose TAP results that comes after a late
plan. Add a couple of blank lines, for clarity.
(handle_tap_plan): Set `$tap_stopped' to true after a late plan
is encountered. Do not complain anymore for extra non-TAP lines
preceding or following the plan. Adjust comments.
(main): Don't increment $lineno anymore.
* tests/tap-plan.test: Extend a bit, and remove stale comment.
* tests/tap-color.test: Adjust.
* tests/tap-passthrough.test: Likewise.
* tests/tap-plan-corner.test: Adjust and extend.
* tests/tap-plan-errors.test: Likewise.
* tests/tap-plan-middle.test: New test.
* tests/tap-plan-corner2.test: Delete, it's obsolete now.
* tests/Makefile.am (XFAIL_TESTS): Remove it.
(tap_with_common_setup_tests): Likewise, and add
`tap-plan-corner.test'.
Stefano Lattarini [Sun, 7 Aug 2011 16:17:27 +0000 (18:17 +0200)]
testsuite: remove now-passing test from XFAIL_TESTS
* tests/Makefile.am (XFAIL_TESTS): Remove `tap-signal.test',
which is passing since previous commit `v1.11-974-gc7fa872'.
Stefano Lattarini [Sat, 6 Aug 2011 19:41:37 +0000 (21:41 +0200)]
tap driver: handle signals received by the tests being run
* lib/tap-driver (get_test_exit_message): Also deal with signals,
by using the `wait' method of the TAP::Parser object instead of
the `exit' method. This required the use of the standard perl
module `POSIX'.
* doc/automake.texi (Use TAP with the Automake test harness):
Document that `--ignore-exit' has effect also on terminating
signals. Add a "synchronizing" comment that references the tests
'tap-exit.test' and 'tap-signal.test'.
* tests/tap-signal.test: Extend and adjust.
Stefano Lattarini [Sat, 6 Aug 2011 18:54:26 +0000 (20:54 +0200)]
test driver: a preparatory refactoring (2)
* lib/tap-driver (finish): Move code to fetch the message about
the exit status of the test being run to ...
(get_test_exit_message): ... this new subroutine.
Stefano Lattarini [Sat, 6 Aug 2011 18:48:09 +0000 (20:48 +0200)]
tap driver: a preparatory refactoring (1)
* lib/tap-driver (finish): Do not flush the TAP stream nor
fetch the exit status of the TAP producer if running with
the `--ignore-exit' option. Obviously, don't try to write
such exit status in the test log anymore.
Stefano Lattarini [Sat, 6 Aug 2011 18:23:54 +0000 (20:23 +0200)]
cosmetics: typofix in the name of a function in the TAP driver
* lib/tap-driver (testuite_error): Renamed ...
(testsuite_error): ... to this, and minor related reformatting.
Stefano Lattarini [Sat, 6 Aug 2011 18:21:18 +0000 (20:21 +0200)]
tap: scripts with a SKIP plan but with exit status != 0 must error
This change has been motivated by Automake's own testsuite. Some
TAP tests there were erroring out (due to unexpected, unhandled
failures) before having encountered TAP result, so that the
simple-minded shell library implementing TAP generation ended up
printing a "1..0" trailing test plan; this caused the script to be
reported as a SKIP rather than an ERROR -- a nasty false negative.
* lib/tap-driver: Add prototypes for each subroutine, to free up
the order in which they can be defined and called.
(main): Move the code checking for a bad exit status of the TAP
producer ...
(finish): ... here, and flush the TAP stream to ensure that the
parser always obtains the producer's exit status.
* tests/tap-skip-whole-badexit.test: New test.
* tests/Makefile.am (tap_with_common_setup_tests): Add it.
Stefano Lattarini [Sat, 6 Aug 2011 15:42:38 +0000 (17:42 +0200)]
tap: fix whitespace munging of diagnostic messages
* lib/tap-driver (extract_tap_comment): Pass the `g' flag to the
substitution operator, to strip also trailing whitespaces. Fixes
a failure in test `tap-whitespace-normalization.test'.
Stefano Lattarini [Fri, 5 Aug 2011 18:59:03 +0000 (20:59 +0200)]
tap: fix typo in TAP driver version message
* lib/tap-driver (anonymous subroutine printing the script version
message): Add missing trailing newline to the version message.
Stefano Lattarini [Fri, 5 Aug 2011 18:52:34 +0000 (20:52 +0200)]
tap docs: we don't support pragma or version directives (yet)
* doc/automake.texi (Incompatibilities with other TAP parsers
and drivers): Document that limitation. Normalize an unrelated
"TODO" item.
Stefano Lattarini [Fri, 5 Aug 2011 17:08:16 +0000 (19:08 +0200)]
tap: new option to change the string designating TAP diagnostic
Here we introduce a new option `--diagnostic-string' in our TAP
test driver, that allows the user to specify which string should
denote the beginning of a TAP diagnostic line. This change is
not gratuitous, nor result if over-engineering: it is motivated
by real issues that have emerged during the use of TAP in the
Automake's own testsuite (see the commit `v1.11-1082-g9b967c2'
"testsuite: yet more use of TAP, and related extensions").
* doc/automake.texi (Use TAP with Automake test harness): Document
the new option.
(Incompatibilities with other TAP parsers and drivers): Report it
as a potential source of incompatibility.
* lib/tap-driver ($diag_string): New global variable, defaulting
to "#", and whose value can be changed ...
(Getopt::Long::GetOptions): ... by the newly recognized option
`--diagnostic-string'.
(handle_tap_comment): Subroutine removed, some of its simple logic
inlined ...
(main): ... in here, where now ...
(extract_tap_comment): ... this new subroutine is used.
($USAGE): Adjust.
* tests/tap-diagnostic.test: Make one check slightly stricter.
* tests/tap-diag-custom.test: New test.
* tests/Makefile.am (tap_other_tests): Add it.
Stefano Lattarini [Fri, 5 Aug 2011 14:48:32 +0000 (16:48 +0200)]
check: add small "synchronization" comment to `test-driver'
* lib/test-driver: Add comment to the code initializing ANSI color
escapes, telling to keep it in sync with the similar initialization
in `lib/am/check.am'.
Stefano Lattarini [Fri, 5 Aug 2011 12:27:12 +0000 (14:27 +0200)]
maintcheck: fix spurious failure w.r.t. use of bare "exit"
* tests/check12.test: Cosmetic changes to avoid triggering a
spurious failure of the `sc_tests_Exit_not_exit' maintainer
check.
Stefano Lattarini [Fri, 5 Aug 2011 12:16:19 +0000 (14:16 +0200)]
check: update comments to reflect recent heavy changes
* lib/am/check.am [%?PARALLEL_TESTS%]: Update the comments to
reflect the recent changes. Remove quite many comments that
were merely duplicating excerpts from the Automake manual.
Stefano Lattarini [Fri, 5 Aug 2011 10:08:20 +0000 (12:08 +0200)]
check: small cleanups and refactorings in test harness and drivers
* lib/tap-driver (yn): New subroutine, converts a boolean value to
a "yes" or "no" string.
(write_test_results): Use it, and related reformatting.
* lib/test-driver (trap): Also remove the `.trs' file on signals,
for extra safety.
(fatal): Remove this function, it's never used.
($logfile, $trsfile): Renamed ...
($log_file, $trs_file): ... to these, for clarity, and in order to
be more consistent with the `tap-driver' script.
Improve a couple of comments.
* automake.in (handle_tests): Don't define anymore the now-obsolete
make macro `$(TEST_LOGS_TMP)', nor add it to the list of files to
be removed upon "make mostlyclean".
* lib/am/check.am ($(TEST_SUITE_LOG)): New shell function `f_ok',
tells whether a path refers to an existing, regular, readable file.
Use it throughout.
(recheck): Be safer w.r.t. make implementation that run recipes
with `errexit' shell flag active.
Stefano Lattarini [Fri, 5 Aug 2011 08:36:06 +0000 (10:36 +0200)]
testsuite: use AM_TESTS_FD_REDIRECT where appropriate
* parallel-tests-interrupt.test: Use AM_TESTS_FD_REDIRECT, not
AM_TESTS_ENVIRONMENT, to define file descriptors' redirection
for "make check". Fix botched comment.
Stefano Lattarini [Fri, 5 Aug 2011 08:17:01 +0000 (10:17 +0200)]
docs: some fixlets in documentation on testsuites support
* doc/automake.texi (Script-based Testsuites): Rename node ...
(Scripts-based Testsuites): ... to this. Break overly long lines
in an example (were causing problems with PDF output).
(Parallel Test Harness): Don't intend paragraphs that come after
examples or bulleted list.
(Overview of Custom Test Drivers Support): Fix typo s/the the/the/.
(Declaring Custom Test Drivers): When giving an aside, prefer comma
to parentheses, as it disrupts the reading flow less.
(API for Custom Test Drivers): Remove a sentence that was basically
duplicated from the previous subsection.
(Command-line arguments for test drivers): Don't intend paragraph
coming after an itemized list. Clarify wording and fix grammaros.
Add a couple of cross-references to earlier explanations of `.log'
and `.trs' files.
(Links and external resources): Node renamed ...
(Links and external resources on TAP): ... to this, for clarity.
Slightly extend and clarify introductory sentence.
Stefano Lattarini [Fri, 5 Aug 2011 07:28:17 +0000 (09:28 +0200)]
test harness: be aware of more metadata, simplify test drivers
* lib/am/check.am ($(TEST_SUITE_LOG)): When producing the
global test log, take into account the new metadata field
`:global-test-result:, to write nicely formatted RST section
titles, instead of leaving this chore to the individual test
drivers.
(am_rst_section): Re-introduce this variable, as removed in
commit `v1.11-870-ga27c9c4'.
* lib/test-driver, lib/tap-driver: Write the global test result
as metadata in the `.trs' file, not as (part of) a formatted RST
subsection title in the `.log' file. Related simplifications;
in particular, get rid of temporary files usage.
* doc/automake.texi (Log files generation and test results
recording): Document the new metadata.
* tests/check12.test: Update.
* tests/parallel-tests-harderror.test: Likewise.
* tests/parallel-tests-interrupt.test: Likewise.
* tests/parallel-tests-log-override-2.test: Likewise.
* tests/parallel-tests-log-override-recheck.test: Likewise.
* tests/parallel-tests-unreadable.test: Likewise.
* tests/tap-global-result.test: Likewise.
* tests/test-metadata-results.test: Likewise.
* tests/test-log.test: Likewise, and another minor unrelated fix.
* tests/test-metadata-global-result.test: New test.
* tests/Makefile.am (TESTS): Update.
Stefano Lattarini [Wed, 3 Aug 2011 18:22:50 +0000 (20:22 +0200)]
tap: check that also a trailing TAP plan can hold a skip directive
* tests/tap-skip-whole-lastline.test: New test.
* tests/Makefile.am (tap_with_common_setup_tests): Add it.
Stefano Lattarini [Wed, 3 Aug 2011 15:54:18 +0000 (17:54 +0200)]
maintcheck: fix more maintainer-check failures
* tests/test-driver-custom-no-html.test (no-rst): Use `EOF',
not `EoL', as the here-document delimiter.
* tests/test-trs-basic.test: Use `cat + here-doc' rather
than `echo' when creating the dummy test scripts, to please
maintainer-check.
* tests/test-trs-recover.test: Use creative quoting where
needed, to please maintainer-check.
* tests/parallel-tests-no-color-in-log.test: Likewise.
* tests/parallel-tests-dry-run.test: Likewise.
Stefano Lattarini [Wed, 3 Aug 2011 15:24:19 +0000 (17:24 +0200)]
Merge branch 'master' into test-protocols
* master:
maintcheck: fix maintainer-check failures, both real and spurious
docs: how to use '-I' option in AM_CPPFLAGS for best VPATH support
tests: relax a test on amhello examples to cater to Solaris tar
test defs: function 'is_newest' now works also with directories
docs: add references between the 2 sections on java support
Sync auxiliary files from upstream.
fix typo in recent ChangeLog entry
docs, tests: synchronize examples on silent-rules from config.site
tests: fix weakness in 'tests-environment-backcompat.test'
tests: portability fixes in tests on amhello examples
remake: add test ensuring that slower remakes don't hang
remake: fix outdated comment in configure.am
docs, tests: synchronize examples from docs to tests
coverage: new test on parallel-tests TESTS runtime overriding
cosmetics: fix typos in recent ChangeLog entries
Stefano Lattarini [Wed, 3 Aug 2011 14:40:15 +0000 (16:40 +0200)]
Merge branch 'maint'
* maint:
maintcheck: fix maintainer-check failures, both real and spurious
Stefano Lattarini [Wed, 3 Aug 2011 14:35:49 +0000 (16:35 +0200)]
maintcheck: fix maintainer-check failures, both real and spurious
* tests/amhello-binpkg.test: Use "$MAKE", not bare "make".
* Makefile.am (sc_perl_local): Also allow perl special variable
`$~' to be localized. And be slightly laxer in the regexp, to
allow for usages like "local $_ = $foo;".
(sc_tests_overriding_macros_on_cmdline): Also allow for command
line overriding of the `DISABLE_HARD_ERRORS' make variable. Try
to avoid false positives for usages like "$MAKE || st=$?".
Stefano Lattarini [Wed, 3 Aug 2011 14:12:34 +0000 (16:12 +0200)]
gitignore: ignore `*.trs' files in lib/Automake/tests
* lib/Automake/tests/.gitignore: Add `*.trs' pattern.
Stefano Lattarini [Wed, 3 Aug 2011 14:09:18 +0000 (16:09 +0200)]
testsuite: list another "forgotten" test script in Makefile.am
* tests/Makefile.am (TESTS): Also list test script
`test-driver-trs-suffix-registered.test'; it had been introduced
in commit `v1.11-910-g0c81b43', but by mistake it wasn't added
to the list of tests in the Makefile back then.
Stefano Lattarini [Wed, 3 Aug 2011 13:58:14 +0000 (15:58 +0200)]
testsuite: list "forgotten" test script in Makefile.am
* tests/Makefile.am (TESTS): Also list the test script
`parallel-tests-no-color-in-log.test'; it had been introduced in
commit `v1.11-899-geaac33f', but by mistake it wasn't added to
the list of tests in the Makefile back then.
Stefano Lattarini [Wed, 3 Aug 2011 13:52:05 +0000 (15:52 +0200)]
testsuite: remove duplicated mention of a test in Makefile.am
* tests/Makefile.am (tap_other_tests): Remove `tap-empty.test',
it is already listed in `$(tap_with_common_setup_tests)'.
Stefano Lattarini [Wed, 3 Aug 2011 13:09:43 +0000 (15:09 +0200)]
Merge branch 'maint'
* maint:
docs: how to use '-I' option in AM_CPPFLAGS for best VPATH support
Bruno Haible [Wed, 3 Aug 2011 13:05:22 +0000 (15:05 +0200)]
docs: how to use '-I' option in AM_CPPFLAGS for best VPATH support
* doc/automake.texi (Program Variables): Recommend -I options to
both the build directory and the source directory when needed.
Stefano Lattarini [Wed, 3 Aug 2011 12:29:52 +0000 (14:29 +0200)]
testsuite: one more use of TAP in our own tests
* tests/tap-bad-prog.test, tests/tap-bad-prog2.test: Merged
into ...
* tests/tap-bad-prog.tap: ... this TAP-generating test.
* tests/Makefile.am (tap_other_tests): Remove plan-bad-prog.test
and plan-bad-prog2.test.
(XFAIL_TESTS): Remove plan-bad-prog2.test.
(plan-bad-prog2.log): Remove.
Stefano Lattarini [Sun, 17 Jul 2011 11:38:42 +0000 (13:38 +0200)]
testsuite: scaffolding to allow use of TAP in our own tests
Now that Automake has initial support for the TAP test protocol,
we can start "eating our own dog food" and rewrite some tests to
use TAP; this should allow for better granularity, especially in
the control of skips and expected failures. With this change, we
set up the initial scaffolding required by the planned TAP tests,
and convert few older tests to use TAP, mostly in order to verify
that there are no obvious errors. The tests in our testsuite that
use the TAP protocol will be marked by the new `.tap' extension.
* tests/plain-functions.sh: New file containing definition of
helper shell functions used by the "protocol-less" tests of the
Automake testsuite.
* tests/tap-functions.sh: New file containing definition of
helper shell functions used by the TAP-producing tests of the
Automake testsuite.
* tests/defs: Remove definitions of some functions that are
now defined in `plain-functions.sh'. Instead, source one of
`plain-functions.sh' or `tap-functions.sh', depending on whether
`$use_tap' is set to "no" or "yes".
(exit trap): Call `late_plan_' if `$use_tap' is true. Also,
unset shell traces before issuing the latest commands, to avoid
confusing the tap driver with spurious output.
* tests/defs-static.in ($use_tap): New variable, by default set
to "yes" if the calling test script has a `.tap' suffix, and to
"no" otherwise. The individual scripts can override it though.
In code sanity-checking the environment, verify that `$use_tap'
is not exported.
* tests/self-check-env-sanitize.test: Update, and small related
reformatting.
* tests/self-check-tap.test: New very minimal self test.
* tests/acoutbs.test, tests/acoutbs2.test, tests/acoutnoq.test,
tests/acoutpt.test, tests/acoutpt2.test, tests/acoutqnl.test:
Removed, merged into ...
* tests/ac-output-old.tap: ... this new test, that uses TAP.
* tests/Makefile.am (TAP_LOG_DRIVER): Define to invoke our own
`tap-driver' script.
(TAP_LOG_DRIVER_FLAGS): Define to `--merge', so that the stdout
and stderr of the test scripts remains synced.
(TEST_EXTENSIONS): Define, to add the `.tap' suffix; also list
the `.test' suffix explicitly.
(EXTRA_DIST): Distribute the new files `plain-functions.sh' and
`tap-functions.sh'.
($(TEST_LOGS)): Depends on them.
(AM_TESTS_ENVIRONMENT): Ensure that `use_tap' is not exported in
the environment of the tests.
(TESTS): Update.
Stefano Lattarini [Tue, 2 Aug 2011 21:36:17 +0000 (23:36 +0200)]
* NEWS: Fix typo, and related reformatting.
Stefano Lattarini [Tue, 2 Aug 2011 21:34:04 +0000 (23:34 +0200)]
* NEWS: add mention of new experimental TAP support
Stefano Lattarini [Tue, 2 Aug 2011 20:56:13 +0000 (22:56 +0200)]
testsuite: rename some test scripts
* tests/parallel-tests-am_tests_environment.test: Renamed ...
* tests/am-tests-environment.test: ... to this.
* tests/check-tests_environment.test: Renamed ...
* tests/tests-environment.test: ... to this.
* tests/parallel-test-driver-install.test: Renamed ...
* tests/parallel-tests-driver-install.test: ... to this.
* tests/parallel-tests-make-n.test: Renamed ...
* tests/parallel-tests-dry-run.test: ... to this.
* tests/Makefile.am (TESTS): Updated.
Stefano Lattarini [Tue, 2 Aug 2011 20:24:28 +0000 (22:24 +0200)]
coverage: parallel-tests and lazy dependencies on EXTRA_PROGRAMS
* tests/parallel-tests-extra-programs.test: New test.
* tests/Makefile.am (TESTS): Update.
* doc/automake.texi (Parallel Test Harness): Add a comment
pointing to the new test.
Stefano Lattarini [Tue, 2 Aug 2011 10:05:20 +0000 (12:05 +0200)]
docs: improve, extend and fix documentation on TAP support
* doc/automake.texi ("Using the TAP test protocol"): Divide this
section into ...
("Introduction to TAP", "Use TAP with the Automake test harness",
"Incompatibilities with other TAP parsers and drivers", "Links
and external resources"): ... these subsections, extend them by
adding more information and examples, and improve them by removing
incomplete and/or temporary wordings and TODO items.
("Script-based Testsuites", "Parallel Test Harness"): Add a couple
of anchors to improve the granularity of cross-references.
* tests/tap-doc2.test: New test, verifying the correctness of the
new examples given in the manual.
* tests/Makefile.am (tap_other_tests): Add the new test.
Stefano Lattarini [Mon, 1 Aug 2011 17:10:41 +0000 (19:10 +0200)]
testsuite: separate the only failing check of an xfailing test
* tests/plan-bad-prog.test: Move the only failing check of this
test (i.e., the one about the total number of "ERROR" outcomes)
into ...
* tests/plan-bad-prog2.test: ... this new test.
* tests/Makefile.am (XFAIL_TESTS): Remove `plan-bad-prog.test',
add `plan-bad-prog2.test'.
(tap_other_tests): Add `plan-bad-prog2.test'.
(plan-bad-prog2.log): Depend on `plan-bad-prog.test'.
Stefano Lattarini [Mon, 1 Aug 2011 16:43:30 +0000 (18:43 +0200)]
testsuite: fix spurious errors in an xfailing test on TAP support
* tests/plan-bad-prog.test: Fix typo in the name of the test
being made unreadable. Correct the wording of the potential
skip message, and simplify the condition under which the test
is to be skipped. Escape literal dots in grep regexps.
Stefano Lattarini [Mon, 1 Aug 2011 16:32:17 +0000 (18:32 +0200)]
testsuite: better granularity in a couple of tests on TAP support
* tests/tap-summary-aux.sh: New auxiliary script, filled with code
moved out from ...
* tests/tap-summary.test: ... this test, from which the checks on
colored testsuite have further been moved into ...
* tests/tap-summary-color.test: ... this new test, which in turn
* tests/Makefile.am (tap_other_tests): Add `tap-summary-color.test'.
(EXTRA_DIST): Distribute `tap-summary-aux.sh'
(tap-summary.log): Depend on `tap-summary-aux.sh'.
(tap-summary-color.log): Likewise.
Stefano Lattarini [Mon, 1 Aug 2011 14:14:44 +0000 (16:14 +0200)]
testsuite: optimize tests on TAP for speed and against duplication
The creation and configuration of common files and data used by
many tests on TAP is, with this change, factored out into a new
dedicated auxiliary test, from which those tests will depend upon.
This reduces code duplication in tests and, more importantly,
offers a noticeable speedup in the involved tests (30-40%).
* tests/tap-common-setup.test: New test, setting up the common
files and data used by various tests on TAP support.
* tests/tap-setup.sh: New file, to be sourced by tests wanting to
bring in data generated by `tap-common-setup.test'.
* tests/tap-autonumber.test: Update to use the pre-computed data
files.
* tests/tap-bailout.test: Likewise.
* tests/tap-color.test: Likewise.
* tests/tap-deps.test: Likewise.
* tests/tap-diagnostic.test: Likewise.
* tests/tap-empty-diagnostic.test: Likewise.
* tests/tap-empty.test: Likewise.
* tests/tap-escape-directive.test: Likewise.
* tests/tap-exit.test: Likewise.
* tests/tap-signal.test: Likewise.
* tests/tap-fancy.test: Likewise.
* tests/tap-fancy2.test: Likewise.
* tests/tap-global-log.test: Likewise.
* tests/tap-global-result.test: Likewise.
* tests/tap-html.test: Likewise.
* tests/tap-log.test: Likewise.
* tests/tap-merge-stdout-stderr.test: Likewise.
* tests/tap-no-merge-stdout-stderr.test: Likewise.
* tests/tap-message-0.test: Likewise.
* tests/tap-no-disable-hard-error.test: Likewise.
* tests/tap-no-spurious-summary.test: Likewise.
* tests/tap-no-spurious.test: Likewise.
* tests/tap-not-ok-skip.test: Likewise.
* tests/tap-numeric-description.test: Likewise.
* tests/tap-out-of-order.test: Likewise.
* tests/tap-passthrough.test: Likewise.
* tests/tap-passthrough-exit.test: Likewise.
* tests/tap-plan.test: Likewise.
* tests/tap-plan-corner.test: Likewise.
* tests/tap-plan-corner2.test: Likewise.
* tests/tap-plan-errors.test: Likewise.
* tests/tap-realtime.test: Likewise.
* tests/tap-recheck-logs.test: Likewise.
* tests/tap-skip-whole.test: Likewise.
* tests/tap-todo-skip-together.test: Likewise.
* tests/tap-todo-skip-whitespace.test: Likewise.
* tests/tap-todo-skip.test: Likewise.
* tests/tap-unplanned.test: Likewise.
* tests/tap-with-and-without-number.test: Likewise.
* tests/tap-xfail-tests.test: Likewise.
* tests/tap-skip-whole-whitespace.test: Likewise, and remove
redundant definitions of `$sp' and `$tab' (they are already
defined in `tests/defs').
* tests/tap-whitespace-normalization.test: Likewise.
* tests/Makefile.am (TESTS): Update.
(EXTRA_DIST): Distribute `tap-setup.sh'.
(tap_with_common_setup_tests, tap_with_common_setup_logs)): New
variables, holding respectively the list of tests using the files
pre-computed by `tap-common-setup.test', and the list of their
corresponding log files.
(tap_other_tests): New variable, holding the list of other tests
on TAP support.
($(tap_with_common_setup_logs)): Depend on `tap-common-setup.log'
and `tap-setup.sh'.
Stefano Lattarini [Thu, 28 Jul 2011 10:25:22 +0000 (12:25 +0200)]
test harness: use new `.trs' files to hold test metadata
With this change, the test harness will keep test metadata in
dedicated `.trs' files, instead of having them embedded into the
`.log' files. This allows for easier forward-compatibility and
extension of test metadata, and for more flexibility in the
format of the `.log' files. Note that this change makes the
`:end-metadata:' field obsolete.
* doc/automake.texi (Parallel Test Harness, Log files generation
and test results recording): Document the new APIs and behaviour;
some related minor rewordings and fixlets.
* NEWS: Update.
* automake.in (handle_tests): When bringing in the content of
`check2.am', substitute %BASE% with the basename of the `.log'
file being created by a rule. Add the generated `.trs' files
to the list of files to be cleaned by "make mostlyclean".
* lib/am/check.am (am__test_driver_flags): Rename ...
(am__common_driver_flags): ... to this, and remove the flags
`--test-name' and `--log-file' from it: they are now define in
the proper rules in `check2.am'.
(am__TEST_BASES): New internal variable, holding the names of
the tests, with any registered extension removed.
(am__stealth_MAKE): New internal variable, can be used instead of
$(MAKE) in recipes requiring a recursive call to make, but which
are not intended to be executed by "make -n".
(.log.trs): New suffix rule, to recover from deletion of `.trs'
files.
($(TEST_SUITE_LOG)): Almost completely rewritten to follow the
new API of "test logs in `.log' files, test metadata in `.trs'
files". It goes to some length to work correctly in face of
unreadable or missing `.log' and `.trs' files, and to error out
with proper error messages when this is not possible.
[%?PARALLEL_TESTS%] (check-TESTS): Also remove relevant "stale"
`.trs' files (in addition to `.log files) before remaking the
$(TEST_SUITE_LOG).
(recheck, recheck-html): Look for the `:recheck:' field in the
`.trs' files, not in the `.log' files.
* lib/am/check2.am (?GENERIC?%EXT%.log, ?!GENERIC?%OBJ%): Adjust
the call to the test driver, in particularly passing the new
option `--trs-file'.
[%am__EXEEXT%] (?GENERIC?%EXT%$(EXEEXT).log): Likewise.
* lib/tap-driver ($USAGE): Adjust the help screen.
(Getopt::Long::GetOptions): Handle the `--trs-file' option,
through the use of ...
($trs_file): ... this new global variable.
(finish): Write metadata for the test run to `$trs_file' rather
then to `$log_file', through the use of ...
(write_test_results): ... this new function.
* lib/test-driver (print_usage): Update the help screen.
(Option parsing): Handle the `--trs-file' option, through the
use of ...
($resfile): ... this new global variable.
(Main code): Write metadata for the test run to `$trsfile' rather
than to `$logfile'.
Minor related adjustments to comments.
* tests/.gitignore: Ignore `*.trs' files.
* tests/parallel-tests-unreadable-log.test: Moved ...
* tests/parallel-tests-unreadable.test: ... to this, and extended
to also check the semantics for unreadable `.trs' files.
* tests/test-driver-end-metadata.test: Deleted as obsolete.
* tests/test-driver-metadata-no-leading-space.test: Likewise.
* tests/test-driver-global-log.test: Renamed ...
* tests/test-metadata-global-log.test: ... to this, and modified
as to verify the new APIs and semantics.
* tests/test-driver-recheck: Renamed ...
* tests/test-metadata-recheck.test: ... to this, and modified
likewise.
* tests/parallel-tests-once.test: New test.
* tests/parallel-tests-make-n.test: Likewise.
* test-metadata-results.test: Likewise.
* test-missing.test: Likewise.
* test-missing2.test: Likewise.
* test-trs-basic.test: Likewise.
* test-trs-recover.test: Likewise.
* test-trs-recover2.test: Likewise.
* tests/Makefile.am (TESTS): Update.
Stefano Lattarini [Tue, 26 Jul 2011 12:51:20 +0000 (14:51 +0200)]
test harness: allow more metadata in log files
This change reworks and improves the parallel test harness to use
more specialized reStructuredText fields in the log files (instead
of relying on specially-placed of "magic lines" and more indirect
semantical formatting); the new fields are the following:
- ":recheck:": tell whether the associated test will have to be
re-run by "make recheck";
- ":copy-in-global-log:": tell whether the content of the log
file should be copied in the "global log" `test-suite.log';
- ":end-metadata:", which inhibits the scanning of the rest of
the log file (for what concerns test metadata).
Also, the special `:test-result:' value "END" has been removed,
superseded by the new `:end-metadata:' field.
* doc/automake.texi (Log files generation and test results
recording): Document the new API and semantics. Remove or fix
some obsolete comments.
* lib/am/check.am ($(TEST_SUITE_LOG), recheck, recheck-html):
Adjust comments and code.
* lib/tap-driver (finish): Adjust, with the help of ...
(must_recheck, copy_in_global_log): ... these new functions.
* lib/test-driver (Main code): Adjust, with the help of ...
($recheck, $gcopy): ... these new variables.
* tests/trivial-test-driver: Update to obey the new APIs.
* tests/test-driver-recheck.test: Likewise.
* tests/test-driver-global-log.test: Likewise.
* tests/tap-passthrough.test: Relax the test, avoiding to check
what is written in `test-suite.log'; such check has been moved ...
* tests/tap-global-log.test: ... in this new test, and extended.
* tests/test-driver-metadata-no-leading-space.test: New test.
* tests/test-driver-end-test-results.test: Removed, it checked
the old APIs; superseded by ...
* tests/test-driver-end-metadata.test: ... this new test.
* tests/tap-log.test: Improve syncing with ...
* tests/test-log.test: ... this new test.
* tests/parallel-tests.test: Remove some duplication w.r.t. this
last new test. Updated heading comments.
* tests/Makefile.am (TESTS): Update.
Stefano Lattarini [Wed, 27 Jul 2011 14:34:30 +0000 (16:34 +0200)]
testsuite: fix a spurious failure with non-bash shells
* tests/tap-fancy2.test: Remove an unportable use of backslashes
with the `echo' builtin, which was causing some shells (among them
bash and Debian /bin/ksh) to print a `\\' string, while other
shells (among them zsh, dash, and Solaris /bin/sh and /bin/ksh)
were unexpectedly printing a single `\' character. Since we are
at it, add a sanity check to ensure that this issue does not
resurface.
Stefano Lattarini [Wed, 27 Jul 2011 13:33:13 +0000 (15:33 +0200)]
testsuite: fix spurious failures with Solaris /bin/sh
* tests/tap-more.test: Use `echo > file', not `: > file', to
create empty files in the "for" loops; this is required since,
as documented in the autoconf manual, Solaris 10 /bin/sh
"optimizes" away the `:' command after the first iteration,
even if it is redirected.
* tests/test-driver-custom-multitest-recheck2.test: Likewise.
* tests/tap-numeric-description.test: Partial rewrite to avoid
using positional parameters from the 10th onward, which are
unportable to Solaris /bin/sh (using `${10}' causes the shell
to die with "bad substitution").
Stefano Lattarini [Sun, 24 Jul 2011 14:47:24 +0000 (16:47 +0200)]
simple tests: support developer-defined fd redirections
Motivated by coreutils bug#8846, and related discussions:
<http://debbugs.gnu.org/cgi/bugreport.cgi?bug=8846>
<http://lists.gnu.org/archive/html/bug-autoconf/2011-06/msg00002.html>
In those threads it has been shown how problematic it can be to do
portable file descriptor redirections in the testsuite when using
the Automake testsuite harness. This change should remedy to that
situation.
* lib/am/check2.am (?GENERIC?%EXT%$(EXEEXT).log,
?GENERIC?%EXT%.log, ?!GENERIC?%OBJ%): Append redirection defined
in $(AM_TESTS_FD_REDIRECT) to the command-line invocations of the
test scripts.
* lib/am/check.am [!%?PARALLEL_TESTS%] $(check-TESTS): Likewise.
* NEWS: Update.
* doc/automake.texi (Script-based Testsuites): Document the new
feature.
* tests/check-fd-redirect.test: New test.
* tests/parallel-tests-fd-redirect.test: Likewise.
* tests/parallel-tests-am_tests_environment.test: Remove checks
about the use of redirections in AM_TESTS_ENVIRONMENT: they would
check deprecated (if not undefined) behaviour now. Strengthen a
couple of still valid checks, to keep the test more in sync with
the documentation. Improve debugging information.
* tests/Makefile.am (TESTS): Update.
Stefano Lattarini [Thu, 21 Jul 2011 15:07:54 +0000 (17:07 +0200)]
docs: rework and extend documentation on testsuites support
* doc/automake.texi (Generalities about Testing): New section.
(Simple Tests using parallel-tests): Section removed, merged
into ...
(Simple Tests): ... this one, which on the other hand has been
subdivided into ...
(Script-based Testsuites, Serial Test Harness, Parallel Test
Harness): ... these new subsection.
(DejaGnu Tests): Minor adjustments.
Other related typofixes and rewordings throughout the manual;
in particular, avoid to use the term "test driver" for three
different concepts (and use instead "test harness" and "test
runner" where appropriate).
* tests/tap-doc.test: New test.
* tests/tap-no-disable-hard-error.test: Likewise.
* tests/Makefile.am (TESTS): Update.
Stefano Lattarini [Sun, 17 Jul 2011 14:46:41 +0000 (16:46 +0200)]
tap: support colorization of testsuite progress output
* lib/tap-driver (%COLORS): New variable (definition extracted
from `lib/am/check.am:$(am__tty_colors)', with some obvious
adjustments.
(report): Adjust to colorize console output when required,
using ...
(decorate_result): ... this new function.
(colored): New function, used by the one above.
* tests/tap-summary.test: Also run the checks when `color-tests'
is in use.
* tests/Makefile.am (XFAIL_TESTS): Remove `tap-color.test'.
Stefano Lattarini [Sun, 17 Jul 2011 14:04:06 +0000 (16:04 +0200)]
tap: some preparatory refactoring (2)
This is a follow-up simplification.
* lib/tap-driver (console_output): Renamed ...
(report): ... to this, and extended to appropriately register
the test results when needed.
(testsuite_error, handle_tap_comment, handle_tap_test,
handle_tap_plan): Adjusted accordingly.
Stefano Lattarini [Sun, 17 Jul 2011 13:18:51 +0000 (15:18 +0200)]
tap: some preparatory refactoring (1)
This refactoring is aimed at simplifying the introduction of
colored console output for the TAP driver.
* lib/tap-driver (console_output): Now accept two arguments, the
first one indicating which kind of thing is to be displayed (for
now only a test result or a diagnostic comment), and the second
one (if present) the message associated to it.
(handle_tap_test, handle_tap_comment, handle_tap_plan,
testsuite_error): Adapt to the new `console_output' interface.
Stefano Lattarini [Sun, 3 Jul 2011 17:20:22 +0000 (19:20 +0200)]
tap: add experimental TAP-aware driver
* doc/automake.texi (Using the TAP test protocol): New section.
(Overview of Custom Test Drivers Support): Minor updates.
* lib/tap-driver: New script, TAP-aware test driver for Automake;
implemented in perl and based on TAP::Parser.
* lib/Makefile.am (dist_script_DATA): Add it.
* tests/tap-autonumber.test: New test.
* tests/tap-bailout.test: Likewise.
* tests/tap-basic.test: Likewise.
* tests/tap-deps.test: Likewise.
* tests/tap-diagnostic.test: Likewise.
* tests/tap-empty.test: Likewise.
* tests/tap-empty-diagnostic.test: Likewise.
* tests/tap-escape-directive.test: Likewise.
* tests/tap-exit.test: Likewise.
* tests/tap-fancy.test: Likewise.
* tests/tap-fancy2.test: Likewise.
* tests/tap-global-result.test: Likewise.
* tests/tap-html.test: Likewise.
* tests/tap-log.test: Likewise.
* tests/tap-merge-stdout-stderr.test: Likewise.
* tests/tap-more.test: Likewise.
* tests/tap-more2.test: Likewise.
* tests/tap-no-merge-stdout-stderr.test: Likewise.
* tests/tap-no-spurious-summary.test: Likewise.
* tests/tap-no-spurious.test: Likewise.
* tests/tap-not-ok-skip.test: Likewise.
* tests/tap-numeric-description.test: Likewise.
* tests/tap-out-of-order.test: Likewise.
* tests/tap-passthrough.test: Likewise.
* tests/tap-plan.test: Likewise.
* tests/tap-plan-errors.test: Likewise.
* tests/tap-plan-corner.test: Likewise.
* tests/tap-realtime.test: Likewise.
* tests/tap-recheck-logs.test: Likewise.
* tests/tap-recheck.test: Likewise.
* tests/tap-skip-whole.test: Likewise.
* tests/tap-summary.test: Likewise.
* tests/tap-todo-skip.test: Likewise.
* tests/tap-todo-skip-together.test: Likewise.
* tests/tap-todo-skip-whitespace.test: Likewise.
* tests/tap-skipall-whitespace.test: Likewise.
* tests/tap-unplanned.test: Likewise.
* tests/tap-whitespace-normalization.test: Likewise.
* tests/tap-with-and-without-number.test: Likewise.
* tests/tap-xfail-tests.test: Likewise.
* tests/tap-bad-prog.test: New xfailing test.
* tests/tap-color.test: Likewise.
* tests/tap-plan-corner2.test: Likewise.
* tests/tap-message-0.test: Likewise.
* tests/tap-signal.test: Likewise.
* tests/Makefile.am (TESTS, XFAIL_TESTS): Update.
Stefano Lattarini [Mon, 4 Jul 2011 16:12:07 +0000 (18:12 +0200)]
test defs: new auxiliary function 'count_test_results'
* tests/defs (count_test_results): New function.
* tests/check11.test: Use it.
* tests/test-driver-custom-multitest.test: Likewise.
* tests/test-driver-custom-multitest-recheck.test: Likewise.
* tests/test-driver-custom-multitest-recheck2.test: Likewise.
* tests/parallel-tests-log-override-recheck.test: Likewise.
* tests/parallel-tests-log-override-recheck.test: Likewise.
* tests/parallel-tests-no-spurious-summary.test: Likewise, and
slightly improve debugging output.
* tests/parallel-tests.test: Make use of `count_test_results'.
Also, make grepping of "make check" output slightly stricter
* tests/parallel-tests9.test: Likewise.
* tests/parallel-tests-log-override-2.test: Likewise, and throw
in a small optimization.
Stefano Lattarini [Sun, 24 Jul 2011 09:17:11 +0000 (11:17 +0200)]
Merge branch 'maint'
* maint:
tests: relax a test on amhello examples to cater to Solaris tar
Stefano Lattarini [Sat, 23 Jul 2011 13:10:07 +0000 (15:10 +0200)]
Merge branch 'fix-amhello-examples-tests' into maint
* fix-amhello-examples-tests:
tests: relax a test on amhello examples to cater to Solaris tar
Stefano Lattarini [Sat, 23 Jul 2011 13:09:23 +0000 (15:09 +0200)]
tests: relax a test on amhello examples to cater to Solaris tar
* tests/amhello-binpkg.test: When the tar implementation in use
is not GNU tar, relax the tests on tar output, to avoid spurious
failures. For example, "tar cvf ..." with GNU tar can output
lines like "./usr/bin/hello" on the standard output, while with
Solaris tar it can output lines like "a ./usr/bin/hello 8K" on
standard output, and with Heirloom tar it can output lines like
"a ./usr/bin/hello 15 tape blocks" on standard error.
Stefano Lattarini [Sat, 23 Jul 2011 11:55:20 +0000 (13:55 +0200)]
test defs: function 'is_newest' now works also with directories
* tests/defs (is_newest): Call `find' with the `-prune' option,
so that it won't descend in the directories (which could cause
spurious results).
* tests/self-check-is_newest: Extend accordingly.
From a report by Jim Meyering, see automake bug#9147.
Stefano Lattarini [Fri, 22 Jul 2011 08:27:13 +0000 (10:27 +0200)]
Merge branch 'maint'
* maint:
docs: add references between the 2 sections on java support
Benoit Sigoure [Fri, 15 Jul 2011 23:49:45 +0000 (16:49 -0700)]
docs: add references between the 2 sections on java support
* doc/automake.texi (Java Support, Java): Add cross-references.
Stefano Lattarini [Fri, 1 Jul 2011 09:12:31 +0000 (11:12 +0200)]
parallel-tests: simplify testsuite summary
Prefer a more deterministic, "tabular" format for the testsuite
summary, always listing the numbers of passed, failed, xfailed,
xpassed, skipped and errored tests, even when these numbers are
zero. This simplify the logic of testsuite summary creation,
makes it more easily machine-parseable, and will probably allow
for easier addition of new kinds of test results in the future.
* lib/am/check.am (am__tty_colors_dummy): New make variable, to
reduce code duplication. Extracted from previous versions of
$(am__tty_colors), and extended by defining two new variables
`$mgn' and `$brg'.
[%?COLOR%, %!?COLOR%] (am__tty_colors): Use that new variable.
(am__text_box): Delete, is not needed anymore.
($(TEST_SUITE_LOG)): Rewrite associated rules to implement the
new testsuite summary format.
* NEWS: Update.
* tests/check10.test: Don't run with the parallel-tests harness
too, that makes no sense anymore.
* tests/color.test: Update and adjust.
* tests/color2.test: Likewise.
* tests/parallel-tests.test: Likewise.
* tests/parallel-tests3.test: Likewise.
* tests/parallel-tests6.test: Likewise.
* tests/parallel-tests9.test: Likewise.
* tests/parallel-tests-unreadable-log.test: Likewise.
* tests/parallel-tests-empty-testlogs.test: Likewise.
* tests/parallel-tests-log-override-recheck.test: Likewise.
* tests/parallel-tests-no-spurious-summary.test: Likewise.
* tests/test-driver-custom-multitest.test: Likewise.
* tests/test-driver-end-test-results.test: Likewise.
* tests/parallel-tests-no-color-in-log.test: New test.
* tests/testsuite-summary-color.test: Likewise.
* tests/testsuite-summary-count.test: Likewise.
* tests/testsuite-summary-count-many.test: Likewise.
* tests/testsuite-summary-reference-log.test: Likewise.
* tests/testsuite-summary-checks.sh: New auxiliary script, used
by the new tests above.
* tests/extract-testsuite-summary: Likewise.
* tests/trivial-test-driver: Optimize for speed when there are
lots of of tests.
* tests/Makefile.am (EXTRA_DIST): Distribute them.
(testsuite-summary-color.log, testsuite-summary-count.log): Depend
on them.
(testsuite-summary-count-many.log): Depend on the auxiliary scripts
'trivial-test-driver' and 'extract-testsuite-summary'.
(TESTS): Update.
Stefano Lattarini [Thu, 30 Jun 2011 12:55:43 +0000 (14:55 +0200)]
parallel-tests: new recognized test result 'ERROR'
* lib/am/check.am ($(TEST_SUITE_LOG)): Recognize a new test result
`ERROR'. Use it when encountering unreadable test logs (previously
a simple `FAIL' was used in this situations).
* lib/test-driver: Set the global test result to `ERROR' when the
test exit status is 99. When doing colorized output, color `ERROR'
results in magenta.
* doc/automake.texi (Log files generation and test results
recording): Update by also listing `ERROR' among the list of valid
`:test-results:' arguments.
* NEWS: Update.
* tests/trivial-test-driver: Update.
* tests/parallel-tests.test: Likewise.
* tests/parallel-tests-harderror.test: Likewise.
* tests/parallel-tests-no-spurious-summary.test: Likewise.
* tests/test-driver-global-log.test: Likewise.
* tests/test-driver-recheck.test: Likewise.
* tests/test-driver-custom-multitest-recheck.test: Likewise.
* tests/test-driver-custom-multitest-recheck2.test: Likewise.
* tests/test-driver-custom-multitest.test: Likewise.
* tests/test-driver-custom-no-html.test: Likewise.
* tests/test-driver-end-test-results.test: Likewise.
* tests/color.test: Likewise. Also, make stricter, and also test
from VPATH.
* tests/color2.test: Likewise, and improve syncing with color.test.
* tests/parallel-tests-exit-statuses.test: New test.
* tests/parallel-tests-console-output.test: Likewise.
* tests/Makefile.am (TESTS): Update.
Ralf Wildenhues [Wed, 20 Jul 2011 21:35:32 +0000 (23:35 +0200)]
Merge branch 'maint'
Ralf Wildenhues [Wed, 20 Jul 2011 21:31:54 +0000 (23:31 +0200)]
Sync auxiliary files from upstream.
* INSTALL, lib/INSTALL, lib/config.guess, lib/config.sub,
lib/texinfo.tex: Sync from upstream.
Signed-off-by: Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
Stefano Lattarini [Fri, 8 Jul 2011 09:57:19 +0000 (11:57 +0200)]
Merge branch 'maint'
* maint:
fix typo in recent ChangeLog entry
docs, tests: synchronize examples on silent-rules from config.site
tests: fix weakness in 'tests-environment-backcompat.test'
Stefano Lattarini [Fri, 8 Jul 2011 08:48:23 +0000 (10:48 +0200)]
Merge branch 'fix-tests-environment-backcompat-example' into maint
* fix-tests-environment-backcompat-example:
tests: fix weakness in 'tests-environment-backcompat.test'
Stefano Lattarini [Fri, 8 Jul 2011 08:48:22 +0000 (10:48 +0200)]
Merge branch 'fix-changelog-typo' into maint
* fix-changelog-typo:
fix typo in recent ChangeLog entry
Stefano Lattarini [Fri, 8 Jul 2011 08:48:22 +0000 (10:48 +0200)]
Merge branch 'doc-silent-rules-reference-config-site-test' into maint
* doc-silent-rules-reference-config-site-test:
docs, tests: synchronize examples on silent-rules from config.site
Stefano Lattarini [Fri, 8 Jul 2011 08:45:53 +0000 (10:45 +0200)]
fix typo in recent ChangeLog entry
Stefano Lattarini [Fri, 8 Jul 2011 08:43:33 +0000 (10:43 +0200)]
docs, tests: synchronize examples on silent-rules from config.site
* doc/automake.texi (Automake silent-rules Option): Reference test
'silent-configsite.test' in comments.
Stefano Lattarini [Fri, 8 Jul 2011 08:34:54 +0000 (10:34 +0200)]
tests: fix weakness in 'tests-environment-backcompat.test'
* tests/tests-environment-backcompat.test: Do not override the
content of xfailing test `baz.test' with a "weaker" version that
fails unconditionally: the test must fail only when the 'strict'
pragma is in use, in order not to reduce coverage.
Stefano Lattarini [Fri, 8 Jul 2011 07:54:38 +0000 (09:54 +0200)]
Merge branch 'maint'
* maint:
tests: portability fixes in tests on amhello examples
remake: add test ensuring that slower remakes don't hang
remake: fix outdated comment in configure.am
Stefano Lattarini [Fri, 8 Jul 2011 07:41:27 +0000 (09:41 +0200)]
Merge branch 'fix-amhello-examples-tests' into maint
* fix-amhello-examples-tests:
tests: portability fixes in tests on amhello examples
Stefano Lattarini [Wed, 6 Jul 2011 08:31:03 +0000 (10:31 +0200)]
tests: portability fixes in tests on amhello examples
* tests/amhello-binpkg.test: Don't use "tar xzf too.tag.gz" to
extract a gzip-compressed tarball, that's unportable to some
tar implementations; use the "gzip -dc fo.tar.gz | tar xf -"
idiom instead.
* tests/amhello-cflags.test: Likewise.
* tests/amhello-cross-compile.test: Likewise.
Suggestion from Ralf Wildenhues.