Stefano Lattarini [Tue, 2 Aug 2011 21:34:04 +0000 (23:34 +0200)]
* NEWS: add mention of new experimental TAP support
Stefano Lattarini [Tue, 2 Aug 2011 20:56:13 +0000 (22:56 +0200)]
testsuite: rename some test scripts
* tests/parallel-tests-am_tests_environment.test: Renamed ...
* tests/am-tests-environment.test: ... to this.
* tests/check-tests_environment.test: Renamed ...
* tests/tests-environment.test: ... to this.
* tests/parallel-test-driver-install.test: Renamed ...
* tests/parallel-tests-driver-install.test: ... to this.
* tests/parallel-tests-make-n.test: Renamed ...
* tests/parallel-tests-dry-run.test: ... to this.
* tests/Makefile.am (TESTS): Updated.
Stefano Lattarini [Tue, 2 Aug 2011 20:24:28 +0000 (22:24 +0200)]
coverage: parallel-tests and lazy dependencies on EXTRA_PROGRAMS
* tests/parallel-tests-extra-programs.test: New test.
* tests/Makefile.am (TESTS): Update.
* doc/automake.texi (Parallel Test Harness): Add a comment
pointing to the new test.
Stefano Lattarini [Tue, 2 Aug 2011 10:05:20 +0000 (12:05 +0200)]
docs: improve, extend and fix documentation on TAP support
* doc/automake.texi ("Using the TAP test protocol"): Divide this
section into ...
("Introduction to TAP", "Use TAP with the Automake test harness",
"Incompatibilities with other TAP parsers and drivers", "Links
and external resources"): ... these subsections, extend them by
adding more information and examples, and improve them by removing
incomplete and/or temporary wordings and TODO items.
("Script-based Testsuites", "Parallel Test Harness"): Add a couple
of anchors to improve the granularity of cross-references.
* tests/tap-doc2.test: New test, verifying the correctness of the
new examples given in the manual.
* tests/Makefile.am (tap_other_tests): Add the new test.
Stefano Lattarini [Mon, 1 Aug 2011 17:10:41 +0000 (19:10 +0200)]
testsuite: separate the only failing check of an xfailing test
* tests/plan-bad-prog.test: Move the only failing check of this
test (i.e., the one about the total number of "ERROR" outcomes)
into ...
* tests/plan-bad-prog2.test: ... this new test.
* tests/Makefile.am (XFAIL_TESTS): Remove `plan-bad-prog.test',
add `plan-bad-prog2.test'.
(tap_other_tests): Add `plan-bad-prog2.test'.
(plan-bad-prog2.log): Depend on `plan-bad-prog.test'.
Stefano Lattarini [Mon, 1 Aug 2011 16:43:30 +0000 (18:43 +0200)]
testsuite: fix spurious errors in an xfailing test on TAP support
* tests/plan-bad-prog.test: Fix typo in the name of the test
being made unreadable. Correct the wording of the potential
skip message, and simplify the condition under which the test
is to be skipped. Escape literal dots in grep regexps.
Stefano Lattarini [Mon, 1 Aug 2011 16:32:17 +0000 (18:32 +0200)]
testsuite: better granularity in a couple of tests on TAP support
* tests/tap-summary-aux.sh: New auxiliary script, filled with code
moved out from ...
* tests/tap-summary.test: ... this test, from which the checks on
colored testsuite have further been moved into ...
* tests/tap-summary-color.test: ... this new test, which in turn
* tests/Makefile.am (tap_other_tests): Add `tap-summary-color.test'.
(EXTRA_DIST): Distribute `tap-summary-aux.sh'
(tap-summary.log): Depend on `tap-summary-aux.sh'.
(tap-summary-color.log): Likewise.
Stefano Lattarini [Mon, 1 Aug 2011 14:14:44 +0000 (16:14 +0200)]
testsuite: optimize tests on TAP for speed and against duplication
The creation and configuration of common files and data used by
many tests on TAP is, with this change, factored out into a new
dedicated auxiliary test, from which those tests will depend upon.
This reduces code duplication in tests and, more importantly,
offers a noticeable speedup in the involved tests (30-40%).
* tests/tap-common-setup.test: New test, setting up the common
files and data used by various tests on TAP support.
* tests/tap-setup.sh: New file, to be sourced by tests wanting to
bring in data generated by `tap-common-setup.test'.
* tests/tap-autonumber.test: Update to use the pre-computed data
files.
* tests/tap-bailout.test: Likewise.
* tests/tap-color.test: Likewise.
* tests/tap-deps.test: Likewise.
* tests/tap-diagnostic.test: Likewise.
* tests/tap-empty-diagnostic.test: Likewise.
* tests/tap-empty.test: Likewise.
* tests/tap-escape-directive.test: Likewise.
* tests/tap-exit.test: Likewise.
* tests/tap-signal.test: Likewise.
* tests/tap-fancy.test: Likewise.
* tests/tap-fancy2.test: Likewise.
* tests/tap-global-log.test: Likewise.
* tests/tap-global-result.test: Likewise.
* tests/tap-html.test: Likewise.
* tests/tap-log.test: Likewise.
* tests/tap-merge-stdout-stderr.test: Likewise.
* tests/tap-no-merge-stdout-stderr.test: Likewise.
* tests/tap-message-0.test: Likewise.
* tests/tap-no-disable-hard-error.test: Likewise.
* tests/tap-no-spurious-summary.test: Likewise.
* tests/tap-no-spurious.test: Likewise.
* tests/tap-not-ok-skip.test: Likewise.
* tests/tap-numeric-description.test: Likewise.
* tests/tap-out-of-order.test: Likewise.
* tests/tap-passthrough.test: Likewise.
* tests/tap-passthrough-exit.test: Likewise.
* tests/tap-plan.test: Likewise.
* tests/tap-plan-corner.test: Likewise.
* tests/tap-plan-corner2.test: Likewise.
* tests/tap-plan-errors.test: Likewise.
* tests/tap-realtime.test: Likewise.
* tests/tap-recheck-logs.test: Likewise.
* tests/tap-skip-whole.test: Likewise.
* tests/tap-todo-skip-together.test: Likewise.
* tests/tap-todo-skip-whitespace.test: Likewise.
* tests/tap-todo-skip.test: Likewise.
* tests/tap-unplanned.test: Likewise.
* tests/tap-with-and-without-number.test: Likewise.
* tests/tap-xfail-tests.test: Likewise.
* tests/tap-skip-whole-whitespace.test: Likewise, and remove
redundant definitions of `$sp' and `$tab' (they are already
defined in `tests/defs').
* tests/tap-whitespace-normalization.test: Likewise.
* tests/Makefile.am (TESTS): Update.
(EXTRA_DIST): Distribute `tap-setup.sh'.
(tap_with_common_setup_tests, tap_with_common_setup_logs)): New
variables, holding respectively the list of tests using the files
pre-computed by `tap-common-setup.test', and the list of their
corresponding log files.
(tap_other_tests): New variable, holding the list of other tests
on TAP support.
($(tap_with_common_setup_logs)): Depend on `tap-common-setup.log'
and `tap-setup.sh'.
Stefano Lattarini [Thu, 28 Jul 2011 10:25:22 +0000 (12:25 +0200)]
test harness: use new `.trs' files to hold test metadata
With this change, the test harness will keep test metadata in
dedicated `.trs' files, instead of having them embedded into the
`.log' files. This allows for easier forward-compatibility and
extension of test metadata, and for more flexibility in the
format of the `.log' files. Note that this change makes the
`:end-metadata:' field obsolete.
* doc/automake.texi (Parallel Test Harness, Log files generation
and test results recording): Document the new APIs and behaviour;
some related minor rewordings and fixlets.
* NEWS: Update.
* automake.in (handle_tests): When bringing in the content of
`check2.am', substitute %BASE% with the basename of the `.log'
file being created by a rule. Add the generated `.trs' files
to the list of files to be cleaned by "make mostlyclean".
* lib/am/check.am (am__test_driver_flags): Rename ...
(am__common_driver_flags): ... to this, and remove the flags
`--test-name' and `--log-file' from it: they are now define in
the proper rules in `check2.am'.
(am__TEST_BASES): New internal variable, holding the names of
the tests, with any registered extension removed.
(am__stealth_MAKE): New internal variable, can be used instead of
$(MAKE) in recipes requiring a recursive call to make, but which
are not intended to be executed by "make -n".
(.log.trs): New suffix rule, to recover from deletion of `.trs'
files.
($(TEST_SUITE_LOG)): Almost completely rewritten to follow the
new API of "test logs in `.log' files, test metadata in `.trs'
files". It goes to some length to work correctly in face of
unreadable or missing `.log' and `.trs' files, and to error out
with proper error messages when this is not possible.
[%?PARALLEL_TESTS%] (check-TESTS): Also remove relevant "stale"
`.trs' files (in addition to `.log files) before remaking the
$(TEST_SUITE_LOG).
(recheck, recheck-html): Look for the `:recheck:' field in the
`.trs' files, not in the `.log' files.
* lib/am/check2.am (?GENERIC?%EXT%.log, ?!GENERIC?%OBJ%): Adjust
the call to the test driver, in particularly passing the new
option `--trs-file'.
[%am__EXEEXT%] (?GENERIC?%EXT%$(EXEEXT).log): Likewise.
* lib/tap-driver ($USAGE): Adjust the help screen.
(Getopt::Long::GetOptions): Handle the `--trs-file' option,
through the use of ...
($trs_file): ... this new global variable.
(finish): Write metadata for the test run to `$trs_file' rather
then to `$log_file', through the use of ...
(write_test_results): ... this new function.
* lib/test-driver (print_usage): Update the help screen.
(Option parsing): Handle the `--trs-file' option, through the
use of ...
($resfile): ... this new global variable.
(Main code): Write metadata for the test run to `$trsfile' rather
than to `$logfile'.
Minor related adjustments to comments.
* tests/.gitignore: Ignore `*.trs' files.
* tests/parallel-tests-unreadable-log.test: Moved ...
* tests/parallel-tests-unreadable.test: ... to this, and extended
to also check the semantics for unreadable `.trs' files.
* tests/test-driver-end-metadata.test: Deleted as obsolete.
* tests/test-driver-metadata-no-leading-space.test: Likewise.
* tests/test-driver-global-log.test: Renamed ...
* tests/test-metadata-global-log.test: ... to this, and modified
as to verify the new APIs and semantics.
* tests/test-driver-recheck: Renamed ...
* tests/test-metadata-recheck.test: ... to this, and modified
likewise.
* tests/parallel-tests-once.test: New test.
* tests/parallel-tests-make-n.test: Likewise.
* test-metadata-results.test: Likewise.
* test-missing.test: Likewise.
* test-missing2.test: Likewise.
* test-trs-basic.test: Likewise.
* test-trs-recover.test: Likewise.
* test-trs-recover2.test: Likewise.
* tests/Makefile.am (TESTS): Update.
Stefano Lattarini [Tue, 26 Jul 2011 12:51:20 +0000 (14:51 +0200)]
test harness: allow more metadata in log files
This change reworks and improves the parallel test harness to use
more specialized reStructuredText fields in the log files (instead
of relying on specially-placed of "magic lines" and more indirect
semantical formatting); the new fields are the following:
- ":recheck:": tell whether the associated test will have to be
re-run by "make recheck";
- ":copy-in-global-log:": tell whether the content of the log
file should be copied in the "global log" `test-suite.log';
- ":end-metadata:", which inhibits the scanning of the rest of
the log file (for what concerns test metadata).
Also, the special `:test-result:' value "END" has been removed,
superseded by the new `:end-metadata:' field.
* doc/automake.texi (Log files generation and test results
recording): Document the new API and semantics. Remove or fix
some obsolete comments.
* lib/am/check.am ($(TEST_SUITE_LOG), recheck, recheck-html):
Adjust comments and code.
* lib/tap-driver (finish): Adjust, with the help of ...
(must_recheck, copy_in_global_log): ... these new functions.
* lib/test-driver (Main code): Adjust, with the help of ...
($recheck, $gcopy): ... these new variables.
* tests/trivial-test-driver: Update to obey the new APIs.
* tests/test-driver-recheck.test: Likewise.
* tests/test-driver-global-log.test: Likewise.
* tests/tap-passthrough.test: Relax the test, avoiding to check
what is written in `test-suite.log'; such check has been moved ...
* tests/tap-global-log.test: ... in this new test, and extended.
* tests/test-driver-metadata-no-leading-space.test: New test.
* tests/test-driver-end-test-results.test: Removed, it checked
the old APIs; superseded by ...
* tests/test-driver-end-metadata.test: ... this new test.
* tests/tap-log.test: Improve syncing with ...
* tests/test-log.test: ... this new test.
* tests/parallel-tests.test: Remove some duplication w.r.t. this
last new test. Updated heading comments.
* tests/Makefile.am (TESTS): Update.
Stefano Lattarini [Wed, 27 Jul 2011 14:34:30 +0000 (16:34 +0200)]
testsuite: fix a spurious failure with non-bash shells
* tests/tap-fancy2.test: Remove an unportable use of backslashes
with the `echo' builtin, which was causing some shells (among them
bash and Debian /bin/ksh) to print a `\\' string, while other
shells (among them zsh, dash, and Solaris /bin/sh and /bin/ksh)
were unexpectedly printing a single `\' character. Since we are
at it, add a sanity check to ensure that this issue does not
resurface.
Stefano Lattarini [Wed, 27 Jul 2011 13:33:13 +0000 (15:33 +0200)]
testsuite: fix spurious failures with Solaris /bin/sh
* tests/tap-more.test: Use `echo > file', not `: > file', to
create empty files in the "for" loops; this is required since,
as documented in the autoconf manual, Solaris 10 /bin/sh
"optimizes" away the `:' command after the first iteration,
even if it is redirected.
* tests/test-driver-custom-multitest-recheck2.test: Likewise.
* tests/tap-numeric-description.test: Partial rewrite to avoid
using positional parameters from the 10th onward, which are
unportable to Solaris /bin/sh (using `${10}' causes the shell
to die with "bad substitution").
Stefano Lattarini [Sun, 24 Jul 2011 14:47:24 +0000 (16:47 +0200)]
simple tests: support developer-defined fd redirections
Motivated by coreutils bug#8846, and related discussions:
<http://debbugs.gnu.org/cgi/bugreport.cgi?bug=8846>
<http://lists.gnu.org/archive/html/bug-autoconf/2011-06/msg00002.html>
In those threads it has been shown how problematic it can be to do
portable file descriptor redirections in the testsuite when using
the Automake testsuite harness. This change should remedy to that
situation.
* lib/am/check2.am (?GENERIC?%EXT%$(EXEEXT).log,
?GENERIC?%EXT%.log, ?!GENERIC?%OBJ%): Append redirection defined
in $(AM_TESTS_FD_REDIRECT) to the command-line invocations of the
test scripts.
* lib/am/check.am [!%?PARALLEL_TESTS%] $(check-TESTS): Likewise.
* NEWS: Update.
* doc/automake.texi (Script-based Testsuites): Document the new
feature.
* tests/check-fd-redirect.test: New test.
* tests/parallel-tests-fd-redirect.test: Likewise.
* tests/parallel-tests-am_tests_environment.test: Remove checks
about the use of redirections in AM_TESTS_ENVIRONMENT: they would
check deprecated (if not undefined) behaviour now. Strengthen a
couple of still valid checks, to keep the test more in sync with
the documentation. Improve debugging information.
* tests/Makefile.am (TESTS): Update.
Stefano Lattarini [Thu, 21 Jul 2011 15:07:54 +0000 (17:07 +0200)]
docs: rework and extend documentation on testsuites support
* doc/automake.texi (Generalities about Testing): New section.
(Simple Tests using parallel-tests): Section removed, merged
into ...
(Simple Tests): ... this one, which on the other hand has been
subdivided into ...
(Script-based Testsuites, Serial Test Harness, Parallel Test
Harness): ... these new subsection.
(DejaGnu Tests): Minor adjustments.
Other related typofixes and rewordings throughout the manual;
in particular, avoid to use the term "test driver" for three
different concepts (and use instead "test harness" and "test
runner" where appropriate).
* tests/tap-doc.test: New test.
* tests/tap-no-disable-hard-error.test: Likewise.
* tests/Makefile.am (TESTS): Update.
Stefano Lattarini [Sun, 17 Jul 2011 14:46:41 +0000 (16:46 +0200)]
tap: support colorization of testsuite progress output
* lib/tap-driver (%COLORS): New variable (definition extracted
from `lib/am/check.am:$(am__tty_colors)', with some obvious
adjustments.
(report): Adjust to colorize console output when required,
using ...
(decorate_result): ... this new function.
(colored): New function, used by the one above.
* tests/tap-summary.test: Also run the checks when `color-tests'
is in use.
* tests/Makefile.am (XFAIL_TESTS): Remove `tap-color.test'.
Stefano Lattarini [Sun, 17 Jul 2011 14:04:06 +0000 (16:04 +0200)]
tap: some preparatory refactoring (2)
This is a follow-up simplification.
* lib/tap-driver (console_output): Renamed ...
(report): ... to this, and extended to appropriately register
the test results when needed.
(testsuite_error, handle_tap_comment, handle_tap_test,
handle_tap_plan): Adjusted accordingly.
Stefano Lattarini [Sun, 17 Jul 2011 13:18:51 +0000 (15:18 +0200)]
tap: some preparatory refactoring (1)
This refactoring is aimed at simplifying the introduction of
colored console output for the TAP driver.
* lib/tap-driver (console_output): Now accept two arguments, the
first one indicating which kind of thing is to be displayed (for
now only a test result or a diagnostic comment), and the second
one (if present) the message associated to it.
(handle_tap_test, handle_tap_comment, handle_tap_plan,
testsuite_error): Adapt to the new `console_output' interface.
Stefano Lattarini [Sun, 3 Jul 2011 17:20:22 +0000 (19:20 +0200)]
tap: add experimental TAP-aware driver
* doc/automake.texi (Using the TAP test protocol): New section.
(Overview of Custom Test Drivers Support): Minor updates.
* lib/tap-driver: New script, TAP-aware test driver for Automake;
implemented in perl and based on TAP::Parser.
* lib/Makefile.am (dist_script_DATA): Add it.
* tests/tap-autonumber.test: New test.
* tests/tap-bailout.test: Likewise.
* tests/tap-basic.test: Likewise.
* tests/tap-deps.test: Likewise.
* tests/tap-diagnostic.test: Likewise.
* tests/tap-empty.test: Likewise.
* tests/tap-empty-diagnostic.test: Likewise.
* tests/tap-escape-directive.test: Likewise.
* tests/tap-exit.test: Likewise.
* tests/tap-fancy.test: Likewise.
* tests/tap-fancy2.test: Likewise.
* tests/tap-global-result.test: Likewise.
* tests/tap-html.test: Likewise.
* tests/tap-log.test: Likewise.
* tests/tap-merge-stdout-stderr.test: Likewise.
* tests/tap-more.test: Likewise.
* tests/tap-more2.test: Likewise.
* tests/tap-no-merge-stdout-stderr.test: Likewise.
* tests/tap-no-spurious-summary.test: Likewise.
* tests/tap-no-spurious.test: Likewise.
* tests/tap-not-ok-skip.test: Likewise.
* tests/tap-numeric-description.test: Likewise.
* tests/tap-out-of-order.test: Likewise.
* tests/tap-passthrough.test: Likewise.
* tests/tap-plan.test: Likewise.
* tests/tap-plan-errors.test: Likewise.
* tests/tap-plan-corner.test: Likewise.
* tests/tap-realtime.test: Likewise.
* tests/tap-recheck-logs.test: Likewise.
* tests/tap-recheck.test: Likewise.
* tests/tap-skip-whole.test: Likewise.
* tests/tap-summary.test: Likewise.
* tests/tap-todo-skip.test: Likewise.
* tests/tap-todo-skip-together.test: Likewise.
* tests/tap-todo-skip-whitespace.test: Likewise.
* tests/tap-skipall-whitespace.test: Likewise.
* tests/tap-unplanned.test: Likewise.
* tests/tap-whitespace-normalization.test: Likewise.
* tests/tap-with-and-without-number.test: Likewise.
* tests/tap-xfail-tests.test: Likewise.
* tests/tap-bad-prog.test: New xfailing test.
* tests/tap-color.test: Likewise.
* tests/tap-plan-corner2.test: Likewise.
* tests/tap-message-0.test: Likewise.
* tests/tap-signal.test: Likewise.
* tests/Makefile.am (TESTS, XFAIL_TESTS): Update.
Stefano Lattarini [Mon, 4 Jul 2011 16:12:07 +0000 (18:12 +0200)]
test defs: new auxiliary function 'count_test_results'
* tests/defs (count_test_results): New function.
* tests/check11.test: Use it.
* tests/test-driver-custom-multitest.test: Likewise.
* tests/test-driver-custom-multitest-recheck.test: Likewise.
* tests/test-driver-custom-multitest-recheck2.test: Likewise.
* tests/parallel-tests-log-override-recheck.test: Likewise.
* tests/parallel-tests-log-override-recheck.test: Likewise.
* tests/parallel-tests-no-spurious-summary.test: Likewise, and
slightly improve debugging output.
* tests/parallel-tests.test: Make use of `count_test_results'.
Also, make grepping of "make check" output slightly stricter
* tests/parallel-tests9.test: Likewise.
* tests/parallel-tests-log-override-2.test: Likewise, and throw
in a small optimization.
Stefano Lattarini [Fri, 1 Jul 2011 09:12:31 +0000 (11:12 +0200)]
parallel-tests: simplify testsuite summary
Prefer a more deterministic, "tabular" format for the testsuite
summary, always listing the numbers of passed, failed, xfailed,
xpassed, skipped and errored tests, even when these numbers are
zero. This simplify the logic of testsuite summary creation,
makes it more easily machine-parseable, and will probably allow
for easier addition of new kinds of test results in the future.
* lib/am/check.am (am__tty_colors_dummy): New make variable, to
reduce code duplication. Extracted from previous versions of
$(am__tty_colors), and extended by defining two new variables
`$mgn' and `$brg'.
[%?COLOR%, %!?COLOR%] (am__tty_colors): Use that new variable.
(am__text_box): Delete, is not needed anymore.
($(TEST_SUITE_LOG)): Rewrite associated rules to implement the
new testsuite summary format.
* NEWS: Update.
* tests/check10.test: Don't run with the parallel-tests harness
too, that makes no sense anymore.
* tests/color.test: Update and adjust.
* tests/color2.test: Likewise.
* tests/parallel-tests.test: Likewise.
* tests/parallel-tests3.test: Likewise.
* tests/parallel-tests6.test: Likewise.
* tests/parallel-tests9.test: Likewise.
* tests/parallel-tests-unreadable-log.test: Likewise.
* tests/parallel-tests-empty-testlogs.test: Likewise.
* tests/parallel-tests-log-override-recheck.test: Likewise.
* tests/parallel-tests-no-spurious-summary.test: Likewise.
* tests/test-driver-custom-multitest.test: Likewise.
* tests/test-driver-end-test-results.test: Likewise.
* tests/parallel-tests-no-color-in-log.test: New test.
* tests/testsuite-summary-color.test: Likewise.
* tests/testsuite-summary-count.test: Likewise.
* tests/testsuite-summary-count-many.test: Likewise.
* tests/testsuite-summary-reference-log.test: Likewise.
* tests/testsuite-summary-checks.sh: New auxiliary script, used
by the new tests above.
* tests/extract-testsuite-summary: Likewise.
* tests/trivial-test-driver: Optimize for speed when there are
lots of of tests.
* tests/Makefile.am (EXTRA_DIST): Distribute them.
(testsuite-summary-color.log, testsuite-summary-count.log): Depend
on them.
(testsuite-summary-count-many.log): Depend on the auxiliary scripts
'trivial-test-driver' and 'extract-testsuite-summary'.
(TESTS): Update.
Stefano Lattarini [Thu, 30 Jun 2011 12:55:43 +0000 (14:55 +0200)]
parallel-tests: new recognized test result 'ERROR'
* lib/am/check.am ($(TEST_SUITE_LOG)): Recognize a new test result
`ERROR'. Use it when encountering unreadable test logs (previously
a simple `FAIL' was used in this situations).
* lib/test-driver: Set the global test result to `ERROR' when the
test exit status is 99. When doing colorized output, color `ERROR'
results in magenta.
* doc/automake.texi (Log files generation and test results
recording): Update by also listing `ERROR' among the list of valid
`:test-results:' arguments.
* NEWS: Update.
* tests/trivial-test-driver: Update.
* tests/parallel-tests.test: Likewise.
* tests/parallel-tests-harderror.test: Likewise.
* tests/parallel-tests-no-spurious-summary.test: Likewise.
* tests/test-driver-global-log.test: Likewise.
* tests/test-driver-recheck.test: Likewise.
* tests/test-driver-custom-multitest-recheck.test: Likewise.
* tests/test-driver-custom-multitest-recheck2.test: Likewise.
* tests/test-driver-custom-multitest.test: Likewise.
* tests/test-driver-custom-no-html.test: Likewise.
* tests/test-driver-end-test-results.test: Likewise.
* tests/color.test: Likewise. Also, make stricter, and also test
from VPATH.
* tests/color2.test: Likewise, and improve syncing with color.test.
* tests/parallel-tests-exit-statuses.test: New test.
* tests/parallel-tests-console-output.test: Likewise.
* tests/Makefile.am (TESTS): Update.
Stefano Lattarini [Wed, 29 Jun 2011 14:02:52 +0000 (16:02 +0200)]
parallel-tests: make parsing of test results safer
The new code for parsing the testsuite-generated `.log' files,
as introduced in commit `v1.11-872-gc96b881', considers each
`:test-result:' field anywhere in a `.log' file as a declaration
of a test result, and accounts for it as such in the testsuite
summary. Unfortunately this could easily cause spurious test
failures being reported in the testsuite summary. This happened
in practice with the Automake's own testsuite; for example:
$ make check TESTS='check12-p.test'; echo exit: $?
...
PASS: check12-p.test
=====================================
4 of 5 tests failed
See tests/test-suite.log
Please report to bug-automake@gnu.org
=====================================
make[2]: *** [test-suite.log] Error 1
make: *** [check-am] Error 2
exit: 2
This change introduces a new special `:test-result:' "END", that,
when seen, prevents the rest of the log file from being parsed.
For more information, refer to the thread:
<http://lists.gnu.org/archive/html/automake-patches/2011-06/msg00199.html>
* lib/am/check.am ($(TEST_SUITE_LOG)): Stop the parsing of a log
file as soon as the special ":test-result:END" directive is seen.
Related changes and enhancements.
* lib/test-driver: Protect the rest of the log after the result
lined with a ":test-result:END" directive.
* tests/parallel-tests-no-spurious-summary.test: New test.
* tests/test-driver-end-test-results.test: Likewise.
* tests/Makefile.am (TESTS): Update.
Stefano Lattarini [Wed, 22 Jun 2011 19:59:34 +0000 (21:59 +0200)]
docs: document custom test drivers and protocols
* doc/automake.texi (Simple Tests): Note that the TESTS_ENVIRONMENT
use suggested here is not portable to 'parallel-tests'.
(Simple Tests using parallel-tests): Document new restrictions on
the uses of TESTS_ENVIRONMENT and AM_TESTS_ENVIRONMENT.
(Custom Test Drivers): New section and node.
(Overview of Custom Test Drivers Support): New subsection.
(Declaring Custom Test Drivers in @file{Makefile.am}): Likewise.
(APIs for Custom Test Drivers): Likewise.
(Options): Update description of color-tests.
* lib/am/check ($(TEST_SUITE_LOG)): Remove comments that have been
moved into the manual.
(recheck, recheck-html): Minor adjustments to better conform to the
documentation (this should cause no semantic changes w.r.t. the
former behaviour); minor improvements and extensions to existing
comments.
* tests/test-driver-create-log-dir.test: New test.
* tests/test-driver-strip-vpath.test: Likewise.
* tests/test-driver-global-log.test: Likewise.
* tests/test-driver-recheck.test: Likewise.
* tests/Makefile.am (TESTS): Update.
Stefano Lattarini [Wed, 29 Jun 2011 08:06:20 +0000 (10:06 +0200)]
Merge branch 'master' into test-protocols
* master:
docs: explain why AM_TESTS_ENVIRONMENT must be semicolon-terminated
docs: fix unportable example of AM_TESTS_ENVIRONMENT usage
docs: avoid a footnote, some related rewordings and improvements
Revert "docs: parallel-tests is not experimental anymore"
docs: minor cosmetic fixes
help: improve text about automatically-distributed files
refactor: split 'usage' subroutine in automake
tests: fix bug in 'autodist.test'
parallel-tests: stricter checks on DISABLE_HARD_ERRORS support
docs: parallel-tests is not experimental anymore
check: document and test $(TEST_SUITE_LOG) overriding
Stefano Lattarini [Wed, 29 Jun 2011 07:53:31 +0000 (09:53 +0200)]
Merge branch 'am-tests-environment'
* am-tests-environment:
docs: explain why AM_TESTS_ENVIRONMENT must be semicolon-terminated
docs: fix unportable example of AM_TESTS_ENVIRONMENT usage
Stefano Lattarini [Wed, 29 Jun 2011 07:44:14 +0000 (09:44 +0200)]
docs: explain why AM_TESTS_ENVIRONMENT must be semicolon-terminated
* doc/automake.texi (Simple Tests using parallel-tests): Ditto, and
related adjustments.
Suggestion by Ralf Wildenhues.
Stefano Lattarini [Thu, 23 Jun 2011 17:05:44 +0000 (19:05 +0200)]
docs: fix unportable example of AM_TESTS_ENVIRONMENT usage
* doc/automake.texi (Simple Tests using parallel-tests): The
old example on AM_TESTS_ENVIRONMENT relied on unportable shell
features, and in particular didn't work with various Korn
Shells (see also commit `v1.11-925-g29ca903'). Give another
example, simpler this time, but still inspired to real-world
usage (the GNU coreutils testsuite).
Stefano Lattarini [Tue, 28 Jun 2011 09:26:15 +0000 (11:26 +0200)]
Merge branch 'maint'
* maint:
Revert "docs: parallel-tests is not experimental anymore"
parallel-tests: stricter checks on DISABLE_HARD_ERRORS support
docs: parallel-tests is not experimental anymore
check: document and test $(TEST_SUITE_LOG) overriding
Stefano Lattarini [Tue, 28 Jun 2011 06:15:21 +0000 (08:15 +0200)]
Merge branch 'parallel-tests-maint' into maint
* parallel-tests-maint:
Revert "docs: parallel-tests is not experimental anymore"
parallel-tests: stricter checks on DISABLE_HARD_ERRORS support
docs: parallel-tests is not experimental anymore
check: document and test $(TEST_SUITE_LOG) overriding
Stefano Lattarini [Mon, 27 Jun 2011 13:36:37 +0000 (15:36 +0200)]
Merge branch 'maint'
* maint:
docs: avoid a footnote, some related rewordings and improvements
docs: minor cosmetic fixes
Stefano Lattarini [Thu, 23 Jun 2011 21:31:27 +0000 (23:31 +0200)]
docs: avoid a footnote, some related rewordings and improvements
* doc/automake.texi (Dist): Reword the part about automatically
distributed files to avoid a footnote. Since we are at it, extend
a bit, and add an example and a reference to a relevant test case.
Stefano Lattarini [Fri, 24 Jun 2011 07:15:14 +0000 (09:15 +0200)]
Revert "docs: parallel-tests is not experimental anymore"
This reverts commit
a9eef973b5ea47cc3495f1a8307d4f7b85aea46f.
It turned out that the current work to introduce TAP and SubUnit
support in Automake-generated testsuite harnesses will probably
require the introduction of slight incompatibilities in the
'parallel-tests' behaviour, starting from release 1.12 onward.
So it's advisable to continue to characterize the 'parallel-tests'
support as "experimental" in maintenance release 1.11.2.
Suggestion from Ralf Wildenhues.
Stefano Lattarini [Thu, 23 Jun 2011 16:38:34 +0000 (18:38 +0200)]
docs: minor cosmetic fixes
* doc/automake.texi: Break few overly long lines, throughout the
file.
("Simple Tests"): Move @vindex for XFAIL_TESTS to the correct
position, i.e., before and not after the paragraph where it is
introduced.
("Options" @item ansi2knr): Use @pxref instead of @xref. This
fixes a texinfo warning.
("Other things Automake recognizes" @item AM_C_PROTOTYPES): Use
@pxref instead of @ref.
Stefano Lattarini [Thu, 23 Jun 2011 08:52:08 +0000 (10:52 +0200)]
Merge branch 'maint'
* maint:
help: improve text about automatically-distributed files
refactor: split 'usage' subroutine in automake
Stefano Lattarini [Wed, 12 Jan 2011 00:27:07 +0000 (01:27 +0100)]
help: improve text about automatically-distributed files
This change fixes automake bug#7819.
* automake.in (usage): Distinguish between files that are always
automatically distributed when found, and those which are only
"under certain conditions".
* doc/automake.texi (Basics of Distribution): Update accordingly.
* tests/autodist-subdir.test: Update.
* tests/autodist-no-duplicate.test: Likewise.
* tests/autodist.test: Likewise.
(configure.in): Remove useless call to AM_MAINTAINER_MODE.
Stefano Lattarini [Wed, 12 Jan 2011 00:06:33 +0000 (01:06 +0100)]
refactor: split 'usage' subroutine in automake
This change is related to automake bug#7819.
* automake.in (print_autodist_files): New subroutine,
extracted from ...
(usage): ... this, which now uses it.
* tests/autodist-no-duplicate.test: New test.
* tests/Makefile.am (TESTS): Update.
Stefano Lattarini [Thu, 23 Jun 2011 08:34:39 +0000 (10:34 +0200)]
Merge branch 'maint'
* maint:
tests: fix bug in 'autodist.test'
Stefano Lattarini [Thu, 23 Jun 2011 08:32:12 +0000 (10:32 +0200)]
tests: fix bug in 'autodist.test'
* tests/autodist.test: Avoid spurious failure due to no
`defs-static' file being found in the parent directory.
Stefano Lattarini [Fri, 3 Jun 2011 20:50:56 +0000 (22:50 +0200)]
parallel-tests: allow each test to have multiple results
With this change, we improve the code creating the `test-suite.log'
global log and the console testsuite summary to make it able to
grasp multiple results per test script. This is required in order
to introduce the planned support for test protocols, like TAP and
SubUnit, which can indeed run multiple tests per test script, each
with its individual result.
The implementation makes use of a custom reStructuredText field
`:test-result:'.
Note that no new documentation is added by this change; that is
be left for follow-up changes.
* lib/check.am ($(TEST_SUITE_LOG)): When processing .log files,
recognize a report of a test's result only if it is declared with
the custom `:test-result:' reStructuredText field placed at the
beginning of a line. Extend and add explanatory comments.
(recheck, recheck-html): Add explanatory comments.
* lib/test-driver: Write an appropriate reStructuredText field
`:test-result:' in the generated log file. Use a reStructuredText
transition to better separate the test outcome report from the
test script's registered output. Improve comments.
* tests/test-driver-custom-xfail-tests.test: Adjust.
* tests/parallel-tests7.test: Adjust.
* tests/parallel-tests-empty-testlogs.test: New test.
* tests/parallel-tests-recheck-override.test: Likewise.
* tests/parallel-tests2.test: Extend and keep more in-sync with ...
* tests/test-driver-custom-html.test: ... this new related test.
* tests/test-driver-custom-no-html.test: New test.
* tests/test-driver-custom-multitest.test: Likewise.
* tests/test-driver-custom-multitest-recheck.test: Likewise.
* tests/test-driver-custom-multitest-recheck2.test: Likewise.
* tests/trivial-test-driver: New file, used by the last four tests
above.
* tests/Makefile.am (TESTS): Update.
(EXTRA_DIST): Distribute `trivial-test-driver'.
(test-driver-custom-multitest.log): Depend on `trivial-test-driver'.
(test-driver-custom-multitest-recheck.log): Likewise.
(test-driver-custom-multitest-recheck2.log): Likewise.
(test-driver-custom-html.log): Likewise.
Stefano Lattarini [Fri, 20 May 2011 19:45:51 +0000 (21:45 +0200)]
parallel-tests: allow custom driver scripts
Allow suffix-based definition of custom "driver script" for the
test scripts. These driver scripts will be responsible of
launching the tests (or their corresponding $(LOG_COMPILER), if
they have an associated one), interpreting and displaying the
test results, and writing the `.log' files.
This new API should allow easy and flexible use of different
test protocols in the future; in particular, we plan to use it
to implement TAP and SubUnit harnesses.
Note that no new documentation is added by this change; that is
be left for follow-up changes.
* automake.in (handle_tests): Define default for $(LOG_DRIVER),
and, for any registered test extension `<ext>', define defaults
for $(<ext>_LOG_DRIVER). Substitute %DRIVER% using these new
variables, instead of the old internal $(am__test_driver). When
processing check2.am, also substitute %DRIVER_FLAGS%.
Require auxiliary script `test-driver' only if no driver has been
explicitly defined for the test script kinds.
* am/check2.am (?GENERIC?%EXT%$(EXEEXT).log, ?GENERIC?%EXT%.log,
?!GENERIC?%OBJ%): Pass the %DRIVER_FLAGS% to the %DRIVER% call.
* tests/parallel-tests-no-extra-driver.test: New test.
* tests/test-driver-custom.test: Likewise.
* tests/test-driver-custom-xfail-tests.test: Likewise.
* tests/test-driver-fail.test: Likewise.
* tests/Makefile.am: Update.
* NEWS: Update.
Stefano Lattarini [Tue, 10 May 2011 09:30:05 +0000 (11:30 +0200)]
parallel-tests: add auxiliary script 'test-driver', refactor
This refactoring should cause no API of functionality change,
and is meant only to simplify the future implementation of TAP
and SubUnit testsuite drivers. More precisely, our roadmap is
to move most of the "testsuite driving" features out of the
Automake-generated Makefiles, and into external scripts with
well-defined interfaces. This will allow the user to define
its own personalized testsuite drivers, and will also offer us
a framework upon which to implement our new TAP and SubUnit
drivers, all in a very unobtrusive way and retaining an high
degree of code reuse and backward-compatibility.
* lib/test-driver: New auxiliary script.
* lib/Makefile.am (dist_SCRIPT_DATA): Add it.
* automake.in (handle_tests): Require the new auxiliary script
`test-driver', and define a new internal makefile variable
`$(am__test_driver)', used to call it. Perform new substitution
on `DRIVER' when processing the `check2.am' file.
* lib/check.am (am__tty_colors): Define new shell variable
`$am__color_tests'.
(am__rst_section): Removed, its role taken over by the new
`test-driver' script.
(am__test_driver_flags): New variable, contains the command
line options passed to `test-driver'.
(am__check_pre): Do not deal with temporary files and exit
traps anymore, as the `test-driver' script takes care of that
now. Define shell variable `$am__enable_hard_errors', used by
`$(am__test_driver_flags)'. Reorder so that we don't need to
save and restore the value of the `TERM' environment variable
anymore.
Other related adjustments.
(am__check_post): Remove, as its role has been completely taken
over by the `test-driver' script.
* am/check2.am (?GENERIC?%EXT%$(EXEEXT).log, ?GENERIC?%EXT%.log,
?!GENERIC?%OBJ%): Call the test script through the Automake
substituted `%DRIVER%', and honor the command-line options
in `$(am__test_driver_flags)'. Do not call the obsoleted
`$(am__check_post)' anymore.
* doc/automake.texi (Auxiliary Programs): Mention the new
`test-driver' script.
(Optional): Mention `test-driver' in AC_CONFIG_AUX_DIR.
Since we are at it, break the list of auxiliary scripts by
placing one per line, to simplify potential future additions
of new scripts.
* tests/check.test: Adjust.
* tests/check2.test : Likewise.
* tests/check3.test : Likewise.
* tests/check4.test : Likewise.
* tests/check10.test: Likewise.
* tests/color.test: Likewise.
* tests/color2.test: Likewise.
* tests/comment9.test: Likewise.
* tests/dejagnu.test: Likewise.
* tests/exeext4.test: Likewise.
* tests/maken3.test: Likewise.
* tests/maken4.test: Likewise.
* tests/parallel-tests-interrupt.test: Likewise.
* tests/posixsubst-tests.test: Likewise.
* tests/repeated-options.test: Likewise.
* tests/check-no-test-driver.test: New test.
* tests/parallel-test-driver-install.test: Likewise.
* tests/Makefile.am (TESTS): Update.
* NEWS: Update.
Stefano Lattarini [Tue, 21 Jun 2011 20:24:26 +0000 (22:24 +0200)]
Merge branch 'maint'
Stefano Lattarini [Mon, 20 Jun 2011 21:52:34 +0000 (23:52 +0200)]
maintcheck: extend 'sc_tests_plain_*' checks
* Makefile.am (sc_tests_plain_autom4te): New check.
(sc_tests_plain_autoreconf): Likewise.
(sc_tests_plain_autoheader): Likewise.
(syntax_check_rules): Update.
Stefano Lattarini [Tue, 21 Jun 2011 11:17:50 +0000 (13:17 +0200)]
tests: interactions between TESTS_ENVIRONMENT and LOG_COMPILER
* tests/tests-environment-and-log-compiler.test: New test,
checking that we can use variables and functions set by
TESTS_ENVIRONMENT and AM_TESTS_ENVIRONMENT in LOG_COMPILER
and LOG_FLAGS (for tests both with and without registered
extensions).
* tests/Makefile.am (TESTS): Update.
Stefano Lattarini [Mon, 20 Jun 2011 15:00:37 +0000 (17:00 +0200)]
Merge branch 'maint'
* maint:
maintcheck: avoid few spurious failures
Stefano Lattarini [Mon, 20 Jun 2011 14:42:10 +0000 (16:42 +0200)]
maintcheck: avoid few spurious failures
* Makefile.am (sc_tests_plain_aclocal, sc_tests_plain_perl,
sc_tests_plain_autoconf, sc_tests_plain_automake,
sc_tests_plain_autoupate): Be stricter in matching an erroneous
literal command, i.e., `aclocal', `automake', `perl', etc.
Stefano Lattarini [Mon, 20 Jun 2011 08:49:09 +0000 (10:49 +0200)]
Merge branch 'maint' into parallel-tests-maint
* maint:
check: don't use multi-line coloring for the report
ansi2knr: deprecate, it will go away in the next major release
docs: primary/prefix combination "pkglib_PROGRAMS" is now invalid
docs: replace obsolete @vindex entry with a useful one
docs: AM_DISTCHECK_CONFIGURE_FLAGS is for corner cases
Stefano Lattarini [Mon, 20 Jun 2011 08:46:09 +0000 (10:46 +0200)]
Merge branch 'maint'
* maint:
check: don't use multi-line coloring for the report
ansi2knr: deprecate, it will go away in the next major release
docs: primary/prefix combination "pkglib_PROGRAMS" is now invalid
docs: replace obsolete @vindex entry with a useful one
docs: AM_DISTCHECK_CONFIGURE_FLAGS is for corner cases
Bert Wesarg [Fri, 17 Jun 2011 19:59:52 +0000 (21:59 +0200)]
check: don't use multi-line coloring for the report
"less -R" can't handle multi-line coloring as it is done for the
check reports of the serial and parallel testsuite, because of
performance reasons. Thus, color each line of the check report
by its own.
* lib/am/check.am (am__text_box): Accept colors for lines, and
color each line by its own.
[%?PARALLEL_TESTS%] $(TEST_SUITE_LOG): Let am__text_box handle
the line coloring.
[!%?PARALLEL_TESTS%] $(check-TESTS): Color each report line by
its own.
* THANKS: Update.
Stefano Lattarini [Mon, 20 Jun 2011 07:43:25 +0000 (09:43 +0200)]
Merge branch 'am-distcheck-configure-flags' into maint
Stefano Lattarini [Mon, 13 Jun 2011 19:27:35 +0000 (21:27 +0200)]
ansi2knr: deprecate, it will go away in the next major release
* doc/automake.texi: Loudly and repeatedly state that the old
de-ANSI-fication features are now deprecated and will be removed
in the next major Automake release. Other related adjustments.
* lib/Automake/Options.pm (_process_option_list ): Give a warning
in the `obsolete' category when the `ansi2knr' option is used.
* m4/protos.m4 (AM_C_PROTOTYPES): Deprecate this macro: a warning
in the `obsolete' category will be emitted it if is used.
* tests/ansi2knr-deprecation.test: New test.
* tests/Makefile.am (TESTS): Update.
* tests/ansi.test: Adjust, by calling autoconf and/or automake
with the `-Wno-obsolete' flag.
* tests/ansi10.test: Likewise.
* tests/ansi2.test: Likewise.
* tests/ansi3.test: Likewise.
* tests/ansi3b.test: Likewise.
* tests/ansi4.test: Likewise.
* tests/ansi5.test: Likewise.
* tests/ansi6.test: Likewise.
* tests/ansi7.test: Likewise.
* tests/ansi8.test: Likewise.
* tests/ansi9.test: Likewise.
* tests/cxxansi.test: Likewise.
* tests/libobj8.test: Likewise.
* NEWS: Update about the future planned backward-incompatibility
due to the removal of de-ANSI-fication feature.
Stefano Lattarini [Sun, 19 Jun 2011 11:15:49 +0000 (13:15 +0200)]
docs: primary/prefix combination "pkglib_PROGRAMS" is now invalid
* doc/automake.texi (Program Sources): pkglib_PROGRAMS is not a
valid combination anymore, so don't document it. Inconsistency
introduced in commit `v1.11-373-g9ca6326'.
Jim Meyering [Sun, 19 Jun 2011 10:32:59 +0000 (12:32 +0200)]
docs: replace obsolete @vindex entry with a useful one
* doc/automake.texi (Program Sources): Do not index obsolete
pkglib_PROGRAMS here. Do index pkglibexec_PROGRAMS.
Stefano Lattarini [Sat, 18 Jun 2011 17:16:41 +0000 (19:16 +0200)]
Merge branch 'test-fd-redirect'
* test-fd-redirect:
tests: more checks on portable fd redirection in TESTS_ENVIRONMENT
Stefano Lattarini [Sat, 18 Jun 2011 12:53:08 +0000 (14:53 +0200)]
tests: more checks on portable fd redirection in TESTS_ENVIRONMENT
* tests/tests-environment-fd-redirect.test: Extend by also using
a perl script among the tests. Run the test shell script with
the `errexit' flag active. Export `VERBOSE' to yes when running
"make check", to give more debugging information in case of
failures. Look for a Korn Shell also in `/usr/bin', not on only
in `/bin'.
Stefano Lattarini [Wed, 15 Jun 2011 08:50:03 +0000 (10:50 +0200)]
docs: AM_DISTCHECK_CONFIGURE_FLAGS is for corner cases
* doc/automake.texi (Checking the Distribution): Explain that the
developers should take care of making their code buildable without
requiring any special configure options, so that in general
AM_DISTCHECK_CONFIGURE_FLAGS shouldn't be used. Give an example
of where its use is legitimate.
Stefano Lattarini [Thu, 16 Jun 2011 15:21:00 +0000 (17:21 +0200)]
Merge branch 'maint' into parallel-tests-maint
* maint:
news: update w.r.t. introduction of AM_DISTCHECK_CONFIGURE_FLAGS
tests: optimize tests on primary/prefix mismatch for speed
Warnings about primary/prefix mismatch fixed and extended.
maintcheck: DISTCHECK_CONFIGURE_FLAGS can be defined on make cmdline
distcheck: add support for AM_DISTCHECK_CONFIGURE_FLAGS
docs: better documentation for silent make rules
Stefano Lattarini [Thu, 16 Jun 2011 14:56:36 +0000 (16:56 +0200)]
Merge branch 'maint'
* maint:
news: update w.r.t. introduction of AM_DISTCHECK_CONFIGURE_FLAGS
tests: optimize tests on primary/prefix mismatch for speed
Warnings about primary/prefix mismatch fixed and extended.
maintcheck: DISTCHECK_CONFIGURE_FLAGS can be defined on make cmdline
distcheck: add support for AM_DISTCHECK_CONFIGURE_FLAGS
docs: better documentation for silent make rules
Stefano Lattarini [Tue, 14 Jun 2011 07:41:14 +0000 (09:41 +0200)]
tests: check portable fd redirection in TESTS_ENVIRONMENT
* tests/tests-environment-fd-redirect.test: New test.
* tests/Makefile.am (TESTS): Update.
Motivated by coreutils bug#8846:
<http://debbugs.gnu.org/cgi/bugreport.cgi?bug=8846>
See also following CC:ed thread on bug-autoconf list:
<http://lists.gnu.org/archive/html/bug-autoconf/2011-06/msg00002.html>
Stefano Lattarini [Thu, 16 Jun 2011 08:29:32 +0000 (10:29 +0200)]
Merge branch 'more-prefix-primary-diagnostic' into maint
Stefano Lattarini [Mon, 13 Jun 2011 19:42:06 +0000 (21:42 +0200)]
Merge branch 'am-distcheck-configure-flags' into maint
* am-distcheck-configure-flags:
news: update w.r.t. introduction of AM_DISTCHECK_CONFIGURE_FLAGS
Stefano Lattarini [Mon, 13 Jun 2011 19:40:57 +0000 (21:40 +0200)]
news: update w.r.t. introduction of AM_DISTCHECK_CONFIGURE_FLAGS
* NEWS (Miscellaneous changes): Update.
Stefano Lattarini [Mon, 13 Jun 2011 18:16:16 +0000 (20:16 +0200)]
Merge branch 'am-distcheck-configure-flags' into maint
* am-distcheck-configure-flags:
maintcheck: DISTCHECK_CONFIGURE_FLAGS can be defined on make cmdline
distcheck: add support for AM_DISTCHECK_CONFIGURE_FLAGS
Stefano Lattarini [Mon, 13 Jun 2011 09:07:20 +0000 (11:07 +0200)]
Merge branch 'silent-rules-doc' into maint
Stefano Lattarini [Mon, 13 Jun 2011 08:51:57 +0000 (10:51 +0200)]
tests: optimize tests on primary/prefix mismatch for speed
* tests/primary-prefix-invalid-couples.test: Partial rewrite, in
order to use just a single automake invocation rather than one
invocation for each invalid primary/prefix couple. This improves
the test script execution time by an order of magnitude.
Since we are at it, throw in some other improvements to avoid
unrelated automake warnings and failures that could potentially
cause false positives w.r.t. the automake exit status.
Stefano Lattarini [Thu, 16 Dec 2010 18:23:06 +0000 (19:23 +0100)]
Warnings about primary/prefix mismatch fixed and extended.
* automake.in (%standard_prefix): Add `doc' and `locale'.
Rename `pkgdatadir' to `pkgdata'. Similarly for`pkglibdir',
`pkgincludedir' and `pkglibexecdir'.
(handle_programs): List `pkglibexec', not `pkglib', among the
prefixes valid for the `PROGRAMS' primary.
(handle_data): List also `doc' among the prefixes valid for
the `DATA' primary. This is required by automake's own build
system.
* tests/dirforbid.test: Test removed, superseded by ...
* tests/primary-prefix-invalid-couples.test: ... this new test.
* tests/primary-prefix-valid-couples.test: New test.
* tests/primary-prefix-documented-valid.test: Likewise.
* tests/primary-prefix-force-valid.test: Likewise.
* tests/java3.test: Adjusted, and extended a bit.
* tests/Makefile.am (TESTS): Updated.
* NEWS: Updated.
From a report by Eric Blake.
Stefano Lattarini [Sat, 11 Jun 2011 10:52:51 +0000 (12:52 +0200)]
Merge branch 'maint' into parallel-tests-maint
* maint:
test defs: new function 'fatal_', for hard errors
Stefano Lattarini [Sat, 11 Jun 2011 10:47:56 +0000 (12:47 +0200)]
Merge branch 'maint'
* maint:
test defs: new function 'fatal_', for hard errors
Stefano Lattarini [Fri, 10 Jun 2011 15:13:17 +0000 (17:13 +0200)]
maintcheck: DISTCHECK_CONFIGURE_FLAGS can be defined on make cmdline
* Makefile.am (sc_tests_overriding_macros_on_cmdline): It's now
acceptable that the test scripts override DISTCHECK_CONFIGURE_FLAGS
on the make command line. Update comments accordingly. Since we
are at it, make the relevant grepping rules slightly tighter.
Stefano Lattarini [Fri, 10 Jun 2011 10:26:42 +0000 (12:26 +0200)]
distcheck: add support for AM_DISTCHECK_CONFIGURE_FLAGS
* doc/automake.texi (Checking the Distribution): Suggest to use
AM_DISTCHECK_CONFIGURE_FLAGS, not DISTCHECK_CONFIGURE_FLAGS, to
define (in the top-level Makefile.am) extra flags to be passed
to configure at "make distcheck" time; DISTCHECK_CONFIGURE_FLAGS
should be reserved for the user. Add proper `@vindex' directive.
Document that AM_DISTCHECK_CONFIGURE_FLAGS is not honoured in a
subpackage Makefile.am, but the flags in it are passed down to
the configure script of the subpackage.
* lib/am/distdir.am (distcheck): Also pass the flags in
$(AM_DISTCHECK_CONFIGURE_FLAGS) to the configure invocation.
Update comments.
* tests/defs.in.test (AM_DISTCHECK_CONFIGURE_FLAGS,
DISTCHECK_CONFIGURE_FLAGS): Unset in case they are exported in
the environment, they might improperly influence our testsuite.
* tests/distcheck-configure-flags.test: New test.
* tests/distcheck-configure-flags-am.test: Likewise.
* tests/distcheck-configure-flags-subpkg.test: Likewise.
* distcheck-hook.test: Likewise.
* distcheck-hook2.test: Likewise.
* tests/Makefile.am (TESTS): Update.
Closes automake bug#8487.
Stefano Lattarini [Fri, 12 Nov 2010 19:26:59 +0000 (20:26 +0100)]
docs: better documentation for silent make rules
* doc/automake.texi (Options): Detailed description of the
automake option `silent-rules' moved from here ...
(Silent Make): ... into this new chapter, expanded, improved,
and subdivided into ...
(Make verbosity, Tricks For Silencing Make,
Automake silent-rules Option): ... these new sections.
(@menu, @detailmenu): Update.
* tests/silent-configsite.test: New test, checking that the
user can control default mode of silent-rules from config.site,
as is documented in the manual.
* tests/Makefile.am (TESTS): Updated.
Stefano Lattarini [Tue, 7 Jun 2011 13:24:11 +0000 (15:24 +0200)]
test defs: new function 'fatal_', for hard errors
Before this patch, the only way offered by tests/defs to
properly signal a hard error was the `framework_failure_'
function. But the error message issued by that function,
as its name would suggest, refers to a set-up failure in the
testsuite, while hard errors can obviously also be due to
other reasons. The best way to fix this inconsistency is to
introduce a new function with a more general error message.
Inspired by a recent similar change to Gnulib's tests/init.sh.
* tests/defs.in (fatal_): New function.
* tests/README (Section "Writing test cases" subsection "Do"):
Suggest the use of `fatal_', not of `framework_failure_', for
generic hard errors. The latter should be reserved for "real"
set-up failures.
Stefano Lattarini [Wed, 8 Jun 2011 08:03:29 +0000 (10:03 +0200)]
Merge branch 'fix-auxdir2-test'
Stefano Lattarini [Sun, 5 Jun 2011 19:44:53 +0000 (21:44 +0200)]
tests: fix typo-related error in auxdir2.test
* tests/auxdir2.test (configure.in): Close m4 quoting in the
argument to AC_CONFIG_AUX_DIR. Without this, aclocal fails with
"ERROR: end of file in string". This problem hasn't been exposed
by the testsuite before because this test is in XFAIL_TESTS, so
its failure went unnoticed, even if it was due to a wrong cause.
Bug introduced in commit v1.11-249-g49ac3de.
Stefano Lattarini [Fri, 3 Jun 2011 09:37:46 +0000 (11:37 +0200)]
Merge branch 'maint' into parallel-tests-maint
* maint:
maintcheck: fix some failures, extend some checks
automake, aclocal: honour configure-time AUTOCONF and AUTOM4TE
build: the user can override AUTOM4TE, AUTORECONF and AUTOUPDATE too
remake: behave better with non-GNU make in subdirectories
tests/README: fix example about `make -e' usage
Stefano Lattarini [Thu, 2 Jun 2011 10:47:58 +0000 (12:47 +0200)]
maintcheck: fix some more failures
* tests/instdir-ltlib.test: Use creative quoting to avoid
spuriously triggering the `sc_rm_minus_f' maintainer check.
* tests/instdir-prog.test: Likewise.
* tests/instspc-data.test: Use creative quoting to avoid
spuriously triggering the `sc_tests_Exit_not_exit' maintainer
check.
Stefano Lattarini [Thu, 2 Jun 2011 10:17:40 +0000 (12:17 +0200)]
Merge branch 'maint'
* maint:
maintcheck: fix some failures, extend some checks
Stefano Lattarini [Thu, 2 Jun 2011 10:15:52 +0000 (12:15 +0200)]
maintcheck: fix some failures, extend some checks
* Makefile.am (sc_diff_automake_in_automake): Update, as we
now expect 9 lines, not 8, to be changed from `automake.in'
to `automake'.
(sc_diff_aclocal_in_aclocal): New maintainer check, similar to
the above, and checking that only 10 lines are changed from
`aclocal.in' to `aclocal'.
(syntax_check_rules): Update.
(sc_tests_Exit_not_exit): Exempt self tests `self-check-*.test'
from this check, as they can legitimately use the bare `exit'
builtin in various places.
* doc/automake.texi (Python): Remove stray `@' from the end of
a line. Typo introduced in commit `v1.11-312-g5bf7af6'.
* tests/depcomp8a.test: Pass DISTCHECK_CONFIGURE_FLAGS to make
from the environment rather than from the command line, to
pacify the `sc_tests_overriding_macros_on_cmdline' maintainer
check.
* tests/depcomp8b.test: Likewise.
Stefano Lattarini [Wed, 1 Jun 2011 17:37:18 +0000 (19:37 +0200)]
Merge branch 'maint'
* maint:
remake: behave better with non-GNU make in subdirectories
Stefano Lattarini [Wed, 1 Jun 2011 16:58:35 +0000 (18:58 +0200)]
Merge branch 'remake-rules-non-GNU-make' into maint
Stefano Lattarini [Tue, 31 May 2011 16:03:48 +0000 (18:03 +0200)]
Merge branch 'maint'
* maint:
automake, aclocal: honour configure-time AUTOCONF and AUTOM4TE
build: the user can override AUTOM4TE, AUTORECONF and AUTOUPDATE too
tests/README: fix example about `make -e' usage
Stefano Lattarini [Sun, 29 May 2011 08:42:00 +0000 (10:42 +0200)]
automake, aclocal: honour configure-time AUTOCONF and AUTOM4TE
Currently, the Automake's own configure script allow definition
of AUTOCONF and AUTOM4TE, expected to point respectively to an
autoconf and autom4te programs. But while these definitions are
honoured in the Automake's build systems and test suite, they
were *not* honoured in the generated `automake' and `aclocal'
scripts. This behaviour, apart from being wrong in that it does
not allow the user enough freedom in choosing his tools, also
caused inconsistencies in the test suite, brining to spurious
failures.
Problem reported by Graham Reitz on the automake list; see thread:
<http://lists.gnu.org/archive/html/automake/2011-05/msg00022.html>
* automake.in ($traces): Use `@am_AUTOCONF', not simply `autoconf'.
* aclocal.in ($traces): Use `@am_AUTOM4TE', not simply `autom4te'.
* Makefile.am (do_subst): Substitute also `@am_AUTOCONF' and
`@am_AUTOM4TE'.
* NEWS: Update.
* THANKS: Update.
Stefano Lattarini [Sun, 29 May 2011 09:04:08 +0000 (11:04 +0200)]
build: the user can override AUTOM4TE, AUTORECONF and AUTOUPDATE too
Our build system allows the user to override AUTOCONF and AUTOHEADER
at configure time, and honours these overrides in our testsuite.
But it didn't do the same with AUTOM4TE, AUTORECONF and AUTOUPDATE.
This change fixes that inconsistency.
* configure.ac (am_AUTOM4TE, am_AUTOUPDATE, am_AUTORECONF): New
AC_SUBSTitutions. Update comments.
* tests/defs.in ($AUTOUPDATE): Default to `@am_AUTOUPDATE@' now.
($AUTOM4TE): New variable, defaulting to `@am_AUTOM4TE@'.
($AUTORECONF): New variable, defaulting to `@am_AUTORECONF@'.
* doc/Makefile.am ($(srcdir)/amhello-1.0.tar.gz): Update.
Stefano Lattarini [Sun, 29 May 2011 13:24:58 +0000 (15:24 +0200)]
remake: behave better with non-GNU make in subdirectories
Currently, with every decent make program, it is possible to
rebuild out-of-date autotools-generated files with a simple
"make Makefile" -- but for this to work reliably with non-GNU
make implementations, the command must be issued from the
top-level directory. This patch removes such limitation.
* lib/am/configure.am (am--refresh): Depend on `%MAKEFILE%'.
* tests/defs.in (using_gmake): New function, backported from the
`master' branch (and simplified).
* tests/remake-subdir.test: New test.
* tests/remake-subdir2.test: Likewise.
* tests/remake-subdir-gnu.test: Likewise.
* tests/remake-subdir-from-subdir.test: Likewise.
* tests/Makefile.am (TESTS): Update.
Stefano Lattarini [Fri, 27 May 2011 08:20:25 +0000 (10:20 +0200)]
testsuite: each test case depends on `defs-static'
* tests/Makefile.am ($(TEST_LOGS)): Depends on `defs-static' too.
Simplify comments.
Ralf Wildenhues [Sun, 22 May 2011 17:02:27 +0000 (19:02 +0200)]
tests/README: fix example about `make -e' usage
* tests/README (Section "Writing test cases" subsection "Do"): When
some variable is never initialized in the Makefile, `-e' is not
necessary in order to override it. DESTDIR is such a variable: we
ensure that we do not ever initialize it. And as such, it is quite
portable to use:
$ make DESTDIR=/foo/bar install
and in fact, quite widely used.
So our example about when `make -e' is required, which references
the `DESTDIR' variable, is poorly chosen, if not downright wrong.
Rewrite it to use `prefix' as the overridden variable instead.
Stefano Lattarini [Sun, 22 May 2011 07:35:17 +0000 (09:35 +0200)]
parallel-tests: stricter checks on DISABLE_HARD_ERRORS support
* tests/parallel-tests-harderror.test: New test, doing more
in-depth checks on DISABLE_HARD_ERRORS.
* tests/parallel-tests.test: Remove tests on DISABLE_HARD_ERRORS,
now redundant.
* tests/Makefile.am (TESTS): Update.
Stefano Lattarini [Fri, 20 May 2011 22:16:25 +0000 (00:16 +0200)]
Merge branch 'maint' into parallel-tests-maint
* maint:
testsuite: avoid re-running few tests with 'parallel-tests' option
testsuite: allow tests to avoid the use of 'parallel-tests' option
tests/README: update obsoleted advice
test defs: rename requirement 'non-cross' -> 'native'
testsuite: be more cross-compile friendly
Stefano Lattarini [Fri, 20 May 2011 21:33:09 +0000 (23:33 +0200)]
Merge branch 'maint'
* maint:
testsuite: avoid re-running few tests with 'parallel-tests' option
testsuite: allow tests to avoid the use of 'parallel-tests' option
tests/README: update obsoleted advice
Stefano Lattarini [Fri, 20 May 2011 21:18:39 +0000 (23:18 +0200)]
testsuite: avoid re-running few tests with 'parallel-tests' option
Some tests in our testsuite use the 'simple-tests' driver only
marginally, or simply as a mean to conveniently check unrelated
invariants. It makes little sense to force these tests to also
run with the 'parallel-tests' Automake option active, as doing so
offers no real gain in coverage, while often causing a measurable
overhead in execution time (for an already too-slow testsuite).
* tests/pr401.test (parallel_tests): Define to "no", to prevent
the generation of a sibling test script using the 'parallel-tests'
driver.
* tests/pr401b.test: Likewise.
* tests/pr401c.test: Likewise.
Stefano Lattarini [Fri, 20 May 2011 20:31:40 +0000 (22:31 +0200)]
testsuite: allow tests to avoid the use of 'parallel-tests' option
* tests/gen-parallel-tests: Do not generate "siblings" for tests
that explicitly define the `parallel_tests', whether to "yes" or
to any other value. Extend heading comments to give a rationale
for this behaviour.
* tests/README: Update.
Stefano Lattarini [Fri, 20 May 2011 20:15:32 +0000 (22:15 +0200)]
tests/README: update obsoleted advice
* tests/README (Section "Writing test cases" subsection "Do"):
Do not suggest to use the `*-p.test' pattern for the names of
hand-written tests which use the `parallel-tests' Automake option.
Not only is this not respected by the existing tests, but it is
more likely to cause conflicts with auto-generated tests.
So, suggest to *avoid* using the `*-p.test' pattern in names
of hand-written tests instead.
(Section "Writing test cases" subsection "Do not"): When
suggesting not to override Makefile variables using command
line arguments, do not use the badly outdated variables `U'
and 'ANSI2KNR' in the example; instead, use the more common
and typical `DESTDIR'.
Stefano Lattarini [Thu, 19 May 2011 19:53:02 +0000 (21:53 +0200)]
Merge branch 'maint'
* maint:
test defs: rename requirement 'non-cross' -> 'native'
Stefano Lattarini [Thu, 19 May 2011 19:48:27 +0000 (21:48 +0200)]
test defs: rename requirement 'non-cross' -> 'native'
* tests/defs.in (non-cross): Rename requirement ...
(native): ... to this, which is clearer and fits the
existing lingo better.
Suggestion by Ralf Wildenhues.
Stefano Lattarini [Sun, 15 May 2011 13:35:27 +0000 (15:35 +0200)]
Merge branch 'maint'
* maint:
testsuite: be more cross-compile friendly
Stefano Lattarini [Sun, 15 May 2011 12:30:43 +0000 (14:30 +0200)]
testsuite: be more cross-compile friendly
* tests/defs.in (cross_compiling): New subroutine.
(am__tool_prefix): New internal variable.
(gcc, g++, gcj): Force the use of the correct "tool prefix"
when cross compiling.
(gfortran, g77, non-cross): New requirements.
Stefano Lattarini [Sun, 15 May 2011 12:41:57 +0000 (14:41 +0200)]
Merge branch 'test-repeated-options-portability-fix'
Stefano Lattarini [Sun, 15 May 2011 12:41:35 +0000 (14:41 +0200)]
tests: fix portability issues in 'repeated-options.test'
* tests/repeated-options.test: Do not assume that object files
have `.o' suffix and executables have no default suffix; instead,
use `$(OBJEXT)' and `$(EXEEXT)'.
Stefano Lattarini [Wed, 11 May 2011 14:01:52 +0000 (16:01 +0200)]
docs: parallel-tests is not experimental anymore
The parallel-tests driver has now been used quite extensively
by a fair number of real-world applications (e.g., GNU coreutils,
GNU libtool, GNU grep, and various packages using Gnulib), and
thus exposed to adequate on-field testing. So there's no point
in declaring it experimental anymore (which would risk to make
potential users shy away from it).
* doc/automake.texi (Simple Tests using parallel-tests): Do not
declare the parallel-tests driver as "experimental" anymore.
Stefano Lattarini [Thu, 21 Apr 2011 08:25:01 +0000 (10:25 +0200)]
check: document and test $(TEST_SUITE_LOG) overriding
* doc/automake.texi (Simple Tests using parallel-tests): Explain
how and why TEST_SUITE_LOG can be overridden at runtime.
* tests/parallel-tests-log-override-1.test: New test, check that
the newly documented idiom and similar usages are truly supported.
* tests/parallel-tests-log-override-2.test: Likewise.
* tests/parallel-tests-log-override-recheck.test: Likewise.
* tests/Makefile.am (TESTS): Update.