+ 2011-08-03 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: one more use of TAP in our own tests
+ * tests/tap-bad-prog.test, tests/tap-bad-prog2.test: Merged
+ into ...
+ * tests/tap-bad-prog.tap: ... this TAP-generating test.
+ * tests/Makefile.am (tap_other_tests): Remove plan-bad-prog.test
+ and plan-bad-prog2.test.
+ (XFAIL_TESTS): Remove plan-bad-prog2.test.
+ (plan-bad-prog2.log): Remove.
+
+ 2011-08-03 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: scaffolding to allow use of TAP in our own tests
+ Now that Automake has initial support for the TAP test protocol,
+ we can start "eating our own dog food" and rewrite some tests to
+ use TAP; this should allow for better granularity, especially in
+ the control of skips and expected failures. With this change, we
+ set up the initial scaffolding required by the planned TAP tests,
+ and convert few older tests to use TAP, mostly in order to verify
+ that there are no obvious errors. The tests in our testsuite that
+ use the TAP protocol will be marked by the new `.tap' extension.
+ * tests/plain-functions.sh: New file containing definition of
+ helper shell functions used by the "protocol-less" tests of the
+ Automake testsuite.
+ * tests/tap-functions.sh: New file containing definition of
+ helper shell functions used by the TAP-producing tests of the
+ Automake testsuite.
+ * tests/defs: Remove definitions of some functions that are
+ now defined in `plain-functions.sh'. Instead, source one of
+ `plain-functions.sh' or `tap-functions.sh', depending on whether
+ `$use_tap' is set to "no" or "yes".
+ (exit trap): Call `late_plan_' if `$use_tap' is true. Also,
+ unset shell traces before issuing the latest commands, to avoid
+ confusing the tap driver with spurious output.
+ * tests/defs-static.in ($use_tap): New variable, by default set
+ to "yes" if the calling test script has a `.tap' suffix, and to
+ "no" otherwise. The individual scripts can override it though.
+ In code sanity-checking the environment, verify that `$use_tap'
+ is not exported.
+ * tests/self-check-env-sanitize.test: Update, and small related
+ reformatting.
+ * tests/self-check-tap.test: New very minimal self test.
+ * tests/acoutbs.test, tests/acoutbs2.test, tests/acoutnoq.test,
+ tests/acoutpt.test, tests/acoutpt2.test, tests/acoutqnl.test:
+ Removed, merged into ...
+ * tests/ac-output-old.tap: ... this new test, that uses TAP.
+ * tests/Makefile.am (TAP_LOG_DRIVER): Define to invoke our own
+ `tap-driver' script.
+ (TAP_LOG_DRIVER_FLAGS): Define to `--merge', so that the stdout
+ and stderr of the test scripts remains synced.
+ (TEST_EXTENSIONS): Define, to add the `.tap' suffix; also list
+ the `.test' suffix explicitly.
+ (EXTRA_DIST): Distribute the new files `plain-functions.sh' and
+ `tap-functions.sh'.
+ ($(TEST_LOGS)): Depends on them.
+ (AM_TESTS_ENVIRONMENT): Ensure that `use_tap' is not exported in
+ the environment of the tests.
+ (TESTS): Update.
+
+ 2011-08-03 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ * NEWS: Fix typo, and related reformatting.
+
+ 2011-08-03 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ * NEWS: add mention of new experimental TAP support
+
+ 2011-08-02 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: rename some test scripts
+ * tests/parallel-tests-am_tests_environment.test: Renamed ...
+ * tests/am-tests-environment.test: ... to this.
+ * tests/check-tests_environment.test: Renamed ...
+ * tests/tests-environment.test: ... to this.
+ * tests/parallel-test-driver-install.test: Renamed ...
+ * tests/parallel-tests-driver-install.test: ... to this.
+ * tests/parallel-tests-make-n.test: Renamed ...
+ * tests/parallel-tests-dry-run.test: ... to this.
+ * tests/Makefile.am (TESTS): Updated.
+
+ 2011-08-02 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ coverage: parallel-tests and lazy dependencies on EXTRA_PROGRAMS
+ * tests/parallel-tests-extra-programs.test: New test.
+ * tests/Makefile.am (TESTS): Update.
+ * doc/automake.texi (Parallel Test Harness): Add a comment
+ pointing to the new test.
+
+ 2011-08-02 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ docs: improve, extend and fix documentation on TAP support
+ * doc/automake.texi ("Using the TAP test protocol"): Divide this
+ section into ...
+ ("Introduction to TAP", "Use TAP with the Automake test harness",
+ "Incompatibilities with other TAP parsers and drivers", "Links
+ and external resources"): ... these subsections, extend them by
+ adding more information and examples, and improve them by removing
+ incomplete and/or temporary wordings and TODO items.
+ ("Script-based Testsuites", "Parallel Test Harness"): Add a couple
+ of anchors to improve the granularity of cross-references.
+ * tests/tap-doc2.test: New test, verifying the correctness of the
+ new examples given in the manual.
+ * tests/Makefile.am (tap_other_tests): Add the new test.
+
+ 2011-08-01 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: separate the only failing check of an xfailing test
+ * tests/plan-bad-prog.test: Move the only failing check of this
+ test (i.e., the one about the total number of "ERROR" outcomes)
+ into ...
+ * tests/plan-bad-prog2.test: ... this new test.
+ * tests/Makefile.am (XFAIL_TESTS): Remove `plan-bad-prog.test',
+ add `plan-bad-prog2.test'.
+ (tap_other_tests): Add `plan-bad-prog2.test'.
+ (plan-bad-prog2.log): Depend on `plan-bad-prog.test'.
+
+ 2011-08-01 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: fix spurious errors in an xfailing test on TAP support
+ * tests/plan-bad-prog.test: Fix typo in the name of the test
+ being made unreadable. Correct the wording of the potential
+ skip message, and simplify the condition under which the test
+ is to be skipped. Escape literal dots in grep regexps.
+
+ 2011-08-01 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: better granularity in a couple of tests on TAP support
+ * tests/tap-summary-aux.sh: New auxiliary script, filled with code
+ moved out from ...
+ * tests/tap-summary.test: ... this test, from which the checks on
+ colored testsuite have further been moved into ...
+ * tests/tap-summary-color.test: ... this new test, which in turn
+ * tests/Makefile.am (tap_other_tests): Add `tap-summary-color.test'.
+ (EXTRA_DIST): Distribute `tap-summary-aux.sh'
+ (tap-summary.log): Depend on `tap-summary-aux.sh'.
+ (tap-summary-color.log): Likewise.
+
+ 2011-08-01 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: optimize tests on TAP for speed and against duplication
+ The creation and configuration of common files and data used by
+ many tests on TAP is, with this change, factored out into a new
+ dedicated auxiliary test, from which those tests will depend upon.
+ This reduces code duplication in tests and, more importantly,
+ offers a noticeable speedup in the involved tests (30-40%).
+ * tests/tap-common-setup.test: New test, setting up the common
+ files and data used by various tests on TAP support.
+ * tests/tap-setup.sh: New file, to be sourced by tests wanting to
+ bring in data generated by `tap-common-setup.test'.
+ * tests/tap-autonumber.test: Update to use the pre-computed data
+ files.
+ * tests/tap-bailout.test: Likewise.
+ * tests/tap-color.test: Likewise.
+ * tests/tap-deps.test: Likewise.
+ * tests/tap-diagnostic.test: Likewise.
+ * tests/tap-empty-diagnostic.test: Likewise.
+ * tests/tap-empty.test: Likewise.
+ * tests/tap-escape-directive.test: Likewise.
+ * tests/tap-exit.test: Likewise.
+ * tests/tap-signal.test: Likewise.
+ * tests/tap-fancy.test: Likewise.
+ * tests/tap-fancy2.test: Likewise.
+ * tests/tap-global-log.test: Likewise.
+ * tests/tap-global-result.test: Likewise.
+ * tests/tap-html.test: Likewise.
+ * tests/tap-log.test: Likewise.
+ * tests/tap-merge-stdout-stderr.test: Likewise.
+ * tests/tap-no-merge-stdout-stderr.test: Likewise.
+ * tests/tap-message-0.test: Likewise.
+ * tests/tap-no-disable-hard-error.test: Likewise.
+ * tests/tap-no-spurious-summary.test: Likewise.
+ * tests/tap-no-spurious.test: Likewise.
+ * tests/tap-not-ok-skip.test: Likewise.
+ * tests/tap-numeric-description.test: Likewise.
+ * tests/tap-out-of-order.test: Likewise.
+ * tests/tap-passthrough.test: Likewise.
+ * tests/tap-passthrough-exit.test: Likewise.
+ * tests/tap-plan.test: Likewise.
+ * tests/tap-plan-corner.test: Likewise.
+ * tests/tap-plan-corner2.test: Likewise.
+ * tests/tap-plan-errors.test: Likewise.
+ * tests/tap-realtime.test: Likewise.
+ * tests/tap-recheck-logs.test: Likewise.
+ * tests/tap-skip-whole.test: Likewise.
+ * tests/tap-todo-skip-together.test: Likewise.
+ * tests/tap-todo-skip-whitespace.test: Likewise.
+ * tests/tap-todo-skip.test: Likewise.
+ * tests/tap-unplanned.test: Likewise.
+ * tests/tap-with-and-without-number.test: Likewise.
+ * tests/tap-xfail-tests.test: Likewise.
+ * tests/tap-skip-whole-whitespace.test: Likewise, and remove
+ redundant definitions of `$sp' and `$tab' (they are already
+ defined in `tests/defs').
+ * tests/tap-whitespace-normalization.test: Likewise.
+ * tests/Makefile.am (TESTS): Update.
+ (EXTRA_DIST): Distribute `tap-setup.sh'.
+ (tap_with_common_setup_tests, tap_with_common_setup_logs)): New
+ variables, holding respectively the list of tests using the files
+ pre-computed by `tap-common-setup.test', and the list of their
+ corresponding log files.
+ (tap_other_tests): New variable, holding the list of other tests
+ on TAP support.
+ ($(tap_with_common_setup_logs)): Depend on `tap-common-setup.log'
+ and `tap-setup.sh'.
+
+ 2011-08-01 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ test harness: use new `.trs' files to hold test metadata
+ With this change, the test harness will keep test metadata in
+ dedicated `.trs' files, instead of having them embedded into the
+ `.log' files. This allows for easier forward-compatibility and
+ extension of test metadata, and for more flexibility in the
+ format of the `.log' files. Note that this change makes the
+ `:end-metadata:' field obsolete.
+ * doc/automake.texi (Parallel Test Harness, Log files generation
+ and test results recording): Document the new APIs and behaviour;
+ some related minor rewordings and fixlets.
+ * NEWS: Update.
+ * automake.in (handle_tests): When bringing in the content of
+ `check2.am', substitute %BASE% with the basename of the `.log'
+ file being created by a rule. Add the generated `.trs' files
+ to the list of files to be cleaned by "make mostlyclean".
+ * lib/am/check.am (am__test_driver_flags): Rename ...
+ (am__common_driver_flags): ... to this, and remove the flags
+ `--test-name' and `--log-file' from it: they are now define in
+ the proper rules in `check2.am'.
+ (am__TEST_BASES): New internal variable, holding the names of
+ the tests, with any registered extension removed.
+ (am__stealth_MAKE): New internal variable, can be used instead of
+ $(MAKE) in recipes requiring a recursive call to make, but which
+ are not intended to be executed by "make -n".
+ (.log.trs): New suffix rule, to recover from deletion of `.trs'
+ files.
+ ($(TEST_SUITE_LOG)): Almost completely rewritten to follow the
+ new API of "test logs in `.log' files, test metadata in `.trs'
+ files". It goes to some length to work correctly in face of
+ unreadable or missing `.log' and `.trs' files, and to error out
+ with proper error messages when this is not possible.
+ [%?PARALLEL_TESTS%] (check-TESTS): Also remove relevant "stale"
+ `.trs' files (in addition to `.log files) before remaking the
+ $(TEST_SUITE_LOG).
+ (recheck, recheck-html): Look for the `:recheck:' field in the
+ `.trs' files, not in the `.log' files.
+ * lib/am/check2.am (?GENERIC?%EXT%.log, ?!GENERIC?%OBJ%): Adjust
+ the call to the test driver, in particularly passing the new
+ option `--trs-file'.
+ [%am__EXEEXT%] (?GENERIC?%EXT%$(EXEEXT).log): Likewise.
+ * lib/tap-driver ($USAGE): Adjust the help screen.
+ (Getopt::Long::GetOptions): Handle the `--trs-file' option,
+ through the use of ...
+ ($trs_file): ... this new global variable.
+ (finish): Write metadata for the test run to `$trs_file' rather
+ then to `$log_file', through the use of ...
+ (write_test_results): ... this new function.
+ * lib/test-driver (print_usage): Update the help screen.
+ (Option parsing): Handle the `--trs-file' option, through the
+ use of ...
+ ($resfile): ... this new global variable.
+ (Main code): Write metadata for the test run to `$trsfile' rather
+ than to `$logfile'.
+ Minor related adjustments to comments.
+ * tests/.gitignore: Ignore `*.trs' files.
+ * tests/parallel-tests-unreadable-log.test: Moved ...
+ * tests/parallel-tests-unreadable.test: ... to this, and extended
+ to also check the semantics for unreadable `.trs' files.
+ * tests/test-driver-end-metadata.test: Deleted as obsolete.
+ * tests/test-driver-metadata-no-leading-space.test: Likewise.
+ * tests/test-driver-global-log.test: Renamed ...
+ * tests/test-metadata-global-log.test: ... to this, and modified
+ as to verify the new APIs and semantics.
+ * tests/test-driver-recheck: Renamed ...
+ * tests/test-metadata-recheck.test: ... to this, and modified
+ likewise.
+ * tests/parallel-tests-once.test: New test.
+ * tests/parallel-tests-make-n.test: Likewise.
+ * test-metadata-results.test: Likewise.
+ * test-missing.test: Likewise.
+ * test-missing2.test: Likewise.
+ * test-trs-basic.test: Likewise.
+ * test-trs-recover.test: Likewise.
+ * test-trs-recover2.test: Likewise.
+ * tests/Makefile.am (TESTS): Update.
+
+ 2011-07-27 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ test harness: allow more metadata in log files
+ This change reworks and improves the parallel test harness to use
+ more specialized reStructuredText fields in the log files (instead
+ of relying on specially-placed of "magic lines" and more indirect
+ semantical formatting); the new fields are the following:
+ - ":recheck:": tell whether the associated test will have to be
+ re-run by "make recheck";
+ - ":copy-in-global-log:": tell whether the content of the log
+ file should be copied in the "global log" `test-suite.log';
+ - ":end-metadata:", which inhibits the scanning of the rest of
+ the log file (for what concerns test metadata).
+ Also, the special `:test-result:' value "END" has been removed,
+ superseded by the new `:end-metadata:' field.
+ * doc/automake.texi (Log files generation and test results
+ recording): Document the new API and semantics. Remove or fix
+ some obsolete comments.
+ * lib/am/check.am ($(TEST_SUITE_LOG), recheck, recheck-html):
+ Adjust comments and code.
+ * lib/tap-driver (finish): Adjust, with the help of ...
+ (must_recheck, copy_in_global_log): ... these new functions.
+ * lib/test-driver (Main code): Adjust, with the help of ...
+ ($recheck, $gcopy): ... these new variables.
+ * tests/trivial-test-driver: Update to obey the new APIs.
+ * tests/test-driver-recheck.test: Likewise.
+ * tests/test-driver-global-log.test: Likewise.
+ * tests/tap-passthrough.test: Relax the test, avoiding to check
+ what is written in `test-suite.log'; such check has been moved ...
+ * tests/tap-global-log.test: ... in this new test, and extended.
+ * tests/test-driver-metadata-no-leading-space.test: New test.
+ * tests/test-driver-end-test-results.test: Removed, it checked
+ the old APIs; superseded by ...
+ * tests/test-driver-end-metadata.test: ... this new test.
+ * tests/tap-log.test: Improve syncing with ...
+ * tests/test-log.test: ... this new test.
+ * tests/parallel-tests.test: Remove some duplication w.r.t. this
+ last new test. Updated heading comments.
+ * tests/Makefile.am (TESTS): Update.
+
+ 2011-07-27 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: fix a spurious failure with non-bash shells
+ * tests/tap-fancy2.test: Remove an unportable use of backslashes
+ with the `echo' builtin, which was causing some shells (among them
+ bash and Debian /bin/ksh) to print a `\\' string, while other
+ shells (among them zsh, dash, and Solaris /bin/sh and /bin/ksh)
+ were unexpectedly printing a single `\' character. Since we are
+ at it, add a sanity check to ensure that this issue does not
+ resurface.
+
+ 2011-07-27 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: fix spurious failures with Solaris /bin/sh
+ * tests/tap-more.test: Use `echo > file', not `: > file', to
+ create empty files in the "for" loops; this is required since,
+ as documented in the autoconf manual, Solaris 10 /bin/sh
+ "optimizes" away the `:' command after the first iteration,
+ even if it is redirected.
+ * tests/test-driver-custom-multitest-recheck2.test: Likewise.
+ * tests/tap-numeric-description.test: Partial rewrite to avoid
+ using positional parameters from the 10th onward, which are
+ unportable to Solaris /bin/sh (using `${10}' causes the shell
+ to die with "bad substitution").
+
+ 2011-07-31 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ simple tests: support developer-defined fd redirections
+ Motivated by coreutils bug#8846, and related discussions:
+ <http://debbugs.gnu.org/cgi/bugreport.cgi?bug=8846>
+ <http://lists.gnu.org/archive/html/bug-autoconf/2011-06/msg00002.html>
+ In those threads it has been shown how problematic it can be to do
+ portable file descriptor redirections in the testsuite when using
+ the Automake testsuite harness. This change should remedy to that
+ situation.
+ * lib/am/check2.am (?GENERIC?%EXT%$(EXEEXT).log,
+ ?GENERIC?%EXT%.log, ?!GENERIC?%OBJ%): Append redirection defined
+ in $(AM_TESTS_FD_REDIRECT) to the command-line invocations of the
+ test scripts.
+ * lib/am/check.am [!%?PARALLEL_TESTS%] $(check-TESTS): Likewise.
+ * NEWS: Update.
+ * doc/automake.texi (Script-based Testsuites): Document the new
+ feature.
+ * tests/check-fd-redirect.test: New test.
+ * tests/parallel-tests-fd-redirect.test: Likewise.
+ * tests/parallel-tests-am_tests_environment.test: Remove checks
+ about the use of redirections in AM_TESTS_ENVIRONMENT: they would
+ check deprecated (if not undefined) behaviour now. Strengthen a
+ couple of still valid checks, to keep the test more in sync with
+ the documentation. Improve debugging information.
+ * tests/Makefile.am (TESTS): Update.
+
+ 2011-07-22 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ docs: rework and extend documentation on testsuites support
+ * doc/automake.texi (Generalities about Testing): New section.
+ (Simple Tests using parallel-tests): Section removed, merged
+ into ...
+ (Simple Tests): ... this one, which on the other hand has been
+ subdivided into ...
+ (Script-based Testsuites, Serial Test Harness, Parallel Test
+ Harness): ... these new subsection.
+ (DejaGnu Tests): Minor adjustments.
+ Other related typofixes and rewordings throughout the manual;
+ in particular, avoid to use the term "test driver" for three
+ different concepts (and use instead "test harness" and "test
+ runner" where appropriate).
+ * tests/tap-doc.test: New test.
+ * tests/tap-no-disable-hard-error.test: Likewise.
+ * tests/Makefile.am (TESTS): Update.
+
+ 2011-07-18 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tap: support colorization of testsuite progress output
+ * lib/tap-driver (%COLORS): New variable (definition extracted
+ from `lib/am/check.am:$(am__tty_colors)', with some obvious
+ adjustments.
+ (report): Adjust to colorize console output when required,
+ using ...
+ (decorate_result): ... this new function.
+ (colored): New function, used by the one above.
+ * tests/tap-summary.test: Also run the checks when `color-tests'
+ is in use.
+ * tests/Makefile.am (XFAIL_TESTS): Remove `tap-color.test'.
+
+ 2011-07-18 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tap: some preparatory refactoring (2)
+ This is a follow-up simplification.
+ * lib/tap-driver (console_output): Renamed ...
+ (report): ... to this, and extended to appropriately register
+ the test results when needed.
+ (testsuite_error, handle_tap_comment, handle_tap_test,
+ handle_tap_plan): Adjusted accordingly.
+
+ 2011-07-18 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tap: some preparatory refactoring (1)
+ This refactoring is aimed at simplifying the introduction of
+ colored console output for the TAP driver.
+ * lib/tap-driver (console_output): Now accept two arguments, the
+ first one indicating which kind of thing is to be displayed (for
+ now only a test result or a diagnostic comment), and the second
+ one (if present) the message associated to it.
+ (handle_tap_test, handle_tap_comment, handle_tap_plan,
+ testsuite_error): Adapt to the new `console_output' interface.
+
+ 2011-07-18 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tap: add experimental TAP-aware driver
+ * doc/automake.texi (Using the TAP test protocol): New section.
+ (Overview of Custom Test Drivers Support): Minor updates.
+ * lib/tap-driver: New script, TAP-aware test driver for Automake;
+ implemented in perl and based on TAP::Parser.
+ * lib/Makefile.am (dist_script_DATA): Add it.
+ * tests/tap-autonumber.test: New test.
+ * tests/tap-bailout.test: Likewise.
+ * tests/tap-basic.test: Likewise.
+ * tests/tap-deps.test: Likewise.
+ * tests/tap-diagnostic.test: Likewise.
+ * tests/tap-empty.test: Likewise.
+ * tests/tap-empty-diagnostic.test: Likewise.
+ * tests/tap-escape-directive.test: Likewise.
+ * tests/tap-exit.test: Likewise.
+ * tests/tap-fancy.test: Likewise.
+ * tests/tap-fancy2.test: Likewise.
+ * tests/tap-global-result.test: Likewise.
+ * tests/tap-html.test: Likewise.
+ * tests/tap-log.test: Likewise.
+ * tests/tap-merge-stdout-stderr.test: Likewise.
+ * tests/tap-more.test: Likewise.
+ * tests/tap-more2.test: Likewise.
+ * tests/tap-no-merge-stdout-stderr.test: Likewise.
+ * tests/tap-no-spurious-summary.test: Likewise.
+ * tests/tap-no-spurious.test: Likewise.
+ * tests/tap-not-ok-skip.test: Likewise.
+ * tests/tap-numeric-description.test: Likewise.
+ * tests/tap-out-of-order.test: Likewise.
+ * tests/tap-passthrough.test: Likewise.
+ * tests/tap-plan.test: Likewise.
+ * tests/tap-plan-errors.test: Likewise.
+ * tests/tap-plan-corner.test: Likewise.
+ * tests/tap-realtime.test: Likewise.
+ * tests/tap-recheck-logs.test: Likewise.
+ * tests/tap-recheck.test: Likewise.
+ * tests/tap-skip-whole.test: Likewise.
+ * tests/tap-summary.test: Likewise.
+ * tests/tap-todo-skip.test: Likewise.
+ * tests/tap-todo-skip-together.test: Likewise.
+ * tests/tap-todo-skip-whitespace.test: Likewise.
+ * tests/tap-skipall-whitespace.test: Likewise.
+ * tests/tap-unplanned.test: Likewise.
+ * tests/tap-whitespace-normalization.test: Likewise.
+ * tests/tap-with-and-without-number.test: Likewise.
+ * tests/tap-xfail-tests.test: Likewise.
+ * tests/tap-bad-prog.test: New xfailing test.
+ * tests/tap-color.test: Likewise.
+ * tests/tap-plan-corner2.test: Likewise.
+ * tests/tap-message-0.test: Likewise.
+ * tests/tap-signal.test: Likewise.
+ * tests/Makefile.am (TESTS, XFAIL_TESTS): Update.
+
+ 2011-07-07 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ test defs: new auxiliary function 'count_test_results'
+ * tests/defs (count_test_results): New function.
+ * tests/check11.test: Use it.
+ * tests/test-driver-custom-multitest.test: Likewise.
+ * tests/test-driver-custom-multitest-recheck.test: Likewise.
+ * tests/test-driver-custom-multitest-recheck2.test: Likewise.
+ * tests/parallel-tests-log-override-recheck.test: Likewise.
+ * tests/parallel-tests-log-override-recheck.test: Likewise.
+ * tests/parallel-tests-no-spurious-summary.test: Likewise, and
+ slightly improve debugging output.
+ * tests/parallel-tests.test: Make use of `count_test_results'.
+ Also, make grepping of "make check" output slightly stricter
+ * tests/parallel-tests9.test: Likewise.
+ * tests/parallel-tests-log-override-2.test: Likewise, and throw
+ in a small optimization.
+
+ 2011-07-07 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ parallel-tests: simplify testsuite summary
+ Prefer a more deterministic, "tabular" format for the testsuite
+ summary, always listing the numbers of passed, failed, xfailed,
+ xpassed, skipped and errored tests, even when these numbers are
+ zero. This simplify the logic of testsuite summary creation,
+ makes it more easily machine-parseable, and will probably allow
+ for easier addition of new kinds of test results in the future.
+ * lib/am/check.am (am__tty_colors_dummy): New make variable, to
+ reduce code duplication. Extracted from previous versions of
+ $(am__tty_colors), and extended by defining two new variables
+ `$mgn' and `$brg'.
+ [%?COLOR%, %!?COLOR%] (am__tty_colors): Use that new variable.
+ (am__text_box): Delete, is not needed anymore.
+ ($(TEST_SUITE_LOG)): Rewrite associated rules to implement the
+ new testsuite summary format.
+ * NEWS: Update.
+ * tests/check10.test: Don't run with the parallel-tests harness
+ too, that makes no sense anymore.
+ * tests/color.test: Update and adjust.
+ * tests/color2.test: Likewise.
+ * tests/parallel-tests.test: Likewise.
+ * tests/parallel-tests3.test: Likewise.
+ * tests/parallel-tests6.test: Likewise.
+ * tests/parallel-tests9.test: Likewise.
+ * tests/parallel-tests-unreadable-log.test: Likewise.
+ * tests/parallel-tests-empty-testlogs.test: Likewise.
+ * tests/parallel-tests-log-override-recheck.test: Likewise.
+ * tests/parallel-tests-no-spurious-summary.test: Likewise.
+ * tests/test-driver-custom-multitest.test: Likewise.
+ * tests/test-driver-end-test-results.test: Likewise.
+ * tests/parallel-tests-no-color-in-log.test: New test.
+ * tests/testsuite-summary-color.test: Likewise.
+ * tests/testsuite-summary-count.test: Likewise.
+ * tests/testsuite-summary-count-many.test: Likewise.
+ * tests/testsuite-summary-reference-log.test: Likewise.
+ * tests/testsuite-summary-checks.sh: New auxiliary script, used
+ by the new tests above.
+ * tests/extract-testsuite-summary: Likewise.
+ * tests/trivial-test-driver: Optimize for speed when there are
+ lots of of tests.
+ * tests/Makefile.am (EXTRA_DIST): Distribute them.
+ (testsuite-summary-color.log, testsuite-summary-count.log): Depend
+ on them.
+ (testsuite-summary-count-many.log): Depend on the auxiliary scripts
+ 'trivial-test-driver' and 'extract-testsuite-summary'.
+ (TESTS): Update.
+
+ 2011-07-07 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ parallel-tests: new recognized test result 'ERROR'
+ * lib/am/check.am ($(TEST_SUITE_LOG)): Recognize a new test result
+ `ERROR'. Use it when encountering unreadable test logs (previously
+ a simple `FAIL' was used in this situations).
+ * lib/test-driver: Set the global test result to `ERROR' when the
+ test exit status is 99. When doing colorized output, color `ERROR'
+ results in magenta.
+ * doc/automake.texi (Log files generation and test results
+ recording): Update by listing `ERROR' too among the list of valid
+ `:test-results:' arguments.
+ * NEWS: Update.
+ * tests/trivial-test-driver: Update.
+ * tests/parallel-tests.test: Likewise.
+ * tests/parallel-tests-harderror.test: Likewise.
+ * tests/parallel-tests-no-spurious-summary.test: Likewise.
+ * tests/test-driver-global-log.test: Likewise.
+ * tests/test-driver-recheck.test: Likewise.
+ * tests/test-driver-custom-multitest-recheck.test: Likewise.
+ * tests/test-driver-custom-multitest-recheck2.test: Likewise.
+ * tests/test-driver-custom-multitest.test: Likewise.
+ * tests/test-driver-custom-no-html.test: Likewise.
+ * tests/test-driver-end-test-results.test: Likewise.
+ * tests/color.test: Likewise. Also, make stricter, and also test
+ from VPATH.
+ * tests/color2.test: Likewise, and improve syncing with color.test.
+ * tests/parallel-tests-exit-statuses.test: New test.
+ * tests/parallel-tests-console-output.test: Likewise.
+ * tests/Makefile.am (TESTS): Update.
+
+ 2011-07-07 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ parallel-tests: make parsing of test results safer
+ The new code for parsing the testsuite-generated `.log' files,
+ as introduced in commit `v1.11-872-gc96b881', considers each
+ `:test-result:' field anywhere in a `.log' file as a declaration
+ of a test result, and accounts for it as such in the testsuite
+ summary. Unfortunately this could easily cause spurious test
+ failures being reported in the testsuite summary. This happened
+ in practice with the Automake's own testsuite; for example:
+ $ make check TESTS='check12-p.test'; echo exit: $?
+ ...
+ PASS: check12-p.test
+ =====================================
+ 4 of 5 tests failed
+ See tests/test-suite.log
+ Please report to bug-automake@gnu.org
+ =====================================
+ make[2]: *** [test-suite.log] Error 1
+ make: *** [check-am] Error 2
+ exit: 2
+ This change introduces a new special `:test-result:' "END", that,
+ when seen, prevents the rest of the log file from being parsed.
+ For more information, refer to the thread:
+ <http://lists.gnu.org/archive/html/automake-patches/2011-06/msg00199.html>
+ * lib/am/check.am ($(TEST_SUITE_LOG)): Stop the parsing of a log
+ file as soon as the special ":test-result:END" directive is seen.
+ Related changes and enhancements.
+ * lib/test-driver: Protect the rest of the log after the result
+ lined with a ":test-result:END" directive.
+ * doc/automake.texi (Log files generation and test results
+ recording): Update, and related improvements.
+ * tests/parallel-tests-no-spurious-summary.test: New test.
+ * tests/test-driver-end-test-results.test: Likewise.
+ * tests/Makefile.am (TESTS): Update.
+
+ 2011-06-29 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ docs: document custom test drivers and protocols
+ * doc/automake.texi (Simple Tests): Note that the TESTS_ENVIRONMENT
+ use suggested here is not portable to 'parallel-tests'.
+ (Simple Tests using parallel-tests): Document new restrictions on
+ the uses of TESTS_ENVIRONMENT and AM_TESTS_ENVIRONMENT.
+ (Custom Test Drivers): New section and node.
+ (Overview of Custom Test Drivers Support): New subsection.
+ (Declaring Custom Test Drivers in @file{Makefile.am}): Likewise.
+ (APIs for Custom Test Drivers): Likewise.
+ (Options): Update description of color-tests.
+ * lib/am/check ($(TEST_SUITE_LOG)): Remove comments that have been
+ moved into the manual.
+ (recheck, recheck-html): Minor adjustments to better conform to the
+ documentation (this should cause no semantic changes w.r.t. the
+ former behaviour); minor improvements and extensions to existing
+ comments.
+ * tests/test-driver-create-log-dir.test: New test.
+ * tests/test-driver-strip-vpath.test: Likewise.
+ * tests/test-driver-global-log.test: Likewise.
+ * tests/test-driver-recheck.test: Likewise.
+ * tests/Makefile.am (TESTS): Update.
+
+2011-08-03 Bruno Haible <bruno@clisp.org>
+
+ docs: how to use '-I' option in AM_CPPFLAGS for best VPATH support
+ * doc/automake.texi (Program Variables): Recommend -I options to
+ both the build directory and the source directory when needed.
+
+2011-07-23 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: relax a test on amhello examples to cater to Solaris tar
+ * tests/amhello-binpkg.test: When the tar implementation in use
+ is not GNU tar, relax the tests on tar output, to avoid spurious
+ failures. For example, "tar cvf ..." with GNU tar can output
+ lines like "./usr/bin/hello" on the standard output, while with
+ Solaris tar it can output lines like "a ./usr/bin/hello 8K" on
+ standard output, and with Heirloom tar it can output lines like
+ "a ./usr/bin/hello 15 tape blocks" on standard error.
+
+2011-07-23 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ test defs: function 'is_newest' now works also with directories
+ * tests/defs (is_newest): Call `find' with the `-prune' option,
+ so that it won't descend in the directories (which could cause
+ spurious results).
+ * tests/self-check-is_newest: Extend accordingly.
+ From a report by Jim Meyering, see automake bug#9147.
+
+2011-07-15 Benoit Sigoure <tsunanet@gmail.com>
+
+ docs: add references between the 2 sections on java support
+ * doc/automake.texi (Java Support, Java): Add cross-references.
+
+2011-07-20 Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
+
+ Sync auxiliary files from upstream.
+ * INSTALL, lib/INSTALL, lib/config.guess, lib/config.sub,
+ lib/texinfo.tex: Sync from upstream.
+
+2011-07-24 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: small enhancement to 'primary3.test'
+ * tests/primary3.test: Also test that the program named `foo.la'
+ has been effectively created.
+
+2011-07-24 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: a small simplification in 'pr8365-remake-timing.test'
+ * tests/pr8365-remake-timing.test: Avoid redundant definition
+ and use of `$save_AUTOCONF' variable.
+
+2011-07-24 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: remove an extra leading blank line
+ * tests/override-conditional-1.test: Remove extra blank line
+ placed before the shebang line.
+
+2011-07-18 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: fix typos, grammaros and other blunders in comments
+ All affected files changed.
+
+2011-07-16 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ test defs: new subroutine 'seq_', simulating GNU seq(1)
+ * tests/defs (seq_): New subroutine.
+ * tests/instmany.test: Use it.
+ * tests/instmany-mans.test: Likewise.
+ * tests/instmany-python.test: Likewise.
+ * tests/self-check-seq.test: New self test.
+ * tests/Makefile.am (TESTS): Update.
+
+2011-07-16 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: remove duplication about testing of config.* aux files
+ * tests/add-missing.test: Also check that the `AC_CANONICAL_SYSTEM'
+ autoconf macro causes the `config.sub' and `config.guess' scripts
+ to be installed by `automake --add-missing'. Since we are at it,
+ fix minor buglets and cosmetic issues in the `check_' function.
+ * tests/hosts.test: Removed, completely subsumed by the previous
+ test now.
+ * tests/Makefile.am (TESTS): Update.
+
+2011-07-16 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: more uses of the 'unindent' subroutine
+ * tests/backcompat.test: Prefer `unindent' over plain `cat' for
+ here documents created from within a loop of if/else body.
+ * tests/backcompat2.test: Likewise.
+ * tests/backcompat4.test: Likewise.
+ * tests/init.test: Likewise.
+ * tests/instmany.test: Likewise.
+ * tests/instmany-mans.test: Likewise.
+ * tests/instmany-python.test: Likewise.
+ * tests/missing5.test: Likewise.
+ * tests/parallel-am2.test : Likewise.
+ * tests/parallel-am3.test : Likewise.
+ * tests/pr307.test: Likewise.
+ * tests/depend4.test: Likewise. Since we are at it, normalize
+ other code formatting.
+ * tests/instfail-java.test: Prefer `echo' over `cat' + here-doc
+ inside a for loop.
+
+2011-07-16 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: avoid '##'-style comments inside recipe commands
+ * tests/autodist.test: Do not whitespace-indent `##' comments when
+ they are embedded in a makefile rule: having them indented is not
+ part of the Automake API, and might cause failures with e.g., Tru64
+ make.
+ * tests/autodist-subdir.test: Likewise.
+ * tests/backcompat.test: Likewise.
+ * tests/backcompat6.test: Likewise.
+ * tests/colon7.test: Likewise.
+ * tests/posixsubst-scripts.test: Likewise.
+ * tests/posixsubst-sources.test: Likewise.
+
+2011-07-16 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: tweak, improve and extend tests on lisp support
+ * tests/lisp2.test: Prettify, and improve debugging output. Make
+ grepping of automake stderr stricter.
+ * tests/lisp7.test: Ensure verbose printing of captured make
+ output. Add trailing `:' command.
+ * tests/lisp8.test: Likewise.
+ * tests/lisp3.test: Likewise. Also, check that `.el' files and
+ compiled `.elc' files get installed by "make install", and
+ uninstalled by "make uninstall".
+ * tests/lisp4.test: Ensure installed `.el' files gets removed by
+ "make uninstall". Related changes. Add trailing `:' command.
+ * tests/lisp5.test: Likewise.
+ * tests/lisp6.test: Use proper m4 quoting in configure.in. Fix
+ use of blank lines, to improve clarity and symmetry. Fix typo in
+ comment. Add trailing `:' command.
+
+2011-07-16 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: drop useless requirement in 'license.test'
+ * tests/license.test: Drop useless requirement "makeinfo". Add
+ an extra blank line, for clarity.
+
+2011-07-08 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: fix weakness in 'tests-environment-backcompat.test'
+ * tests/tests-environment-backcompat.test: Do not override the
+ content of xfailing test `baz.test' with a "weaker" version that
+ fails unconditionally: the test must fail only when the 'strict'
+ pragma is in use, in order not to reduce coverage.
+
+2011-07-08 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ docs, tests: synchronize examples on silent-rules from config.site
+ * doc/automake.texi (Automake silent-rules Option): Reference test
+ 'silent-configsite.test' in comments.
+
+2011-07-08 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests defs: more uses of $top_testsrcdir
+ * tests/autodist-stamp-vti.test: Use `$top_testsrcdir' instead
+ of `$testsrcdir/..'.
+ * tests/repeated-options.test: Likewise.
+ * tests/suffix5.test: Likewise.
+ * tests/vtexi3.test: Likewise.
+
+2011-07-08 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ self tests: fix typo in comment
+ * tests/self-check-exit.test: Fix typo in heading comments.
+
+2011-07-08 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: remove redundant settings of `errexit' shell flag
+ * tests/amhello-binpkg.test: Do not set the `errexit' shell
+ flag, as it is already set by `tests/defs'.
+ * tests/amhello-cflags.test: Likewise.
+ * tests/amhello-cross-compile.test: Likewise.
+ * tests/ansi2knr-deprecation.test: Likewise.
+ * tests/autodist-no-duplicate.test: Likewise.
+ * tests/distcheck-configure-flags-am.test: Likewise.
+ * tests/distcheck-configure-flags-subpkg.test: Likewise.
+ * tests/distcheck-configure-flags.test: Likewise.
+ * tests/distcheck-hook.test: Likewise.
+ * tests/distcheck-hook2.test: Likewise.
+ * tests/parallel-tests-am_tests_environment.test: Likewise.
+ * tests/parallel-tests-harderror.test: Likewise.
+ * tests/parallel-tests-log-compiler-example.test: Likewise.
+ * tests/parallel-tests-log-override-1.test: Likewise.
+ * tests/parallel-tests-log-override-2.test: Likewise.
+ * tests/parallel-tests-log-override-recheck.test: Likewise.
+ * tests/primary-prefix-couples-documented-valid.test: Likewise.
+ * tests/primary-prefix-couples-force-valid.test: Likewise.
+ * tests/primary-prefix-invalid-couples.test: Likewise.
+ * tests/primary-prefix-valid-couples.test: Likewise.
+ * tests/remake-subdir-from-subdir.test: Likewise.
+ * tests/remake-subdir-gnu.test: Likewise.
+ * tests/remake-subdir-long-time.test: Likewise.
+ * tests/remake-subdir.test: Likewise.
+ * tests/remake-subdir2.test: Likewise.
+ * tests/silent-configsite.test: Likewise.
+ * tests/tests-environment-backcompat.test: Likewise.
+
+2011-07-08 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: portability fixes in tests on amhello examples
+ * tests/amhello-binpkg.test: Don't use "tar xzf too.tag.gz" to
+ extract a gzip-compressed tarball, that's unportable to some
+ tar implementations; use the "gzip -dc fo.tar.gz | tar xf -"
+ idiom instead.
+ * tests/amhello-cflags.test: Likewise.
+ * tests/amhello-cross-compile.test: Likewise.
+ Suggestion from Ralf Wildenhues.
+
+2011-07-04 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ remake: add test ensuring that slower remakes don't hang
+ * tests/remake-subdir-long-time.test: New test.
+ * tests/Makefile.am (TESTS): Update.
+ Suggestion by Ralf Wildenhues.
+
+2011-07-04 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ remake: fix outdated comment in configure.am
+ * lib/am/configure.am: Fix comment falsified by changes in
+ commit `v1.11-366-gbee9871'.
+ Suggestion by Ralf Wildenhues.
+
+2011-07-01 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ docs, tests: synchronize examples from docs to tests
+ * tests/README (Writing test cases): Give suggestions on how to
+ keep test cases and examples in the documentation synchronized.
+ * doc/automake.texi: Improve or fix existing testcase-referencing
+ comments, and add many new ones.
+ * HACKING (Administrivia): Suggest to test complex examples and
+ idioms from the manual.
+ * tests/specflg8.test: Improve synchronization with the example
+ in the manual.
+ * tests/output11.test:Likewise.
+ * tests/txinfo21.test:Likewise.
+ * tests/interp.test: Likewise.
+ * tests/amhello-cflags.test: New test.
+ * tests/amhello-cross-compile.test: Likewise.
+ * tests/amhello-binpkg.test: Likewise.
+ * tests/tests-environment-backcompat.test: Likewise.
+ * tests/parallel-tests-log-compiler-example.test: Likewise.
+ * tests/Makefile.am (TESTS): Update.
+
+2011-06-30 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ coverage: new test on parallel-tests TESTS runtime overriding
+ * tests/parallel-tests-cmdline-override.test: New test, check that
+ we can use indirections when overriding TESTS and TEST_LOGS from
+ the command line.
+ * tests/Makefile.am (TESTS): Update.
+
2011-06-29 Stefano Lattarini <stefano.lattarini@gmail.com>
docs: explain why AM_TESTS_ENVIRONMENT must be semicolon-terminated
Installation Instructions
*************************
--Copyright (C) 1994, 1995, 1996, 1999, 2000, 2001, 2002, 2004, 2005,
--2006, 2007, 2008, 2009, 2010 Free Software Foundation, Inc.
++Copyright (C) 1994-1996, 1999-2002, 2004-2011 Free Software Foundation,
++Inc.
Copying and distribution of this file, with or without modification,
are permitted in any medium without royalty provided the copyright
and if that doesn't work, install pre-built binaries of GCC for HP-UX.
++ HP-UX `make' updates targets which have the same time stamps as
++their prerequisites, which makes it generally unusable when shipped
++generated files such as `configure' are involved. Use GNU `make'
++instead.
++
On OSF/1 a.k.a. Tru64, some versions of the default C compiler cannot
parse its `<wchar.h>' header file. The option `-nodtk' can be used as
a workaround. If GNU CC is not installed, it is therefore recommended
@vindex TESTS
@vindex TESTS_ENVIRONMENT
- The variable @code{TESTS_ENVIRONMENT} can be used to set environment
- variables for the test run; the environment variable @env{srcdir} is
- set in the rule. If all your test programs are scripts, you can also
- set @code{TESTS_ENVIRONMENT} to an invocation of the shell (e.g.
- @samp{$(SHELL) -x} can be useful for debugging the tests), or any other
- interpreter. For instance, the following setup may be used to run tests
- with Perl:
+ @vindex AM_TESTS_ENVIRONMENT
+ The @code{AM_TESTS_ENVIRONMENT} and @code{TESTS_ENVIRONMENT} variables can
+ be used to run initialization code and set environment variables for the
+ test scripts. The former variable is developer-reserved, and can be
+ defined in the @file{Makefile.am}, while the latter is reserved for the
+ user, which can employ it to extend or override the settings in the
+ former; for this to work portably, however, the contents of a non-empty
+ @code{AM_TESTS_ENVIRONMENT} @emph{must} be terminated by a semicolon.
+
+ @vindex AM_TESTS_FD_REDIRECT
+ The @code{AM_TESTS_FD_REDIRECT} variable can be used to define file
+ descriptor redirections for the test scripts. One might think that
+ @code{AM_TESTS_ENVIRONMENT} could be used for this purpose, but experience
+ has shown that doing so portably is practically impossible. The main
+ hurdle is constituted by Korn shells, which usually set the close-on-exec
+ flag on file descriptors opened with the @command{exec} builtin, thus
+ rendering an idiom like @code{AM_TESTS_ENVIRONMENT = exec 9>&2;}
+ ineffectual. This issue also affects some Bourne shells, such as the
+ HP-UX's @command{/bin/sh},
+ @c FIXME: should we offer a link to the relevant discussions on the
+ @c bug-autoconf list?
+@c Keep in sync with tests-environment-backcompat.test.
@example
- TESTS_ENVIRONMENT = $(PERL) -Mstrict -w
- TESTS = foo.pl bar.pl baz.pl
+ AM_TESTS_ENVIRONMENT = \
+ ## Some environment initializations are kept in a separate shell file
+ ## `tests-env.sh', which can make it easier to also run tests from
+ ## the command line.
+ . $(srcdir)/tests-env.sh; \
+ ## On Solaris, prefer more POSIX-compliant versions of the standard
+ ## tools by default.
+ if test -d /usr/xpg4/bin; then \
+ PATH=/usr/xpg4/bin:$$PATH; export PATH; \
+ fi;
+ @c $$ restore font-lock
+ ## With this, the test scripts will be able to print diagnostic messages
+ ## to the original standard error stream, even if the test driver
+ ## redirects the stderr of the test scripts to a log file before executing
+ ## them.
+ AM_TESTS_FD_REDIRECT = 9>&2
@end example
- Note that the @option{parallel-tests} driver provides a more elegant
- way to achieve the same effect, freeing the @code{TESTS_ENVIRONMENT}
- variable for the user to override (@pxref{Simple Tests using
- parallel-tests}).
-
-
- @cindex Tests, expected failure
- @cindex Expected test failure
-
- @vindex XFAIL_TESTS
- You may define the variable @code{XFAIL_TESTS} to a list of tests
- (usually a subset of @code{TESTS}) that are expected to fail. This will
- reverse the result of those tests.
+ @noindent
+ Note however that @code{AM_TESTS_ENVIRONMENT} is, for historical and
+ implementation reasons, @emph{not} supported by the serial harness
+ (@pxref{Serial Test Harness}).
Automake ensures that each file listed in @code{TESTS} is built before
- any tests are run; you can list both source and derived programs (or
- scripts) in @code{TESTS}; the generated rule will look both in
- @code{srcdir} and @file{.}. For instance, you might want to run a C
- program as a test. To do this you would list its name in @code{TESTS}
- and also in @code{check_PROGRAMS}, and then specify it as you would
- any other program.
+ it is run; you can list both source and derived programs (or scripts)
+ in @code{TESTS}; the generated rule will look both in @code{srcdir} and
+ @file{.}. For instance, you might want to run a C program as a test.
+ To do this you would list its name in @code{TESTS} and also in
+ @code{check_PROGRAMS}, and then specify it as you would any other
+ program.
Programs listed in @code{check_PROGRAMS} (and @code{check_LIBRARIES},
@code{check_LTLIBRARIES}...) are only built during @code{make check},
@vindex AM_@var{ext}_LOG_FLAGS
@vindex AM_LOG_FLAGS
For tests that match an extension @code{.@var{ext}} listed in
- @code{TEST_EXTENSIONS}, you can provide a test driver using the variable
- @code{@var{ext}_LOG_COMPILER} (note the upper-case extension) and pass
- options in @code{AM_@var{ext}_LOG_FLAGS} and allow the user to pass
- options in @code{@var{ext}_LOG_FLAGS}. It will cause all tests with
- this extension to be called with this driver. For all tests without a
- registered extension, the variables @code{LOG_COMPILER},
+ @code{TEST_EXTENSIONS}, you can provide a custom ``test runner'' using
+ the variable @code{@var{ext}_LOG_COMPILER} (note the upper-case
+ extension) and pass options in @code{AM_@var{ext}_LOG_FLAGS} and allow
+ the user to pass options in @code{@var{ext}_LOG_FLAGS}. It will cause
+ all tests with this extension to be called with this runner. For all
+ tests without a registered extension, the variables @code{LOG_COMPILER},
@code{AM_LOG_FLAGS}, and @code{LOG_FLAGS} may be used. For example,
+@c Keep in sync with parallel-tests-log-compiler-example.test.
@example
TESTS = foo.pl bar.py baz
TEST_EXTENSIONS = .pl .py
all: all-am
.SUFFIXES:
- .SUFFIXES: .html .log .pl
+ .SUFFIXES: .html .log .pl .trs
-$(srcdir)/Makefile.in: $(srcdir)/Makefile.am $(am__configure_deps)
+$(srcdir)/Makefile.in: $(srcdir)/Makefile.am $(top_srcdir)/CheckListOfTests.am $(am__configure_deps)
@for dep in $?; do \
case '$(am__configure_deps)' in \
*$$dep*) \
defs-static
instspc-tests.am
parallel-tests.am
+config-shell-tests.am
+depmod-tests.am
*.dir
*.log
+ *.trs
*.log-t
*-p.test
instspc-*-build.test
MAINTAINERCLEANFILES =
EXTRA_DIST = ChangeLog-old
- TEST_EXTENSIONS = .test
+# Run the tests with the shell detected at configure time.
+TEST_LOG_COMPILER = $(SHELL)
+
XFAIL_TESTS = \
all.test \
auxdir2.test \
AM_TESTS_ENVIRONMENT = \
test x"$$me" = x || unset me; \
test x"$$required" = x || unset required; \
+ test x"$$use_tap" = x || unset use_tap; \
test x"$$parallel_tests" = x || unset parallel_tests; \
+ test x"$$test_prefer_config_shell" || unset test_prefer_config_shell; \
test x"$$original_AUTOMAKE" = x || unset original_AUTOMAKE; \
- test x"$$original_ACLOCAL" = x || unset original_ACLOCAL;
+ test x"$$original_ACLOCAL" = x || unset original_ACLOCAL; \
+ AM_TESTS_REEXEC=no; export AM_TESTS_REEXEC; \
+ stderr_fileno_=9; export stderr_fileno_; 9>&2
TESTS = \
+ $(handwritten_tests) \
+ $(config_shell_tests) \
+ $(parallel_tests) \
+ $(depmod_tests) \
+ $(instspc_tests)
+
+EXTRA_DIST += $(handwritten_tests)
+
+handwritten_tests = \
aclocal.test \
aclocal3.test \
aclocal4.test \
check10.test \
check11.test \
check12.test \
+check-subst.test \
+check-subst-prog.test \
check-exported-srcdir.test \
+ check-fd-redirect.test \
check-tests-in-builddir.test \
- check-tests_environment.test \
- tests-environment-backcompat.test \
+ check-no-test-driver.test \
checkall.test \
clean.test \
clean2.test \
parallel-tests8.test \
parallel-tests9.test \
parallel-tests10.test \
- parallel-tests-am_tests_environment.test \
- parallel-tests-unreadable-log.test \
+parallel-tests-suffix.test \
+parallel-tests-suffix-prog.test \
+parallel-tests-ext-driver.test \
+parallel-tests-ext-driver-prog.test \
+ parallel-tests-dry-run.test \
+ parallel-tests-fd-redirect.test \
+ parallel-tests-extra-programs.test \
+ parallel-tests-unreadable.test \
parallel-tests-subdir.test \
parallel-tests-interrupt.test \
parallel-tests-reset-term.test \
parallel-tests-log-override-1.test \
parallel-tests-log-override-2.test \
parallel-tests-log-override-recheck.test \
- parallel-tests-cmdline-override.test \
+parallel-tests-log-compiler-example.test \
++parallel-tests-cmdline-override.test \
+ parallel-tests-empty-testlogs.test \
+ parallel-tests-driver-install.test \
+ parallel-tests-no-spurious-summary.test \
+ parallel-tests-exit-statuses.test \
+ parallel-tests-console-output.test \
+ parallel-tests-once.test \
+ tests-environment.test \
+ am-tests-environment.test \
++tests-environment-backcompat.test \
+ testsuite-summary-color.test \
+ testsuite-summary-count.test \
+ testsuite-summary-count-many.test \
+ testsuite-summary-reference-log.test \
+ test-driver-custom-no-extra-driver.test \
+ test-driver-custom.test \
+ test-driver-custom-xfail-tests.test \
+ test-driver-custom-multitest.test \
+ test-driver-custom-multitest-recheck.test \
+ test-driver-custom-multitest-recheck2.test \
+ test-driver-custom-html.test \
+ test-driver-custom-no-html.test \
+ test-driver-create-log-dir.test \
+ test-driver-strip-vpath.test \
+ test-driver-fail.test \
+ test-log.test \
+ test-metadata-global-log.test \
+ test-metadata-recheck.test \
+ test-metadata-results.test \
+ test-missing.test \
+ test-missing2.test \
+ test-trs-basic.test \
+ test-trs-recover.test \
+ test-trs-recover2.test \
parse.test \
percent.test \
percent2.test \
self-check-dir.test \
self-check-env-sanitize.test \
self-check-exit.test \
+self-check-explicit-skips.test \
self-check-is_newest.test \
self-check-me.test \
+self-check-reexec.test \
self-check-report.test \
self-check-sanity.test \
+self-check-seq.test \
+ self-check-tap.test \
self-check-unindent.test \
sanity.test \
scripts.test \
yflags-d-false-positives.test \
yflags-force-override.test \
yflags-force-conditional.test \
-yflags-var-expand.test \
-$(parallel_tests)
-
-EXTRA_DIST += $(TESTS)
+yflags-var-expand.test
+ ## Files containing auxiliary functions used by our test cases.
+ EXTRA_DIST += tap-functions.sh plain-functions.sh
+
+ test-driver-custom-multitest.log: trivial-test-driver
+ test-driver-custom-multitest-recheck.log: trivial-test-driver
+ test-driver-custom-multitest-recheck2.log: trivial-test-driver
+ test-driver-custom-html.log: trivial-test-driver
+ EXTRA_DIST += trivial-test-driver
+
+ testsuite-summary-color.log testsuite-summary-count.log: \
+ testsuite-summary-checks.sh extract-testsuite-summary
+ EXTRA_DIST += testsuite-summary-checks.sh
+ EXTRA_DIST += extract-testsuite-summary
+
+ testsuite-summary-count-many.log: trivial-test-driver
+ testsuite-summary-count-many.log: extract-testsuite-summary
+
+ # List of tests on TAP support that use the files pre-computed by
+ # `tap-common-setup.test', and sources the `tap-setup.sh' helper
+ # script.
+ tap_with_common_setup_tests = \
+ tap-autonumber.test \
+ tap-bailout.test \
+ tap-color.test \
+ tap-deps.test \
+ tap-diagnostic.test \
+ tap-empty-diagnostic.test \
+ tap-empty.test \
+ tap-escape-directive.test \
+ tap-exit.test \
+ tap-signal.test \
+ tap-fancy.test \
+ tap-fancy2.test \
+ tap-global-log.test \
+ tap-global-result.test \
+ tap-html.test \
+ tap-log.test \
+ tap-merge-stdout-stderr.test \
+ tap-no-merge-stdout-stderr.test \
+ tap-message-0.test \
+ tap-no-disable-hard-error.test \
+ tap-no-spurious-summary.test \
+ tap-no-spurious.test \
+ tap-not-ok-skip.test \
+ tap-numeric-description.test \
+ tap-out-of-order.test \
+ tap-passthrough.test \
+ tap-passthrough-exit.test \
+ tap-plan.test \
+ tap-plan-corner.test \
+ tap-plan-corner2.test \
+ tap-plan-errors.test \
+ tap-realtime.test \
+ tap-recheck-logs.test \
+ tap-skip-whole-whitespace.test \
+ tap-skip-whole.test \
+ tap-todo-skip-together.test \
+ tap-todo-skip-whitespace.test \
+ tap-todo-skip.test \
+ tap-unplanned.test \
+ tap-whitespace-normalization.test \
+ tap-with-and-without-number.test \
+ tap-xfail-tests.test
+
+ # Their log files.
+ tap_with_common_setup_logs = $(tap_with_common_setup_tests:.test=.log)
+
+ # Their dependencies.
+ $(tap_with_common_setup_logs): tap-common-setup.log tap-setup.sh
+ EXTRA_DIST += tap-setup.sh
+
+ # Other tests on TAP support.
+ tap_other_tests = \
+ tap-common-setup.test \
+ tap-bad-prog.tap \
+ tap-basic.test \
+ tap-doc.test \
+ tap-doc2.test \
+ tap-empty.test \
+ tap-more.test \
+ tap-more2.test \
+ tap-recheck.test \
+ tap-summary.test \
+ tap-summary-color.test
+
+ tap-summary.log tap-summary-color.log: tap-summary-aux.sh
+ EXTRA_DIST += tap-summary-aux.sh
- ## Checking the test of tests.
- checked_test_extensions = .test
++## Checking the list of tests.
++checked_test_extensions = .test .tap
+expected_list_of_tests = $(handwritten_tests)
+include $(top_srcdir)/CheckListOfTests.am
+
# Dependencies valid for each test case.
$(TEST_LOGS): defs defs-static aclocal-$(APIVERSION) automake-$(APIVERSION)
+ # FIXME: this should be made more granular once we have a cleaner
+ # subdivision of the tests.
+ $(TEST_LOGS): plain-functions.sh tap-functions.sh
clean-local: clean-local-check
.PHONY: clean-local-check
TEST_SUITE_HTML = $(TEST_SUITE_LOG:.log=.html)
am__test_logs1 = $(TESTS:=.log)
am__test_logs2 = $(am__test_logs1:.test.log=.log)
+ TEST_LOG_DRIVER = $(SHELL) $(top_srcdir)/lib/test-driver
TEST_LOG_COMPILE = $(TEST_LOG_COMPILER) $(AM_TEST_LOG_FLAGS) \
$(TEST_LOG_FLAGS)
- am__test_logs3 = $(am__test_logs2:.ptest.log=.log)
-TEST_LOGS = $(am__test_logs2:.tap.log=.log)
++am__test_logs3 = $(am__test_logs2:.tap.log=.log)
+ TAP_LOG_COMPILE = $(TAP_LOG_COMPILER) $(AM_TAP_LOG_FLAGS) \
+ $(TAP_LOG_FLAGS)
++am__test_logs4 = $(am__test_logs3:.ptest.log=.log)
++PTEST_LOG_DRIVER = $(SHELL) $(top_srcdir)/lib/test-driver
+PTEST_LOG_COMPILE = $(PTEST_LOG_COMPILER) $(AM_PTEST_LOG_FLAGS) \
+ $(PTEST_LOG_FLAGS)
- am__test_logs4 = $(am__test_logs3:.instspc.log=.log)
++am__test_logs5 = $(am__test_logs4:.instspc.log=.log)
++INSTSPC_LOG_DRIVER = $(SHELL) $(top_srcdir)/lib/test-driver
+INSTSPC_LOG_COMPILE = $(INSTSPC_LOG_COMPILER) $(AM_INSTSPC_LOG_FLAGS) \
+ $(INSTSPC_LOG_FLAGS)
- am__test_logs5 = $(am__test_logs4:.depmod.log=.log)
++am__test_logs6 = $(am__test_logs5:.depmod.log=.log)
++DEPMOD_LOG_DRIVER = $(SHELL) $(top_srcdir)/lib/test-driver
+DEPMOD_LOG_COMPILE = $(DEPMOD_LOG_COMPILER) $(AM_DEPMOD_LOG_FLAGS) \
+ $(DEPMOD_LOG_FLAGS)
- TEST_LOGS = $(am__test_logs5:.shtst.log=.log)
++TEST_LOGS = $(am__test_logs6:.shtst.log=.log)
++SHTST_LOG_DRIVER = $(SHELL) $(top_srcdir)/lib/test-driver
+SHTST_LOG_COMPILE = $(SHTST_LOG_COMPILER) $(AM_SHTST_LOG_FLAGS) \
+ $(SHTST_LOG_FLAGS)
TEST_LOGS_TMP = $(TEST_LOGS:.log=.log-t)
DISTFILES = $(DIST_COMMON) $(DIST_SOURCES) $(TEXINFOS) $(EXTRA_DIST)
ACLOCAL = @ACLOCAL@
top_build_prefix = @top_build_prefix@
top_builddir = @top_builddir@
top_srcdir = @top_srcdir@
-TEST_EXTENSIONS = .test .tap
++TEST_EXTENSIONS = .test .tap .ptest .instspc .depmod .shtst
+ TAP_LOG_DRIVER = $(PERL) $(top_srcdir)/lib/tap-driver
+ TAP_LOG_DRIVER_FLAGS = --merge
-MAINTAINERCLEANFILES = $(parallel_tests) $(instspc_tests)
-EXTRA_DIST = ChangeLog-old gen-parallel-tests instspc-tests.sh \
- $(TESTS) tap-functions.sh plain-functions.sh \
- trivial-test-driver testsuite-summary-checks.sh \
- extract-testsuite-summary tap-setup.sh tap-summary-aux.sh
+MAINTAINERCLEANFILES =
+EXTRA_DIST = ChangeLog-old gen-parallel-tests parallel-tests.sh \
+ instspc-tests.sh depmod-tests.sh gen-config-shell-tests \
- config-shell-tests.sh $(handwritten_tests)
- TEST_EXTENSIONS = .test .ptest .instspc .depmod .shtst
++ config-shell-tests.sh $(handwritten_tests) tap-functions.sh \
++ plain-functions.sh trivial-test-driver \
++ testsuite-summary-checks.sh extract-testsuite-summary \
++ tap-setup.sh tap-summary-aux.sh
++
+# Run the tests with the shell detected at configure time.
+TEST_LOG_COMPILER = $(SHELL)
XFAIL_TESTS = all.test auxdir2.test cond17.test gcj6.test \
override-conditional-2.test pr8365-remake-timing.test \
- yacc-dist-nobuild-subdir.test txinfo5.test \
+ yacc-dist-nobuild-subdir.test tap-plan-corner2.test \
+ tap-message-0.test tap-signal.test txinfo5.test \
$(instspc_xfail_tests)
-parallel_tests = backcompat5-p.test check-exported-srcdir-p.test \
- check-fd-redirect-p.test check-tests-in-builddir-p.test \
- check-p.test check11-p.test check12-p.test check2-p.test \
- check3-p.test check4-p.test check5-p.test check6-p.test \
- check7-p.test check8-p.test check9-p.test color-p.test \
- color2-p.test comment9-p.test dejagnu-p.test exeext4-p.test \
- maken3-p.test maken4-p.test posixsubst-tests-p.test \
- tests-environment-p.test
-instspc_tests = instspc-squote-build.test instspc-squote-install.test \
- instspc-dquote-build.test instspc-dquote-install.test \
- instspc-bquote-build.test instspc-bquote-install.test \
- instspc-sharp-build.test instspc-sharp-install.test \
- instspc-dollar-build.test instspc-dollar-install.test \
- instspc-bang-build.test instspc-bang-install.test \
- instspc-bslash-build.test instspc-bslash-install.test \
- instspc-ampersand-build.test instspc-ampersand-install.test \
- instspc-percent-build.test instspc-percent-install.test \
- instspc-leftpar-build.test instspc-leftpar-install.test \
- instspc-rightpar-build.test instspc-rightpar-install.test \
- instspc-pipe-build.test instspc-pipe-install.test \
- instspc-caret-build.test instspc-caret-install.test \
- instspc-tilde-build.test instspc-tilde-install.test \
- instspc-qmark-build.test instspc-qmark-install.test \
- instspc-star-build.test instspc-star-install.test \
- instspc-plus-build.test instspc-plus-install.test \
- instspc-minus-build.test instspc-minus-install.test \
- instspc-comma-build.test instspc-comma-install.test \
- instspc-colon-build.test instspc-colon-install.test \
- instspc-semicol-build.test instspc-semicol-install.test \
- instspc-equal-build.test instspc-equal-install.test \
- instspc-less-build.test instspc-less-install.test \
- instspc-more-build.test instspc-more-install.test \
- instspc-at-build.test instspc-at-install.test \
- instspc-lqbrack-build.test instspc-lqbrack-install.test \
- instspc-rqbrack-build.test instspc-rqbrack-install.test \
- instspc-lcbrack-build.test instspc-lcbrack-install.test \
- instspc-rcbrack-build.test instspc-rcbrack-install.test \
- instspc-space-build.test instspc-space-install.test \
- instspc-tab-build.test instspc-tab-install.test \
- instspc-linefeed-build.test instspc-linefeed-install.test \
- instspc-backspace-build.test instspc-backspace-install.test \
- instspc-formfeed-build.test instspc-formfeed-install.test \
- instspc-carriageret-build.test \
- instspc-carriageret-install.test \
- instspc-quadrigraph0-build.test \
- instspc-quadrigraph0-install.test \
- instspc-quadrigraph1-build.test \
- instspc-quadrigraph1-install.test \
- instspc-quadrigraph2-build.test \
- instspc-quadrigraph2-install.test \
- instspc-quadrigraph3-build.test \
- instspc-quadrigraph3-install.test \
- instspc-quadrigraph4-build.test \
- instspc-quadrigraph4-install.test instspc-a_b-build.test \
- instspc-a_b-install.test instspc-a__b-build.test \
- instspc-a__b-install.test instspc-a_lf_b-build.test \
- instspc-a_lf_b-install.test instspc-dotdotdot-build.test \
- instspc-dotdotdot-install.test instspc-dosdrive-build.test \
- instspc-dosdrive-install.test instspc-miscglob1-build.test \
- instspc-miscglob1-install.test instspc-miscglob2-build.test \
- instspc-miscglob2-install.test
-instspc_xfail_tests = instspc-squote-build.test \
- instspc-dquote-build.test instspc-bquote-build.test \
- instspc-sharp-build.test instspc-dollar-build.test \
- instspc-bslash-build.test instspc-ampersand-build.test \
- instspc-linefeed-build.test instspc-quadrigraph0-build.test \
- instspc-a_lf_b-build.test instspc-squote-install.test \
- instspc-dquote-install.test instspc-bquote-install.test \
- instspc-sharp-install.test instspc-dollar-install.test \
- instspc-linefeed-install.test instspc-a_lf_b-install.test
+parallel_tests = backcompat5-p.ptest check-exported-srcdir-p.ptest \
- check-subst-prog-p.ptest check-subst-p.ptest \
- check-tests-in-builddir-p.ptest \
- check-tests_environment-p.ptest check-p.ptest check10-p.ptest \
- check11-p.ptest check12-p.ptest check2-p.ptest check3-p.ptest \
- check4-p.ptest check5-p.ptest check6-p.ptest check7-p.ptest \
- check8-p.ptest color-p.ptest color2-p.ptest comment9-p.ptest \
- dejagnu-p.ptest exeext4-p.ptest maken3-p.ptest maken4-p.ptest \
- posixsubst-tests-p.ptest
++ check-fd-redirect-p.ptest check-subst-prog-p.ptest \
++ check-subst-p.ptest check-tests-in-builddir-p.ptest \
++ check-p.ptest check11-p.ptest check12-p.ptest check2-p.ptest \
++ check3-p.ptest check4-p.ptest check5-p.ptest check6-p.ptest \
++ check7-p.ptest check8-p.ptest color-p.ptest color2-p.ptest \
++ comment9-p.ptest dejagnu-p.ptest exeext4-p.ptest \
++ maken3-p.ptest maken4-p.ptest posixsubst-tests-p.ptest \
++ tests-environment-p.ptest
+PTEST_LOG_COMPILER = $(SHELL) $(srcdir)/parallel-tests.sh
+instspc_tests = build-squote.instspc install-squote.instspc \
+ build-dquote.instspc install-dquote.instspc \
+ build-bquote.instspc install-bquote.instspc \
+ build-sharp.instspc install-sharp.instspc build-dollar.instspc \
+ install-dollar.instspc build-bang.instspc install-bang.instspc \
+ build-bslash.instspc install-bslash.instspc \
+ build-ampersand.instspc install-ampersand.instspc \
+ build-percent.instspc install-percent.instspc \
+ build-leftpar.instspc install-leftpar.instspc \
+ build-rightpar.instspc install-rightpar.instspc \
+ build-pipe.instspc install-pipe.instspc build-caret.instspc \
+ install-caret.instspc build-tilde.instspc \
+ install-tilde.instspc build-qmark.instspc \
+ install-qmark.instspc build-star.instspc install-star.instspc \
+ build-plus.instspc install-plus.instspc build-minus.instspc \
+ install-minus.instspc build-comma.instspc \
+ install-comma.instspc build-colon.instspc \
+ install-colon.instspc build-semicol.instspc \
+ install-semicol.instspc build-equal.instspc \
+ install-equal.instspc build-less.instspc install-less.instspc \
+ build-more.instspc install-more.instspc build-at.instspc \
+ install-at.instspc build-lqbrack.instspc \
+ install-lqbrack.instspc build-rqbrack.instspc \
+ install-rqbrack.instspc build-lcbrack.instspc \
+ install-lcbrack.instspc build-rcbrack.instspc \
+ install-rcbrack.instspc build-space.instspc \
+ install-space.instspc build-tab.instspc install-tab.instspc \
+ build-linefeed.instspc install-linefeed.instspc \
+ build-backspace.instspc install-backspace.instspc \
+ build-formfeed.instspc install-formfeed.instspc \
+ build-carriageret.instspc install-carriageret.instspc \
+ build-quadrigraph0.instspc install-quadrigraph0.instspc \
+ build-quadrigraph1.instspc install-quadrigraph1.instspc \
+ build-quadrigraph2.instspc install-quadrigraph2.instspc \
+ build-quadrigraph3.instspc install-quadrigraph3.instspc \
+ build-quadrigraph4.instspc install-quadrigraph4.instspc \
+ build-a_b.instspc install-a_b.instspc build-a__b.instspc \
+ install-a__b.instspc build-a_lf_b.instspc \
+ install-a_lf_b.instspc build-dotdotdot.instspc \
+ install-dotdotdot.instspc build-dosdrive.instspc \
+ install-dosdrive.instspc build-miscglob1.instspc \
+ install-miscglob1.instspc build-miscglob2.instspc \
+ install-miscglob2.instspc
+instspc_xfail_tests = build-squote.instspc build-dquote.instspc \
+ build-bquote.instspc build-sharp.instspc build-dollar.instspc \
+ build-bslash.instspc build-ampersand.instspc \
+ build-linefeed.instspc build-quadrigraph0.instspc \
+ build-a_lf_b.instspc install-squote.instspc \
+ install-dquote.instspc install-bquote.instspc \
+ install-sharp.instspc install-dollar.instspc \
+ install-linefeed.instspc install-a_lf_b.instspc
+INSTSPC_LOG_COMPILER = $(SHELL) $(srcdir)/instspc-tests.sh
+depmod_tests = depcomp-gcc3.depmod depcomp-gcc.depmod \
+ depcomp-hp.depmod depcomp-sgi.depmod depcomp-aix.depmod \
+ depcomp-icc.depmod depcomp-hp2.depmod depcomp-tru64.depmod \
+ depcomp-msvc7.depmod depcomp-msvc7msys.depmod \
+ depcomp-nosideeffect.depmod depcomp-dashmstdout.depmod \
+ depcomp-dashXmstdout.depmod depcomp-makedepend.depmod \
+ depcomp-cpp.depmod depcomp-msvisualcpp.depmod \
+ depcomp-msvcmsys.depmod
+DEPMOD_LOG_COMPILER = $(SHELL) $(srcdir)/depmod-tests.sh
+config_shell_tests = ar-lib-w.shtst compile-w.shtst compile2-w.shtst \
+ compile3-w.shtst compile4-w.shtst compile5-w.shtst \
+ compile6-w.shtst instsh2-w.shtst instsh3-w.shtst \
+ mdate5-w.shtst mdate6-w.shtst missing-w.shtst missing2-w.shtst \
+ missing3-w.shtst missing5-w.shtst mkinst3-w.shtst
+SHTST_LOG_COMPILER = $(SHELL) $(srcdir)/config-shell-tests.sh
# Some testsuite-influential variables should be overridable from the
# test scripts, but not from the environment.
AM_TESTS_ENVIRONMENT = \
test x"$$me" = x || unset me; \
test x"$$required" = x || unset required; \
+ test x"$$use_tap" = x || unset use_tap; \
test x"$$parallel_tests" = x || unset parallel_tests; \
+ test x"$$test_prefer_config_shell" || unset test_prefer_config_shell; \
test x"$$original_AUTOMAKE" = x || unset original_AUTOMAKE; \
- test x"$$original_ACLOCAL" = x || unset original_ACLOCAL;
+ test x"$$original_ACLOCAL" = x || unset original_ACLOCAL; \
+ AM_TESTS_REEXEC=no; export AM_TESTS_REEXEC; \
+ stderr_fileno_=9; export stderr_fileno_; 9>&2
TESTS = \
+ $(handwritten_tests) \
+ $(config_shell_tests) \
+ $(parallel_tests) \
+ $(depmod_tests) \
+ $(instspc_tests)
+
+handwritten_tests = \
aclocal.test \
aclocal3.test \
aclocal4.test \
check10.test \
check11.test \
check12.test \
+check-subst.test \
+check-subst-prog.test \
check-exported-srcdir.test \
+ check-fd-redirect.test \
check-tests-in-builddir.test \
- check-tests_environment.test \
- tests-environment-backcompat.test \
+ check-no-test-driver.test \
checkall.test \
clean.test \
clean2.test \
parallel-tests8.test \
parallel-tests9.test \
parallel-tests10.test \
- parallel-tests-am_tests_environment.test \
- parallel-tests-unreadable-log.test \
+parallel-tests-suffix.test \
+parallel-tests-suffix-prog.test \
+parallel-tests-ext-driver.test \
+parallel-tests-ext-driver-prog.test \
+ parallel-tests-dry-run.test \
+ parallel-tests-fd-redirect.test \
+ parallel-tests-extra-programs.test \
+ parallel-tests-unreadable.test \
parallel-tests-subdir.test \
parallel-tests-interrupt.test \
parallel-tests-reset-term.test \
parallel-tests-log-override-1.test \
parallel-tests-log-override-2.test \
parallel-tests-log-override-recheck.test \
- parallel-tests-cmdline-override.test \
+parallel-tests-log-compiler-example.test \
++parallel-tests-cmdline-override.test \
+ parallel-tests-empty-testlogs.test \
+ parallel-tests-driver-install.test \
+ parallel-tests-no-spurious-summary.test \
+ parallel-tests-exit-statuses.test \
+ parallel-tests-console-output.test \
+ parallel-tests-once.test \
+ tests-environment.test \
+ am-tests-environment.test \
++tests-environment-backcompat.test \
+ testsuite-summary-color.test \
+ testsuite-summary-count.test \
+ testsuite-summary-count-many.test \
+ testsuite-summary-reference-log.test \
+ test-driver-custom-no-extra-driver.test \
+ test-driver-custom.test \
+ test-driver-custom-xfail-tests.test \
+ test-driver-custom-multitest.test \
+ test-driver-custom-multitest-recheck.test \
+ test-driver-custom-multitest-recheck2.test \
+ test-driver-custom-html.test \
+ test-driver-custom-no-html.test \
+ test-driver-create-log-dir.test \
+ test-driver-strip-vpath.test \
+ test-driver-fail.test \
+ test-log.test \
+ test-metadata-global-log.test \
+ test-metadata-recheck.test \
+ test-metadata-results.test \
+ test-missing.test \
+ test-missing2.test \
+ test-trs-basic.test \
+ test-trs-recover.test \
+ test-trs-recover2.test \
parse.test \
percent.test \
percent2.test \
self-check-dir.test \
self-check-env-sanitize.test \
self-check-exit.test \
+self-check-explicit-skips.test \
self-check-is_newest.test \
self-check-me.test \
+self-check-reexec.test \
self-check-report.test \
self-check-sanity.test \
+self-check-seq.test \
+ self-check-tap.test \
self-check-unindent.test \
sanity.test \
scripts.test \
yflags-d-false-positives.test \
yflags-force-override.test \
yflags-force-conditional.test \
-yflags-var-expand.test \
-$(parallel_tests)
+yflags-var-expand.test
- checked_test_extensions = .test
+
+ # List of tests on TAP support that use the files pre-computed by
+ # `tap-common-setup.test', and sources the `tap-setup.sh' helper
+ # script.
+ tap_with_common_setup_tests = \
+ tap-autonumber.test \
+ tap-bailout.test \
+ tap-color.test \
+ tap-deps.test \
+ tap-diagnostic.test \
+ tap-empty-diagnostic.test \
+ tap-empty.test \
+ tap-escape-directive.test \
+ tap-exit.test \
+ tap-signal.test \
+ tap-fancy.test \
+ tap-fancy2.test \
+ tap-global-log.test \
+ tap-global-result.test \
+ tap-html.test \
+ tap-log.test \
+ tap-merge-stdout-stderr.test \
+ tap-no-merge-stdout-stderr.test \
+ tap-message-0.test \
+ tap-no-disable-hard-error.test \
+ tap-no-spurious-summary.test \
+ tap-no-spurious.test \
+ tap-not-ok-skip.test \
+ tap-numeric-description.test \
+ tap-out-of-order.test \
+ tap-passthrough.test \
+ tap-passthrough-exit.test \
+ tap-plan.test \
+ tap-plan-corner.test \
+ tap-plan-corner2.test \
+ tap-plan-errors.test \
+ tap-realtime.test \
+ tap-recheck-logs.test \
+ tap-skip-whole-whitespace.test \
+ tap-skip-whole.test \
+ tap-todo-skip-together.test \
+ tap-todo-skip-whitespace.test \
+ tap-todo-skip.test \
+ tap-unplanned.test \
+ tap-whitespace-normalization.test \
+ tap-with-and-without-number.test \
+ tap-xfail-tests.test
+
+
+ # Their log files.
+ tap_with_common_setup_logs = $(tap_with_common_setup_tests:.test=.log)
+
+ # Other tests on TAP support.
+ tap_other_tests = \
+ tap-common-setup.test \
+ tap-bad-prog.tap \
+ tap-basic.test \
+ tap-doc.test \
+ tap-doc2.test \
+ tap-empty.test \
+ tap-more.test \
+ tap-more2.test \
+ tap-recheck.test \
+ tap-summary.test \
+ tap-summary-color.test
+
++checked_test_extensions = .test .tap
+expected_list_of_tests = $(handwritten_tests)
+am__tmk = tests-in-makefile-list.tmp
+am__tfs = tests-on-filesystem-list.tmp
+am__tdf = diff-in-tests-lists.tmp
all: all-am
.SUFFIXES:
- .SUFFIXES: .depmod .html .instspc .log .ptest .shtst .test
-.SUFFIXES: .html .log .tap .test .trs
-$(srcdir)/Makefile.in: $(srcdir)/Makefile.am $(srcdir)/parallel-tests.am $(srcdir)/instspc-tests.am $(am__configure_deps)
++.SUFFIXES: .depmod .html .instspc .log .ptest .shtst .tap .test .trs
+$(srcdir)/Makefile.in: $(srcdir)/Makefile.am $(srcdir)/parallel-tests.am $(srcdir)/instspc-tests.am $(srcdir)/depmod-tests.am $(srcdir)/config-shell-tests.am $(top_srcdir)/CheckListOfTests.am $(am__configure_deps)
@for dep in $?; do \
case '$(am__configure_deps)' in \
*$$dep*) \
list=`echo "$$list" | sed 's/ *$$//'`; \
$(MAKE) $(AM_MAKEFLAGS) $$target AM_MAKEFLAGS='$(AM_MAKEFLAGS) TEST_LOGS="'"$$list"'"'
.test.log:
- @p='$<'; $(am__check_pre) $(TEST_LOG_COMPILE) "$$tst" $(am__check_post)
+ @p='$<'; $(am__check_pre) $(TEST_LOG_DRIVER) --test-name "$$f" \
+ --log-file '$*.log' --trs-file '$*.trs' \
+ $(am__common_driver_flags) $(AM_TEST_LOG_DRIVER_FLAGS) $(TEST_LOG_DRIVER_FLAGS) -- $(TEST_LOG_COMPILE) "$$tst" \
+ $(AM_TESTS_FD_REDIRECT)
+ .tap.log:
+ @p='$<'; $(am__check_pre) $(TAP_LOG_DRIVER) --test-name "$$f" \
+ --log-file '$*.log' --trs-file '$*.trs' \
+ $(am__common_driver_flags) $(AM_TAP_LOG_DRIVER_FLAGS) $(TAP_LOG_DRIVER_FLAGS) -- $(TAP_LOG_COMPILE) "$$tst" \
+ $(AM_TESTS_FD_REDIRECT)
+.ptest.log:
- @p='$<'; $(am__check_pre) $(PTEST_LOG_COMPILE) "$$tst" $(am__check_post)
++ @p='$<'; $(am__check_pre) $(PTEST_LOG_DRIVER) --test-name "$$f" \
++ --log-file '$*.log' --trs-file '$*.trs' \
++ $(am__common_driver_flags) $(AM_PTEST_LOG_DRIVER_FLAGS) $(PTEST_LOG_DRIVER_FLAGS) -- $(PTEST_LOG_COMPILE) "$$tst" \
++ $(AM_TESTS_FD_REDIRECT)
+.instspc.log:
- @p='$<'; $(am__check_pre) $(INSTSPC_LOG_COMPILE) "$$tst" $(am__check_post)
++ @p='$<'; $(am__check_pre) $(INSTSPC_LOG_DRIVER) --test-name "$$f" \
++ --log-file '$*.log' --trs-file '$*.trs' \
++ $(am__common_driver_flags) $(AM_INSTSPC_LOG_DRIVER_FLAGS) $(INSTSPC_LOG_DRIVER_FLAGS) -- $(INSTSPC_LOG_COMPILE) "$$tst" \
++ $(AM_TESTS_FD_REDIRECT)
+.depmod.log:
- @p='$<'; $(am__check_pre) $(DEPMOD_LOG_COMPILE) "$$tst" $(am__check_post)
++ @p='$<'; $(am__check_pre) $(DEPMOD_LOG_DRIVER) --test-name "$$f" \
++ --log-file '$*.log' --trs-file '$*.trs' \
++ $(am__common_driver_flags) $(AM_DEPMOD_LOG_DRIVER_FLAGS) $(DEPMOD_LOG_DRIVER_FLAGS) -- $(DEPMOD_LOG_COMPILE) "$$tst" \
++ $(AM_TESTS_FD_REDIRECT)
+.shtst.log:
- @p='$<'; $(am__check_pre) $(SHTST_LOG_COMPILE) "$$tst" $(am__check_post)
++ @p='$<'; $(am__check_pre) $(SHTST_LOG_DRIVER) --test-name "$$f" \
++ --log-file '$*.log' --trs-file '$*.trs' \
++ $(am__common_driver_flags) $(AM_SHTST_LOG_DRIVER_FLAGS) $(SHTST_LOG_DRIVER_FLAGS) -- $(SHTST_LOG_COMPILE) "$$tst" \
++ $(AM_TESTS_FD_REDIRECT)
distdir: $(DISTFILES)
@srcdirstrip=`echo "$(srcdir)" | sed 's/[].[^$$\\*]/\\\\&/g'`; \
backcompat5-p.log: backcompat5.test
check-exported-srcdir-p.log: check-exported-srcdir.test
+ check-fd-redirect-p.log: check-fd-redirect.test
+check-subst-prog-p.log: check-subst-prog.test
+check-subst-p.log: check-subst.test
check-tests-in-builddir-p.log: check-tests-in-builddir.test
- check-tests_environment-p.log: check-tests_environment.test
check-p.log: check.test
- check10-p.log: check10.test
check11-p.log: check11.test
check12-p.log: check12.test
check2-p.log: check2.test
$(AM_V_GEN)($(am__cd) $(srcdir) \
&& $(SHELL) ./instspc-tests.sh --generate-makefile) >$@
-$(instspc_tests): Makefile.am
- $(AM_V_at)rm -f $@ $@-t
- $(AM_V_GEN) :; \
- base=`expr 'x/$@' : 'x.*/instspc-\(.*\)\.test$$'`; \
- name=`expr x"$$base" : x'\(.*\)-'`; \
- action=`expr x"$$base" : x'.*-\(.*\)'`; \
- { \
- echo '#!/bin/sh'; \
- echo '# DO NOT EDIT! GENERATED AUTOMATICALLY!'; \
- echo; \
- echo '# Ensure proper definition of $$testsrcdir.'; \
- echo '. ./defs-static || exit 99'; \
- echo 'test -n "$$testsrcdir" || exit 99 # sanity check'; \
- echo; \
- echo "instspc_test_name='$$name'"; \
- echo "instspc_action='test-$$action'"; \
- echo ". \$$testsrcdir/instspc-tests.sh"; \
- } > $@-t
- $(AM_V_at)chmod a+rx $@-t && mv -f $@-t $@
-
-# All instspc-*.test tests work by sourcing the `instspc-tests.sh'
-# script. Also, they all use shared data generated by the helper
-# test `instspc-data.test', for reasons of speed.
+# All `*.instspc' tests work by calling the script `instspc-tests.sh'.
+# Also, they all use shared data generated by the `instspc-data.test'
+# helper test (for reasons of speed).
instspc-data.log: instspc-tests.sh
-$(instspc_tests:.test=.log): instspc-tests.sh instspc-data.log
+$(instspc_tests:.instspc=.log): instspc-tests.sh instspc-data.log
+# This dummy declaration is required to have make actually produce
+# expected log files from the `.instspc.log' suffix rule.
+$(instspc_tests):
+
+$(srcdir)/depmod-tests.am: depmod-tests.sh Makefile.am
+ $(AM_V_GEN)($(am__cd) $(srcdir) \
+ && $(SHELL) ./depmod-tests.sh --generate-makefile) >$@
+
+# All `*.depmod' tests work by calling the script `depmod-tests.sh'.
+# Also, they all use shared data generated by the `depmod-data.test'
+# helper test (for reasons of speed).
+depmod-data.log: depmod-tests.sh
+$(depmod_tests:.depmod=.log): depmod-tests.sh depmod-data.log
+# This dummy declaration is required to have make actually produce
+# expected log files from the `.depmod.log' suffix rule.
+$(depmod_tests):
+ar-lib-w.log: ar-lib.test
+compile-w.log: compile.test
+compile2-w.log: compile2.test
+compile3-w.log: compile3.test
+compile4-w.log: compile4.test
+compile5-w.log: compile5.test
+compile6-w.log: compile6.test
+instsh2-w.log: instsh2.test
+instsh3-w.log: instsh3.test
+mdate5-w.log: mdate5.test
+mdate6-w.log: mdate6.test
+missing-w.log: missing.test
+missing2-w.log: missing2.test
+missing3-w.log: missing3.test
+missing5-w.log: missing5.test
+mkinst3-w.log: mkinst3.test
+
+$(srcdir)/config-shell-tests.am: gen-config-shell-tests Makefile.am
+ $(AM_V_GEN)($(am__cd) $(srcdir) && $(SHELL) ./gen-config-shell-tests) >$@
+
+# All `*-w.shtst' tests work by sourcing the script `config-shell-tests.sh'.
+$(config_shell_tests:.shtst=.log): config-shell-tests.sh
+# This dummy declaration is required to have make actually produce
+# expected log files from the `.shtst.log' suffix rule.
+$(config_shell_tests):
+ test-driver-custom-multitest.log: trivial-test-driver
+ test-driver-custom-multitest-recheck.log: trivial-test-driver
+ test-driver-custom-multitest-recheck2.log: trivial-test-driver
+ test-driver-custom-html.log: trivial-test-driver
+
+ testsuite-summary-color.log testsuite-summary-count.log: \
+ testsuite-summary-checks.sh extract-testsuite-summary
+
+ testsuite-summary-count-many.log: trivial-test-driver
+ testsuite-summary-count-many.log: extract-testsuite-summary
+
+ # Their dependencies.
+ $(tap_with_common_setup_logs): tap-common-setup.log tap-setup.sh
+
+ tap-summary.log tap-summary-color.log: tap-summary-aux.sh
+
+.PHONY: maintainer-check-list-of-tests
+maintainer-check-list-of-tests:
+ @if diff -u /dev/null /dev/null >/dev/null 2>&1; then \
+ diff='diff -u'; \
+ else \
+ diff='diff'; \
+ fi; \
+ set X $(checked_test_extensions); shift; \
+ if test $$# -eq 0; then \
+ set X $(TEST_EXTENSIONS); shift; \
+ fi; \
+ if test $$# -eq 0; then \
+ echo '$@: internal error: $$(checked_test_extensions) and' \
+ '$$(TEST_EXTENSIONS) are both undefined or empty' >&2; \
+ exit 255; \
+ fi; \
+ lst='$(expected_list_of_tests)'; \
+ test -n "$$lst" || lst='$(TESTS)'; \
+ for t in $$lst; do \
+ echo "$$t"; \
+ done | sort >$(am__tmk); \
+ for ext in $$*; do \
+ ls *$$ext 2>/dev/null; \
+ if test $(srcdir) != $(builddir); then \
+ (cd $(srcdir) && ls *$$ext 2>/dev/null); \
+ fi; \
+ done | sort | uniq >$(am__tfs); \
+ if $$diff $(am__tmk) $(am__tfs) >$(am__tdf); then \
+ result=0; \
+ else \
+ echo '$@: list of tests in Makefile an on filesystem differ' >&2; \
+ echo "+ $$diff in-makefile on-filesystem" >&2; \
+ cat $(am__tdf) >&2; \
+ result=1; \
+ fi; \
+ rm -f $(am__tmk) $(am__tfs) $(am__tdf); \
+ exit $$result;
+
+.PHONY: clean-maintcheck-testslist-tmp
+clean-local: clean-maintcheck-testslist-tmp
+clean-maintcheck-testslist-tmp:
+ rm -f $(am__tmk) $(am__tfs) $(am__tdf)
+
# Dependencies valid for each test case.
$(TEST_LOGS): defs defs-static aclocal-$(APIVERSION) automake-$(APIVERSION)
+ # FIXME: this should be made more granular once we have a cleaner
+ # subdivision of the tests.
+ $(TEST_LOGS): plain-functions.sh tap-functions.sh
clean-local: clean-local-check
.PHONY: clean-local-check
$AUTOMAKE -a
./configure
-env TESTS=skip $MAKE -e check >stdout
+
+env TESTS=skip $MAKE -e check >stdout || { cat stdout; Exit 1; }
cat stdout
- grep '1.*passed' stdout && Exit 1
+ if test x"$parallel_tests" = x"yes"; then
+ count_test_results total=1 pass=0 fail=0 skip=1 xfail=0 xpass=0 error=0
+ else
+ grep '1.*passed' stdout && Exit 1
+ : For shells with buggy 'set -e'.
+ fi
-env TESTS="skip skip2" $MAKE -e check >stdout
+env TESTS="skip skip2" $MAKE -e check >stdout || { cat stdout; Exit 1; }
cat stdout
- grep '2.*passed' stdout && Exit 1
+ if test x"$parallel_tests" = x"yes"; then
+ count_test_results total=2 pass=0 fail=0 skip=2 xfail=0 xpass=0 error=0
+ else
+ grep '2.*passed' stdout && Exit 1
+ : For shells with buggy 'set -e'.
+ fi
:
$ACLOCAL
$AUTOCONF
- $AUTOMAKE
+ $AUTOMAKE -a
./configure --prefix "`pwd`/inst"
-$MAKE check >stdout
+
+$MAKE check >stdout || { cat stdout; Exit 1; }
cat stdout
-grep 'PASS: subrun.sh' stdout
+grep '^PASS: subrun\.sh *$' stdout
+grep 'PASS.*echo\.sh' stdout && Exit 1
-# check should depend directly on check-am (similar tests are
-# in check.test and check2.test).
-$FGREP 'check: $(BUILT_SOURCES)' Makefile.in
-$FGREP 'check: $(BUILT_SOURCES)' dir/Makefile.in
+# check should depend directly on $(BUILT_SOURCES) (similar tests
+# are in check.test and check2.test).
+$EGREP '^check:.* \$\(BUILT_SOURCES\)( |$)' Makefile.in
+$EGREP '^check:.* \$\(BUILT_SOURCES\)( |$)' dir/Makefile.in
$MAKE clean
# Sanity checks
Exit ()
{
set +e
- (exit $1)
- exit $1
+ # See comments in the exit trap for the reason we do this.
+ test 77 = $1 && am__test_skipped=yes
+ (exit $1); exit $1
}
- # Print warnings (e.g., about skipped and failed tests) to this file
- # number. Override by putting, say:
- # stderr_fileno_=9; export stderr_fileno_; 9>&2
- # at the *end* (yes, this is mandatory; see comments in Makefile.am) of
- # the definition of AM_TESTS_ENVIRONMENT.
- # This is useful when using automake's parallel tests mode, to print the
- # reason for skip/failure to console, rather than to the *.log files.
- : ${stderr_fileno_=2}
-
- # Copied from Gnulib's `tests/init.sh'.
- warn_ () { echo "$@" 1>&$stderr_fileno_; }
- fail_ () { warn_ "$me: failed test: $@"; Exit 1; }
- skip_ () { warn_ "$me: skipped test: $@"; Exit 77; }
- fatal_ () { warn_ "$me: hard error: $@"; Exit 99; }
- framework_failure_ () { warn_ "$me: set-up failure: $@"; Exit 99; }
+ if test $using_tap = yes; then
+ funcs_file_=tap-functions.sh
+ else
+ funcs_file_=plain-functions.sh
+ fi
+
+ if test -f "$testsrcdir/$funcs_file_"; then
+ . "$testsrcdir/$funcs_file_"
+ else
+ echo "$me: $testsrcdir/$funcs_file_ not found, check \$testsrcdir" >&2
+ Exit 99
+ fi
+ unset funcs_file_
# cross_compiling
# ---------------
Exit 99;;
esac
}
+am__using_gmake="" # Avoid interferences from the environment.
+
+# seq_ - print a sequence of numbers
+# ----------------------------------
+# This function simulates GNU seq(1) portably. Valid usages:
+# - seq LAST
+# - seq FIRST LAST
+# - seq FIRST INCREMENT LAST
+seq_ ()
+{
+ case $# in
+ 0) fatal_ "seq_: missing argument";;
+ 1) seq_first=1 seq_incr=1 seq_last=$1;;
+ 2) seq_first=$1 seq_incr=1 seq_last=$2;;
+ 3) seq_first=$1 seq_incr=$2 seq_last=$3;;
+ *) fatal_ "seq_: too many arguments";;
+ esac
+ # Try to avoid forks if possible.
+ case "$BASH_VERSION" in
+ ""|[12].*)
+ : Not bash, or a too old bash version. ;;
+ *)
+ # Use eval to protect dumber shells from parsing errors.
+ eval 'for ((i = seq_first; i <= seq_last; i += seq_incr)); do
+ echo $i
+ done'
+ return 0;;
+ esac
+ # Else, use GNU seq if available.
+ seq "$@" && return 0
+ # Otherwise revert to a slower loop using expr(1).
+ i=$seq_first
+ while test $i -le $seq_last; do
+ echo $i
+ i=`expr $i + $seq_incr`
+ done
+}
+ # count_test_results total=N pass=N fail=N xpass=N xfail=N skip=N error=N
+ # -----------------------------------------------------------------------
+ # Check that a testsuite run driven by the parallel-tests harness has
+ # had the specified numbers of test results (specified by kind).
+ # This function assumes that the output of "make check" or "make recheck"
+ # has been saved in the `stdout' file in the current directory, and its
+ # log in the `test-suite.log' file.
+ count_test_results ()
+ {
+ # Use a subshell so that we won't pollute the script namespace.
+ (
+ # TODO: Do proper checks on the arguments?
+ total=ERR pass=ERR fail=ERR xpass=ERR xfail=ERR skip=ERR error=ERR
+ eval "$@"
+ # For debugging.
+ $EGREP -i '(total|x?pass|x?fail|skip|error)' stdout || :
+ rc=0
+ # Avoid spurious failures with shells with "overly sensible"
+ # `errexit' shell flag, such as e.g., Solaris /bin/sh.
+ set +e
+ test `grep -c '^PASS:' stdout` -eq $pass || rc=1
+ test `grep -c '^XFAIL:' stdout` -eq $xfail || rc=1
+ test `grep -c '^SKIP:' stdout` -eq $skip || rc=1
+ test `grep -c '^FAIL:' stdout` -eq $fail || rc=1
+ test `grep -c '^XPASS:' stdout` -eq $xpass || rc=1
+ test `grep -c '^ERROR:' stdout` -eq $error || rc=1
+ grep "^# TOTAL: *$total$" stdout || rc=1
+ grep "^# PASS: *$pass$" stdout || rc=1
+ grep "^# XFAIL: *$xfail$" stdout || rc=1
+ grep "^# SKIP: *$skip$" stdout || rc=1
+ grep "^# FAIL: *$fail$" stdout || rc=1
+ grep "^# XPASS: *$xpass$" stdout || rc=1
+ grep "^# ERROR: *$error$" stdout || rc=1
+ test $rc -eq 0
+ )
+ }
+
commented_sed_unindent_prog='
/^$/b # Nothing to do for empty lines.
x # Get x<indent> into pattern space.
# Check that the environment is properly sanitized.
# Having variables exported to the empty string is OK, since our code
# treats such variables as if they were unset.
- for var in me required parallel_tests test_prefer_config_shell \
- original_AUTOMAKE original_ACLOCAL
-for var in me required use_tap parallel_tests \
++for var in me required use_tap parallel_tests test_prefer_config_shell \
+ original_AUTOMAKE original_ACLOCAL;
do
if eval "test x\"\$$var\" != x" && env | grep "^$var=" >/dev/null; then
echo "$argv0: variable \`$var' is set in the environment:" \
$AUTOMAKE -a
./configure
-$MAKE check
++
+ $MAKE
- grep 'chk-driver *1 *2 ' foo.log
- grep 'test-driver *3 *4 ' bar.log
- grep 'noext-driver *5 *6 ' baz.log
- grep 'test-driver *3 *4 ' bla.log
- grep 'noext-driver *5 *6 ' bli.suff.log
- grep 'noext-driver *5 *6 ' sub/test.log
+$MAKE check || { cat test-suite.log; Exit 1; }
+ls -l . sub
+cat test-suite.log
+ grep 'chk-compiler *1 *2' foo.log
+ grep 'test-compiler *3 *4' bar.log
+ grep 'noext-compiler *5 *6' baz.log
+ grep 'test-compiler *3 *4' bla.log
+ grep 'noext-compiler *5 *6' bli.suff.log
+ grep 'noext-compiler *5 *6' sub/test.log
+
:
set -x
exec 5>&1
- for var in me parallel_tests required original_AUTOMAKE original_ACLOCAL \
- test_prefer_config_shell
- do
+ vars='
+ me
+ parallel_tests
+ required
+ use_tap
++ test_prefer_config_shell
+ original_AUTOMAKE
+ original_ACLOCAL
+ '
+
+ for var in $vars; do
env "$var=foo" $SHELL -c '. ./defs' foo.test && exit 1
env "$var=foo" $SHELL -c '. ./defs' foo.test 2>&1 1>&5 \
- | grep "foo\.test:.* variable \`$var'.* in the environment.*unsafe" || exit 1
+ | grep "foo\.test:.* variable \`$var'.* in the environment.*unsafe" \
+ || exit 1
done
:
projects / platform / upstream / automake.git / commitdiff