@noindent
will invoke @samp{$(PERL) -w foo.pl}, @samp{$(PYTHON) -v bar.py},
and @samp{./wrapper-script -d baz} to produce @file{foo.log},
-@file{bar.log}, and @file{baz.log}, respectively. The
-@samp{TESTS_ENVIRONMENT} variable is still expanded before the driver,
-but should be reserved for the user.
+@file{bar.log}, and @file{baz.log}, respectively. The @file{foo.trs},
+@file{bar.trs} and @file{baz.trs} files will be automatically produced
+as a side-effect.
-@vindex VERBOSE
-As with the simple driver above, by default one status line is printed
-per completed test, and a short summary after the suite has completed.
-However, standard output and standard error of the test are redirected
-to a per-test log file, so that parallel execution does not produce
-intermingled output. The output from failed tests is collected in the
-@file{test-suite.log} file. If the variable @samp{VERBOSE} is set, this
-file is output after the summary. For best results, the tests should be
-verbose by default now.
+It's important to note that, differently from what we've seen for the
+serial test harness (@pxref{Parallel Test Harness}), the
+@code{AM_TESTS_ENVIRONMENT} and @code{TESTS_ENVIRONMENT} variables
+@emph{cannot} be use to define a custom test runner; the
+@code{LOG_COMPILER} and @code{LOG_FLAGS} (or their extension-specific
+counterparts) should be used instead:
+
+@example
+## This is WRONG!
+AM_TESTS_ENVIRONMENT = PERL5LIB='$(srcdir)/lib' $(PERL) -Mstrict -w
+@end example
+
+@example
+## Do this instead.
+AM_TESTS_ENVIRONMENT = PERL5LIB='$(srcdir)/lib'; export PERL5LIB;
+LOG_COMPILER = $(PERL)
+AM_LOG_FLAGS = -Mstrict -w
+@end example
- @trindex mostlyclean
@trindex check-html
@vindex RST2HTML
@vindex TEST_SUITE_HTML
- With @code{make check-html}, the log files may be converted from RST
- (reStructuredText, see @uref{http://docutils.sourceforge.net/@/rst.html})
- to HTML using @samp{RST2HTML}, which defaults to @command{rst2html} or
- @command{rst2html.py}. The variable @samp{TEST_SUITE_HTML} contains the
- set of converted log files. The log and HTML files are removed upon
- @code{make mostlyclean}.
+@noindent
+ Previous versions of automake used to provide a @code{check-html} target
+ to convert the log files to HTML. This feature is now deprecated, and
+ @emph{will be removed} in the next major Automake release, so don't rely
+ on it anymore.
-@vindex DISABLE_HARD_ERRORS
-@cindex Exit status 99, special interpretation
-@cindex hard error
-Even in the presence of expected failures (see @code{XFAIL_TESTS}), there
-may be conditions under which a test outcome needs attention. For
-example, with test-driven development, you may write tests for features
-that you have not implemented yet, and thus mark these tests as expected
-to fail. However, you may still be interested in exceptional conditions,
-for example, tests that fail due to a segmentation violation or another
-error that is independent of the feature awaiting implementation.
-Tests can exit with an exit status of 99 to signal such a @emph{hard
-error}. Unless the variable @code{DISABLE_HARD_ERRORS} is set to a
-nonempty value, such tests will be counted as failed.
-
-By default, the test suite driver will run all tests, but there are
+By default, the test suite harness will run all tests, but there are
several ways to limit the set of tests that are run:
@itemize @bullet
You can ensure that all tests are rerun which have failed or passed
unexpectedly, by running @code{make recheck} in the test directory.
This convenience target will set @code{RECHECK_LOGS} appropriately
- before invoking the main test harness. The @code{recheck-html} target
- does the same as @code{recheck} but again converts the resulting
- @file{.log} file in HTML format, like the @code{check-html} target.
-before invoking the main test driver.
++before invoking the main test harness.
@end itemize
+@noindent
In order to guarantee an ordering between tests even with @code{make
--j@var{N}}, dependencies between the corresponding log files may be
-specified through usual @command{make} dependencies. For example, the
-following snippet lets the test named @file{foo-execute.test} depend
+-j@var{N}}, dependencies between the corresponding @file{.log} files
+may be specified through usual @command{make} dependencies. For example,
+the following snippet lets the test named @file{foo-execute.test} depend
upon completion of the test @file{foo-compile.test}:
@example
if test -f "./$$f"; then dir=./; \
elif test -f "$$f"; then dir=; \
else dir="$(srcdir)/"; fi; \
-tst=$$dir$$f; log='$@'; __SAVED_TERM=$$TERM; \
-$(TESTS_ENVIRONMENT)
-
-# To be appended to the command running the test. Handle the stdout
-# and stderr redirection, and catch the exit status.
-am__check_post = \
->$@-t 2>&1; \
-estatus=$$?; \
-if test -n '$(DISABLE_HARD_ERRORS)' \
- && test $$estatus -eq 99; then \
- estatus=1; \
-fi; \
-TERM=$$__SAVED_TERM; export TERM; \
-$(am__tty_colors); \
-xfailed=PASS; \
+tst=$$dir$$f; log='$@'; \
+if test -n '$(DISABLE_HARD_ERRORS)'; then \
+ am__enable_hard_errors=no; \
+else \
+ am__enable_hard_errors=yes; \
+fi; \
+## The use of $dir below is required to account for VPATH
+## rewriting done by Sun make.
case " $(XFAIL_TESTS) " in \
*[\ \ ]$$f[\ \ ]* | *[\ \ ]$$dir$$f[\ \ ]*) \
- xfailed=XFAIL;; \
-esac; \
-case $$estatus.$$xfailed in \
- 0.XFAIL) col=$$red; res=XPASS;; \
- 0.*) col=$$grn; res=PASS ;; \
- 77.*) col=$$blu; res=SKIP ;; \
- 99.*) col=$$red; res=FAIL ;; \
- *.XFAIL) col=$$lgn; res=XFAIL;; \
- *.*) col=$$red; res=FAIL ;; \
-esac; \
-echo "$${col}$$res$${std}: $$f"; \
-echo "$$res: $$f (exit: $$estatus)" | \
- $(am__rst_section) >$@; \
-cat $@-t >>$@; \
-rm -f $@-t
+ am__expect_failure=yes;; \
+ *) \
+ am__expect_failure=no;; \
+esac; \
+$(AM_TESTS_ENVIRONMENT) $(TESTS_ENVIRONMENT)
+
+# A shell command to get the names of the tests scripts with any registered
+# extension removed (i.e., equivalently, the names of the test logs, with
+# the `.log' extension removed). The result is saved in the shell variable
+# `$bases'. This honors runtime overriding of TESTS and TEST_LOGS. Sadly,
+# we cannot use something simpler, involving e.g., "$(TEST_LOGS:.log=)",
+# since that might cause problem with VPATH rewrites for suffix-less tests.
+# See also 'test-harness-vpath-rewrite.test' and 'test-trs-basic.test'.
+am__set_TESTS_bases = \
+ bases='$(TEST_LOGS)'; \
+ bases=`for i in $$bases; do echo $$i; done | sed 's/\.log$$//'`; \
- ## Trim away any extra whitespace. This has already proved useful in
- ## avoiding weird bug on lesser make implementations.
++## Trim away any extra whitespace. This has already proved useful
++## in avoiding weird bug on lesser make implementations. It also
++## works around the GNU make 3.80 bug where trailing whitespace in
++## "TESTS = foo.test $(empty)" causes $(TESTS_LOGS) to erroneously
++## expand to "foo.log .log".
+ bases=`echo $$bases`
+
+# Recover from deleted `.trs' file; this should ensure that
+# "rm -f foo.log; make foo.trs" re-run `foo.test', and re-create
+# both `foo.log' and `foo.trs'. Break the recipe in two subshells
+# to avoid problems with "make -n".
+.log.trs:
+ rm -f $< $@
+ $(MAKE) $(AM_MAKEFLAGS) $<
+
+# Helper recipe used by $(TEST_SUITE_LOG) below, to avoid problems with
+# "make -n". Break this recipe in multiple shell invocations too, to
+# really work as expected with "make -n".
+am--redo-logs:
+ @rm -f $$redo_logs
+ @rm -f $$redo_results
+## The use of the `am__remaking_logs' environment variable below is
+## required to ensure that we don't go into an infinite recursion in
+## case a test log in $(TEST_LOGS) is the same as $(TEST_SUITE_LOG).
+## Yes, this has already happened in practice. Sigh!
+ @if test -n "$$am__remaking_logs"; then \
+ echo "fatal: making $(TEST_SUITE_LOG): possible infinite" \
+ "recursion detected" >&2; \
+ else \
+ am__remaking_logs=yes $(MAKE) $(AM_MAKEFLAGS) $$redo_logs; \
+ fi;
+## Sanity check: each unreadable or non-existent test result file should
+## has been properly remade at this point, as should the corresponding log
+## file.
+ @st=0; \
+ errmsg="fatal: making $(TEST_SUITE_LOG): failed to create"; \
+ for i in $$redo_bases; do \
+ test -f $$i.trs && test -r $$i.trs \
+ || { echo "$$errmsg $$i.trs"; st=1; }; \
+ test -f $$i.log && test -r $$i.log \
+ || { echo "$$errmsg $$i.log"; st=1; }; \
+ done; \
+ test $$st -eq 0
+.PHONY: am--redo-logs
$(TEST_SUITE_LOG): $(TEST_LOGS)
- @$(am__sh_e_setup); \
- list='$(TEST_LOGS)'; \
- results=`for f in $$list; do \
- test -r $$f && read line < $$f && echo "$$line" \
- || echo FAIL; \
- done`; \
- all=`echo "$$results" | sed '/^$$/d' | wc -l | sed -e 's/^[ ]*//'`; \
- fail=`echo "$$results" | grep -c '^FAIL'`; \
- pass=`echo "$$results" | grep -c '^PASS'`; \
- skip=`echo "$$results" | grep -c '^SKIP'`; \
- xfail=`echo "$$results" | grep -c '^XFAIL'`; \
- xpass=`echo "$$results" | grep -c '^XPASS'`; \
- failures=`expr $$fail + $$xpass`; \
- all=`expr $$all - $$skip`; \
- if test "$$all" -eq 1; then tests=test; All=; \
- else tests=tests; All="All "; fi; \
- case fail=$$fail:xpass=$$xpass:xfail=$$xfail in \
- fail=0:xpass=0:xfail=0) \
- msg="$$All$$all $$tests passed. "; \
- exit=true;; \
- fail=0:xpass=0:xfail=*) \
- msg="$$All$$all $$tests behaved as expected"; \
- if test "$$xfail" -eq 1; then xfailures=failure; \
- else xfailures=failures; fi; \
- msg="$$msg ($$xfail expected $$xfailures). "; \
- exit=true;; \
- fail=*:xpass=0:xfail=*) \
- msg="$$fail of $$all $$tests failed. "; \
- exit=false;; \
- fail=*:xpass=*:xfail=*) \
- msg="$$failures of $$all $$tests did not behave as expected"; \
- if test "$$xpass" -eq 1; then xpasses=pass; \
- else xpasses=passes; fi; \
- msg="$$msg ($$xpass unexpected $$xpasses). "; \
- exit=false;; \
- *) \
- echo >&2 "incorrect case"; exit 4;; \
- esac; \
- if test "$$skip" -ne 0; then \
- if test "$$skip" -eq 1; then \
- msg="$$msg($$skip test was not run). "; \
- else \
- msg="$$msg($$skip tests were not run). "; \
- fi; \
- fi; \
+ @$(am__set_TESTS_bases); \
+## Helper shell function, tells whether a path refers to an existing,
+## regular, readable file.
+ am__f_ok () { test -f "$$1" && test -r "$$1"; }; \
+## We need to ensures that all the required `.trs' and `.log' files will
+## be present and readable. The direct dependencies of $(TEST_SUITE_LOG)
+## only ensure that all the `.log' files exists; they don't ensure that
+## the `.log' files are readable, and worse, they don't ensure that the
+## `.trs' files even exist.
+ redo_bases=`for i in $$bases; do \
+ am__f_ok $$i.trs && am__f_ok $$i.log || echo $$i; \
+ done`; \
+ if test -n "$$redo_bases"; then \
+## Uh-oh, either some `.log' files were unreadable, or some `.trs' files
+## were missing (or unreadable). We need to re-run the corresponding
+## tests in order to re-create them.
+ redo_logs=`for i in $$redo_bases; do echo $$i.log; done`; \
+ redo_results=`for i in $$redo_bases; do echo $$i.trs; done`; \
+## The exported variables are needed by the helper hook.
+ redo_bases="$$redo_bases" \
+ redo_logs="$$redo_logs" \
+ redo_results="$$redo_results" \
+ $(MAKE) $(AM_MAKEFLAGS) am--redo-logs || exit 1; \
+ else :; fi;
+## We need a new subshell to work portably with "make -n", since the
+## previous part of the recipe contained a $(MAKE) invocation.
+ @$(am__sh_e_setup); $(am__tty_colors); $(am__set_TESTS_bases); \
+ ws='[ ]'; \
+## List of test result files.
+ results=`for b in $$bases; do echo $$b.trs; done`; \
+ test -n "$$results" || results=/dev/null; \
+## Prepare data for the test suite summary. These do not take into account
+## unreadable test results, but they'll be appropriately updated later if
+## needed.
+ all=` grep "^$$ws*:test-result:" $$results | wc -l`; \
+ pass=` grep "^$$ws*:test-result:$$ws*PASS" $$results | wc -l`; \
+ fail=` grep "^$$ws*:test-result:$$ws*FAIL" $$results | wc -l`; \
+ skip=` grep "^$$ws*:test-result:$$ws*SKIP" $$results | wc -l`; \
+ xfail=`grep "^$$ws*:test-result:$$ws*XFAIL" $$results | wc -l`; \
+ xpass=`grep "^$$ws*:test-result:$$ws*XPASS" $$results | wc -l`; \
+ error=`grep "^$$ws*:test-result:$$ws*ERROR" $$results | wc -l`; \
+## Whether the testsuite was successful or not.
+ if test `expr $$fail + $$xpass + $$error` -eq 0; then \
+ success=true; \
+ else \
+ success=false; \
+ fi; \
+## Make $br a line of exactly 76 `=' characters, that will be used to
+## enclose the testsuite summary report when displayed on the console.
+ br='==================='; br=$$br$$br$$br$$br; \
+## When writing the test summary to the console, we want to color a line
+## reporting the count of some result *only* if at least one test
+## experienced such a result. This function is handy in this regard.
+ result_count () \
+ { \
+ if test x"$$1" = x"--maybe-color"; then \
+ maybe_colorize=yes; \
+ elif test x"$$1" = x"--no-color"; then \
+ maybe_colorize=no; \
+ else \
+ echo "$@: invalid 'result_count' usage" >&2; exit 4; \
+ fi; \
+ shift; \
+ desc=$$1 count=$$2; \
+ if test $$maybe_colorize = yes && test $$count -gt 0; then \
+ color_start=$$3 color_end=$$std; \
+ else \
+ color_start= color_end=; \
+ fi; \
+ echo "$${color_start}# $$desc $$count$${color_end}"; \
+ }; \
+## A shell function that creates the testsuite summary. We need it
+## because we have to create *two* summaries, one for test-suite.log,
+## and a possibly-colorized one for console output.
+ create_testsuite_report () \
+ { \
+ result_count $$1 "TOTAL:" $$all "$$brg"; \
+ result_count $$1 "PASS: " $$pass "$$grn"; \
+ result_count $$1 "SKIP: " $$skip "$$blu"; \
+ result_count $$1 "XFAIL:" $$xfail "$$lgn"; \
+ result_count $$1 "FAIL: " $$fail "$$red"; \
+ result_count $$1 "XPASS:" $$xpass "$$red"; \
+ result_count $$1 "ERROR:" $$error "$$mgn"; \
+ }; \
+## Write "global" testsuite log.
{ \
echo "$(PACKAGE_STRING): $(subdir)/$(TEST_SUITE_LOG)" | \
$(am__rst_title); \
RECHECK_LOGS = $(TEST_LOGS)
- # Run all the tests.
- check-TESTS:
- ## Expand $(RECHECK_LOGS) only once, to avoid exceeding line length limits.
- @list='$(RECHECK_LOGS)'; test -z "$$list" || rm -f $$list
- @list='$(RECHECK_LOGS:.log=.trs)'; test -z "$$list" || rm -f $$list
++## ------------------------------------------ ##
++## Running all tests, or rechecking failures. ##
++## ------------------------------------------ ##
++
+ check-TESTS recheck:
+ ## If we are running "make recheck", it's not the user which can decide
+ ## which tests to consider for re-execution, so we must ignore the value
+ ## of $(RECHECK_LOGS).
++## Here and below, we expand $(RECHECK_LOGS) only once, to avoid exceeding
++## line length limits.
++ @if test $@ != recheck; then \
++ list='$(RECHECK_LOGS)'; \
++ test -z "$$list" || rm -f $$list; \
++ fi
+ @if test $@ != recheck; then \
-## Expand $(RECHECK_LOGS) only once, to avoid exceeding line length limits.
- list='$(RECHECK_LOGS)'; test -z "$$list" || rm -f $$list; \
++ list='$(RECHECK_LOGS:.log=.trs)'; \
++ test -z "$$list" || rm -f $$list; \
+ fi
## We always have to remove TEST_SUITE_LOG, to ensure its rule is run
## in any case even in lazy mode: otherwise, if no test needs rerunning,
- ## or a prior run plus reruns all happen within the same timestamp
- ## (can happen with a prior `make TESTS=<subset>'),
- ## then we get no log output.
+ ## or a prior run plus reruns all happen within the same timestamp (can
+ ## happen with a prior `make TESTS=<subset>'), then we get no log output.
## OTOH, this means that, in the rule for `$(TEST_SUITE_LOG)', we
## cannot use `$?' to compute the set of lazily rerun tests, lest
## we rely on .PHONY to work portably.
@test -z "$(TEST_SUITE_LOG)" || rm -f $(TEST_SUITE_LOG)
- @list='' list2='$(TEST_LOGS)'; for i in $$list2; do \
- @list='' list2='$(TEST_LOGS)'; for f in $$list2; do \
--## Trailing whitespace in `TESTS = foo.test $(empty)' causes GNU make
--## 3.80 to erroneously expand $(TESTS_LOGS) to `foo.log .log'.
--## Work around this bug.
- test .log = $$i && continue; \
- ## Be careful to avoid extra whitespace in the definition of $list. See
- ## comments in `recheck' below for why this might be useful.
- if test -z "$$list"; then list=$$i; else list="$$list $$i"; fi; \
- done; \
- $(MAKE) $(AM_MAKEFLAGS) $(TEST_SUITE_LOG) TEST_LOGS="$$list"
-
- AM_RECURSIVE_TARGETS += check
-
- ## -------------- ##
- ## Produce HTML. ##
- ## -------------- ##
-
- .log.html:
- @list='$(RST2HTML) $$RST2HTML rst2html rst2html.py'; \
- for r2h in $$list; do \
- if ($$r2h --version) >/dev/null 2>&1; then \
- R2H=$$r2h; \
- fi; \
- done; \
- if test -z "$$R2H"; then \
- echo >&2 "cannot find rst2html, cannot create $@"; \
- exit 2; \
- fi; \
- $$R2H $< >$@.tmp
- @mv $@.tmp $@
-
- # Be sure to run check first, and then to convert the result.
- # Beware of concurrent executions. Run "check" not "check-TESTS", as
- # check-SCRIPTS and other dependencies are rebuilt by the former only.
- # And expect check to fail.
- check-html:
- @if $(MAKE) $(AM_MAKEFLAGS) check; then \
- rv=0; else rv=$$?; \
- fi; \
- $(MAKE) $(AM_MAKEFLAGS) $(TEST_SUITE_HTML) || exit 4; \
- exit $$rv
-
- .PHONY: check-html
- .MAKE: check-html
-
- AM_RECURSIVE_TARGETS += check-html
-
- ## -------------------- ##
- ## Rechecking failures. ##
- ## -------------------- ##
-
- ## Rerun all tests that experienced an error or an unexpected failure.
- recheck recheck-html:
- test .log = $$f && continue; \
-## If running a "make recheck", we must only consider tests that had
-## an unexpected outcome (FAIL or XPASS) in the earlier run.
+ @ws='[ ]'; \
- target=`echo $@ | sed 's,^re,,'`; \
+ list=''; $(am__set_TESTS_bases); \
+ for i in $$bases; do \
- ## Skip tests that haven't been run, but recover gracefully from deleted
++## If running a "make recheck", we must only consider tests that had an
++## unexpected outcome (FAIL or XPASS) in the earlier run. In particular,
++## skip tests that haven't been run. But recover gracefully from deleted
+## `.trs' files.
- test -f $$i.trs || test -f $$i.log || continue; \
+ if test $@ = recheck; then \
- test -f $$f || continue; \
- if test -r $$f && read line < $$f; then \
- case $$line in FAIL*|XPASS*) : ;; *) continue;; esac; \
- fi; \
- fi; \
++ test -f $$i.trs || test -f $$i.log || continue; \
+## FIXME: one fork per test -- this is horrendously inefficient!
- grep "^$$ws*:recheck:$$ws*no$$ws*$$" $$i.trs \
- >/dev/null 2>&1 && continue; \
++ grep "^$$ws*:recheck:$$ws*no$$ws*$$" $$i.trs \
++ >/dev/null 2>&1 && continue; \
++ else :; fi; \
## Be careful to avoid extra whitespace in the definition of $list, since
## its value will be passed to the recursive make invocation below through
## the TEST_LOGS macro, and leading/trailing white space in a make macro
## definition can be problematic. In this particular case, trailing white
## space was known to cause a segmentation fault on Solaris 10 XPG4 make:
## <http://lists.gnu.org/archive/html/bug-automake/2010-08/msg00004.html>
- if test -z "$$list"; then list=$$f; else list="$$list $$f"; fi; \
+ if test -z "$$list"; then list=$$i.log; else list="$$list $$i.log"; fi; \
done; \
- $(MAKE) $(AM_MAKEFLAGS) $$target AM_MAKEFLAGS='$(AM_MAKEFLAGS) TEST_LOGS="'"$$list"'"'
-## Under "make recheck", remove the logs of the files to recheck, so that
-## those will be rerun by the "make test-suite.log" recursive invocation
-## below. But use a proper hack to avoid extra files removal when running
-## under "make -n".
++## Under "make recheck", remove the .log and .trs files associated
++## with the files to recheck, so that those will be rerun by the
++## "make test-suite.log" recursive invocation below. But use a proper
++## hack to avoid extra files removal when running under "make -n".
+ if test $@ = recheck && test -n "$$list"; then \
- echo "am--clean: ; rm -f $$list" \
- | $(MAKE) $(AM_MAKEFLAGS) -f - am--clean || exit 1; \
++ { \
++ echo 'am--clean: am--clean-log am--clean-trs'; \
++ echo 'am--clean-log: ; rm -f $$(am__logs)'; \
++ echo 'am--clean-trs: ; rm -f $$(am__logs:.log=.trs)'; \
++ } | $(MAKE) $(AM_MAKEFLAGS) -f - am--clean am__logs="$$list" \
++ || exit 1; \
+ fi; \
+ $(MAKE) $(AM_MAKEFLAGS) $(TEST_SUITE_LOG) TEST_LOGS="$$list"
+
+ ## Recheck must depend on $(check_SCRIPTS), $(check_PROGRAMS), etc.
+ recheck: %CHECK_DEPS%
+
+ AM_RECURSIVE_TARGETS += check recheck
+
+ .PHONY: recheck
- .PHONY: recheck recheck-html
- .MAKE: recheck recheck-html
+ ## ----------------------------------------------- ##
+ ## Produce HTML. To be removed in automake 1.12. ##
+ ## ----------------------------------------------- ##
- AM_RECURSIVE_TARGETS += recheck recheck-html
+ include check-html.am
else !%?PARALLEL_TESTS%
projects / platform / upstream / automake.git / commitdiff