(extract_program_version):... this new function, and saved ...
($autopoint_version): ... in this new variable.
+2011-12-23 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: automatically handle deps also for generated tests
+ * tests/gen-testsuite-part: The test scripts are now scanned for
+ automatic dependency generation *after* the auto-generated tests
+ have been created, so they too will be scanned. It is a little
+ tricky to ensure that the freshly-generated tests are correctly
+ scanned, and we do that with the help of ...
+ (@generated_tests): ... this new variable.
+ Other related minor changes and refactorings.
+
+2011-12-23 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: auto-generate deps for tests requiring libtool/gettext
+ * tests/gen-testsuite-part: Tests requiring libtool (or related
+ programs) will need libtool-provided m4 macros, so they should
+ be run after `libtool-macros.test'. Similarly for gettext tests.
+ So, generate proper declarations of such dependencies.
+ * tests/Makefile.am: Remove now-unneeded hand-written declaration
+ of those dependencies.
+
+2011-12-23 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: remove FIXME comments from tricks to pacify syntax checks
+ Originally, I myself had insisted on adding those "FIXME" comments
+ near every unusual construct or use of "creative quoting" whose
+ only purpose was to avoid false positive from out syntax checks.
+ I did that with the expectation that we could have easily and soon
+ added a whitelisting capability to our maintainer-specific syntax
+ checks. This hasn't been the case unfortunately, and now those
+ "FIXME" in the comments are just distracting noise, making it more
+ difficult to grep the test cases for stuff that could really use a
+ fix or an improvement. So just get rid of them.
+ * tests/instdir-ltlib.test: Remove that extra "FIXME".
+ * tests/instdir-prog.test: Likewise.
+ * tests/parallel-tests3.test: Likewise.
+ * tests/parallel-tests-dry-run.test: Likewise.
+ * tests/parallel-tests-console-output.test: Likewise.
+ * tests/parallel-tests-no-color-in-log.test: Likewise.
+ * tests/tap-realtime.test: Likewise.
+ * tests/test-trs-recover.test: Likewise.
+ * tests/tap-deps.test: Likewise.
+
+2011-12-23 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: few minor fixlets, improvements and tweakings
+ * tests/defs-static.in ($top_testbuilddir): New, for consistency
+ and completeness.
+ * tests/Makefile.am (do_subst): Substitute @abs_top_testbuilddir@
+ as well.
+ * tests/java-compile-run-mested.test: Prefer AM_TESTS_ENVIRONMENT
+ over TESTS_ENVIRONMENT, now that the former has become available.
+ * tests/java-compile-run-flat.test: Correct a botched "FIXME"
+ comment.
+ * tests/tap-realtime.test: Remove extra whitespace in comments.
+ * tests/missing-tar.test: Use `get_shell_script' to bring in the
+ `missing' script, to increase coverage. Remove redundant call to
+ `set -e'. Fix a typo in comments.
+ * tests/get-sysconf.test: Remove redundant definitions of
+ `$top_testsrcdir', `testbuilddir' and `$top_testbuilddir'.
+
+2011-12-23 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: remove redundant uses of `set -e'
+ * tests/deleted-am.test: Do not set the `errexit' shell flag, as
+ it is already set by `tests/defs'.
+ * tests/deleted-m4.test: Likewise.
+ * tests/dist-missing-am.test: Likewise.
+ * tests/dist-missing-am.test: Likewise.
+ * tests/dist-missing-m4.test: Likewise.
+ * tests/dist-missing-included-m4.test: Likewise.
+ * tests/get-sysconf.test: Likewise.
+ * tests/makefile-deps.test: Likewise.
+ * tests/remake-m4-pr10111.test: Likewise.
+ * tests/remake-am-pr10111.test: Likewise.
+ * tests/remake-deleted-am.test: Likewise.
+ * tests/remake-deleted-am2.test: Likewise.
+ * tests/remake-deleted-am-subdir.test: Likewise.
+ * tests/remake-renamed-am.test: Likewise.
+
2011-12-22 Stefano Lattarini <stefano.lattarini@gmail.com>
tests: fix failure due to debugging code forgotten into a test
2011-12-23 Stefano Lattarini <stefano.lattarini@gmail.com>
+ Merge branch 'master' into testsuite-work
+ * tests/gettext-macros.test: In the generated `get.sh' file,
+ use `skip_all_', not `skip_'.
+ * tests/libtool-macros.test: Likewise.
+ * tests/list-of-tests.mk: Update.
+
+2011-12-23 Stefano Lattarini <stefano.lattarini@gmail.com>
+
tests: avoid spurious failure of libtool and gettext tests
On Solaris 10 (and presumably earlier), /bin/sh trips up on
unistd.h to be present (it is not present when compiling with,
e.g., MSVC 9).
+2011-12-20 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: fix spurious failure on MSYS/MinGW and Cygwin
+ * tests/primary-prefix-couples-force-valid.test (Makefile.am):
+ Correctly append $(EXEEXT) to the path of built binaries.
+
2011-12-16 Stefano Lattarini <stefano.lattarini@gmail.com>
test defs: hack to support autoconf-wrapper programs
* HACKING (Release procedure): ... state here that "make check"
and "make distcheck" should be run before calling "make git-dist".
+2011-12-15 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: fix spurious failure in 'color2.test'
+ * tests/color2.test: (expect-make): Add an "expect eof" directive,
+ so that the collected output from the spawned make program will be
+ displayed on stdout, as desired.
+ Since we are at it, also correctly remove a temporary file which
+ we was trying to delete using a wrong filename.
+ Problem introduced in merge `v1.11-1579-g8d3466c', probably by a
+ botched edit or conflict resolution.
+
+2011-12-15 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: fix spurious failure in 'lex3.test'
+ * tests/lex3.test (foo.l): Remove duplicated definition of
+ `yywrap'. Problem introduced in merge `v1.11-1579-g8d3466c'.
+
2011-12-09 Jim Meyering <meyering@redhat.com>
Stefano Lattarini <stefano.lattarini@gmail.com>
2011-11-05 Stefano Lattarini <stefano.lattarini@gmail.com>
+ tests: more faithful 'installcheck' support in few tests
+ * tests/ar-lib5a.test: Use the `$am_scriptdir' variable instead
+ of "$testsrcdir/../lib" or "$top_testsrcdir/lib", to test more
+ faithfully under "make installcheck".
+ * tests/ar-lib5a.test: Likewise.
+ * tests/libobj-basic.test: Likewise.
+ * tests/libobj19.test: Likewise.
+ * tests/suffix5.test: Likewise.
+
+2011-11-05 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: remove redundant settings of `errexit' shell flag
+ * tests/aclocal-path-install-serial.test: Do not set the
+ `errexit' shell flag, as it is already set by `tests/defs'.
+ * tests/ar-lib2.test: Likewise.
+ * tests/ar-lib3.test: Likewise.
+ * tests/ar-lib4.test: Likewise.
+ * tests/ar-lib5a.test: Likewise.
+ * tests/ar-lib5b.test: Likewise.
+ * tests/ar-lib6a.test: Likewise.
+ * tests/ar-lib6b.test: Likewise.
+ * tests/ar-lib7.test: Likewise.
+ * tests/ar3.test: Likewise.
+ * tests/ar4.test: Likewise.
+ * tests/ar5.test: Likewise.
+ * tests/extra-portability.test: Likewise.
+ * tests/extra-portability2.test: Likewise.
+ * tests/extra-portability3.test: Likewise.
+ * tests/install-info-dir.test: Likewise.
+ * tests/maintmode-configure-msg.test: Likewise.
+
+2011-11-05 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ generated tests: avoid extra "FIXME" comments
+ * tests/gen-testsuite-part: Avoid putting too much "FIXME"
+ comments in the generated test scripts, since they end up
+ being just counter-productive noise when one has to grep
+ the test scripts looking for real "FIXME" items.
+
+2011-09-12 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ cosmetics: fix typo in 'prove-runner' script
+ * tests/prove-runner: Remove erroneously-repeated word.
+ * THANKS: Update.
+ Reported by Dave Hart.
+
+2011-11-05 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ Merge branch 'master' into testsuite-work
+ * tests/extra-portability.test: Use `$am_original_AUTOMAKE'
+ instead of the obsolete `$original_AUTOMAKE'.
+ * tests/extra-portability3.test: Likewise.
+ * tests/list-of-tests.mk: Update.
+
+2011-11-05 Stefano Lattarini <stefano.lattarini@gmail.com>
+
Merge branch 'msvc' into master
* tests/extra-portability3.test: Use `$original_AUTOMAKE' instead
of hackishly extracting the first component of `$AUTOMAKE'.
2011-11-03 Stefano Lattarini <stefano.lattarini@gmail.com>
+ Merge branch 'master' into testsuite-work
+
+ * tests/instspc.tap: Adjust to new portability requirements
+ due to the new AM_PROG_AR macro.
+ * tests/lex-lib.test: Likewise.
+ * tests/extra-portability2.test: Use `$am_original_AUTOMAKE'
+ instead of the obsolete `$original_AUTOMAKE'.
+ * tests/list-of-tests.mk: Update.
+
+2011-11-03 Stefano Lattarini <stefano.lattarini@gmail.com>
+
tests: various minor tweakings, mostly related to AM_PROG_AR
* tests/alloca.test: Adjust to new portability requirements due
to the new AM_PROG_AR macro.
* THANKS: Likewise.
Report by Jonathan Nieder.
+2011-10-25 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ test defs: new function 'am_keeping_testdirs', for better APIs
+
+ This change is meant to make it easier for TAP-based test scripts
+ to remove files left over by completed (sub)tests, while still
+ honoring the user-provided `$keep_testdirs' setting.
+
+ * tests/defs (am_keeping_testdirs): New function, telling whether
+ we should keep the test directories around, even in case of
+ success (by default, we don't).
+ (exit trap): Use it.
+ * tests/depmod.tap: Likewise.
+ * tests/instspc.tap: Likewise. Also, fix some pre-existing
+ "copy & paste" bugs, and a typo in comments.
+
+2011-10-25 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: remove redundant `set -e', and other minor improvements
+ * tests/dejagnu-absolute-builddir.test: Do not set the `errexit'
+ shell flag, as it is already set by `tests/defs'.
+ * tests/dejagnu-relative-srcdir.test: Likewise.
+ * tests/dejagnu-siteexp-append.test: Likewise.
+ * tests/dejagnu-siteexp-useredit.test: Likewise, and fix a typo
+ in comments.
+ * tests/dejagnu-siteexp-extend.test: Likewise, and prefer the
+ use of `$distdir' over hard-coded `$me-1.0'. Also ...
+ (write_check_for): ... use `unindent' when writing from an
+ here-doc, for better code indentation.
+
2011-10-20 Stefano Lattarini <stefano.lattarini@gmail.com>
deps: partially revert commit `v1.11-512-geeee551'
* tests/silent-lex-generic.test: Likewise.
* tests/silent-lex-gcc.test: Likewise.
+2011-10-21 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: more faithful 'installcheck' support in few tests
+ * tests/test-driver-acsubst.test: Use the `$am_scriptdir'
+ variable instead of `$testsrcdir/../lib', to test more
+ faithfully under "make installcheck".
+ * tests/test-driver-cond.test: Likewise.
+ * tests/dist-auxfile.test: Likewise, and add one more use
+ of `fatal_' to report hard errors.
+ * tests/tests/dist-auxdir-many-subdirs.test: Likewise.
+
+2011-10-21 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: remove redundant settings of `errexit' shell flag
+ * tests/aclocal-path-install-serial.test: Do not set the
+ `errexit' shell flag, as it is already set by `tests/defs'.
+ * tests/aclocal-path-install.test: Likewise.
+ * tests/aclocal-path-nonexistent.test: Likewise.
+ * tests/aclocal-path-precedence.test: Likewise.
+ * tests/aclocal-path.test: Likewise.
+ * tests/dist-auxfile-2.test: Likewise.
+ * tests/dist-auxfile.test: Likewise.
+ * tests/distcheck-pr9579.test: Likewise.
+ * tests/javadir-undefined.test: Likewise.
+ * tests/subpkg-yacc.test: Likewise.
+ * tests/test-extensions-cond.test: Likewise.
+ * tests/test-extensions.test: Likewise.
+ * tests/uninstall-fail.test: Likewise.
+ * tests/uninstall-pr9578.test: Likewise.
+ * tests/distcheck-override-infodir.test: Likewise. Also,
+ fix typo in heading comments since we are at it.
+
+2011-10-20 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ fixup: add forgotten test scripts to $(TESTS)
+ * tests/list-of-tests.mk: Add various test scripts that are in
+ the git repository but weren't correctly listed in $(TESTS).
+ The inconsistency had been caused by botched/incomplete merges,
+ and revealed by the "maintainer-check-list-of-tests" target.
+
2011-10-18 Stefano Lattarini <stefano.lattarini@gmail.com>
tests: fix spurious failure with FreeBSD make and Yacc in VPATH
variable `$relative_dir'.
(scan_autoconf_traces, check_directories_in_var): Adjust.
+2011-10-17 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ cosmetics: remove obsolete comment
+ * automake.in (push_required_file): Remove "FIXME" comment stating
+ that the special handling of `$config_aux_dir' breaks threaded
+ automake: this is not true anymore after the recent commits (see
+ in particular commit `v1.11-1225-gf672944').
+
+2011-10-08 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: fix spurious failure with Solaris /bin/sh
+ * tests/dist-auxfile.test: Don't use redirected `:' in a loop,
+ it hits a bug in Solaris 10 /bin/sh.
+
+2011-10-08 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ threaded-am: better serialization for required config files
+ With this change, we make serialization/de-serialization of
+ required config files installs more granular, and in the process
+ fix the bug introduced by commit `v1.11-1219-g326ecba'.
+ * automake.in ($required_conf_file_queue): Move its declaration
+ earlier.
+ (require_file_internal): Add a new argument telling whether the
+ function should act immediately or queue its action for the master
+ thread to handle.
+ (queue_required_conf_file): Renamed ...
+ (queue_required_file_check_or_copy): ... to this.
+ (require_queued_conf_file): Renamed ...
+ (require_queued_file_check_or_copy): ... to this, and make it call
+ `required_file_check_or_copy' instead of `require_file_internal'.
+ (require_conf_file, handle_makefiles_threaded): Adjust and simplify
+ accordingly.
+ * tests/Makefile.am (XFAIL_TESTS): Remove `parallel-am.test'.
+
+2011-10-08 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ automake: refactor, break up 'require_file_internal'
+ This refactoring is only required in view of future changes.
+ * automake.in (require_file_internal): Move the guts of this
+ function ...
+ (required_file_check_or_copy): ... into this new function. This
+ ensures that calls to `push_required_file' and code that copies
+ required files are placed in separate functions; this will be
+ very useful for reorganizing de-serialization of file installs
+ in future changes.
+
+2011-10-08 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ dist: separate auxiliary file instantiation from DIST_COMMON update
+ This change simplifies the automake internals dealing with the
+ checking, copying and distributing of required auxiliary files.
+ With this change, a required auxiliary file is *unconditionally*
+ added to the contents of the DIST_COMMON variable in the generated
+ Makefile.in, before checking whether it exists, or trying to copy
+ it (if `--add-missing' is in use). This shouldn't be a problem,
+ since if the checking or copying of the file fails, automake will
+ bail out, the Makefile.in won't be created, and thus its content
+ will not matter. OK, this is not completely true when threaded
+ automake is in use, but then, such a situation was also possible
+ before this patch, so no regression here, which is enough for us.
+ This change is the first of a series of steps aimed at fixing
+ the regression introduced in threaded automake usage by commit
+ `v1.11-1219-g326ecba'.
+ * automake.in (require_file_internal): Add the required file to
+ DIST_COMMON unconditionally.
+ * tests/missing-auxfile-stops-makefiles-creation.test: New test.
+ * tests/Makefile.am (TESTS): Add it.
+
+2011-10-08 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ refactor: rename a subroutine to a more proper name
+ * automake.in (maybe_push_require_file): Renamed ...
+ (push_required_file): ... to this.
+ All callers adjusted.
+
+2011-10-08 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ dist: truly always distribute files in AC_REQUIRE_AUX_FILE
+ This change fixes automake bug#9651.
+ * automake.in (handle_dist): Files whose distribution is required
+ by configure.ac are now correctly distributed even if the build-aux
+ directory coincides with the top-level directory.
+ * tests/Makefile.am (XFAIL_TESTS): Remove `dist-auxfile.test'.
+ * NEWS: Update.
+
+2011-10-08 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ dist: simplify automake code accordingly to the previous changes
+ After the previous change `v1.11-1219-g326ecba', every Makefile.am
+ requiring a file in the "config auxdir" will cause it to be listed
+ in the DIST_COMMON variable of the corresponding generated
+ Makefile.in, not only of the top-level Makefile.in. Thus we don't
+ need to worry anymore about trying to distribute files in the
+ config auxdir from the top-level Makefile or the Makefile in
+ config auxdir (if any). This allows us to carry out some useful
+ simplifications in the automake script.
+ This will however cause some changes in the generated Makefile;
+ for example, if the `PROGRAMS' primary is used only in a subdir
+ Makefile.am, the `depcomp' script will be distributed only by
+ `subdir/Makefile', not by the top-level Makefile. This change
+ can be seen as an internal detail though, and definitely warranted
+ by the nice simplifications the current commit provides.
+ * automake.in ($automake_will_process_aux_dir): Variable removed.
+ (scan_autoconf_files): Adjust and simplify.
+ (handle_makefiles_threaded): Likewise.
+ (get_number_of_threads): Likewise.
+ (require_file_internal): Likewise.
+ (maybe_push_required_file): Its return value is not used anymore,
+ so return nothing. Add special handling for distributed files from
+ `$config_libobj_dir'. Also, catch "this can't happen" situations,
+ and abort accordingly. Fix function description to better fit its
+ new role.
+ * tests/libobj19.test: Extend to also check situations where the
+ Makefile.am using $(LIBOBJS) is not the top-level one.
+ * tests/distcom6.test: Delete, merged into ...
+ * tests/distcom2.test: ... this test, which has been update to
+ account for the changed automake semantics.
+ * tests/Makefile.am (TESTS): Update.
+ * NEWS: Update.
+
+2011-10-08 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ dist: auxiliary files can be distributed from subdir Makefiles
+ With this change, we make it possible for a subdir Makefile.am
+ to distribute files in the config auxdir; while this means that
+ some files might be copied multiple times, it simplify some logic
+ in the automake script, and fix at least one important bug. In
+ fact, before this change, the auxiliary script `test-driver' was
+ not being distributed as expected when TESTS was defined only in
+ a subdir Makefile (which is a pretty common setup indeed). Now
+ this does not happen anymore: so the present change fixes automake
+ bug#9546.
+ Another welcome collateral effect is that `dist-auxfile-2.test'
+ now passes.
+ OTOH, the present changes *breaks threaded automake*. The reason
+ is that automake needs to serialize file installations spawned
+ by `--add-missing' (this isn't for cosmetic reasons, but is
+ really needed to avoid possible race conditions and botched output
+ files). Currently the code that installs required files is
+ intertwined with the code that declares the DIST_COMMON variables;
+ so, upon de-serialization, the definition of DIST_COMMON might get
+ emitted in the wrong Makefile.in.
+ Some follow-up refactoring in automake will soon take care of
+ remedying this situation. For the moment, we just declare some
+ "parallel-am" tests as xfailing.
+ * automake.in (maybe_push_required_file): Add ad-hoc handling for
+ the case where the directory part of the required file is different
+ from the subdir where the current Makefile.am resides, but is equal
+ to the "config-aux directory" ($config_auxdir). This is needed to
+ allow a construct in a non-top-level Makefile.am to require a file
+ in the config-aux directory.
+ * tests/test-driver-is-distributed.test: Extend and adjust. This
+ test now passes.
+ * tests/java-compile-run-nested.test: Call automake with the `-a'
+ option to ensure that the `test-driver' auxiliary script gets
+ correctly installed. This test now passes.
+ * tests/distcom-subdir.test: New test.
+ * tests/dist-readonly.test: Likewise.
+ * tests/dist-repeated.test: Likewise.
+ * tests/dist-auxdir-many-subdirs.test: Likewise.
+ * tests/distcom7.test: Removed, it's obsolete now.
+ * tests/Makefile.am (TESTS, XFAIL_TESTS): Update.
+
+2011-10-11 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ coverage: conditional contents in LOG_DRIVER variables
+ * tests/test-driver-cond.test: New test.
+ * tests/Makefile.am (TESTS): Add it.
+
+2011-10-11 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ coverage: LOG_DRIVER variables can be defined through AC_SUBST
+ * tests/test-driver-acsubst.test: New test.
+ * tests/Makefile.am (TESTS): Add it.
+
2011-10-07 Stefano Lattarini <stefano.lattarini@gmail.com>
parallel-tests: warn on conditional TEST_EXTENSIONS definition
(is_valid_test_extension): ... into this subroutine. Don't allow
generic @substitutions@ anymore (possibly making an exception for
`@EXEEXT@' under the proper circumstances).
- * tests/test-extensions.test: Adjust and extend.
+ * tests/test-extensions.test: Adjust and extend. This test does
+ not cause any internal error in automake anymore now.
2011-10-06 Stefano Lattarini <stefano.lattarini@gmail.com>
2011-10-06 Stefano Lattarini <stefano.lattarini@gmail.com>
+ tests: fix spurious error in 'test-extensions.test'
+ * tests/test-extensions.test: Call automake with the option
+ `--add-missing', to ensure the required script `test-driver'
+ gets installed. Note that the test still fails due to an
+ internal error in automake, though.
+
+2011-10-06 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: avoid few useless uses of the `-e' make option
+ * tests/python12.test: Using `make -e' is not required to override
+ the DESTDIR variable, since this variable shouldn't be defined in
+ any automake-generated Makefile.in file.
+ * tests/man2.test: Likewise.
+ * tests/man5.test: Likewise.
+
+2011-10-06 Stefano Lattarini <stefano.lattarini@gmail.com>
+
fix: make a test script executable
* tests/nobase-nodist.test: Make executable.
2011-09-28 Stefano Lattarini <stefano.lattarini@gmail.com>
+ tap/awk: improve comments about Korn shell signal handling issues
+ * lib/tap-driver.sh: Link an Austin Group discussion about how
+ shells are allowed, and even encouraged, to set the special
+ variable `$?' to values greater than 256 to report termination
+ of a child by a signal. Improve and extend comments about our
+ workarounds for unusual korn shell signals' propagation.
+ Thanks to Eric Blake for the pointers.
+
+2011-09-28 Stefano Lattarini <stefano.lattarini@gmail.com>
+
maintcheck: fix usage of `cd' instead of `$(am__cd)'
* lib/am/inst-vars.am (am__uninstall_files_from_dir): Use
`$(am__cd)', not plain `cd'.
+2011-09-28 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tap/awk: account for unusual korn shell signal handling behaviour
+ This change has been motivated by a testsuite failure on Debian
+ with the AT&T Korn Shell version 93u-1.
+ * lib/tap-driver.sh: Temporarily ignore some common signals when
+ waiting for the test command to complete, to avoid premature exit
+ in Korn shells that propagate to themselves signals that have
+ killed a child process.
+ See also related commit `v1.11-1342-g6321ad3'.
+
+2011-09-28 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tap/awk: handle exit statuses > 256 (seen on few korn shells)
+ Some Korn shells, when a child process die due to signal number
+ n, can leave in $? an exit status of 256+n instead of the more
+ standard 128+n. Apparently, both behaviours are allowed by
+ POSIX, so be prepared to handle them both.
+ This change has been motivated by a testsuite failure on Debian
+ with the AT&T Korn Shell version 93u-1.
+ * lib/tap-driver.sh (get_test_exit_message): Handle the described
+ Korn Shell behaviour too.
+ ($scriptversion): Update.
+
+2011-09-28 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: remove redundant workaround for Solaris /bin/sh
+ After previous commit `v1.11-1424-g27a399a', some workarounds for
+ Solaris 10 /bin/sh are no more required, and in fact they have
+ started causing XPASS results on Solaris.
+ * tests/tap-signal.tap: Remove unneeded workarounds for Solaris 10
+ /bin/sh. Some related minor code reorganizations.
+
2011-09-24 Stefano Lattarini <stefano.lattarini@gmail.com>
uninstall: "make uninstall" before "make install" works
* tests/aclocal-path-precedence.test: Call `$ACLOCAL' with the
proper overridden system acdir.
+2011-09-28 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: fix spurious failures in tests on TAP + signals
+ * tests/tap-signal.tap: Write the dummy test scripts as perl
+ scripts, not as shell scripts, to work around unportabilities
+ in the handling of signals. In fact, even with bash, the older
+ script were unable to properly deliver a SIGQUIT to themselves
+ consistently.
+
+2011-09-26 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: fix spurious failure in test on ACLOCAL_PATH precedences
+ * tests/aclocal-path-precedence.test: Do not merely append to
+ `configure.in', it should be created from scratch. Pass package
+ name and version to AC_INIT as arguments, otherwise autoconf will
+ complain.
+
2011-09-20 Stefano Lattarini <stefano.lattarini@gmail.com>
docs: remove mention of deprecated option `--acdir'
* tests/Makefile.am (TESTS): Add it.
* NEWS: Update.
+2011-09-18 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ parallel-tests: expose automake bug#9546
+ * tests/test-driver-is-distributed.test: New test.
+ * tests/Makefile.am (TESTS, XFAIL_TESTS): Add it.
+
+2011-09-18 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ py-compile tests: more faithful 'installcheck' support
+ * tests/py-compile-basedir.test: Use the `$am_scriptsdir' variable
+ instead of `$testsrcdir/../lib', to test more faithfully under
+ "make installcheck". More uses of the `fatal_' subroutine to
+ diagnose hard errors.
+ * tests/py-compile-basic.test: Likewise.
+ * tests/py-compile-basic2.test: Likewise.
+ * tests/py-compile-destdir.test: Likewise.
+ * tests/py-compile-env.test: Likewise.
+ * tests/py-compile-option-terminate.test: Likewise.
+ * tests/py-compile-usage.test: Likewise.
+
+2011-09-18 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: remove redundant settings of `errexit' shell flag
+ * tests/aclocal-acdir.test: Do not set the `errexit' shell
+ flag, as it is already set by `tests/defs'.
+ * tests/aclocal-print-acdir.test: Likewise.
+ * tests/distcheck-missing-m4.test: Likewise.
+ * tests/distcheck-outdated-m4.test: Likewise.
+ * tests/java-clean.test: Likewise.
+ * tests/java-compile-install.test: Likewise.
+ * tests/java-compile-run-flat.test: Likewise.
+ * tests/java-compile-run-nested.test: Likewise.
+ * tests/java-mix.test: Likewise.
+ * tests/java-no-duplicate.test: Likewise.
+ * tests/java-nobase.test: Likewise.
+ * tests/java-rebuild.test: Likewise.
+ * tests/java-sources.test: Likewise.
+ * tests/java-uninstall.test: Likewise.
+ * tests/javaflags.test: Likewise.
+ * tests/nobase-nodist.test: Likewise.
+ * tests/py-compile-basedir.test: Likewise.
+ * tests/py-compile-basic.test: Likewise.
+ * tests/py-compile-basic2.test: Likewise.
+ * tests/py-compile-destdir.test: Likewise.
+ * tests/py-compile-env.test: Likewise.
+ * tests/py-compile-option-terminate.test: Likewise.
+ * tests/py-compile-usage.test: Likewise. Since we are at it,
+ add an extra blank line, for clarity.
+
2011-09-16 Stefano Lattarini <stefano.lattarini@gmail.com>
automake: remove special handling of `ANSI2KNR' make variable
* automake.in (define_configure_variable): Remove special-cased
handling of `ANSI2KNR' variable. Related minor simplifications.
-2011-09-12 Stefano Lattarini <stefano.lattarini@gmail.com>
-
- cosmetics: fix various typos and grammaros
- * NEWS: Fix typos.
- * tests/py-compile-basic.test: Likewise.
- * tests/py-compile-basedir.test: Fix botched wording in comments.
- * tests/py-compile-option-terminate.test: Fix typo and incomplete
- comment.
- Suggestions by Peter Rosin.
-
2011-09-18 Stefano Lattarini <stefano.lattarini@gmail.com>
coverage: test interactions of `nobase_' and `nodist_'
of the `.java' files. Correct other minor blunders. Improve some
comments.
+2011-09-15 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ automake: remove code duplication in 'handle_tests'
+ * automake.in (handle_tests): Factor out some code dealing with
+ test extensions and rules for generation of `.log' files into ...
+ (handle_per_suffix_test): ... this new subroutine.
+ * tests/parallel-tests-exeext.test: New test.
+ * tests/Makefile.am (TESTS): Update.
+
+2011-09-13 Peter Rosin <peda@lysator.liu.se>
+
+ * tests/defs [$sh_errexit_works = yes]: Output 'SIGINT' for signal 2.
+
+2011-09-13 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ self tests: work around 'unset' spurious non-zero return value
+ * tests/self-check-exit.tap ($init): Unset the possibly already
+ unset variable `am_explicit_skips' when the `errexit' shell flag
+ is not yet active, for the benefit of shell that return a non-zero
+ status when unsetting an already-unset variable.
+
+2011-09-13 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: an explicative comment on signal-related woes
+ * tests/self-check-exit.tap: Add pointer to a bug-autoconf
+ discussion explaining why this test needs to temporarily block
+ SIGINT.
+
+2011-09-13 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: also trap SIGQUIT; other fixes
+ Some shells seems to just ignore SIGQUIT under some circumstances,
+ even when it is not blocked; however, if the signal it trapped,
+ the trap gets correctly executed.
+ * tests/defs [$sh_errexit_works = yes]: Also trap SIGQUIT. Add
+ extensive explanation of why we do so.
+ * tests/self-check-cleanup.tap: Improve and fix to ensure that an
+ earlier failure does not prevent the rest of the test to execute.
+ Problem revealed by a failure in `self-check-cleanup.tap'.
+
+2011-09-13 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ test defs: work around weird ksh behaviour w.r.t. signal handling
+ * tests/defs (is_blocked_signal): Use perl to determine whether a
+ signal is trapped, since trying to do it portably within the shell
+ means opening a nasty can of worms.
+ For more information and background, see:
+ <http://lists.gnu.org/archive/html/bug-autoconf/2011-09/msg00004.html>
+ <http://mail.opensolaris.org/pipermail/ksh93-integration-discuss/2009-February/004121.html>
+ <http://www.cons.org/cracauer/sigint.html>
+
+2011-09-12 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ cosmetics: fix various typos and grammaros
+ * NEWS: Fix typos.
+ * tests/py-compile-basic.test: Likewise.
+ * tests/py-compile-basedir.test: Fix botched wording in comments.
+ * tests/py-compile-option-terminate.test: Fix typo and incomplete
+ comment.
+ Suggestions by Peter Rosin.
+
+2011-09-12 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: remove redundant line from a test script
+ * tests/test-log.test (fail.test): Remove extra `echo' command,
+ probably due to a past copy&paste blunder.
+
+2011-09-11 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: convert another test to TAP
+ * tests/primary-prefix-invalid-couples.test: Renamed ...
+ * tests/primary-prefix-invalid-couples.tap: ... to this, and
+ converted to the use of the TAP protocols (this requires only
+ very minimal changes this time).
+ * tests/list-of-tests.mk: Update.
+
2011-09-11 Stefano Lattarini <stefano.lattarini@gmail.com>
tests: fix spurious failure on fast machines
invocations, and by always calling autoconf with the `--force'
flag.
+2011-09-10 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: minor optimizations/simplifications in some aclocal tests
+ * acloca11.test: Instead of running ./configure and then grepping
+ the file it has produced, directly grep configure. Add trailing
+ `:' command.
+ * acloca12.test: Likewise.
+ * acloca10.test: Likewise. Also, remove the autom4te caches
+ by hand, instead of sleeping to ensure they get outdated and
+ ignored.
+ * aclocal9.test: Likewise.
+ * acloca18.test: Likewise. Also, do not remove file `foo',
+ which is not being created anymore.
+ * acloca20.test: The test makes sense only if the system support
+ real symlinks, so skip it if this is not the case. This allows
+ for various simplifications.
+
+2011-09-10 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ prove: avoid weird signal-related bug
+ * tests/defs (trap): In the trap code dealing with SIGPIPE
+ signal, ignore further signals of the same kind.
+ See also Test::Harness issue [rt.cpan.org #70855], archived at
+ <https://rt.cpan.org/Ticket/Display.html?id=70855>
+
+2011-09-10 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: allow use of `prove' to run automake testsuite
+ * tests/prove-runner: New helper script; mostly it wraps
+ our `*.test' test scripts to make them runnable by the
+ `prove' utility.
+ * tests/Makefile.am (EXTRA_DIST): Distribute it.
+ (PROVE, AM_PROVEFLAGS, AM_PROVECMD): New variables.
+ (prove, installprove): New targets, used run the automake
+ testsuite through the `prove' utility.
+ * tests/gen-testsuite-part: For any considered dependency,
+ generate a variables that groups all the tests depending on
+ it. This is especially useful for use in the new `prove'
+ and `installprove' targets, which cannot rely on make for
+ inter-test dependencies, and have to handle them "by hand".
+
+2011-09-10 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests defs: more namespace-safeness
+ ($using_tap): Renamed ...
+ ($am_using_tap): ... to this.
+ ($parallel_tests): Renamed ...
+ ($am_parallel_tests): ... to this.
+ ($test_prefer_config_shell): Renamed ...
+ ($am_test_prefer_config_shell): ... to this.
+ ($original_AUTOMAKE): Renamed ...
+ ($am_original_AUTOMAKE): ... to this.
+ ($original_ACLOCAL): Renamed ...
+ ($am_original_ACLOCAL): ... to this.
+ * tests/defs, tests/defs-static.in, tests/gen-testsuite-part,
+ tests/README, and all callers: Adjusted.
+ * tests/Makefile.am (AM_TESTS_ENVIRONMENT): Adjust, and rewrite
+ to make future adjustments easier.
+ * tests/defs (funcs_file_): Renamed ...
+ (am_funcs_file): ... to this.
+ * Makefile.am (sc_tests_obsolete_variables): New maintainer
+ check.
+ (syntax_check_rules): Add it.
+
2011-09-09 Stefano Lattarini <stefano.lattarini@gmail.com>
aclocal: better URL reference in error message
definitions, reference the automake page at `www.gnu.org', not
at `sources.redhat.com'.
+2011-09-10 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: avoid creating useless files in some test directories
+ * tests/aclocal.test: Define `$am_create_testdir' to "empty"
+ before sourcing ./defs, to avoid bringing in unused auxiliary
+ files. Related adjustments.
+ * tests/aclocal3.test: Likewise.
+ * tests/aclocal8.test: Likewise.
+ * tests/aclocal9.test: Likewise.
+ * tests/acloca10.test: Likewise.
+ * tests/acloca11.test: Likewise.
+ * tests/acloca12.test: Likewise.
+ * tests/acloca15.test: Likewise.
+ * tests/acloca16.test: Likewise.
+ * tests/acloca17.test: Likewise.
+ * tests/acloca18.test: Likewise.
+ * tests/acloca20.test: Likewise.
+ * tests/acloca21.test: Likewise.
+ * tests/acloca23.test: Likewise.
+ * tests/aclocal-print-acdir.test: Likewise.
+ * tests/ar-lib.test: Likewise.
+ * tests/compile.test: Likewise.
+ * tests/compile2.test: Likewise.
+ * tests/compile3.test: Likewise.
+ * tests/compile6.test: Likewise.
+ * tests/instsh2.test: Likewise.
+ * tests/instsh3.test: Likewise.
+ * tests/mdate5.test: Likewise.
+ * tests/mdate6.test: Likewise.
+ * tests/missing3.test: Likewise.
+ * tests/missing5.test: Likewise.
+ * tests/mkinst3.test: Likewise.
+ * tests/self-check-explicit-skips.test: Likewise.
+ * tests/self-check-report.test: Likewise.
+ * tests/self-check-sanity.test: Likewise.
+ * tests/tap-driver-stderr.test: Likewise.
+
+2011-09-09 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ test defs: more environment cleanup
+ * tests/defs: Also unset the XFAIL_TESTS, and various
+ LOG_DRIVER, LOG_COMPILER variables (and corresponding
+ flags variable).
+
+2011-09-09 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: allow depmod.tap to only run a subset of its tests
+ * tests/depmod.tap: If command-line arguments are passed to the
+ tests, they are now taken as the list of test names for which the
+ checks are to be run.
+
+2011-09-09 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: allow instspc.tap to only run a subset of its tests
+ * tests/instspc.tap: If command-line arguments are passed to the
+ tests, they are now taken as the list of test names for which the
+ checks are to be run.
+
2011-09-09 Stefano Lattarini <stefano.lattarini@gmail.com>
coverage: distcheck-hook to catch missing/outdated *.m4 files
(EXTRA_DIST): Distribute it.
(TESTS): Add the new tests.
+2011-09-08 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: more granularity in variables for tools overriding
+ Among the other things, this change makes it possible to use, at
+ the same time, an autoconf version in the automake testsuite and
+ another one in the automake build system; even more importantly, it
+ makes it possible to use, say, a non-GNU make implementation in the
+ test scripts, while at the same time using GNU make to drive the
+ execution of the automake's test suite.
+ * tests/defs-static.in (TEX, EGREP, FGREP, SHELL, PERL, MAKE,
+ AUTOCONF, AUTOM4TE, AUTORECONF, AUTOHEADER, AUTOUPDATE,
+ original_AUTOMAKE, original_ACLOCAL): Allow and prefer overriding
+ through corresponding `AM_TESTSUITE_..' variables; for example,
+ $GREP can now be overridden by an exported `$AM_TESTSUITE_GREP'
+
+2011-09-08 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: fix `skip_row_' subroutine in tap-functions.sh
+ * tests/tap-functions.sh (skip_row_): Use `seq_', not `seq', to
+ be portable to non-GNU systems.
+ Bug revealed by a testsuite error on Solaris.
+
+2011-09-08 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ installcheck: support it in our own testsuite
+ * NEWS: Update.
+ * tests/defs-static.in ($am_running_installcheck): New variable.
+ Initialize it to a default of "no", and sanity-check its value.
+ ($am_datadir, $am_docdir, $am_automake_acdir, $am_system_acdir,
+ $am_bindir, $am_scriptdir, $am_pkgvdir): New variables,
+ initialized from AC_SUBST'ed values which are adaptively chosen
+ depending on whether the testsuite is running in "simple" mode
+ or in "installcheck" mode.
+ ($MISSING): Remove, it's not really required by the testsuite.
+ ($PATH): Update using `$am_bindir', not `$testbuilddir'.
+ * Makefile.am (installcheck-local): New target.
+ * lib/Automake/tests/Makefile.am (installcheck-local): New target.
+ (AM_TESTS_ENVIRONMENT, AM_PL_LOG_FLAGS): Adaptively choose the
+ `-I' flags for perl based on whether the tests are being run by
+ "make check" or "make installcheck".
+ * tests/Makefile.am (installcheck-local): New target.
+ (defs-static): Generate it at make time, not at configure time,
+ in order to correctly resolve all the indirections in the
+ AC_SUBST'd variables; done with the help of ...
+ (do_subst): ... this new macro.
+ (EXTRA_DIST): Explicitly distribute `defs-static.in'.
+ (CLEANFILES): Remove `defs-static'.
+ * configure.ac (AC_CONFIG_FILES): Remove `tests/defs-static'.
+ (AC_SUBST): Substitute also `automake_acdir', `system_acdir',
+ `scriptdir' and `amdir' to proper values.
+ * lib/Makefile.am (scriptdir): Don't explicitly define anymore.
+ * lib/am/Makefile.am (amdir): Likewise.
+ * m4/acdir/dirlist: Remove, it's not needed anymore. Instead
+ use ...
+ * m4/acdir/.placeholder: ... this file.
+ * m4/Makefile.am (EXTRA_DIST): Update.
+ (automake_acdir, system_acdir): Don't explicitly define anymore.
+ * tests/aclocal-print-acdir.test: Use new `$am_...dir' variables,
+ instead of hard-coding values with $testsrcdir and $testbuilddir,
+ to test more faithfully under "make installcheck".
+ * tests/amhello-binpkg.test: Likewise.
+ * tests/amhello-cflags.test: Likewise.
+ * tests/amhello-cross-compile.test: Likewise.
+ * tests/ansi2knr-no-more.test: Likewise.
+ * tests/autodist-stamp-vti.test: Likewise.
+ * tests/auxdir.test: Likewise.
+ * tests/check2.test: Likewise.
+ * tests/copy.test: Likewise.
+ * tests/multlib.test: Likewise.
+ * tests/obsolete.test: Likewise.
+ * tests/parallel-tests-interrupt.test: Likewise.
+ * tests/repeated-options.test: Likewise.
+ * tests/suffix5.test: Likewise.
+ * tests/tap-doc2.test: Likewise.
+ * tests/txinfo22.test: Likewise.
+ * tests/vtexi3.test: Likewise.
+ * tests/defs: Likewise. Also, print the values of `$using_tap'
+ and `$am_running_installcheck', to simplify debugging, and do
+ some related code movings and tweakings.
+ * tests/aclocal3.test: Update to avoid possible spurious failures
+ when running under "make installcheck".
+ * tests/warnings-unknown.test: Likewise.
+ * tests/location.test: Likewise, and improve debugging input since
+ we are at it.
+ * tests/acloca10.test: Adapt and tweak to work also during a
+ "make installcheck", and not to require a pre-populated `dirlist'
+ file.
+ * tests/acloca18.test: Likewise.
+ * tests/dirlist2.test: Likewise.
+ * tests/dirlist.test: Likewise, and remove an obsolete comment.
+ * tests/README: Update.
+ * tests/dirlist-abspath.test: New test.
+ * tests/list-of-tests.mk: Update.
+
+2011-09-08 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ maintcheck: fix spurious failures
+ * tests/gen-testsuite-part (write_wrapper_script): More uses of
+ creative quoting in the generated tests, to avoid triggering
+ failures in the `sc_tests_Exit_not_exit' maintcheck.
+ * Makefile.am (sc_tests_tap_plan): It's OK for auto-generated
+ TAP tests that sources an hand-written one not to declare a TAP
+ plan: the sourced test will do so.
+ * tests/Makefile.am (checked_test_extensions): Don't define
+ anymore, the default value `$(TEST_EXTENSIONS)' is correct now.
+ (expected_list_of_tests): Don't define anymore, the default
+ value `$(TESTS)' is correct now.
+
2011-09-06 Stefano Lattarini <stefano.lattarini@gmail.com>
aclocal: more granularity in acdir overriding
* NEWS, bootstrap: Update.
* doc/automake.texi (aclocal Options, Macro Search Path): Update.
+2011-09-08 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ cosmetic: rename gen-tests-deps -> gen-testsuite-part
+ * tests/gen-tests-deps: Renamed ...
+ * tests/gen-testsuite-part: ... to this.
+ * bootstrap: Update.
+ * tests/Makefile.am ($(srcdir)/tests-deps.am): Don't generate
+ nor include this ...
+ ($(srcdir)/testsuite-part.am): ... do with this instead.
+ * tests/.gitignore (tests-deps.am): Don't ignore this ...
+ (testsuite-part.am): ... ignore this instead.
+
+2011-09-08 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: revamp generation of autogenerated tests
+ With this change, we make generated tests be real scripts again
+ -- albeit only thin layers around the tests they wrap. It turned
+ out that the apparently clever tricks we played by having lots
+ of custom (and auto-generated) `..._LOG_COMPILER' variables and
+ by having the tests in $(wrap_TESTS) not being real on-file-system
+ scripts caused more pain and confusion than expected, for only
+ marginal gains. Also, that previous setup didn't interact very
+ well with "make check RECHECK_LOGS=", since the log files for
+ the "wrappee" tests were always considered out-of-date (which in
+ hindsight is natural, since these log files depended on wrappee
+ tests that didn't exist -- and weren't created). And finally, it
+ wasn't possible to easily and naturally run the wrappee tests
+ from the command line -- definitely harmful for debugging and
+ sanity checking.
+ * tests/gen-wrap-tests: Deleted, its role subsumed into ...
+ * tests/gen-tests-deps: ... this script, with the help of
+ (unindent, atomic_write, build_matcher, write_wrapper_script,
+ %test_generators): ... these new functions and variables.
+ Other few minor fixlets and improvements.
+ * tests/wrap-tests.sh: Delete, not needed anymore.
+ * tests/Makefile.am ($(srcdir)/wrap-tests.am): Don't generate
+ nor include anymore.
+ (EXTRA_DIST): Update.
+ (wrap_TESTS): Renamed to ...
+ (generated_TESTS): ... this.
+ ($(generated_TESTS)): Must have the same dependencies of
+ `$(srcdir)/tests-deps.am'.
+ Other minor related reordering and adjustments.
+ * bootstrap: Do not generate `wrap-tests.am' anymore.
+ * tests/.gitignore (wrap-tests.am): Don't ignore anymore.
+ (*-w.test, *-w.tap): Ignore.
+ * tests/README: Update paragraph on automatically-generated
+ tests.
+
+2011-09-08 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: automatize generation of dependencies for tests
+ * tests/gen-test-deps: New script, automatically generates
+ dependencies for the tests.
+ * tests/Makefile.am (EXTRA_DIST): Distribute it.
+ ($(srcdir)/tests-deps.am): New rule.
+ (include $(srcdir)/tests-deps.am): New inclusion.
+ Remove hand-written dependencies for tests. Other related
+ updates.
+ * tests/list-of-tests (tap_with_common_setup_TESTS,
+ tap_other_TESTS): Delete, their content merged back into ...
+ (handwritten_TESTS): ... this.
+ * tests/.gitignore (tests-deps.am): New ignored file.
+ * bootstrap: Generate `tests/tests-deps.am'.
+
+2011-09-07 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: avoid spurious failure due to bug in older TAP::Parser
+ Older TAP::Parser versions (e.g., 3.10) erroneously convert a test
+ number `0' into `1'; do not let this bug cause a spurious failure
+ in our testsuite.
+ * tests/tap-test-number-0.test: Skip if the aforementioned
+ TAP::Parser bug is detected.
+
+2011-09-07 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tap/perl: be more portable to older perl versions
+ * lib/tap-driver.pl (trap_perl_warnings_and_errors): Do not use
+ the three-args version of the `open' builtin, it is not well
+ supported by older perl versions (at least perl 5.6.2).
+ (start): Add a no-op statement to pacify a "possible typo ... used
+ only once" perl warning.
+
+2011-09-07 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tap/perl: be more portable to older TAP::Parser versions
+ * lib/tap-driver.pl (start): Do not call the `ignore_exit' method
+ on our TAP parser: it is not supported in older TAP::Parser
+ versions (e.g., 3.10), and is not really required by our usage of
+ the parser object.
+
+2011-09-07 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: fix spurious Cygwin failure of a test on TAP support
+ * tests/tap-bad-prog.tap (noexec.test): Add forgotten `echo'.
+ When trying to run it, redirect its output to /dev/null, to
+ avoid confusing automake's testsuite own TAP driver.
+
+2011-09-07 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ test defs: fix glitch in TAP version of `warn_' function
+ * tests/tap-functions.sh (warn_): Do not display the warning
+ message two times.
+
+2011-09-07 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: fix another spurious failure due to ignored signals
+ * tests/tap-signal.test: Renamed ...
+ * tests/tap-signal.tap: ... to this, converted to the use of the
+ TAP protocol, and improved not to try to use/trap signals that
+ are ignored by the parent shell (they will be forcibly ignored
+ by all the child processes too). Extend the test a bit since we
+ are at it.
+ * tests/list-of-tests.mk: Update.
+
+2011-09-07 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: fix a spurious failure due to lacking Fortran compilers
+ * tests/silent-many-generic.test (configure.in): Cause the test
+ to be skipped if `configure' fails to find working C++, Fortran
+ and Fortran 77 compilers.
+
+2011-09-07 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ self tests: more use of TAP
+ * tests/self-check-reexec.test: Renamed ...
+ * tests/self-check-reexec.tap: ... to this, converted to the
+ use of TAP protocol, and extended a bit.
+ * tests/self-check-seq.test: Renamed ...
+ * tests/self-check-seq.tap: ... to this, and converted to the
+ use of TAP protocol.
+ * tests/list-of-tests.mk: Update.
+
+2011-09-07 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ self tests: fix spurious failures when /bin/ksh do not exists
+ * tests/self-check-reexec.test: Ensure that the non-bash shells
+ we look do really exists. Since we are at it, also try the name
+ `ash', non `dash', for the Almquist Shell, for Cygwin 1.5.
+
+2011-09-06 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: more use of TAP in our own testsuite
+ * tests/parallel-tests-interrupt.test: Renamed ...
+ * tests/parallel-tests-interrupt.tap: ... to this, and converted
+ to the use of TAP protocol.
+ * tests/self-check-cleanup.test: Renamed ...
+ * tests/self-check-cleanup.tap: ... to this, and converted to the
+ use of TAP protocol.
+ * tests/self-check-dir.test: Renamed ...
+ * tests/self-check-dir.tap: ... to this, and converted to the use
+ of TAP protocol.
+ * tests/self-check-is_newest.test: Renamed ...
+ * tests/self-check-is_newest.tap: ... to this, and converted to
+ the use of TAP protocol.
+ * tests/self-check-unindent.test: Renamed ...
+ * tests/self-check-unindent.tap: ... to this, and converted to
+ the use of TAP protocol.
+ * tests/list-of-tests.mk: Update.
+
+2011-09-06 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ self tests: sanity check on 'is_blocked_signal' function
+ * tests/self-check-is-blocked-signal.tap: New test.
+ * tests/list-of-tests.mk: Update.
+
2011-09-06 Stefano Lattarini <stefano.lattarini@gmail.com>
tests: fix spurious failures due to ignored signals
* tests/defs.in: Don't nullify $ACLOCAL_TESTSUITE_FLAGS, and do
not export it.
-2011-09-01 Stefano Lattarini <stefano.lattarini@gmail.com>
+2011-09-02 Stefano Lattarini <stefano.lattarini@gmail.com>
- automake: fix regression due to de-ansification support removal
- The last change `v1.11-947-g136b489' removed code that automake
- was using to decide whether binary objects were built by the
- generated Makefile.in, so that it could avoid to emit unneeded
- code when this was not the case. Re-introduce such code in a
- less-obfuscated form, and add a test to ensure we don't regress
- again.
+ tests: remove redundant settings of `errexit' shell flag
+ * doc-parsing-buglets-colneq-subst.test: Do not set the
+ `errexit' shell flag, as it is already set by `tests/defs'.
+ * doc-parsing-buglets-tabs.test: Likewise.
+ * java-empty-classpath.test: Likewise.
+ * vala-vpath.test: Likewise.
+
+2011-09-02 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: more uses of `$top_testsrcdir', in the `amhello' tests
+ * tests/amhello-binpkg.test: Use `$top_testsrcdir' instead
+ of `$testsrcdir/..'.
+ * tests/amhello-cflags.test: Likewise.
+ * tests/amhello-cross-compile.test: Likewise.
+
+2011-09-02 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: avoid creating useless files in the `amhello' tests
+ * tests/amhello-binpkg.test: Define `$am_create_testdir' to
+ "empty" before sourcing ./defs, to avoid bringing in unused
+ auxiliary files.
+ * tests/amhello-cflags.test: Likewise.
+ * tests/amhello-cross-compile.test: Likewise.
+
+2011-09-02 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ coverage: vala support failing for VPATH from-scratch builds
+ * tests/vala-vpath.test: New test, xfailing.
+ * tests/Makefile.am (TESTS): Update.
+ * THANKS: Update.
+ From a report by Zbigniew Jędrzejewski-Szmek.
+ Related to automake bug#8753.
+
+2011-09-01 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ docs: report few more automake parsing limitations
+ Partly motivated by automake bug#8360.
+ * doc/automake.texi (General Operation): Report few more automake
+ limitations w.r.t. parsing of unusual makefile constructs. Related
+ minor reorderings.
+ * tests/doc-parsing-buglets-colneq-subst.test: New test.
+ * tests/doc-parsing-buglets-tabs.test: Likewise.
+ * tests/Makefile.am (TESTS): Update.
+
+2011-09-01 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ automake: fix regression due to de-ansification support removal
+ The last change `v1.11-947-g136b489' removed code that automake
+ was using to decide whether binary objects were built by the
+ generated Makefile.in, so that it could avoid to emit unneeded
+ code when this was not the case. Re-introduce such code in a
+ less-obfuscated form, and add a test to ensure we don't regress
+ again.
* automake.in ($must_handle_compiled_objects): New global
variable, telling whether the generated Makefile has to build
compiled objects.
(@menu, @detailmenu): Remove entries about de-ANSI-fication
support.
-2011-09-02 Stefano Lattarini <stefano.lattarini@gmail.com>
+2011-08-29 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: better respect of TAP usage in wrapped tests
+ * tests/gen-wrap-tests: Partial rewrite to take into account the
+ possibility that the wrapped test uses the TAP protocol. This
+ complicates the code quite a bit, but is essential to ensure real
+ correctness in all the wrapped scripts.
+ * tests/wrap-tests.sh: Update, and add few more sanity checks.
+
+2011-08-29 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: use both perl and awk implementations for tests on TAP
+ * tests/gen-wrap-tests (get_list_for_pltap): New function, gets
+ the list of tests that check the Automake TAP support.
+ ($defines_for_pltap): New variable.
+ ($wrapper_type_list): Add "pltap".
+ * tests/defs-static.in: Check that `$am_tap_implementation' is
+ not exported in the environment.
+ * tests/self-check-env-sanitize.tap: Update.
+
+2011-08-29 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: two single auxiliary scripts to run/generate wrapped tests
+ * tests/config-shell-tests.sh, tests/parallel-tests.sh,
+ tests/gen-config-shell-tests, tests/gen-parallel-tests: Deleted,
+ their roles taken over by ...
+ * tests/wrap-tests.sh, tests/gen-wrap-tests: ... these new more
+ general scripts.
+ * tests/Makefile.am (EXTRA_DIST): Distribute the new scripts, and
+ do not distribute the obsolete ones anymore.
+ ($(srcdir)/parallel-tests.am, $(srcdir)/config-shell-tests.am): Do
+ not include nor generate anymore, they have been superseded by ...
+ ($(srcdir)/wrap-tests.am): ... this new fragment, automatically
+ generated by `gen-wrap-tests'.
+ (PTEST_LOG_COMPILER, SHTST_LOG_COMPILER): Delete, they are now
+ defined in the new `wrap-tests.am' makefile fragment.
+ (TEST_EXTENSIONS): Do not append to it anymore, that is now done
+ by the new `wrap-tests.am' makefile fragment above.
+ ($(parallel_tests:.ptest=.log), $(config_shell_tests:.shtst=.log),
+ $(parallel_tests), $(config_shell_tests): Remove this dependencies,
+ superseded by ...
+ ($(wrap_TESTS), $(wrap_LOGS)): ... these ones.
+ (MAINTAINERCLEANFILES): Delete, it's not used anymore.
+ (TESTS): Do not contain anymore `$(config_shell_tests)' nor
+ `$(parallel_tests)', contain `$(wrap_TESTS)' instead, as defined
+ by `$(srcdir)/wrap-tests.am'.
+ Other related minor edits and reorderings.
+ * bootstrap, tests/.gitignore, tests/README: Update.
+
+2011-08-29 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: list of tests placed in a separate Makefile fragment
+ This change allows us to easily use make instead of by-hand (and
+ more fragile) grepping to extract the list of hand-written tests
+ from our Makefiles.
+ * bootstrap (BOOTSTRAP_MAKE): New variable.
+ Use it to run make in order to extract the list of hand-written
+ tests from ...
+ * tests/list-of-tests.mk: ... this new file, where they are
+ listed in ...
+ (handwritten_TESTS, tap_with_common_setup_TESTS, tap_other_TESTS):
+ ... these variables, extracted respectively from ...
+ * tests/Makefile.am (handwritten_tests, tap_other_tests,
+ tap_with_common_setup_tests): ... these older variables, now
+ removed.
+ (EXTRA_DIST, expected_list_of_tests): Update w.r.t. the variable
+ names' changes.
+ (include $(srcdir)/list-of-tests.mk): New inclusion.
+ Remove an overkill indirection, and with it ...
+ (tap_with_common_setup_logs): ... this variable.
+ ($(srcdir)/parallel-tests.am, $(srcdir)/config-shell-tests.am):
+ Update to match the new interface of ...
+ (gen-parallel-tests, gen-config-shell-tests): ... these scripts,
+ that now read the list of tests from the standard input.
+
+2011-08-29 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: prefer AM_... variables to user-reserved ones in Makefile
+ * tests/Makefile.am (TAP_LOG_DRIVER_FLAGS): Renamed ...
+ (AM_TAP_LOG_DRIVER_FLAGS): ... to this.
+
+2011-08-29 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: some minor fixlets w.r.t. TAP-based tests
+ The variable that is used by our test scripts to decide whether
+ they are "plain" or TAP-based is `using_tap'; however, some
+ peripheral sanity checks and sanitization code in our testsuite
+ were erroneously using the `use_tap' variable instead. Fix that.
+ * tests/Makefile.am (AM_TESTS_ENVIRONMENT): Nullify `using_tap',
+ not `use_tap'.
+ * tests/defs-static.in: Check that `using_tap', not `use_tap',
+ is not exported.
+ * tests/self-check-env-sanitize.test: Update.
- coverage: vala support failing for VPATH from-scratch builds
- * tests/vala-vpath.test: New test, xfailing.
- * tests/Makefile.am (TESTS): Update.
- * THANKS: Update.
- From a report by Zbigniew Jędrzejewski-Szmek.
- Related to automake bug#8753.
+2011-08-29 Stefano Lattarini <stefano.lattarini@gmail.com>
-2011-09-01 Stefano Lattarini <stefano.lattarini@gmail.com>
+ tests: avoid re-running a test with the 'parallel-tests' option
+ * tests/backcompat5.test (parallel_tests): Define to "no", to
+ prevent the generation of a sibling test script that uses the
+ 'parallel-tests' driver. That would be useless anyway, since
+ we manually override the contents of configure.in, thus clobbering
+ the `parallel-tests' option in AM_INIT_AUTOMAKE.
+ Fix a typo in comments since we are at it.
- docs: report few more automake parsing limitations
- Partly motivated by automake bug#8360.
- * doc/automake.texi (General Operation): Report few more automake
- limitations w.r.t. parsing of unusual makefile constructs. Related
- minor reorderings.
- * tests/doc-parsing-buglets-colneq-subst.test: New test.
- * tests/doc-parsing-buglets-tabs.test: Likewise.
- * tests/Makefile.am (TESTS): Update.
+2011-08-29 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: defaults to awk+shell driver in tests on TAP support
+ * tests/defs ($am_tap_implementation): Default to "shell", not
+ to "perl".
2011-08-25 Stefano Lattarini <stefano.lattarini@gmail.com>
* lib/ylwrap: Convert tabs to spaces.
Reported by Karl Berry.
+2011-08-28 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: different test scripts cannot share the same basename
+ If two test scripts have the same basename, they will end up
+ sharing the same log file, leading to all sort of undefined and
+ undesired behaviours. Add a Makefile check verifying that this
+ is not the case.
+ * tests/test-logs-repeated.test: New test, verifying previously
+ unspecified semantics (in the definition of $(TEST_LOGS) from
+ $(TESTS)) that we use in our new check.
+ * tests/Makefile.am (TESTS): Update.
+ (check-no-repeated-test-name): New .PHONY check.
+ (check-local): Depend on it.
+
+2011-08-26 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: fix spurious failure for awk implementation of TAP driver
+ * tests/tap-realtime.test: The awk+shell implementation of the TAP
+ driver must "read ahead" of one line in order to catch the exit
+ status of the test script it runs. So, in the TAP-producer script
+ here, be sure to echo one "dummy" line after each TAP result line
+ in order not to cause false positives.
+
+2011-08-25 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tap/awk: allow escaping of TAP directives
+ * lib/tap-driver.sh (setup_result_obj): Handle escaping of TAP
+ directives in a way tat is (mostly) compatible by what is done
+ by the TAP::Parser module.
+ With this change, the tests `tap-escape-directive.test' and
+ `tap-escape-directive-2.test' now also pass with the shell/awk
+ implementation of the TAP driver.
+
+2011-08-25 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ coverage: more about escaping of TAP directives
+ * tests/tap-escape-directive.test: Make grepping of testsuite
+ progress output slightly stricter.
+ * tests/tap-escape-directive-2.test: New test (failing with the
+ awk implementation of the TAP driver).
+ * tests/Makefile.am (tap_with_common_setup_tests): Update.
+
+2011-08-25 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tap: improve syncing between awk+shell and perl implementations
+ * lib/tap-driver.pl (stringify_test_result): Renamed ...
+ (stringify_result_obj): ... to this. Break up a clause in the
+ long "if/elsif/.../else" construct to avoid unaesthetic line
+ breaks and to be more synced with the sibling function in
+ `tap-driver.sh'. Rename the `$result', `$PASS' and `$FAIL'
+ variables to respectively `$result_obj', `$COOKED_PASS' and
+ `$COOKED_FAIL', for clarity and better syncing.
+ (handle_tap_test): Renamed ...
+ (handle_tap_result): ... to this, and change the name of the
+ `$test' local variable to `$result_obj'.
+ (extract_comment): Reimplement using the simpler `index' and
+ `substr' builtins, rather than with more advanced uses of
+ regular expressions.
+ (%test_results, @test_results): Renamed respectively ...
+ (%test_results_seen, @test_results_list): ... to these, and
+ related adjustments throughout the `TEST_RESULTS' block.
+ (main, get_global_test_result): Refactor and do some cosmetic
+ changes to make these functions clearer and better synced with
+ sibling code in `tap-driver.sh'.
+ Other minor cosmetic and typo fixes.
+ * lib/tap-driver.sh (extract_tap_comment): Remove outdated
+ "FIXME" comments.
+ (get_global_test_result): Small reordering to make it better
+ synced with its sibling function in `tap-driver.pl'.
+ (stringify_result_obj): Consistently use `result_obj' as the
+ parameter name.
+ Other minor cosmetic and typo fixes.
+
+2011-08-25 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tap/perl: don't redirect perl warnings/errors to log files
+ With this change, the test `tap-driver-stderr.test' also passes
+ with the perl implementation of the TAP driver.
+ * lib/tap-driver.pl (start): Save the original stderr into the
+ `OLDERR' file handle, and call ...
+ (trap_perl_warnings_and_errors): ... this new function, trapping
+ the `__WARN__' and `__DIE__' "pseudo-signals" to ensure that the
+ warning and error messages goes to that original stderr stream.
+ Since we are at it, be sure to prepend all possible "die" message
+ with the name of the script as given by the `$ME' variable.
+
+2011-08-25 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tap/perl: add copyright notice, version string, and emacs stuff
+ * lib/tap-driver.pl: Add proper copyright notice and bug reporting
+ address. Remove obsolete heading comments. Add trailing comments
+ for emacs w.r.t. code-formatting rules and automatic updating of
+ the date-based script version saved into the `$VERSION' variable.
+ ($VERSION): Define to the current UTC date.
+
+2011-08-24 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tap/awk: don't redirect awk stderr to log files
+ * lib/tap-driver.sh: Handle the file descriptors redirections
+ more smartly, to avoid sending error messages from awk (about
+ e.g. syntax or I/O errors) to the log files instead that to the
+ console.
+ * tests/tap-driver-stderr.test: New test, verifying the improved
+ driver behaviour. Notice that this test still fails when using
+ the perl implementation of the TAP driver.
+ * tests/Makefile.am (tap_other_tests): Update.
+
+2011-08-24 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tap/awk: remove obsolete "FIXME" comment
+ * lib/tap-driver.sh: Remove obsolete "FIXME" comment about our
+ driver losing the exit status of the tested program; this issue
+ has been solved in precedent commit `v1.11-1052-gd630a0d'. Fix
+ an unrelated typo in comments since we are at it.
+
+2011-08-22 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: avoid spurious failure on Solaris
+ * tests/tap-signal.test (signal_caught): Fix a regression from
+ commit `v1.11-1052-gd630a0d' that was causing this test to fail
+ spuriously on Solaris (due to an already worked-around Solaris
+ /bin/sh bug, see commit `v1.11-981-g0ef3ef5').
+
+2011-08-22 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: use the new awk+shell TAP driver in our own testsuite
+ * tests/Makefile.am (TAP_LOG_DRIVER): Redefine to use the new
+ awk+shell tap driver, which should be complete enough for use in
+ a "controlled" production environment.
+
+2011-08-22 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tap/awk: handle non-zero exit status from the test command
+ * lib/tap-driver.sh (get_test_exit_message): New function in the
+ awk script, used to extract the exit status of the test program,
+ or at least a good guess of it.
+ (write_test_results): Use it, reporting an ERROR result if it
+ detects that the test program exited with a non-zero status.
+ * tests/tap-signal.test: Account for the differences in the
+ error messages generated by the awk and perl TAP drivers. Fix
+ an unrelated typo in comments since we are at it.
+
+2011-08-22 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tap/awk: prepare to fetch the exit status of the test command
+ * lib/tap-driver.sh: Rewrite some logic in the main parsing loop,
+ to make it possible to read the exit status of the test command
+ from the last line of the input stream.
+
+2011-08-22 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tap/awk: refactor for future changes
+ * lib/tap-driver.sh: Rewrite the awk script so that the parsing
+ of the input stream is implemented as a hand-rolled loop in the
+ BEGIN block, using the `getline' builtin.
+
+2011-08-22 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: avoid spurious failure if rst2html program is missing
+ * tests/tap-html.test ($required): Require `rst2html'.
+
+2011-08-22 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tap/awk: support Solaris /usr/xpg4/bin/awk
+ * lib/tap-driver.sh: Use `\t', not `\\t', to represent tabulation
+ character in regexps, even when inside double-quoted strings; that
+ seems to be more portable to Solaris 10 XPG4 awk.
+
+2011-08-22 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tap/awk: support Solaris /usr/bin/nawk
+ * lib/tap-driver.sh (close_or_die): Remove, the use of the `close'
+ builtin in there was confusing /usr/bin/nawk from Solaris 10 into
+ a syntax error.
+ (write_test_results): Simply use `close' on the `*.trs' file, that
+ should be enough.
+
+2011-08-21 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tap/awk: correctly extract message from SKIP plans
+ * lib/tap-driver.sh (/^1\.\.0+[ \t]*#/): Add a forgotten `?' to
+ the regex used to extract the skip message from a "1..0" TAP plan.
+ * tests/planskip.test: Check for the just-fixed bug.
+ * tests/tap-planskip-unplanned-corner.test: Don't be too strict
+ w.r.t. the formatting of the "skip whole test" message, to avoid
+ failing due to bugs irrelevant for this test (like the one just
+ fixed).
+
+2011-08-21 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tap/awk: support colorized output
+ * lib/tap-driver.sh: Pass definition of colors to the BEGIN block
+ of the awk script, using ...
+ ($init_colors): ... this new variable, which supersedes ...
+ ($red, $grn, $lgn, $blu, $mgn, $std): ... these variables, which
+ have been consequently removed.
+ ($color_tests): We don't have to pass this anymore to the awk
+ script, so do not convert it to a numeric boolean.
+ (decorate_result): Implement for real.
+ With this change, the test 'tap-color.test' now passes.
+
+2011-08-21 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tap: real (but still incomplete) awk implementation of TAP driver
+ * lib/tap-driver.sh: Add an incomplete, but mostly working,
+ implementation of a TAP parser and driver in awk. It doesn't yet
+ support colorized output, fetching of exit status from test
+ programs, nor a way to escape TAP directives in TAP result lines,
+ but passes all the tests of TAP support in the automake testsuite,
+ apart from the following ones:
+ - tap-color.test
+ - tap-escape-directive.test
+ - tap-exit.test
+ - tap-missing-plan-and-bad-exit.test
+ - tap-passthrough-exit.test
+ - tap-planskip-badexit.test
+ - tap-planskip-unplanned-corner.test
+ - tap-signal.test
+ Tested on Debian GNU/Linux with GNU awk 3.1.7 and 3.0.2, "original
+ awk" 2010-05-23-1, and mawk 1.3.3-15 (with which also the test
+ `tap-realtime.test' fails, in addition to those listed above).
+
+2011-08-21 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: fix spurious failure due to Solaris 'wc' incompatibility
+ * tests/tap-number-wordboundary.test ($planned): Strip any extra
+ whitespace from the output of `wc', to cater to Solaris 10.
+
+2011-08-21 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ coverage: keyword "SKIP" in TAP plan is case-insensitive
+ * tests/tap-planskip-case-insensitive.test: New test.
+ * tests/tap-planskip.test: Remove now-duplicated checks.
+ * tests/Makefile.am (tap_with_common_setup_tests): Update.
+
+2011-08-21 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ coverage: ambiguous use of TAP "TODO"/"SKIP" directives
+ * tests/tap-ambiguous-directive.test: New test.
+ * tests/tap-todo-skip-together.test: Reference it in heading
+ comments.
+ * tests/Makefile.am (tap_with_common_setup_tests): Update.
+
+2011-08-21 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ coverage: non-directive comments in TAP results are preserved
+ * tests/tap-result-comment.test: New test.
+ * tests/Makefile.am (tap_with_common_setup_tests): Update.
+
+2011-08-20 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ coverage: TAP result number followed by non-whitespace word boundary
+ * tests/tap-number-wordboundary.test: New test.
+ * tests/Makefile.am (tap_with_common_setup_tests): Update.
+
+2011-08-20 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ coverage: a TAP result numbered "0" is always out-of-order
+ * tests/tap-test-number-0.test: New, checks that a test result
+ numbered as 0 is to be considered out-of-order; this is
+ consistent with the behaviour of the `prove' utility.
+ * tests/Makefile.am (tap_with_common_setup_tests): Update.
+
+2011-08-20 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ cosmetics: fix botched indentation in perl TAP driver
+ * lib/tap-driver.pl (main, Getopt::Long::GetOptions): Fix
+ improper or botched indentation.
+
+2011-08-20 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ coverage: some more tests on corner cases of TAP support
+ * tests/tap-no-spurious.test: Extend checks verifying that a line
+ matching, say, the regex "^ok[a-zA-Z0-9_]" is *not* considered a
+ TAP result.
+ * tests/tap-no-spurious-numbers.test: New test, checking that our
+ TAP driver doesn't spuriously recognize as TAP result numbers what
+ is not (even if it seems pretty close).
+ * tests/tap-negative-numbers.test: New test, checking that our
+ TAP driver doesn't spuriously recognize negative TAP result
+ numbers, but correctly interprets them as test descriptions
+ instead.
+ * tests/tap-plan-leading-zero.test, tap-numbers-leading-zero.test:
+ New tests, checking how our driver fares in recognizing numbers
+ with leading zeroes in TAP results or TAP plans.
+ * tests/tap-planskip-malformed.test: New test, checking that a
+ malformed TAP plan is not recognized as a valid plan.
+ * tests/tap-plan-whitespace.test: New test, checking that a TAP
+ plan line with trailing whitespace is recognized and handled
+ correctly.
+ * tests/Makefile.am (tap_with_common_setup_tests): Update.
+
+2011-08-19 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: various fixlets and tweakings
+ * tests/seenc.test: Make grepping of automake stderr stricter.
+ Add a trailing `:' command.
+ * tests/symlink.test: Likewise. Also, define `$am_create_testdir'
+ to "empty" to avoid bringing in unused auxiliary files (only to
+ have to remove them right away), and use proper m4 quoting in
+ `configure.in'.
+ * tests/vpath.test: Make grepping of generated `Makefile.in'
+ slightly stricter. Prefer trailing `:' over trailing `Exit 0'.
+ * tests/suffix6c.test: Unset OBJEXT to avoid interferences from
+ the environment.
+ * tests/suffix12.test: Do not whitespace-indent `##' comments
+ when they are embedded in a makefile rule: having them indented
+ is not part of the Automake API, and might cause failures with
+ e.g., Tru64 make.
+ * tests/syntax.test: Simplify the `Makefile.am' to ensure that
+ automake doesn't fail for the wrong reasons. Make grepping of
+ automake stderr slightly stricter.
+ * tests/test-harness-vpath-rewrite.test: Remove useless variable
+ definition from `Makefile.am'.
+ * tests/test-driver-custom-multitest.test: Fix typo in comments.
+ * tests/self-check-me.tap: Fix minor typo in test description.
+ * tests/vars3.test: Make grepping of automake stderr stricter
+ and safer.
+ * tests/version6.test: Add sanity check, verifying that the
+ version number extracted from `automake --version' output seems
+ legit.
+ * tests/auxdir2.test: Renamed ...
+ * tests/auxdir-compauted.tap: ... to this, and converted to the
+ use of TAP.
+ * tests/auxdir4.test: Renamed ...
+ * tests/auxdir-unportable.tap: ... to this, and converted to the
+ use of TAP.
+ * tests/auxdir3.test: Renamed ...
+ * tests/auxdir-misplaced.test: ... to this.
+ * tests/auxdir5.test: Renamed ...
+ * tests/auxdir-nonexistent.test: ... to this.
+ * tests/auxdir9.test: Renamed ...
+ * tests/auxdir-autodetect.test: ... to this.
+ * tests/Makefile.am (TESTS): Update.
+ (XFAIL_TESTS): Remove `auxdir2.test'.
+
+2011-08-19 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: fix some redundant autotools calls in tests on TAP
+ * tests/tap-deps.test: Do not call autotools and configure; there
+ is no need to, since this script sources `tap-setup.sh'.
+ * tests/tap-empty-diagnostic.test: Likewise.
+ * tests/tap-escape-directive.test: Likewise.
+ * tests/tap-fancy2.test: Likewise.
+ * tests/tap-no-spurious.test: Likewise.
+ * tests/tap-out-of-order.test: Likewise.
+
2011-08-17 Stefano Lattarini <stefano.lattarini@gmail.com>
* THANKS (Daniel Richard G.): Update e-mail address.
* HACKING ("Working in git"): Use autoconf and autom4te 2.68,
not 2.67, in the examples.
+2011-08-17 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tap: improve granularity for tests on problematic TAP messages
+ * tests/tap-message-0.test: Break up into ...
+ * tests/tap-msg0-result.test, tests/tap-msg0-directive.test,
+ tests/tap-msg0-planskip.test, tests/tap-msg0-misc.test,
+ tests/tap-msg0-bailout.test: ... these new tests, and extend
+ a little.
+ * tests/Makefile.am (tap_with_common_setup_tests): Update.
+
+2011-08-17 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tap: correctly handle string "0" in TAP messages
+ * lib/tap-driver.pl (is_null_string): New function, can be used
+ to determine whether a given string variable is empty or undefined.
+ Useful to avoid pitfalls like:
+ if ($message) { print "$message\n"; }
+ which wouldn't print anything if $message is the literal "0".
+ (handle_tap_test, handle_tap_plan, handle_tap_bailout): Use it,
+ to avoid missing messages composed only by a literal "0" in TAP
+ result descriptions and in skip, todo and bailout explanations.
+ * tests/tap-message-0.test: Enhance.
+ * tests/Makefile.am (XFAIL_TESTS): Remove it, it passes now.
+
+2011-08-17 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tap: a minor simplification in the perl TAP driver
+ * lib/tap-driver.pl: The `--disable-hard-errors' option is a
+ no-op, so just ignore it and its argument.
+
+2011-08-17 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ parallel-tests: fix help screen for test driver scripts
+ * lib/tap-driver.pl ($USAGE): The `--trs-file' option is mandatory
+ too.
+ * lib/tap-driver.sh (print_usage): Likewise.
+ * lib/test-driver (print_usage): Likewise.
+ ($scriptversion): Update.
+
+2011-08-17 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tap: add a dummy TAP driver script implemented in shell + awk
+ The user can also now decide which implementation of the TAP driver
+ to use in the testsuite by defining the `$am_tap_implementation'
+ variable to either "perl" or "shell". Future enhancements will
+ allow the testsuite to automatically run the test scripts on TAP
+ support with both the TAP driver implementations, to improve
+ coverage.
+ * tests/defs (fetch_tap_driver): Honor the `$am_tap_implementation'
+ variable to decide which implementation of the TAP driver to fetch.
+ ($am_tap_implementation): Default to "perl".
+ * tests/tap-common-setup.test: Do not fetch the TAP driver, the
+ code in tap-setup.sh does that already (and respecting runtime
+ overriding of `$am_tap_implementation').
+ * lib/tap-driver: Renamed ...
+ * lib/tap-driver.pl: ... to this, and ...
+ ($ME): ... adjusted this.
+ * doc/automake.texi: Adjust to the renaming.
+ * tests/Makefile.am (TAP_LOG_DRIVER): Likewise.
+ * tests/tap-doc2.test: Likewise.
+ * lib/tap-driver.sh: New script, still mostly dummy.
+ * lib/Makefile.am (dist_script_DATA): Update, and since we are at
+ it, rewrite it to make it easier to add new entries in the future.
+
+2011-08-17 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: refactor tests on TAP support in view of future changes
+ * tests/defs (fetch_tap_driver): New subroutine; it fetches the
+ automake-provided TAP driver from the `lib/' directory into the
+ current directory, and edits its shebang line so that it will be
+ run with the perl interpreter determined at configure time.
+ * tests/tap-setup.sh: Use it.
+ * tests/tap-common-setup.test: There's no need to AC_SUBST `PERL'
+ anymore, nor to use it in the Makefile to run the TAP driver.
+ Also, use the `fetch_tap_driver' function instead of copying the
+ `tap-driver' auxiliary script directly.
+ * tests/tap-bad-prog.tap: Likewise.
+ * tests/tap-diagnostic-custom.test: Likewise.
+ * tests/tap-doc.test: Likewise.
+ * tests/tap-merge-stdout-stderr.test: Likewise.
+ * tests/tap-more.test: Likewise.
+ * tests/tap-more2.test: Likewise.
+ * tests/tap-recheck.test: Likewise.
+ * tests/tap-summary-aux.sh: Likewise.
+ * tests/tap-basic.test: Likewise, and fix a grammaro in comments
+ since we are at it.
+
+2011-08-17 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ test defs: fix typo in comments
+ * tests/defs (get_shell_script): Fix typo in the function
+ description.
+
+2011-08-14 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ coverage: missing tap plan and non-zero exit status
+ * tests/tap-missing-plan-and-bad-exit.test: New test.
+ * tests/Makefile.am (tap_with_common_setup_tests): Update.
+
+2011-08-12 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: fix an unportable use of sed
+ * tests/tap-todo-skip-whitespace.test: Alternation with "\|" in
+ sed regular expressions is not portable to (at least) FreeBSD
+ sed; do not use it.
+
+2011-08-12 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: avoid spurious failure if rst2html program is missing
+ * tests/test-driver-custom-no-html.test: Account for the
+ possibility of a lacking `rst2html' program in our final
+ grepping of make output.
+
+2011-08-12 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tap: improve diagnosing and reporting of plan mismatches
+ Problem exposed by a failure in the test 'tap-plan-errors.test'.
+ * lib/tap-driver (NO_PLAN, EARLY_PLAN, LATE_PLAN): New constants.
+ (Throughout the file): Use more complex semantics for `$plan_seen',
+ which now also remember whether the plan that has been seen is an
+ "early" or "late" plan; in turn, this makes ...
+ ($tap_stopped): ... this variable redundant; remove it.
+ (handle_tap_plan): Adjust to avoid producing spurious or confusing
+ error messages about extra or mismatched "late" TAP plans.
+ * tests/tap-plan-corned.test: Adjust and extend.
+ * tests/tap-color.test: Likewise.
+
+2011-08-12 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tap: log all TAP stream, even after a "Bail out!"
+ * lib/tap-driver ($bailed_out): New global boolean variable,
+ telling whether a "Bail out!" directive has been seen or not.
+ (handle_tap_bailout): This function does not anymore stop the
+ reading from TAP stream; instead, it sets `$bailed_out' to a
+ true value, so that only the subsequent parsing of the input
+ TAP stream is stopped.
+ (finish): Remove, no more needed, its contents inlined into ...
+ (main): ... this function, with related adjustments in the code
+ flow.
+ (get_test_exit_message): Do not "flush" the input TAP stream
+ to fetch the exit status of test script, it is not anymore
+ required. Add a sanity check.
+ * tests/tap-bailout-and-logging.test: New test.
+ * tests/Makefile.am (tap_with_common_setup_tests): Update.
+
+2011-08-12 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ coverage: TAP diagnostics after "Bail out!" aren't reported
+ This is compatible with the behaviour of the `prove' utility.
+ * tests/tap-bailout-suppress-later-diagnostic.test: New test.
+ * tests/Makefile.am (tap_with_common_setup_tests): Update.
+
+2011-08-12 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ coverage: TAP errors after a "Bail out!" directive aren't reported
+ This is compatible with the behaviour of the `prove' utility.
+ * tests/tap-bailout-suppress-later-errors.test: New test.
+ * tests/tap-bailout-badexit.test: Renamed ...
+ * tests/tap-bailout-suppress-badexit.test: ... to this, for
+ consistency.
+ * tests/Makefile.am (tap_with_common_setup_tests): Update.
+
+2011-08-12 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: reorganize tests on TAP plans
+ * tests/tap-skip-whole.test: Rename ...
+ * tests/tap-planskip.test: ... to this
+ * tests/tap-skip-whole-badexit.test: Rename ...
+ * tests/tap-planskip-badexit.test: ... to this.
+ * tests/tap-skip-whole-bailout.test: Rename ...
+ * tests/tap-planskip-bailout.test: ... to this.
+ * tests/tap-skip-whole-bailout.test: Rename ...
+ * tests/tap-planskip-later-errors.test: ... to this.
+ * tests/tap-skip-whole-unplanned.test: Rename ...
+ * tests/tap-planskip-unplanned.test: ... to this.
+ * tests/tap-skip-whole-whitespace.test: Rename ...
+ * tests/tap-planskip-whitespace.test: ... to this.
+ * tests/tap-skip-whole-continue.test: Rename ...
+ * tests/tap-planskip-and-logging.test: ... to this, extend to try
+ more cases, and fix heading comments.
+ * tests/tap-skip-whole-lastline.test: Rename ...
+ * tests/tap-planskip-late.test: ... to this, fix heading comments,
+ and remove some checks that are just a duplication of those already
+ performed in the previous script.
+ * tests/tap-results-with-planskip.test: Rename to ...
+ * tests/tap-planskip-unplanned-corner.test: ... this, and improve
+ hading comments.
+ * tests/tap-planskip-later-errors.test: New test.
+ * tests/Makefile.am (tap_with_common_setup_tests): Update.
+
+2011-08-12 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ coverage: more TAP driver consistency w.r.t. 'prove'
+ * tests/skip-results-with-planskip.test: New test, verifies that
+ test results seen in a TAP stream that has a "plan with SKIP" are
+ flagged as errors.
+ * tests/Makefile.am (tap_with_common_setup_tests): Update.
+
+2011-08-12 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tap: improve some comments in the TAP driver
+ * lib/tap-driver (handle_tap_plan): Improve comments describing
+ possible errors in a "plan with SKIP" directive.
+
+2011-08-12 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tap: non-zero exit status after "Bail out!" should not be reported
+ This new behaviour is more consistent with that of the `prove'
+ utility. Also, experience with the use of TAP in the Automake's
+ own test suite has shown that reporting non-zero exit status from
+ a script after it has issued a "Bail out!" directive is mostly
+ redundant, more introducing noise rather than conveying useful
+ information.
+ * lib/tap-driver (finish): Inline the part of it that was getting
+ the script exit status ...
+ (main): ... in here.
+ * tests/tap-bailout-badexit.test: New test.
+ * tests/Makefile.am (tap_with_common_setup_tests): Update.
+
+2011-08-11 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tap: a "plan with SKIP" after test results is an error
+ This new behaviour is both more natural and more consistent with
+ that of the `prove' utility.
+ * lib/tap-driver (handle_tap_plan): Do not stop TAP parsing when
+ a "plan with SKIP" line (e.g., "1..0 # SKIP") is encountered, and
+ do not print a "SKIP" line if some TAP result has already been
+ seen.
+ * tests/tap-skip-whole.test: Adapt.
+ * tests/tap-skip-whole-lastline.test: Likewise.
+ * tests/tap-global-result.test: Adapt and extend.
+ * tests/tap-skip-plan-errors.test: Likewise, and fix an obsolete
+ small part of a comment.
+ * tests/tap-skip-whole-bailout.test: New test.
+ * tests/tap-skip-whole-unplanned.test: Likewise.
+ * tests/tap-skip-whole-continue.test: Likewise.
+ * tests/Makefile.am (tap_with_common_setup_tests): Update.
+
+2011-08-09 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ test harness: avoid possible fork bomb
+ This fixes a regression w.r.t. the master branch, exposed by
+ test 'parallel-tests-fork-bomb.test'.
+ * lib/am/check.am (am--redo-logs): Detect possible infinite
+ recursion due to a test log in $(TEST_LOGS) being the same
+ as $(TEST_SUITE_LOG).
+
+2011-08-09 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ coverage: enhance test against infinite recursion in test harness
+ * tests/parallel-tests-fork-bomb.test: Enhance and extend a
+ little now, for soon-to-be-implemented more explicit and safer
+ semantics.
+
2011-08-09 Stefano Lattarini <stefano.lattarini@gmail.com>
coverage: possible infinite recursion in the test harness
test defs: yet more environment cleanup
* tests/defs: Also unset the TEST_SUITE_LOG variable.
+2011-08-14 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ self tests: some minor refactoring
+ * tests/self-check-exit.tap: Disable the `errexit' shell flag;
+ this allow to remove some boilerplate idioms and to make some
+ code slightly less cumbersome.
+ * tests/self-check-me.tap: Likewise.
+
+2011-08-14 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ maintcheck: look for TAP-based tests without TAP plan
+ * Makefile.am (sc_tests_tap_plan): New maintainer check.
+ (syntax_check_rules): Add it.
+ * tests/suffix10.tap: Declare a TAP plan.
+ * tests/add-missing.tap: Likewise.
+
+2011-08-14 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: avoid confusing SKIP messages for errored tests
+ * tests/defs [$use_tap] (exit trap): Don't print the test plan
+ in case the client script exited with a non-zero status. This
+ avoids the display of redundant "SKIP" messages for tests with
+ lazy plans that error out before any test is run.
+
+2011-08-13 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: keep more debugging info around in 'add-missing.tap'
+ * tests/add-missing.tap: Use a different directory for each test,
+ so that in case of failure more information is kept around.
+ Since we are at it, remove redundant TAP diagnostic, and define
+ `$am_create_testdir' to "empty" to avoid bringing in unused
+ auxiliary files.
+
+2011-08-13 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: more granular count of test results in our TAP library
+ * tests/tap-functions.sh ($tap_bad_count_): Removed, superseded
+ by ...
+ ($tap_fail_count_, $tap_xpass_count_): ... these new variables,
+ which keep more granular counts.
+ ($tap_pass_count_: New variable.
+ * tests/defs (exit trap): Adjust and simplify accordingly.
+
+2011-08-13 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: fixlets and improvements in two long TAP-based tests
+ * tests/depmod.tap: Clean up the subdirectories used by tests that
+ passed, to avoid ending up with a too big test directory. This is
+ especially important since, on each and every system, at least some
+ tests in this script are expected to be skipped (which ones exactly
+ depends on the system), thus causing the test directory not to be
+ removed when the script terminates.
+ * tests/instspc.tap: Likewise (with the difference that tests here
+ are not expected to be skipped, but to xfail). Give more explicit
+ error message in case of setup failure. Skip the proper number of
+ tests when required, to avoid spurious errors due to mismatched
+ plan. Also, really avoid to run the unsupported test cases when
+ the need to skip them is detected.
+
2011-08-09 Stefano Lattarini <stefano.lattarini@gmail.com>
- tests: fix various blunders in 'suffix-chain.test'
- * tests/suffix-chain.test: Fix various blunders that were causing
- this test to fail spuriously: append to `configure.in', not to
- `configure.ac'; add a `;' character after the `return' statement
- in an input file; use tabs, not spaces, to indent Makefile recipes;
- and use more uncommon suffixes that don't trigger built-in recipes
- with FreeBSD make.
+ test harness: recursive make invocations must use $(AM_MAKEFLAGS)
+ * lib/am/check.am ($(TEST_SUITE_LOG)): Pass $(AM_MAKEFLAGS)
+ to $(MAKE) when issuing a recursive make call.
+ Problem detected by the `sc_tests_make_without_am_makeflags'
+ maintainer check.
+
+2011-08-09 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: fix spurious failure in 'suffix-chain.tap'
+ * tests/suffix-chain.test: Use more uncommon suffixes that
+ don't trigger built-in recipes with FreeBSD make.
+
+2011-08-09 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: improvements to our shell TAP library and APIs
+ These improvements impact only the Automake's own testsuite.
+ * tests/tap-functions.sh (plan_): Allow new special arguments
+ "unknown" a.k.a. "later" and "lazy" a.k.a. "now". Perform
+ sanity check the given arguments. Adjust incorrect comments.
+ (lazy_plan_): Removed, superseded by the new version of `plan_'.
+ (not): New function, run a command and inverts its exit status.
+ (ok_, not_ok_, skip_): These are just shorthands for common
+ `result_' usages.
+ (xfail_, xpass_, reset_test_count_): Removed, they are not used,
+ nor does a legitimate use for them seem in sight.
+ (not): New function, to run a command inverting its exit status.
+ (command_not_ok_): Removed, superseded by `command_ok_' + `not'.
+ (command_ok_): Extended to accept TAP directives.
+ * tests/defs (trap): Don't use `late_plan_' anymore, use
+ `plan_ "now"' instead.
+ * tests/ac-output-old.tap: Update to take advantage of the new
+ APIs.
+ * tests/self-check-env-sanitize.tap: Likewise.
+ * tests/suffix-chain.tap: Likewise.
+ * tests/suffix10.tap: Likewise.
+ * tests/suffix11.tap: Likewise.
+ * tests/suffix3.tap: Likewise.
+ * tests/suffix8.tap: Likewise.
+ * tests/tap-bad-prog.tap: Likewise.
+ * tests/self-check-exit.tap: Likewise, plus a small improvement
+ in the name of a variable.
+
+2011-08-09 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: convert some `suffix*' tests to TAP
+ This will allow us to explicitly declare some expected failures
+ involving Solaris make as such, instead of partly papering over
+ them with skips and partly letting them fail too loudly.
+ * tests/suffix-chain.test: Rename ...
+ * tests/suffix-chain.tap: ... to this, and convert to the use of
+ TAP. Also, fix some bugs since we are at it, and ...
+ ($required): ... require `cc'.
+ * tests/suffix3.test: Rename ...
+ * tests/suffix3.tap: ... to this, and convert to the use of TAP.
+ Also, since we are at it, throw in some formatting fixes.
+ * tests/suffix8.test: Rename ...
+ * tests/suffix8.tap: ... to this, and convert to the use of TAP.
+ Also, since we are at it, throw in some formatting fixes.
+ * tests/suffix10.test: Rename ...
+ * tests/suffix10.tap: ... to this, and convert to the use of TAP.
+ * tests/suffix11.test: Rename ...
+ * tests/suffix11.tap: ... to this, and convert to the use of TAP.
+ * tests/Makefile.am (TESTS): Update.
2011-08-09 Stefano Lattarini <stefano.lattarini@gmail.com>
2011-08-09 Stefano Lattarini <stefano.lattarini@gmail.com>
- testsuite: avoid spurious failure of self checks on Solaris
- This is just a quick workaround to prevent having too much noise
- in the testsuite; more proper fixes have been committed into the
- 'testsuite-work' branch.
- * tests/self-check-dir.test: Relax a test to avoid unimportant
- failure on Solaris 10.
- * tests/self-check-exit.test: Likewise.
+ self tests: avoid spurious output on console, and related fixlets
+ * tests/self-check-cleanup.test: Set `$stderr_fileno_' to `2'
+ in the self tests, to avoid spurious diagnostic on console.
+ * tests/self-check-exit.tap: Only unset `am_explicit_skips' and
+ `$stderr_fileno_' in the self tests, not at the global level, it
+ is more correct.
+ ($init): New variable, to reduce code duplication.
+ Some refactoring and adjustments to the name of the tests.
+
+2011-08-09 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ test defs: few more sanity checks
+ * tests/defs: Catch some more (very unlikely) set-up or internal
+ errors, and diagnose them explicitly. This is intended mostly
+ to be a theoretic-only improvement.
+
+2011-08-09 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ test defs: really make them "set -e" clean
+ Motivated by a spurious failure of test `instsh3-w.shtst' on
+ Solaris 10.
+ * tests/defs: Make really "set -e" clean, to avoid spurious
+ failures in case this file is sourced by a test script that has
+ already set the `errexit' flag. To be sure we don't regress,
+ move the `set -e' call much earlier, and make the comment telling
+ about the expected "set -e" cleanliness more loud and outstanding.
+
+2011-08-09 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ self tests: register an expected failures with Solaris /bin/sh
+ Solaris 10 /bin/sh erroneously exit with success right away when
+ the following three conditions are met at the same time:
+ 1. the `errexit' flag is active,
+ 2. an exit trap is installed, and
+ 3. a non-existing command is issued.
+ * tests/self-exit.tap: When that bug is detected, issue an XFAIL
+ rather than a FAIL. Since we are at it, improve by avoiding
+ creation of stray temporary files when testing for non-executable
+ commands.
+
+2011-08-09 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ gitignore: update and improve for the testsuite files
+ * tests/.gitignore: Don't ignore temporary partial log files,
+ autogenerated tests and makefile fragments that are not used
+ anymore. Anchor all the remaining patterns.
+
+2011-08-09 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ self tests: some new uses of TAP, few extensions
+ * tests/self-check-me.test: Renamed ...
+ * tests/self-check-me.tap: .. to this, and converted to use TAP.
+ * tests/self-check-exit.test: Renamed ...
+ * tests/self-check-exit.tap: .. to this, and converted to use TAP.
+ Set `$stderr_fileno_' to `2', to avoid spurious messages on the
+ console. Add a new test, and relax another one.
+ * tests/self-check-env-sanitize.test: Renamed ...
+ * tests/self-check-env-sanitize.tap: .. to this, and converted to
+ the use TAP.
+ * tests/Makefile.am (TESTS): Update.
+
+2011-08-09 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ test defs: test directory creation more configurable
+ After this change, the individual tests will be able to tell
+ `./defs' whether or not to create a temporary test directory
+ for them, and if yes, whether or not to pre-populate it.
+ * tests/defs: Only create the temporary test directory if the
+ variable `$am_create_testdir' is not set to "no". In this
+ case, only pre-populate it if that variable is not set to
+ `$empty'.
+ * tests/defs-static.in: Check that `$am_create_testdir' is not
+ exported in the environment.
+ * tests/self-check-env-sanitize.test: Update.
+ * tests/self-check-dir.test: Extend, and make deeper and more
+ robust.
+ * tests/README (Section "Writing test cases" subsection "Do"):
+ Update.
+ * tests/autodist.test: Set `$am_create_testdir' instead of
+ removing by hand files created by ./defs.
+ * tests/tap-doc2.test: Likewise.
+
+2011-08-08 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: more uses of the `seq_' function
+ * tests/parallel-tests-exit-statuses.test: Use the `seq_' function
+ from `tests/defs', instead of duplicating its semantics.
+ * tests/testsuite-summary-count.test: Likewise.
+
+2011-08-08 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: drop redundant environment sanitization
+ * tests/Makefile.am (TESTS_ENVIRONMENT): Don't remove `planned'
+ from the environment.
+ * tests/defs-static.in: Don't complain if `planned' is set in
+ the environment.
+ * tests/self-check-env-sanitize.test: Update.
+
+2011-08-08 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ test defs: fix bug in plain version of `skip_all_'
+ * tests/plain-functions.sh (skip_all_): Call `skip_', not `skip'.
+
+2011-08-08 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ test defs: small cleanups and tweakings
+ * tests/defs: Where possible, prefer `framework_failure_' or
+ `fatal_' to `Exit 99'. Try to catch more possible hard errors
+ and setup failures. Do not hide expected error messages. Avoid
+ a couple of forks when copying files from `$(top_srcdir)/lib'.
+ ($signal): Remove, no more needed.
+
+2011-08-08 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: improve interface for giving early TAP plan
+ Since commit `v1.11-976-g4740f56', an early TAP plan can also be
+ specified after input lines have already been seen, as long as
+ no one of them is a TAP result line. We can now take advantage
+ of this in our own testsuite, allowing the TAP-based scripts to
+ specify their earlier plan after having included `./defs'. In
+ particular, this offers the benefit of allowing those scripts to
+ issue a "skip all" plan (e.g., "1..0 # SKIP reason ...") after
+ the inclusion of `./defs', which wasn't possible with the older
+ interface (introduced in commit `v1.11-1114-g0286308').
+ * tests/defs: Don't try to call `plan_' if `$planned' is defined,
+ the calling scripts can do that directly now.
+ * tests/ac-output-old.tap: Use `plan_' directly instead of
+ setting `$planned' before including `./defs'.
+ * tests/automake-cmdline.tap: Likewise.
+ * tests/depmod.tap: Likewise.
+ * tests/instspc.tap: Likewise.
+ * tests/tap-bad-prog.tap: Likewise.
+ * tests/add-missing.tap: Don't set `$planned'.
+
+2011-08-08 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ test defs: fix bug in TAP version of `skip_all_'
+ * tests/tap-functions (skip_all_): Set `$have_tap_plan_' to "yes"
+ when called, so that the exit trap won't write another TAP plan.
+
+2011-08-08 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ test defs: fix skip directives to work also with TAP-based tests
+ * test/defs: Use `skip_all_', not `skip_', to issue a skip of the
+ whole calling test script.
2011-08-08 Stefano Lattarini <stefano.lattarini@gmail.com>
2011-08-08 Stefano Lattarini <stefano.lattarini@gmail.com>
+ test harness: more minor VPATH-rewrite issues
+ * tests/am/check.am (am__set_TESTS_bases, $(TEST_SUITE_LOGS),
+ am--redo-logs, recheck, recheck-html): Cosmetic fixlets to
+ minimize the risk of unwanted VPATH rewrites.
+ (check-TESTS): Likewise, and normalize trailing whitespace
+ since we are at it.
+ Bugs exposed by test cases `check6-p.test' and `check7-p.test'.
+
+2011-08-08 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ test harness: "make -n check" works correctly with Solaris make
+ To avoid some recipes being executed with "make -n", we used the
+ trick of issuing recursive make calls not directly with $(MAKE),
+ but with a simple indirection instead:
+ ...
+ am__stealth_MAKE = $(MAKE)
+ ...
+ target:
+ ... $(am__stealth_MAKE) ...
+ ...
+ This hack worked with GNU make, but, unfortunately, not with
+ Solaris make. The present change removes the hack, and uses more
+ proper idioms to avoid unwanted recipe executions with "make -n".
+ Bug exposed by test case `parallel-tests-dry-run.test'.
+ * lib/am/check.am (am__stealth_MAKE): Remove, no more used.
+ (.log.trs): Break recipe in multiple shell invocations.
+ ($(TEST_SUITE_LOG)): Likewise, and move some corner-case parts of
+ the recipe to ...
+ (am--redo-logs): ... this new internal helper .PHONY target.
+
+2011-08-08 Stefano Lattarini <stefano.lattarini@gmail.com>
+
tests: fix spurious failure with Solaris make
* tests/distcheck-configure-flags-am.test: Avoid using `+=' too
liberally with AM_DISTCHECK_CONFIGURE_FLAGS, since the line breaks
so introduced, in conjunction with single quotes, might confuse
Solaris make.
+2011-08-08 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ test harness: fixlet in workaround for VPATH rewrite issue
+ Fixes a minor in previous commit `v1.11-983-gfda3de5'.
+ * lib/am/check.am (am__set_TESTS_bases): Add forgotten end-of-line
+ anchor to sed expression. In comments, add reference to ...
+ * tests/test-trs-basic.test: ... this test, which has been adapted
+ and tightened (and tweaked).
+
+2011-08-08 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ test harness: work around a VPATH rewrite issue
+ * lib/am/check.am (am__TEST_BASES): Removed, it's role taken
+ over by ...
+ (am__set_TESTS_bases): ... these new variable.
+ ($(TEST_SUITE_LOG): Use it, to avoid VPATH rewrite issues.
+ * automake.in (handle_tests): Update the code for the cleanup
+ of the `.trs' file to use `$(TEST_LOGS)' instead of
+ `$(am__TEST_BASES)'.
+ * tests/test-harness-vpath-rewrite.test: New test, better
+ exposing the problem fixed by this change
+ * tests/Makefile.am (TESTS): Add it.
+ * tests/test-trs-basic.test: Update and extend.
+
+2011-08-07 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ parallel-tests: work around Solaris XPG4 make segfault
+ Issue exposed by test `posixsubst-tests-p.test', and similar to
+ the problem solved by commit `v1.11-159-ge7aa360'.
+ * lib/am/check.am [%?PARALLEL_TESTS%] (check-TESTS): Trim trailing
+ whitespace from $list, to avoid triggering a nasty bug (potential
+ segfault) on Solaris XPG4 make and Heirloom make.
+
+2011-08-07 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: fix weird spurious failure with Solaris /bin/sh
+ Solaris /bin/sh, when killed with a SIGTERM or SIGINT signal, can
+ apparently end up exiting with exit status 208, instead of leaving
+ the correct wide exit status to the parent. See:
+ <http://dbaspot.com/shell/396118-bourne-shell-exit-code-term.html>
+ Work around this incompatibility.
+ * tap-signal.test: Look for the above Solaris bug.
+ (signal_caught): Adapt to handle it.
+
+2011-08-07 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: fix another spurious failure on Solaris make
+ * tests/parallel-tests-log-override-recheck.test: Filter make
+ output before grepping it, for make implementations that, like
+ Solaris' one, print the whole of the failed recipe on failure.
+
+2011-08-07 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: fix two spurious failures on Solaris make
+ * tests/tests-environment-and-log-compiler.test: Do not fail if
+ "make -n check" fails, as that is issued only for debugging, its
+ semantics being tested in other test scripts.
+ * tests/test-driver-strip-vpath.test: Likewise.
+
+2011-08-07 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tap driver: refactor and remove dead code
+ * lib/tap-driver (get_tap_line): Removed, ...
+ (all callers): ... updated to use $parser->next directly instead.
+ (peek_tap_line, unget_tap_line, @tap_lines): Remove, they're not
+ used anymore.
+ (TAP_PEEKING): Block deleted, all its subroutines and variables
+ having been removed.
+
+2011-08-07 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: fix spurious failure in a test on TAP support
+ * tests/tap-passthrough-exit.test: When the `--ignore-exit' flag
+ of the TAP driver is used, don't look for a message reporting the
+ non-zero exit statuses of tests in the log files; such message is
+ not expected to be there anymore. Related simplifications.
+
+2011-08-07 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tap: plan location is more liberal w.r.t. non-TAP lines
+ With this change, only lines that are TAP results will matter
+ w.r.t. the position of the TAP plan in the input; for example,
+ this input:
+ this is a non-TAP line
+ # and this a TAP diagnostic line
+ 1..1
+ ok 1
+ was considered to be an error, diagnosed with a message "test
+ plan in middle of output"; as effect of the current change, such
+ input is now valid. This is more consistent with the behaviour
+ of the `prove' utility.
+ * lib/tap-driver ($lineno): Removed, no more needed.
+ ($tap_stopped): New global variable.
+ (stringify_test_result): Return "ERROR" if a TAP result is found
+ when `$tap_stopped' is set to true.
+ (handle_tap_test): Diagnose TAP results that comes after a late
+ plan. Add a couple of blank lines, for clarity.
+ (handle_tap_plan): Set `$tap_stopped' to true after a late plan
+ is encountered. Do not complain anymore for extra non-TAP lines
+ preceding or following the plan. Adjust comments.
+ (main): Don't increment $lineno anymore.
+ * tests/tap-plan.test: Extend a bit, and remove stale comment.
+ * tests/tap-color.test: Adjust.
+ * tests/tap-passthrough.test: Likewise.
+ * tests/tap-plan-corner.test: Adjust and extend.
+ * tests/tap-plan-errors.test: Likewise.
+ * tests/tap-plan-middle.test: New test.
+ * tests/tap-plan-corner2.test: Delete, it's obsolete now.
+ * tests/Makefile.am (XFAIL_TESTS): Remove it.
+ (tap_with_common_setup_tests): Likewise, and add
+ `tap-plan-corner.test'.
+
+2011-08-07 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: remove now-passing test from XFAIL_TESTS
+ * tests/Makefile.am (XFAIL_TESTS): Remove `tap-signal.test',
+ which is passing since previous commit `v1.11-974-gc7fa872'.
+
+2011-08-07 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ test defs: get name of TAP-using tests correctly
+ This change fixes the names of the temporary directories used
+ by the TAP-based tests (e.g., test `foo.tap' now uses `foo.dir'
+ rather than `foo.tap.dir'); more importantly, this make the
+ variable `$destdir' defined in `tests/defs' usable also in the
+ TAP tests.
+ ($me): Also strip the any `.tap' suffix from the name of the
+ test.
+
+2011-08-07 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: TAP tests properly decide when to remove tempdirs
+ Before this change, the TAP tests in the Automake testsuite were
+ removing the temporary test directory even when they failed or
+ were skipped, thus making debugging more difficult.
+ * tests/tap-functions.sh (incr_tap_count): Removed, superseded
+ by ...
+ (incr_): ... this function, which can increment the value of any
+ variable passed to it.
+ (result_): Updated to use `incr_' instead of the now-removed
+ `incr_tap_count_'. Keep count of failures, xfailures, xpasses,
+ and skips, using ...
+ ($tap_skip_count_, $tap_bad_count, _$tap_xfail_count_): ...
+ these new variables.
+ * tests/defs (trap): Try to use their values to decide whether
+ the temporary directory being used by the test script should be
+ removed or not. Other code reorganizations. And move the code
+ for the removal of the temporary directory out to ...
+ (rm_rf_): ... this new subroutine.
+ (Main code): Use that instead of duplicating the code.
+
+2011-08-07 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: improve and refactor our custom TAP shell library
+ This change is mostly done in preparation for the next one.
+ * tests/tap-functions.sh (result_with_directive_): Merge into ...
+ (result_): ... this function, which has been greatly extended and
+ improved in various ways.
+ (ok_, not_ok_, skip_, xfail_, xpass_): Adjust.
+ Adjust comments. Remove an extra empty line.
+ * tests/instspc.tap: Use the new improved interface of `result_'.
+ * tests/tap-bad-prog.tap: Likewise.
+
+2011-08-07 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: update "user interface" description in tests/README
+ * tests/README (Subsection "Interpretation"): Mention the "ERROR"
+ outcome.
+ (Subsection "Getting details from failures"): Don't tell that the
+ test scripts can be run directly, this is not 100% true; anymore;
+ instead ...
+ (Subsection "About the tests"): ... use this new subsection to
+ explain how to run the tests (either simple or TAP-based) by hand.
+ (Subsection "Supported shells"): Extend to refer to TAP tests too.
+
+2011-08-07 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: TAP tests can have early plan now
+ * tests/defs: When using TAP, allow the user to specify the number
+ of tests through the `$planned' variable.
+ (trap): Do not print the auto-generated plan if an explicit one
+ has already been printed.
+ * tests/defs-static.in: Complain and bail out if the `$planned'
+ variable is exported in the environment. Reformat the code to
+ facilitate future additions.
+ * tests/Makefile.am (TESTS_ENVIRONMENT): Neutralize `$planned'.
+ * tests/self-check-env-sanitize.test: Update.
+ * tests/ac-output-old.tap: Give an explicit TAP plan, by defining
+ `$planned' just before including `./defs'.
+ * tests/depmode.tap: Likewise.
+ * tests/automake-cmdline.tap: Likewise.
+ * tests/tap-bad-prog.tap: Likewise.
+ * tests/instspc.tap: Likewise, and related adjustments.
+ * tests/add-missing.tap: Make it explicit that we have no a-priori
+ plan, and tell why.
+
+2011-08-07 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: run autogenerated tests with $(LOG_COMPILER) too
+ This change enhances consistency in the testsuite.
+ * tests/Makefile.am: Some cosmetic reordering, to keep the
+ definition of FOO_LOG_COMPILER near to the place where `.foo'
+ is added to $(TEST_EXTENSIONS).
+ (PTEST_LOG_COMPILER): Run `$(srcdir)/parallel-tests.sh' with
+ `$(LOG_COMPILER)', not with `$(SHELL)'.
+ (SHTST_LOG_COMPILER): Likewise.
+
+2011-08-07 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: run TAP and "plain" tests with the same $(LOG_COMPILER)
+ With this change, it will be easier to override, at make time, the
+ shell used to run the tests.
+ * tests/Makefile.am (LOG_COMPILER): Define to `$(SHELL)'.
+ (TEST_LOG_COMPILER): Re-define to `$(LOG_COMPILER)'.
+ (TAP_LOG_COMPILER): Likewise.
+
+2011-08-06 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: run TAP-based tests explicitly with $(SHELL)
+ * tests/Makefile.am (TAP_LOG_COMPILER): Define to `$(SHELL)'.
+
+2011-08-06 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tap driver: handle signals received by the tests being run
+ * lib/tap-driver (get_test_exit_message): Also deal with signals,
+ by using the `wait' method of the TAP::Parser object instead of
+ the `exit' method. This required the use of the standard perl
+ module `POSIX'.
+ * doc/automake.texi (Use TAP with the Automake test harness):
+ Document that `--ignore-exit' has effect also on terminating
+ signals. Add a "synchronizing" comment that references the tests
+ 'tap-exit.test' and 'tap-signal.test'.
+ * tests/tap-signal.test: Extend and adjust.
+
+2011-08-06 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ test driver: a preparatory refactoring (2)
+ * lib/tap-driver (finish): Move code to fetch the message about
+ the exit status of the test being run to ...
+ (get_test_exit_message): ... this new subroutine.
+
+2011-08-06 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tap driver: a preparatory refactoring (1)
+ * lib/tap-driver (finish): Do not flush the TAP stream nor
+ fetch the exit status of the TAP producer if running with
+ the `--ignore-exit' option. Obviously, don't try to write
+ such exit status in the test log anymore.
+
+2011-08-06 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ cosmetics: typofix in the name of a function in the TAP driver
+ * lib/tap-driver (testuite_error): Renamed ...
+ (testsuite_error): ... to this, and minor related reformatting.
+
+2011-08-06 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tap: scripts with a SKIP plan but with exit status != 0 must error
+ This change has been motivated by Automake's own testsuite. Some
+ TAP tests there were erroring out (due to unexpected, unhandled
+ failures) before having encountered TAP result, so that the
+ simple-minded shell library implementing TAP generation ended up
+ printing a "1..0" trailing test plan; this caused the script to be
+ reported as a SKIP rather than an ERROR -- a nasty false negative.
+ * lib/tap-driver: Add prototypes for each subroutine, to free up
+ the order in which they can be defined and called.
+ (main): Move the code checking for a bad exit status of the TAP
+ producer ...
+ (finish): ... here, and flush the TAP stream to ensure that the
+ parser always obtains the producer's exit status.
+ * tests/tap-skip-whole-badexit.test: New test.
+ * tests/Makefile.am (tap_with_common_setup_tests): Add it.
+
+2011-08-06 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tap: fix whitespace munging of diagnostic messages
+ * lib/tap-driver (extract_tap_comment): Pass the `g' flag to the
+ substitution operator, to strip also trailing whitespaces. Fixes
+ a failure in test `tap-whitespace-normalization.test'.
+
+2011-08-06 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: refactor and cleanup 'instspc.tap'
+ * tests/instspc.tap (is_in_list): New helper subroutine.
+ (expected_to_fail): Re-implement using it.
+ (define_problematic_string): Likewise. Also, rename the special
+ arguments `build-fail' and 'install-fail' to respectively
+ `builddir-fail' and `destdir-fail', and other related changes.
+ (Test data definition): Adapt.
+ ($instspc_xfail_builds_list): Renamed ...
+ ($builddir_xfails): ... to this.
+ ($instspc_xfail_installs_list): Renamed ...
+ ($destdir_xfails): ... to this.
+ ($instspc_names_list): Renamed ...
+ ($test_names_list): ... to this.
+ ($instspc_test_string): Renamed ...
+ ($test_string): ... to this.
+ Add some explicative and "FIXME" comments.
+
+2011-08-06 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: use TAP for `depmod*' tests, related simplifications
+ * tests/depmod-tests.sh: Delete this complex and multifarious
+ script, moving all it checks it used to perform into ...
+ * tests/depmod.tap: ... this TAP-based test script. And extend
+ it quite a bit, since the new form of the test was papering over
+ some pre-existing failures (this was due to the fact that the
+ values used for the VPATH were shorter in the new test script).
+ * tests/depmod-data.test: Deleted.
+ * tests/Makefile.am ($(srcdir)/depmod-tests.am): Don't include
+ nor generate it anymore.
+ (EXTRA_DIST): Don't distribute depmod-tests.sh anymore.
+ (TEST_EXTENSIONS): Remove '.depmod'.
+ (DEPMOD_LOG_COMPILER): Deleted.
+ (depmod-data.log, $(depmod_tests:.depmod=.log)): Delete
+ dependencies for these files, which are not used anymore.
+ ($(depmod_tests)): Delete this dummy dependency declaration.
+ (TESTS): Remove $(depmod_tests).
+ (handwritten_tests): Remove depmod-data.test, add depmod.tap.
+ * bootstrap: Don't generate tests/depmod-tests.am anymore.
+
+2011-08-06 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: use TAP for `instspc*' tests, related simplifications
+ * tests/instspc-tests.sh: Delete this complex and multifarious
+ script, moving all it checks it used to perform into ...
+ * tests/instspc.tap: ... this TAP-based test script.
+ * tests/instspc-data.test: Deleted.
+ * tests/Makefile.am ($(srcdir)/instspc-tests.am): Don't include
+ nor generate it anymore.
+ (EXTRA_DIST): Don't distribute instspc-tests.sh anymore.
+ (TEST_EXTENSIONS): Remove '.instspc'.
+ (INSTSPC_LOG_COMPILER): Deleted.
+ (XFAIL_TESTS): Remove $(instspc_xfail_tests).
+ (instspc-data.log, $(instspc_tests:.instspc=.log)): Delete
+ dependencies for these files, which are not used anymore.
+ ($(instspc_tests)): Delete this dummy dependency declaration.
+ (TESTS): Remove $(instspc_tests).
+ (handwritten_tests): Remove instspc-data.test, add instspc.tap.
+ * bootstrap: Don't generate tests/instspc-tests.am anymore.
+
+2011-08-06 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: do fd redirections with $(AM_TESTS_FD_REDIRECT)
+ * tests/Makefile.am: The redirections of file descriptors needed
+ by our test scripts is now done using the $(AM_TESTS_FD_REDIRECT)
+ variable (the new blessed way since commit `v1.11-906-gb9e9d54'),
+ not using an hack involving $(AM_TESTS_ENVIRONMENT). This has
+ the further benefit of allowing the use of $(TESTS_ENVIRONMENT)
+ again on part of the users.
+ * tests/Makefile.am (AM_TESTS_ENVIRONMENT): Remove redirection of
+ file descriptors.
+ (AM_TESTS_FD_REDIRECT): Redirect file descriptor 9 to original
+ stderr.
+ Comments adjusted.
+ * tests/plain-functions.sh ($stderr_fileno_): Update comment.
+
+2011-08-05 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: use custom TAP diagnostic in our own tests
+ This change allow us to easily and transparently avoid spurious
+ TAP diagnostic in our own testsuite, with the help of the new
+ `--diagnostic-string' option of our TAP test driver.
+ * tests/Makefile.am (TAP_LOG_DRIVER_FLAGS): Also pass option
+ `--diagnostic-string' with value `#%#' (necessarily obtained
+ through some shell indirections).
+ * tests/tap-functions.sh (diag_): Assume that the string denoting
+ TAP diagnostic is stored in the `$diag_string_' variable. Update
+ comments.
+ ($diag_string_): Initialize to "#".
+ * tests/defs ($diag_string_): Re-define to "#%#".
+ (show_): Remove, it's obsolete now.
+ * add-missing.tap: Don't use `show_' anymore to avoid spurious
+ TAP diagnostic; `cat' should suffice now.
+
+2011-08-05 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tap: fix typo in TAP driver version message
+ * lib/tap-driver (anonymous subroutine printing the script version
+ message): Add missing trailing newline to the version message.
+
+2011-08-05 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tap docs: we don't support pragma or version directives (yet)
+ * doc/automake.texi (Incompatibilities with other TAP parsers
+ and drivers): Document that limitation. Normalize an unrelated
+ "TODO" item.
+
+2011-08-05 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tap: new option to change the string designating TAP diagnostic
+ Here we introduce a new option `--diagnostic-string' in our TAP
+ test driver, that allows the user to specify which string should
+ denote the beginning of a TAP diagnostic line. This change is
+ not gratuitous, nor result if over-engineering: it is motivated
+ by real issues that have emerged during the use of TAP in the
+ Automake's own testsuite (see the commit `v1.11-1082-g9b967c2'
+ "testsuite: yet more use of TAP, and related extensions").
+ * doc/automake.texi (Use TAP with Automake test harness): Document
+ the new option.
+ (Incompatibilities with other TAP parsers and drivers): Report it
+ as a potential source of incompatibility.
+ * lib/tap-driver ($diag_string): New global variable, defaulting
+ to "#", and whose value can be changed ...
+ (Getopt::Long::GetOptions): ... by the newly recognized option
+ `--diagnostic-string'.
+ (handle_tap_comment): Subroutine removed, some of its simple logic
+ inlined ...
+ (main): ... in here, where now ...
+ (extract_tap_comment): ... this new subroutine is used.
+ ($USAGE): Adjust.
+ * tests/tap-diagnostic.test: Make one check slightly stricter.
+ * tests/tap-diag-custom.test: New test.
+ * tests/Makefile.am (tap_other_tests): Add it.
+
+2011-08-05 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ check: add small "synchronization" comment to `test-driver'
+ * lib/test-driver: Add comment to the code initializing ANSI color
+ escapes, telling to keep it in sync with the similar initialization
+ in `lib/am/check.am'.
+
+2011-08-05 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ maintcheck: fix spurious failure w.r.t. use of bare "exit"
+ * tests/check12.test: Cosmetic changes to avoid triggering a
+ spurious failure of the `sc_tests_Exit_not_exit' maintainer
+ check.
+
+2011-08-05 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ check: update comments to reflect recent heavy changes
+ * lib/am/check.am [%?PARALLEL_TESTS%]: Update the comments to
+ reflect the recent changes. Remove quite many comments that
+ were merely duplicating excerpts from the Automake manual.
+
+2011-08-05 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ check: small cleanups and refactorings in test harness and drivers
+ * lib/tap-driver (yn): New subroutine, converts a boolean value to
+ a "yes" or "no" string.
+ (write_test_results): Use it, and related reformatting.
+ * lib/test-driver (trap): Also remove the `.trs' file on signals,
+ for extra safety.
+ (fatal): Remove this function, it's never used.
+ ($logfile, $trsfile): Renamed ...
+ ($log_file, $trs_file): ... to these, for clarity, and in order to
+ be more consistent with the `tap-driver' script.
+ Improve a couple of comments.
+ * automake.in (handle_tests): Don't define anymore the now-obsolete
+ make macro `$(TEST_LOGS_TMP)', nor add it to the list of files to
+ be removed upon "make mostlyclean".
+ * lib/am/check.am ($(TEST_SUITE_LOG)): New shell function `f_ok',
+ tells whether a path refers to an existing, regular, readable file.
+ Use it throughout.
+ (recheck): Be safer w.r.t. make implementation that run recipes
+ with `errexit' shell flag active.
+
+2011-08-05 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: use AM_TESTS_FD_REDIRECT where appropriate
+ * parallel-tests-interrupt.test: Use AM_TESTS_FD_REDIRECT, not
+ AM_TESTS_ENVIRONMENT, to define file descriptors' redirection
+ for "make check". Fix botched comment.
+
+2011-08-05 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ docs: some fixlets in documentation on testsuites support
+ * doc/automake.texi (Script-based Testsuites): Rename node ...
+ (Scripts-based Testsuites): ... to this. Break overly long lines
+ in an example (were causing problems with PDF output).
+ (Parallel Test Harness): Don't intend paragraphs that come after
+ examples or bulleted list.
+ (Overview of Custom Test Drivers Support): Fix typo s/the the/the/.
+ (Declaring Custom Test Drivers): When giving an aside, prefer comma
+ to parentheses, as it disrupts the reading flow less.
+ (API for Custom Test Drivers): Remove a sentence that was basically
+ duplicated from the previous subsection.
+ (Command-line arguments for test drivers): Don't intend paragraph
+ coming after an itemized list. Clarify wording and fix grammaros.
+ Add a couple of cross-references to earlier explanations of `.log'
+ and `.trs' files.
+ (Links and external resources): Node renamed ...
+ (Links and external resources on TAP): ... to this, for clarity.
+ Slightly extend and clarify introductory sentence.
+
+2011-08-04 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ test harness: be aware of more metadata, simplify test drivers
+ * lib/am/check.am ($(TEST_SUITE_LOG)): When producing the
+ global test log, take into account the new metadata field
+ `:global-test-result:, to write nicely formatted RST section
+ titles, instead of leaving this chore to the individual test
+ drivers.
+ (am_rst_section): Re-introduce this variable, as removed in
+ commit `v1.11-870-ga27c9c4'.
+ * lib/test-driver, lib/tap-driver: Write the global test result
+ as metadata in the `.trs' file, not as (part of) a formatted RST
+ subsection title in the `.log' file. Related simplifications;
+ in particular, get rid of temporary files usage.
+ * doc/automake.texi (Log files generation and test results
+ recording): Document the new metadata.
+ * tests/check12.test: Update.
+ * tests/parallel-tests-harderror.test: Likewise.
+ * tests/parallel-tests-interrupt.test: Likewise.
+ * tests/parallel-tests-log-override-2.test: Likewise.
+ * tests/parallel-tests-log-override-recheck.test: Likewise.
+ * tests/parallel-tests-unreadable.test: Likewise.
+ * tests/tap-global-result.test: Likewise.
+ * tests/test-metadata-results.test: Likewise.
+ * tests/test-log.test: Likewise, and another minor unrelated fix.
+ * tests/test-metadata-global-result.test: New test.
+ * tests/Makefile.am (TESTS): Update.
+
+2011-08-03 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tap: check that also a trailing TAP plan can hold a skip directive
+ * tests/tap-skip-whole-lastline.test: New test.
+ * tests/Makefile.am (tap_with_common_setup_tests): Add it.
+
+2011-08-04 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: more correct names for a couple of tests
+ * tests/parallel-tests-ext-driver.test: Rename ...
+ * tests/parallel-tests-log-compiler-1.test: ... to this, and
+ adjust heading comments.
+ * tests/parallel-tests-ext-driver-prog.test: Rename ...
+ * tests/parallel-tests-log-compiler-2.test: ... to this, and
+ adjust heading comments.
+ * tests/Makefile.am (TESTS): Update.
+
+2011-08-04 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: fix another spurious failure
+ * tests/parallel-tests-ext-driver.test: Name out custom/dummy
+ log compilers as `foo-compiler' rather `foo-driver'; not only
+ this is less confusing, it also avoids errors due to the dummy
+ log compiler `test-driver' overriding the `test-driver' helper
+ script installed by Automake.
+ * tests/parallel-tests-ext-driver-prog.test: Adjust heading
+ comments.
+
+2011-08-04 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: fix a spurious failure
+ * tests/parallel-tests-ext-driver.test: Exporting of environment
+ variables directly from $(LOG_COMPILER) variables is not allowed
+ anymore. Adjust to this.
+
+2011-08-04 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: yet more use of TAP, and related extensions
+ * tests/defs (show_): New function, display the contents of one or
+ more files on stdout, with bells & whistles (both for cosmetic and
+ practical reasons, the latter aimed at avoiding spurious TAP
+ diagnostic).
+ (AUTOMAKE_run): Extend and adapt to make it compatible with TAP
+ based tests. Since we are at it, make its implementation more
+ namespace-safe, and improve its command-line interface.
+ (AUTOMAKE_fails): Adapt to the new command-line interface of
+ `AUTOMAKE_run'.
+ * tests/add-missing.test: Renamed ...
+ * tests/add-missing.tap: ... to this, and converted to the use
+ of the TAP protocol, and of the new `show_' function.
+ * tests/automake.test: Renamed ...
+ * tests/automake-cmdline.tap: ... to this, and converted to the
+ use of the TAP protocol.
+ * tests/getopt.test: Adapt to the new command-line interface
+ of `AUTOMAKE_run'.
+ * tests/cond44.test: Likewise.
+ * tests/cond45.test: Likewise.
+ * tests/configure.test: Likewise.
+ * tests/license2.test: Likewise.
+ * tests/parallel-am.test: Likewise.
+ * tests/parallel-am3.test: Likewise.
+ * tests/tags2.test: Likewise.
+ * tests/werror3.test: Likewise.
+ * tests/werror4.test: Likewise.
+ * tests/amopts-variable-expansion.test: Likewise.
+ * tests/warnings-win-over-strictness.test: Likewise.
+ * tests/Makefile.am (TESTS): Update.
+ (TAP_LOG_DRIVER_FLAGS): Add `--merge', so that TAP diagnostic is
+ reported in the testsuite progress output.
+
+2011-08-03 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ maintcheck: fix more maintainer-check failures
+ * tests/test-driver-custom-no-html.test (no-rst): Use `EOF',
+ not `EoL', as the here-document delimiter.
+ * tests/test-trs-basic.test: Use `cat + here-doc' rather
+ than `echo' when creating the dummy test scripts, to please
+ maintainer-check.
+ * tests/test-trs-recover.test: Use creative quoting where
+ needed, to please maintainer-check.
+ * tests/parallel-tests-no-color-in-log.test: Likewise.
+ * tests/parallel-tests-dry-run.test: Likewise.
+
2011-08-03 Stefano Lattarini <stefano.lattarini@gmail.com>
maintcheck: fix maintainer-check failures, both real and spurious
* tests/Makefile.am (TESTS): Update.
Suggestion by Ralf Wildenhues.
-2011-07-04 Stefano Lattarini <stefano.lattarini@gmail.com>
+2011-07-04 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ remake: fix outdated comment in configure.am
+ * lib/am/configure.am: Fix comment falsified by changes in
+ commit `v1.11-366-gbee9871'.
+ Suggestion by Ralf Wildenhues.
+
+2011-07-01 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ docs, tests: synchronize examples from docs to tests
+ * tests/README (Writing test cases): Give suggestions on how to
+ keep test cases and examples in the documentation synchronized.
+ * doc/automake.texi: Improve or fix existing testcase-referencing
+ comments, and add many new ones.
+ * HACKING (Administrivia): Suggest to test complex examples and
+ idioms from the manual.
+ * tests/specflg8.test: Improve synchronization with the example
+ in the manual.
+ * tests/output11.test:Likewise.
+ * tests/txinfo21.test:Likewise.
+ * tests/interp.test: Likewise.
+ * tests/amhello-cflags.test: New test.
+ * tests/amhello-cross-compile.test: Likewise.
+ * tests/amhello-binpkg.test: Likewise.
+ * tests/tests-environment-backcompat.test: Likewise.
+ * tests/parallel-tests-log-compiler-example.test: Likewise.
+ * tests/Makefile.am (TESTS): Update.
+
+2011-06-30 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ coverage: new test on parallel-tests TESTS runtime overriding
+ * tests/parallel-tests-cmdline-override.test: New test, check that
+ we can use indirections when overriding TESTS and TEST_LOGS from
+ the command line.
+ * tests/Makefile.am (TESTS): Update.
+
+2011-08-03 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ gitignore: ignore `*.trs' files in lib/Automake/tests
+ * lib/Automake/tests/.gitignore: Add `*.trs' pattern.
+
+2011-08-03 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: list another "forgotten" test script in Makefile.am
+ * tests/Makefile.am (TESTS): Also list test script
+ `test-driver-trs-suffix-registered.test'; it had been introduced
+ in commit `v1.11-910-g0c81b43', but by mistake it wasn't added
+ to the list of tests in the Makefile back then.
+
+2011-08-03 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: list "forgotten" test script in Makefile.am
+ * tests/Makefile.am (TESTS): Also list the test script
+ `parallel-tests-no-color-in-log.test'; it had been introduced in
+ commit `v1.11-899-geaac33f', but by mistake it wasn't added to
+ the list of tests in the Makefile back then.
+
+2011-08-03 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: remove duplicated mention of a test in Makefile.am
+ * tests/Makefile.am (tap_other_tests): Remove `tap-empty.test',
+ it is already listed in `$(tap_with_common_setup_tests)'.
+
+2011-08-03 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: one more use of TAP in our own tests
+ * tests/tap-bad-prog.test, tests/tap-bad-prog2.test: Merged
+ into ...
+ * tests/tap-bad-prog.tap: ... this TAP-generating test.
+ * tests/Makefile.am (tap_other_tests): Remove plan-bad-prog.test
+ and plan-bad-prog2.test.
+ (XFAIL_TESTS): Remove plan-bad-prog2.test.
+ (plan-bad-prog2.log): Remove.
+
+2011-08-03 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: scaffolding to allow use of TAP in our own tests
+ Now that Automake has initial support for the TAP test protocol,
+ we can start "eating our own dog food" and rewrite some tests to
+ use TAP; this should allow for better granularity, especially in
+ the control of skips and expected failures. With this change, we
+ set up the initial scaffolding required by the planned TAP tests,
+ and convert few older tests to use TAP, mostly in order to verify
+ that there are no obvious errors. The tests in our testsuite that
+ use the TAP protocol will be marked by the new `.tap' extension.
+ * tests/plain-functions.sh: New file containing definition of
+ helper shell functions used by the "protocol-less" tests of the
+ Automake testsuite.
+ * tests/tap-functions.sh: New file containing definition of
+ helper shell functions used by the TAP-producing tests of the
+ Automake testsuite.
+ * tests/defs: Remove definitions of some functions that are
+ now defined in `plain-functions.sh'. Instead, source one of
+ `plain-functions.sh' or `tap-functions.sh', depending on whether
+ `$use_tap' is set to "no" or "yes".
+ (exit trap): Call `late_plan_' if `$use_tap' is true. Also,
+ unset shell traces before issuing the latest commands, to avoid
+ confusing the tap driver with spurious output.
+ * tests/defs-static.in ($use_tap): New variable, by default set
+ to "yes" if the calling test script has a `.tap' suffix, and to
+ "no" otherwise. The individual scripts can override it though.
+ In code sanity-checking the environment, verify that `$use_tap'
+ is not exported.
+ * tests/self-check-env-sanitize.test: Update, and small related
+ reformatting.
+ * tests/self-check-tap.test: New very minimal self test.
+ * tests/acoutbs.test, tests/acoutbs2.test, tests/acoutnoq.test,
+ tests/acoutpt.test, tests/acoutpt2.test, tests/acoutqnl.test:
+ Removed, merged into ...
+ * tests/ac-output-old.tap: ... this new test, that uses TAP.
+ * tests/Makefile.am (TAP_LOG_DRIVER): Define to invoke our own
+ `tap-driver' script.
+ (TAP_LOG_DRIVER_FLAGS): Define to `--merge', so that the stdout
+ and stderr of the test scripts remains synced.
+ (TEST_EXTENSIONS): Define, to add the `.tap' suffix; also list
+ the `.test' suffix explicitly.
+ (EXTRA_DIST): Distribute the new files `plain-functions.sh' and
+ `tap-functions.sh'.
+ ($(TEST_LOGS)): Depends on them.
+ (AM_TESTS_ENVIRONMENT): Ensure that `use_tap' is not exported in
+ the environment of the tests.
+ (TESTS): Update.
+
+2011-08-03 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ * NEWS: Fix typo, and related reformatting.
+
+2011-08-03 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ * NEWS: add mention of new experimental TAP support
+
+2011-08-02 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: rename some test scripts
+ * tests/parallel-tests-am_tests_environment.test: Renamed ...
+ * tests/am-tests-environment.test: ... to this.
+ * tests/check-tests_environment.test: Renamed ...
+ * tests/tests-environment.test: ... to this.
+ * tests/parallel-test-driver-install.test: Renamed ...
+ * tests/parallel-tests-driver-install.test: ... to this.
+ * tests/parallel-tests-make-n.test: Renamed ...
+ * tests/parallel-tests-dry-run.test: ... to this.
+ * tests/Makefile.am (TESTS): Updated.
+
+2011-08-02 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ coverage: parallel-tests and lazy dependencies on EXTRA_PROGRAMS
+ * tests/parallel-tests-extra-programs.test: New test.
+ * tests/Makefile.am (TESTS): Update.
+ * doc/automake.texi (Parallel Test Harness): Add a comment
+ pointing to the new test.
+
+2011-08-02 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ docs: improve, extend and fix documentation on TAP support
+ * doc/automake.texi ("Using the TAP test protocol"): Divide this
+ section into ...
+ ("Introduction to TAP", "Use TAP with the Automake test harness",
+ "Incompatibilities with other TAP parsers and drivers", "Links
+ and external resources"): ... these subsections, extend them by
+ adding more information and examples, and improve them by removing
+ incomplete and/or temporary wordings and TODO items.
+ ("Script-based Testsuites", "Parallel Test Harness"): Add a couple
+ of anchors to improve the granularity of cross-references.
+ * tests/tap-doc2.test: New test, verifying the correctness of the
+ new examples given in the manual.
+ * tests/Makefile.am (tap_other_tests): Add the new test.
+
+2011-08-01 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: separate the only failing check of an xfailing test
+ * tests/plan-bad-prog.test: Move the only failing check of this
+ test (i.e., the one about the total number of "ERROR" outcomes)
+ into ...
+ * tests/plan-bad-prog2.test: ... this new test.
+ * tests/Makefile.am (XFAIL_TESTS): Remove `plan-bad-prog.test',
+ add `plan-bad-prog2.test'.
+ (tap_other_tests): Add `plan-bad-prog2.test'.
+ (plan-bad-prog2.log): Depend on `plan-bad-prog.test'.
+
+2011-08-01 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: fix spurious errors in an xfailing test on TAP support
+ * tests/plan-bad-prog.test: Fix typo in the name of the test
+ being made unreadable. Correct the wording of the potential
+ skip message, and simplify the condition under which the test
+ is to be skipped. Escape literal dots in grep regexps.
+
+2011-08-01 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: better granularity in a couple of tests on TAP support
+ * tests/tap-summary-aux.sh: New auxiliary script, filled with code
+ moved out from ...
+ * tests/tap-summary.test: ... this test, from which the checks on
+ colored testsuite have further been moved into ...
+ * tests/tap-summary-color.test: ... this new test, which in turn
+ * tests/Makefile.am (tap_other_tests): Add `tap-summary-color.test'.
+ (EXTRA_DIST): Distribute `tap-summary-aux.sh'
+ (tap-summary.log): Depend on `tap-summary-aux.sh'.
+ (tap-summary-color.log): Likewise.
+
+2011-08-01 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: optimize tests on TAP for speed and against duplication
+ The creation and configuration of common files and data used by
+ many tests on TAP is, with this change, factored out into a new
+ dedicated auxiliary test, from which those tests will depend upon.
+ This reduces code duplication in tests and, more importantly,
+ offers a noticeable speedup in the involved tests (30-40%).
+ * tests/tap-common-setup.test: New test, setting up the common
+ files and data used by various tests on TAP support.
+ * tests/tap-setup.sh: New file, to be sourced by tests wanting to
+ bring in data generated by `tap-common-setup.test'.
+ * tests/tap-autonumber.test: Update to use the pre-computed data
+ files.
+ * tests/tap-bailout.test: Likewise.
+ * tests/tap-color.test: Likewise.
+ * tests/tap-deps.test: Likewise.
+ * tests/tap-diagnostic.test: Likewise.
+ * tests/tap-empty-diagnostic.test: Likewise.
+ * tests/tap-empty.test: Likewise.
+ * tests/tap-escape-directive.test: Likewise.
+ * tests/tap-exit.test: Likewise.
+ * tests/tap-signal.test: Likewise.
+ * tests/tap-fancy.test: Likewise.
+ * tests/tap-fancy2.test: Likewise.
+ * tests/tap-global-log.test: Likewise.
+ * tests/tap-global-result.test: Likewise.
+ * tests/tap-html.test: Likewise.
+ * tests/tap-log.test: Likewise.
+ * tests/tap-merge-stdout-stderr.test: Likewise.
+ * tests/tap-no-merge-stdout-stderr.test: Likewise.
+ * tests/tap-message-0.test: Likewise.
+ * tests/tap-no-disable-hard-error.test: Likewise.
+ * tests/tap-no-spurious-summary.test: Likewise.
+ * tests/tap-no-spurious.test: Likewise.
+ * tests/tap-not-ok-skip.test: Likewise.
+ * tests/tap-numeric-description.test: Likewise.
+ * tests/tap-out-of-order.test: Likewise.
+ * tests/tap-passthrough.test: Likewise.
+ * tests/tap-passthrough-exit.test: Likewise.
+ * tests/tap-plan.test: Likewise.
+ * tests/tap-plan-corner.test: Likewise.
+ * tests/tap-plan-corner2.test: Likewise.
+ * tests/tap-plan-errors.test: Likewise.
+ * tests/tap-realtime.test: Likewise.
+ * tests/tap-recheck-logs.test: Likewise.
+ * tests/tap-skip-whole.test: Likewise.
+ * tests/tap-todo-skip-together.test: Likewise.
+ * tests/tap-todo-skip-whitespace.test: Likewise.
+ * tests/tap-todo-skip.test: Likewise.
+ * tests/tap-unplanned.test: Likewise.
+ * tests/tap-with-and-without-number.test: Likewise.
+ * tests/tap-xfail-tests.test: Likewise.
+ * tests/tap-skip-whole-whitespace.test: Likewise, and remove
+ redundant definitions of `$sp' and `$tab' (they are already
+ defined in `tests/defs').
+ * tests/tap-whitespace-normalization.test: Likewise.
+ * tests/Makefile.am (TESTS): Update.
+ (EXTRA_DIST): Distribute `tap-setup.sh'.
+ (tap_with_common_setup_tests, tap_with_common_setup_logs)): New
+ variables, holding respectively the list of tests using the files
+ pre-computed by `tap-common-setup.test', and the list of their
+ corresponding log files.
+ (tap_other_tests): New variable, holding the list of other tests
+ on TAP support.
+ ($(tap_with_common_setup_logs)): Depend on `tap-common-setup.log'
+ and `tap-setup.sh'.
+
+2011-08-01 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ test harness: use new `.trs' files to hold test metadata
+ With this change, the test harness will keep test metadata in
+ dedicated `.trs' files, instead of having them embedded into the
+ `.log' files. This allows for easier forward-compatibility and
+ extension of test metadata, and for more flexibility in the
+ format of the `.log' files. Note that this change makes the
+ `:end-metadata:' field obsolete.
+ * doc/automake.texi (Parallel Test Harness, Log files generation
+ and test results recording): Document the new APIs and behaviour;
+ some related minor rewordings and fixlets.
+ * NEWS: Update.
+ * automake.in (handle_tests): When bringing in the content of
+ `check2.am', substitute %BASE% with the basename of the `.log'
+ file being created by a rule. Add the generated `.trs' files
+ to the list of files to be cleaned by "make mostlyclean".
+ * lib/am/check.am (am__test_driver_flags): Rename ...
+ (am__common_driver_flags): ... to this, and remove the flags
+ `--test-name' and `--log-file' from it: they are now define in
+ the proper rules in `check2.am'.
+ (am__TEST_BASES): New internal variable, holding the names of
+ the tests, with any registered extension removed.
+ (am__stealth_MAKE): New internal variable, can be used instead of
+ $(MAKE) in recipes requiring a recursive call to make, but which
+ are not intended to be executed by "make -n".
+ (.log.trs): New suffix rule, to recover from deletion of `.trs'
+ files.
+ ($(TEST_SUITE_LOG)): Almost completely rewritten to follow the
+ new API of "test logs in `.log' files, test metadata in `.trs'
+ files". It goes to some length to work correctly in face of
+ unreadable or missing `.log' and `.trs' files, and to error out
+ with proper error messages when this is not possible.
+ [%?PARALLEL_TESTS%] (check-TESTS): Also remove relevant "stale"
+ `.trs' files (in addition to `.log files) before remaking the
+ $(TEST_SUITE_LOG).
+ (recheck, recheck-html): Look for the `:recheck:' field in the
+ `.trs' files, not in the `.log' files.
+ * lib/am/check2.am (?GENERIC?%EXT%.log, ?!GENERIC?%OBJ%): Adjust
+ the call to the test driver, in particularly passing the new
+ option `--trs-file'.
+ [%am__EXEEXT%] (?GENERIC?%EXT%$(EXEEXT).log): Likewise.
+ * lib/tap-driver ($USAGE): Adjust the help screen.
+ (Getopt::Long::GetOptions): Handle the `--trs-file' option,
+ through the use of ...
+ ($trs_file): ... this new global variable.
+ (finish): Write metadata for the test run to `$trs_file' rather
+ then to `$log_file', through the use of ...
+ (write_test_results): ... this new function.
+ * lib/test-driver (print_usage): Update the help screen.
+ (Option parsing): Handle the `--trs-file' option, through the
+ use of ...
+ ($resfile): ... this new global variable.
+ (Main code): Write metadata for the test run to `$trsfile' rather
+ than to `$logfile'.
+ Minor related adjustments to comments.
+ * tests/.gitignore: Ignore `*.trs' files.
+ * tests/parallel-tests-unreadable-log.test: Moved ...
+ * tests/parallel-tests-unreadable.test: ... to this, and extended
+ to also check the semantics for unreadable `.trs' files.
+ * tests/test-driver-end-metadata.test: Deleted as obsolete.
+ * tests/test-driver-metadata-no-leading-space.test: Likewise.
+ * tests/test-driver-global-log.test: Renamed ...
+ * tests/test-metadata-global-log.test: ... to this, and modified
+ as to verify the new APIs and semantics.
+ * tests/test-driver-recheck: Renamed ...
+ * tests/test-metadata-recheck.test: ... to this, and modified
+ likewise.
+ * tests/parallel-tests-once.test: New test.
+ * tests/parallel-tests-make-n.test: Likewise.
+ * test-metadata-results.test: Likewise.
+ * test-missing.test: Likewise.
+ * test-missing2.test: Likewise.
+ * test-trs-basic.test: Likewise.
+ * test-trs-recover.test: Likewise.
+ * test-trs-recover2.test: Likewise.
+ * tests/Makefile.am (TESTS): Update.
+
+2011-07-27 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ test harness: allow more metadata in log files
+ This change reworks and improves the parallel test harness to use
+ more specialized reStructuredText fields in the log files (instead
+ of relying on specially-placed of "magic lines" and more indirect
+ semantical formatting); the new fields are the following:
+ - ":recheck:": tell whether the associated test will have to be
+ re-run by "make recheck";
+ - ":copy-in-global-log:": tell whether the content of the log
+ file should be copied in the "global log" `test-suite.log';
+ - ":end-metadata:", which inhibits the scanning of the rest of
+ the log file (for what concerns test metadata).
+ Also, the special `:test-result:' value "END" has been removed,
+ superseded by the new `:end-metadata:' field.
+ * doc/automake.texi (Log files generation and test results
+ recording): Document the new API and semantics. Remove or fix
+ some obsolete comments.
+ * lib/am/check.am ($(TEST_SUITE_LOG), recheck, recheck-html):
+ Adjust comments and code.
+ * lib/tap-driver (finish): Adjust, with the help of ...
+ (must_recheck, copy_in_global_log): ... these new functions.
+ * lib/test-driver (Main code): Adjust, with the help of ...
+ ($recheck, $gcopy): ... these new variables.
+ * tests/trivial-test-driver: Update to obey the new APIs.
+ * tests/test-driver-recheck.test: Likewise.
+ * tests/test-driver-global-log.test: Likewise.
+ * tests/tap-passthrough.test: Relax the test, avoiding to check
+ what is written in `test-suite.log'; such check has been moved ...
+ * tests/tap-global-log.test: ... in this new test, and extended.
+ * tests/test-driver-metadata-no-leading-space.test: New test.
+ * tests/test-driver-end-test-results.test: Removed, it checked
+ the old APIs; superseded by ...
+ * tests/test-driver-end-metadata.test: ... this new test.
+ * tests/tap-log.test: Improve syncing with ...
+ * tests/test-log.test: ... this new test.
+ * tests/parallel-tests.test: Remove some duplication w.r.t. this
+ last new test. Updated heading comments.
+ * tests/Makefile.am (TESTS): Update.
+
+2011-07-27 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: fix a spurious failure with non-bash shells
+ * tests/tap-fancy2.test: Remove an unportable use of backslashes
+ with the `echo' builtin, which was causing some shells (among them
+ bash and Debian /bin/ksh) to print a `\\' string, while other
+ shells (among them zsh, dash, and Solaris /bin/sh and /bin/ksh)
+ were unexpectedly printing a single `\' character. Since we are
+ at it, add a sanity check to ensure that this issue does not
+ resurface.
+
+2011-07-27 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: fix spurious failures with Solaris /bin/sh
+ * tests/tap-more.test: Use `echo > file', not `: > file', to
+ create empty files in the "for" loops; this is required since,
+ as documented in the autoconf manual, Solaris 10 /bin/sh
+ "optimizes" away the `:' command after the first iteration,
+ even if it is redirected.
+ * tests/test-driver-custom-multitest-recheck2.test: Likewise.
+ * tests/tap-numeric-description.test: Partial rewrite to avoid
+ using positional parameters from the 10th onward, which are
+ unportable to Solaris /bin/sh (using `${10}' causes the shell
+ to die with "bad substitution").
+
+2011-07-31 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ simple tests: support developer-defined fd redirections
+ Motivated by coreutils bug#8846, and related discussions:
+ <http://debbugs.gnu.org/cgi/bugreport.cgi?bug=8846>
+ <http://lists.gnu.org/archive/html/bug-autoconf/2011-06/msg00002.html>
+ In those threads it has been shown how problematic it can be to do
+ portable file descriptor redirections in the testsuite when using
+ the Automake testsuite harness. This change should remedy to that
+ situation.
+ * lib/am/check2.am (?GENERIC?%EXT%$(EXEEXT).log,
+ ?GENERIC?%EXT%.log, ?!GENERIC?%OBJ%): Append redirection defined
+ in $(AM_TESTS_FD_REDIRECT) to the command-line invocations of the
+ test scripts.
+ * lib/am/check.am [!%?PARALLEL_TESTS%] $(check-TESTS): Likewise.
+ * NEWS: Update.
+ * doc/automake.texi (Script-based Testsuites): Document the new
+ feature.
+ * tests/check-fd-redirect.test: New test.
+ * tests/parallel-tests-fd-redirect.test: Likewise.
+ * tests/parallel-tests-am_tests_environment.test: Remove checks
+ about the use of redirections in AM_TESTS_ENVIRONMENT: they would
+ check deprecated (if not undefined) behaviour now. Strengthen a
+ couple of still valid checks, to keep the test more in sync with
+ the documentation. Improve debugging information.
+ * tests/Makefile.am (TESTS): Update.
+
+2011-07-22 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ docs: rework and extend documentation on testsuites support
+ * doc/automake.texi (Generalities about Testing): New section.
+ (Simple Tests using parallel-tests): Section removed, merged
+ into ...
+ (Simple Tests): ... this one, which on the other hand has been
+ subdivided into ...
+ (Script-based Testsuites, Serial Test Harness, Parallel Test
+ Harness): ... these new subsection.
+ (DejaGnu Tests): Minor adjustments.
+ Other related typofixes and rewordings throughout the manual;
+ in particular, avoid to use the term "test driver" for three
+ different concepts (and use instead "test harness" and "test
+ runner" where appropriate).
+ * tests/tap-doc.test: New test.
+ * tests/tap-no-disable-hard-error.test: Likewise.
+ * tests/Makefile.am (TESTS): Update.
+
+2011-07-18 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tap: support colorization of testsuite progress output
+ * lib/tap-driver (%COLORS): New variable (definition extracted
+ from `lib/am/check.am:$(am__tty_colors)', with some obvious
+ adjustments.
+ (report): Adjust to colorize console output when required,
+ using ...
+ (decorate_result): ... this new function.
+ (colored): New function, used by the one above.
+ * tests/tap-summary.test: Also run the checks when `color-tests'
+ is in use.
+ * tests/Makefile.am (XFAIL_TESTS): Remove `tap-color.test'.
+
+2011-07-18 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tap: some preparatory refactoring (2)
+ This is a follow-up simplification.
+ * lib/tap-driver (console_output): Renamed ...
+ (report): ... to this, and extended to appropriately register
+ the test results when needed.
+ (testsuite_error, handle_tap_comment, handle_tap_test,
+ handle_tap_plan): Adjusted accordingly.
+
+2011-07-18 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tap: some preparatory refactoring (1)
+ This refactoring is aimed at simplifying the introduction of
+ colored console output for the TAP driver.
+ * lib/tap-driver (console_output): Now accept two arguments, the
+ first one indicating which kind of thing is to be displayed (for
+ now only a test result or a diagnostic comment), and the second
+ one (if present) the message associated to it.
+ (handle_tap_test, handle_tap_comment, handle_tap_plan,
+ testsuite_error): Adapt to the new `console_output' interface.
+
+2011-07-18 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tap: add experimental TAP-aware driver
+ * doc/automake.texi (Using the TAP test protocol): New section.
+ (Overview of Custom Test Drivers Support): Minor updates.
+ * lib/tap-driver: New script, TAP-aware test driver for Automake;
+ implemented in perl and based on TAP::Parser.
+ * lib/Makefile.am (dist_script_DATA): Add it.
+ * tests/tap-autonumber.test: New test.
+ * tests/tap-bailout.test: Likewise.
+ * tests/tap-basic.test: Likewise.
+ * tests/tap-deps.test: Likewise.
+ * tests/tap-diagnostic.test: Likewise.
+ * tests/tap-empty.test: Likewise.
+ * tests/tap-empty-diagnostic.test: Likewise.
+ * tests/tap-escape-directive.test: Likewise.
+ * tests/tap-exit.test: Likewise.
+ * tests/tap-fancy.test: Likewise.
+ * tests/tap-fancy2.test: Likewise.
+ * tests/tap-global-result.test: Likewise.
+ * tests/tap-html.test: Likewise.
+ * tests/tap-log.test: Likewise.
+ * tests/tap-merge-stdout-stderr.test: Likewise.
+ * tests/tap-more.test: Likewise.
+ * tests/tap-more2.test: Likewise.
+ * tests/tap-no-merge-stdout-stderr.test: Likewise.
+ * tests/tap-no-spurious-summary.test: Likewise.
+ * tests/tap-no-spurious.test: Likewise.
+ * tests/tap-not-ok-skip.test: Likewise.
+ * tests/tap-numeric-description.test: Likewise.
+ * tests/tap-out-of-order.test: Likewise.
+ * tests/tap-passthrough.test: Likewise.
+ * tests/tap-plan.test: Likewise.
+ * tests/tap-plan-errors.test: Likewise.
+ * tests/tap-plan-corner.test: Likewise.
+ * tests/tap-realtime.test: Likewise.
+ * tests/tap-recheck-logs.test: Likewise.
+ * tests/tap-recheck.test: Likewise.
+ * tests/tap-skip-whole.test: Likewise.
+ * tests/tap-summary.test: Likewise.
+ * tests/tap-todo-skip.test: Likewise.
+ * tests/tap-todo-skip-together.test: Likewise.
+ * tests/tap-todo-skip-whitespace.test: Likewise.
+ * tests/tap-skipall-whitespace.test: Likewise.
+ * tests/tap-unplanned.test: Likewise.
+ * tests/tap-whitespace-normalization.test: Likewise.
+ * tests/tap-with-and-without-number.test: Likewise.
+ * tests/tap-xfail-tests.test: Likewise.
+ * tests/tap-bad-prog.test: New xfailing test.
+ * tests/tap-color.test: Likewise.
+ * tests/tap-plan-corner2.test: Likewise.
+ * tests/tap-message-0.test: Likewise.
+ * tests/tap-signal.test: Likewise.
+ * tests/Makefile.am (TESTS, XFAIL_TESTS): Update.
+
+2011-07-07 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ test defs: new auxiliary function 'count_test_results'
+ * tests/defs (count_test_results): New function.
+ * tests/check11.test: Use it.
+ * tests/test-driver-custom-multitest.test: Likewise.
+ * tests/test-driver-custom-multitest-recheck.test: Likewise.
+ * tests/test-driver-custom-multitest-recheck2.test: Likewise.
+ * tests/parallel-tests-log-override-recheck.test: Likewise.
+ * tests/parallel-tests-log-override-recheck.test: Likewise.
+ * tests/parallel-tests-no-spurious-summary.test: Likewise, and
+ slightly improve debugging output.
+ * tests/parallel-tests.test: Make use of `count_test_results'.
+ Also, make grepping of "make check" output slightly stricter
+ * tests/parallel-tests9.test: Likewise.
+ * tests/parallel-tests-log-override-2.test: Likewise, and throw
+ in a small optimization.
+
+2011-07-07 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ parallel-tests: simplify testsuite summary
+ Prefer a more deterministic, "tabular" format for the testsuite
+ summary, always listing the numbers of passed, failed, xfailed,
+ xpassed, skipped and errored tests, even when these numbers are
+ zero. This simplify the logic of testsuite summary creation,
+ makes it more easily machine-parseable, and will probably allow
+ for easier addition of new kinds of test results in the future.
+ * lib/am/check.am (am__tty_colors_dummy): New make variable, to
+ reduce code duplication. Extracted from previous versions of
+ $(am__tty_colors), and extended by defining two new variables
+ `$mgn' and `$brg'.
+ [%?COLOR%, %!?COLOR%] (am__tty_colors): Use that new variable.
+ (am__text_box): Delete, is not needed anymore.
+ ($(TEST_SUITE_LOG)): Rewrite associated rules to implement the
+ new testsuite summary format.
+ * NEWS: Update.
+ * tests/check10.test: Don't run with the parallel-tests harness
+ too, that makes no sense anymore.
+ * tests/color.test: Update and adjust.
+ * tests/color2.test: Likewise.
+ * tests/parallel-tests.test: Likewise.
+ * tests/parallel-tests3.test: Likewise.
+ * tests/parallel-tests6.test: Likewise.
+ * tests/parallel-tests9.test: Likewise.
+ * tests/parallel-tests-unreadable-log.test: Likewise.
+ * tests/parallel-tests-empty-testlogs.test: Likewise.
+ * tests/parallel-tests-log-override-recheck.test: Likewise.
+ * tests/parallel-tests-no-spurious-summary.test: Likewise.
+ * tests/test-driver-custom-multitest.test: Likewise.
+ * tests/test-driver-end-test-results.test: Likewise.
+ * tests/parallel-tests-no-color-in-log.test: New test.
+ * tests/testsuite-summary-color.test: Likewise.
+ * tests/testsuite-summary-count.test: Likewise.
+ * tests/testsuite-summary-count-many.test: Likewise.
+ * tests/testsuite-summary-reference-log.test: Likewise.
+ * tests/testsuite-summary-checks.sh: New auxiliary script, used
+ by the new tests above.
+ * tests/extract-testsuite-summary: Likewise.
+ * tests/trivial-test-driver: Optimize for speed when there are
+ lots of of tests.
+ * tests/Makefile.am (EXTRA_DIST): Distribute them.
+ (testsuite-summary-color.log, testsuite-summary-count.log): Depend
+ on them.
+ (testsuite-summary-count-many.log): Depend on the auxiliary scripts
+ 'trivial-test-driver' and 'extract-testsuite-summary'.
+ (TESTS): Update.
+
+2011-07-07 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ parallel-tests: new recognized test result 'ERROR'
+ * lib/am/check.am ($(TEST_SUITE_LOG)): Recognize a new test result
+ `ERROR'. Use it when encountering unreadable test logs (previously
+ a simple `FAIL' was used in this situations).
+ * lib/test-driver: Set the global test result to `ERROR' when the
+ test exit status is 99. When doing colorized output, color `ERROR'
+ results in magenta.
+ * doc/automake.texi (Log files generation and test results
+ recording): Update by listing `ERROR' too among the list of valid
+ `:test-results:' arguments.
+ * NEWS: Update.
+ * tests/trivial-test-driver: Update.
+ * tests/parallel-tests.test: Likewise.
+ * tests/parallel-tests-harderror.test: Likewise.
+ * tests/parallel-tests-no-spurious-summary.test: Likewise.
+ * tests/test-driver-global-log.test: Likewise.
+ * tests/test-driver-recheck.test: Likewise.
+ * tests/test-driver-custom-multitest-recheck.test: Likewise.
+ * tests/test-driver-custom-multitest-recheck2.test: Likewise.
+ * tests/test-driver-custom-multitest.test: Likewise.
+ * tests/test-driver-custom-no-html.test: Likewise.
+ * tests/test-driver-end-test-results.test: Likewise.
+ * tests/color.test: Likewise. Also, make stricter, and also test
+ from VPATH.
+ * tests/color2.test: Likewise, and improve syncing with color.test.
+ * tests/parallel-tests-exit-statuses.test: New test.
+ * tests/parallel-tests-console-output.test: Likewise.
+ * tests/Makefile.am (TESTS): Update.
+
+2011-07-07 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ parallel-tests: make parsing of test results safer
+ The new code for parsing the testsuite-generated `.log' files,
+ as introduced in commit `v1.11-872-gc96b881', considers each
+ `:test-result:' field anywhere in a `.log' file as a declaration
+ of a test result, and accounts for it as such in the testsuite
+ summary. Unfortunately this could easily cause spurious test
+ failures being reported in the testsuite summary. This happened
+ in practice with the Automake's own testsuite; for example:
+ $ make check TESTS='check12-p.test'; echo exit: $?
+ ...
+ PASS: check12-p.test
+ =====================================
+ 4 of 5 tests failed
+ See tests/test-suite.log
+ Please report to bug-automake@gnu.org
+ =====================================
+ make[2]: *** [test-suite.log] Error 1
+ make: *** [check-am] Error 2
+ exit: 2
+ This change introduces a new special `:test-result:' "END", that,
+ when seen, prevents the rest of the log file from being parsed.
+ For more information, refer to the thread:
+ <http://lists.gnu.org/archive/html/automake-patches/2011-06/msg00199.html>
+ * lib/am/check.am ($(TEST_SUITE_LOG)): Stop the parsing of a log
+ file as soon as the special ":test-result:END" directive is seen.
+ Related changes and enhancements.
+ * lib/test-driver: Protect the rest of the log after the result
+ lined with a ":test-result:END" directive.
+ * doc/automake.texi (Log files generation and test results
+ recording): Update, and related improvements.
+ * tests/parallel-tests-no-spurious-summary.test: New test.
+ * tests/test-driver-end-test-results.test: Likewise.
+ * tests/Makefile.am (TESTS): Update.
+
+2011-06-29 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ docs: document custom test drivers and protocols
+ * doc/automake.texi (Simple Tests): Note that the TESTS_ENVIRONMENT
+ use suggested here is not portable to 'parallel-tests'.
+ (Simple Tests using parallel-tests): Document new restrictions on
+ the uses of TESTS_ENVIRONMENT and AM_TESTS_ENVIRONMENT.
+ (Custom Test Drivers): New section and node.
+ (Overview of Custom Test Drivers Support): New subsection.
+ (Declaring Custom Test Drivers in @file{Makefile.am}): Likewise.
+ (APIs for Custom Test Drivers): Likewise.
+ (Options): Update description of color-tests.
+ * lib/am/check ($(TEST_SUITE_LOG)): Remove comments that have been
+ moved into the manual.
+ (recheck, recheck-html): Minor adjustments to better conform to the
+ documentation (this should cause no semantic changes w.r.t. the
+ former behaviour); minor improvements and extensions to existing
+ comments.
+ * tests/test-driver-create-log-dir.test: New test.
+ * tests/test-driver-strip-vpath.test: Likewise.
+ * tests/test-driver-global-log.test: Likewise.
+ * tests/test-driver-recheck.test: Likewise.
+ * tests/Makefile.am (TESTS): Update.
+
+2011-07-24 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: small enhancement to 'primary3.test'
+ * tests/primary3.test: Also test that the program named `foo.la'
+ has been effectively created.
+
+2011-07-24 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: a small simplification in 'pr8365-remake-timing.test'
+ * tests/pr8365-remake-timing.test: Avoid redundant definition
+ and use of `$save_AUTOCONF' variable.
+
+2011-07-24 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: remove an extra leading blank line
+ * tests/override-conditional-1.test: Remove extra blank line
+ placed before the shebang line.
+
+2011-07-18 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: fix typos, grammaros and other blunders in comments
+ All affected files changed.
+
+2011-07-16 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ test defs: new subroutine 'seq_', simulating GNU seq(1)
+ * tests/defs (seq_): New subroutine.
+ * tests/instmany.test: Use it.
+ * tests/instmany-mans.test: Likewise.
+ * tests/instmany-python.test: Likewise.
+ * tests/self-check-seq.test: New self test.
+ * tests/Makefile.am (TESTS): Update.
+
+2011-07-16 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: remove duplication about testing of config.* aux files
+ * tests/add-missing.test: Also check that the `AC_CANONICAL_SYSTEM'
+ autoconf macro causes the `config.sub' and `config.guess' scripts
+ to be installed by `automake --add-missing'. Since we are at it,
+ fix minor buglets and cosmetic issues in the `check_' function.
+ * tests/hosts.test: Removed, completely subsumed by the previous
+ test now.
+ * tests/Makefile.am (TESTS): Update.
+
+2011-07-16 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: more uses of the 'unindent' subroutine
+ * tests/backcompat.test: Prefer `unindent' over plain `cat' for
+ here documents created from within a loop of if/else body.
+ * tests/backcompat2.test: Likewise.
+ * tests/backcompat4.test: Likewise.
+ * tests/init.test: Likewise.
+ * tests/instmany.test: Likewise.
+ * tests/instmany-mans.test: Likewise.
+ * tests/instmany-python.test: Likewise.
+ * tests/missing5.test: Likewise.
+ * tests/parallel-am2.test : Likewise.
+ * tests/parallel-am3.test : Likewise.
+ * tests/pr307.test: Likewise.
+ * tests/depend4.test: Likewise. Since we are at it, normalize
+ other code formatting.
+ * tests/instfail-java.test: Prefer `echo' over `cat' + here-doc
+ inside a for loop.
+
+2011-07-16 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: avoid '##'-style comments inside recipe commands
+ * tests/autodist.test: Do not whitespace-indent `##' comments when
+ they are embedded in a makefile rule: having them indented is not
+ part of the Automake API, and might cause failures with e.g., Tru64
+ make.
+ * tests/autodist-subdir.test: Likewise.
+ * tests/backcompat.test: Likewise.
+ * tests/backcompat6.test: Likewise.
+ * tests/colon7.test: Likewise.
+ * tests/posixsubst-scripts.test: Likewise.
+ * tests/posixsubst-sources.test: Likewise.
+
+2011-07-16 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: tweak, improve and extend tests on lisp support
+ * tests/lisp2.test: Prettify, and improve debugging output. Make
+ grepping of automake stderr stricter.
+ * tests/lisp7.test: Ensure verbose printing of captured make
+ output. Add trailing `:' command.
+ * tests/lisp8.test: Likewise.
+ * tests/lisp3.test: Likewise. Also, check that `.el' files and
+ compiled `.elc' files get installed by "make install", and
+ uninstalled by "make uninstall".
+ * tests/lisp4.test: Ensure installed `.el' files gets removed by
+ "make uninstall". Related changes. Add trailing `:' command.
+ * tests/lisp5.test: Likewise.
+ * tests/lisp6.test: Use proper m4 quoting in configure.in. Fix
+ use of blank lines, to improve clarity and symmetry. Fix typo in
+ comment. Add trailing `:' command.
+
+2011-07-16 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: drop useless requirement in 'license.test'
+ * tests/license.test: Drop useless requirement "makeinfo". Add
+ an extra blank line, for clarity.
+
+2011-07-08 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests defs: more uses of $top_testsrcdir
+ * tests/autodist-stamp-vti.test: Use `$top_testsrcdir' instead
+ of `$testsrcdir/..'.
+ * tests/repeated-options.test: Likewise.
+ * tests/suffix5.test: Likewise.
+ * tests/vtexi3.test: Likewise.
+
+2011-07-08 Stefano Lattarini <stefano.lattarini@gmail.com>
- remake: fix outdated comment in configure.am
- * lib/am/configure.am: Fix comment falsified by changes in
- commit `v1.11-366-gbee9871'.
- Suggestion by Ralf Wildenhues.
+ self tests: fix typo in comment
+ * tests/self-check-exit.test: Fix typo in heading comments.
-2011-07-01 Stefano Lattarini <stefano.lattarini@gmail.com>
+2011-07-08 Stefano Lattarini <stefano.lattarini@gmail.com>
- docs, tests: synchronize examples from docs to tests
- * tests/README (Writing test cases): Give suggestions on how to
- keep test cases and examples in the documentation synchronized.
- * doc/automake.texi: Improve or fix existing testcase-referencing
- comments, and add many new ones.
- * HACKING (Administrivia): Suggest to test complex examples and
- idioms from the manual.
- * tests/specflg8.test: Improve synchronization with the example
- in the manual.
- * tests/output11.test:Likewise.
- * tests/txinfo21.test:Likewise.
- * tests/interp.test: Likewise.
- * tests/amhello-cflags.test: New test.
+ tests: remove redundant settings of `errexit' shell flag
+ * tests/amhello-binpkg.test: Do not set the `errexit' shell
+ flag, as it is already set by `tests/defs'.
+ * tests/amhello-cflags.test: Likewise.
* tests/amhello-cross-compile.test: Likewise.
- * tests/amhello-binpkg.test: Likewise.
- * tests/tests-environment-backcompat.test: Likewise.
+ * tests/ansi2knr-deprecation.test: Likewise.
+ * tests/autodist-no-duplicate.test: Likewise.
+ * tests/distcheck-configure-flags-am.test: Likewise.
+ * tests/distcheck-configure-flags-subpkg.test: Likewise.
+ * tests/distcheck-configure-flags.test: Likewise.
+ * tests/distcheck-hook.test: Likewise.
+ * tests/distcheck-hook2.test: Likewise.
+ * tests/parallel-tests-am_tests_environment.test: Likewise.
+ * tests/parallel-tests-harderror.test: Likewise.
* tests/parallel-tests-log-compiler-example.test: Likewise.
- * tests/Makefile.am (TESTS): Update.
-
-2011-06-30 Stefano Lattarini <stefano.lattarini@gmail.com>
-
- coverage: new test on parallel-tests TESTS runtime overriding
- * tests/parallel-tests-cmdline-override.test: New test, check that
- we can use indirections when overriding TESTS and TEST_LOGS from
- the command line.
- * tests/Makefile.am (TESTS): Update.
+ * tests/parallel-tests-log-override-1.test: Likewise.
+ * tests/parallel-tests-log-override-2.test: Likewise.
+ * tests/parallel-tests-log-override-recheck.test: Likewise.
+ * tests/primary-prefix-couples-documented-valid.test: Likewise.
+ * tests/primary-prefix-couples-force-valid.test: Likewise.
+ * tests/primary-prefix-invalid-couples.test: Likewise.
+ * tests/primary-prefix-valid-couples.test: Likewise.
+ * tests/remake-subdir-from-subdir.test: Likewise.
+ * tests/remake-subdir-gnu.test: Likewise.
+ * tests/remake-subdir-long-time.test: Likewise.
+ * tests/remake-subdir.test: Likewise.
+ * tests/remake-subdir2.test: Likewise.
+ * tests/silent-configsite.test: Likewise.
+ * tests/tests-environment-backcompat.test: Likewise.
2011-06-29 Stefano Lattarini <stefano.lattarini@gmail.com>
2011-06-21 Stefano Lattarini <stefano.lattarini@gmail.com>
+ parallel-tests: allow each test to have multiple results
+ With this change, we improve the code creating the `test-suite.log'
+ global log and the console testsuite summary to make it able to
+ grasp multiple results per test script. This is required in order
+ to introduce the planned support for test protocols, like TAP and
+ SubUnit, which can indeed run multiple tests per test script, each
+ with its individual result.
+ The implementation makes use of a custom reStructuredText field
+ `:test-result:'.
+ Note that no new documentation is added by this change; that is
+ be left for follow-up changes.
+ * lib/check.am ($(TEST_SUITE_LOG)): When processing .log files,
+ recognize a report of a test's result only if it is declared with
+ the custom `:test-result:' reStructuredText field placed at the
+ beginning of a line. Extend and add explanatory comments.
+ (recheck, recheck-html): Add explanatory comments.
+ * lib/test-driver: Write an appropriate reStructuredText field
+ `:test-result:' in the generated log file. Use a reStructuredText
+ transition to better separate the test outcome report from the
+ test script's registered output. Improve comments.
+ * tests/test-driver-custom-xfail-tests.test: Adjust.
+ * tests/parallel-tests7.test: Adjust.
+ * tests/parallel-tests-empty-testlogs.test: New test.
+ * tests/parallel-tests-recheck-override.test: Likewise.
+ * tests/parallel-tests2.test: Extend and keep more in-sync with ...
+ * tests/test-driver-custom-html.test: ... this new related test.
+ * tests/test-driver-custom-no-html.test: New test.
+ * tests/test-driver-custom-multitest.test: Likewise.
+ * tests/test-driver-custom-multitest-recheck.test: Likewise.
+ * tests/test-driver-custom-multitest-recheck2.test: Likewise.
+ * tests/trivial-test-driver: New file, used by the last four tests
+ above.
+ * tests/Makefile.am (TESTS): Update.
+ (EXTRA_DIST): Distribute `trivial-test-driver'.
+ (test-driver-custom-multitest.log): Depend on `trivial-test-driver'.
+ (test-driver-custom-multitest-recheck.log): Likewise.
+ (test-driver-custom-multitest-recheck2.log): Likewise.
+ (test-driver-custom-html.log): Likewise.
+
+2011-06-21 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ parallel-tests: allow custom driver scripts
+ Allow suffix-based definition of custom "driver script" for the
+ test scripts. These driver scripts will be responsible of
+ launching the tests (or their corresponding $(LOG_COMPILER), if
+ they have an associated one), interpreting and displaying the
+ test results, and writing the `.log' files.
+ This new API should allow easy and flexible use of different
+ test protocols in the future; in particular, we plan to use it
+ to implement TAP and SubUnit harnesses.
+ Note that no new documentation is added by this change; that is
+ be left for follow-up changes.
+ * automake.in (handle_tests): Define default for $(LOG_DRIVER),
+ and, for any registered test extension `<ext>', define defaults
+ for $(<ext>_LOG_DRIVER). Substitute %DRIVER% using these new
+ variables, instead of the old internal $(am__test_driver). When
+ processing check2.am, also substitute %DRIVER_FLAGS%.
+ Require auxiliary script `test-driver' only if no driver has been
+ explicitly defined for the test script kinds.
+ * am/check2.am (?GENERIC?%EXT%$(EXEEXT).log, ?GENERIC?%EXT%.log,
+ ?!GENERIC?%OBJ%): Pass the %DRIVER_FLAGS% to the %DRIVER% call.
+ * tests/parallel-tests-no-extra-driver.test: New test.
+ * tests/test-driver-custom.test: Likewise.
+ * tests/test-driver-custom-xfail-tests.test: Likewise.
+ * tests/test-driver-fail.test: Likewise.
+ * tests/Makefile.am: Update.
+ * NEWS: Update.
+
+2011-06-21 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ parallel-tests: add auxiliary script 'test-driver', refactor
+ This refactoring should cause no API of functionality change,
+ and is meant only to simplify the future implementation of TAP
+ and SubUnit testsuite drivers. More precisely, our roadmap is
+ to move most of the "testsuite driving" features out of the
+ Automake-generated Makefiles, and into external scripts with
+ well-defined interfaces. This will allow the user to define
+ its own personalized testsuite drivers, and will also offer us
+ a framework upon which to implement our new TAP and SubUnit
+ drivers, all in a very unobtrusive way and retaining an high
+ degree of code reuse and backward-compatibility.
+ * lib/test-driver: New auxiliary script.
+ * lib/Makefile.am (dist_SCRIPT_DATA): Add it.
+ * automake.in (handle_tests): Require the new auxiliary script
+ `test-driver', and define a new internal makefile variable
+ `$(am__test_driver)', used to call it. Perform new substitution
+ on `DRIVER' when processing the `check2.am' file.
+ * lib/check.am (am__tty_colors): Define new shell variable
+ `$am__color_tests'.
+ (am__rst_section): Removed, its role taken over by the new
+ `test-driver' script.
+ (am__test_driver_flags): New variable, contains the command
+ line options passed to `test-driver'.
+ (am__check_pre): Do not deal with temporary files and exit
+ traps anymore, as the `test-driver' script takes care of that
+ now. Define shell variable `$am__enable_hard_errors', used by
+ `$(am__test_driver_flags)'. Reorder so that we don't need to
+ save and restore the value of the `TERM' environment variable
+ anymore.
+ Other related adjustments.
+ (am__check_post): Remove, as its role has been completely taken
+ over by the `test-driver' script.
+ * am/check2.am (?GENERIC?%EXT%$(EXEEXT).log, ?GENERIC?%EXT%.log,
+ ?!GENERIC?%OBJ%): Call the test script through the Automake
+ substituted `%DRIVER%', and honor the command-line options
+ in `$(am__test_driver_flags)'. Do not call the obsoleted
+ `$(am__check_post)' anymore.
+ * doc/automake.texi (Auxiliary Programs): Mention the new
+ `test-driver' script.
+ (Optional): Mention `test-driver' in AC_CONFIG_AUX_DIR.
+ Since we are at it, break the list of auxiliary scripts by
+ placing one per line, to simplify potential future additions
+ of new scripts.
+ * tests/check.test: Adjust.
+ * tests/check2.test : Likewise.
+ * tests/check3.test : Likewise.
+ * tests/check4.test : Likewise.
+ * tests/check10.test: Likewise.
+ * tests/color.test: Likewise.
+ * tests/color2.test: Likewise.
+ * tests/comment9.test: Likewise.
+ * tests/dejagnu.test: Likewise.
+ * tests/exeext4.test: Likewise.
+ * tests/maken3.test: Likewise.
+ * tests/maken4.test: Likewise.
+ * tests/parallel-tests-interrupt.test: Likewise.
+ * tests/posixsubst-tests.test: Likewise.
+ * tests/repeated-options.test: Likewise.
+ * tests/check-no-test-driver.test: New test.
+ * tests/parallel-test-driver-install.test: Likewise.
+ * tests/Makefile.am (TESTS): Update.
+ * NEWS: Update.
+
+2011-06-21 Stefano Lattarini <stefano.lattarini@gmail.com>
+
maintcheck: extend 'sc_tests_plain_*' checks
* Makefile.am (sc_tests_plain_autom4te): New check.
(sc_tests_plain_autoreconf): Likewise.
extensions).
* tests/Makefile.am (TESTS): Update.
+2011-06-24 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: make 'subst-no-trailing-empty-line.test' more robust
+ * tests/subst-no-trailing-empty-line.test: Strengthen existing
+ checks, to try more scenarios and be slightly stricter in some
+ grepping checks.
+
+2011-06-24 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: fix an incomplete comment
+ * tests/subst-no-trailing-empty-line.test: Fix an incomplete
+ comment, and move it to a more proper place. Minor cosmetic
+ fixes to other comments.
+
+2011-06-20 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ maintcheck: avoid few more spurious failures
+ * tests/depmod-data.test: Use creative quoting to avoid
+ spuriously triggering the `sc_tests_Exit_not_exit' maintainer
+ check. Commit `v1.11-900-g3453b8e' attempted to fix it, but
+ succeeded only partially.
+ * tests/cond33.test: Fix header comments, not to reference
+ obsolescent make variable `$(mkdir_p)'.
+ * tests/cond4.test: Rewrite "$MAKE exp=..." as "exp=.. $MAKE -e",
+ to please `sc_tests_overriding_macros_on_cmdline'
+ * tests/cond19.test: Likewise.
+ * tests/cond32.test: Likewise.
+ * tests/add-missing.test: Use "AUTOMAKE_fails" instead of
+ "$AUTOMAKE ... && Exit 1", to please `sc_tests_automake_fails'.
+
2011-06-20 Stefano Lattarini <stefano.lattarini@gmail.com>
maintcheck: avoid few spurious failures
See also following CC:ed thread on bug-autoconf list:
<http://lists.gnu.org/archive/html/bug-autoconf/2011-06/msg00002.html>
+2011-06-16 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: make test 'self-check-reexec.test' more portable
+ * tests/self-check-reexec.test: Rewrite not to require a Korn
+ Shell able to grok variable expansions such as `${.sh.version}';
+ Solaris 10 /bin/ksh fails on this for example. Instead, just
+ require bash and a shell that is not bash.
+
+2011-06-14 Stefano Lattarini <stefano.lattarini@gmail.com>
+ Jim Meyering <meyering@redhat.com>
+
+ test defs: fix ksh-related portability bug in warning messages
+ Running "make check" normally prints a diagnostic to the outermost
+ stderr (usually a tty) to explain why a test is skipped, thus
+ giving better and faster feedback to the user. It used to do
+ so by redirecting file descriptor 9 to stderr (via "exec 9>&2")
+ before invoking the test scripts, which then would write any skip
+ explanation to file descriptor 9 via the `skip_' function defined
+ in `tests/defs'.
+ However, various Korn Shells (at least Solaris 10's /bin/ksh and
+ Debian GNU/Linux's /bin/ksh) and the HP-UX's /bin/sh close open
+ file descriptors > 2 upon an `exec' system call; thus the effects
+ of "exec 9>&2" are cancelled upon fork-and-exec, so we would get
+ a "Bad file number" diagnostic and no skip explanation with those
+ shells.
+ The present change remedies this situation.
+ * tests/Makefile.am (AM_TESTS_ENVIRONMENT): Redirect more portably,
+ via a trailing "9>&2", rather than the prior "exec 9>&2; ...". Add
+ explanatory comments.
+ * tests/defs (stderr_fileno_): Update the advice in comments.
+ Based on commit v8.12-82-g6b68745 "tests: accommodate HP-UX and
+ ksh-derived shells" in GNU coreutils.
+ Further references, with lots of discussion:
+ <http://lists.gnu.org/archive/html/bug-autoconf/2011-06/msg00002.html>
+ <http://thread.gmane.org/gmane.comp.gnu.coreutils.bugs/22488>
+ <http://debbugs.gnu.org/cgi/bugreport.cgi?bug=8846>
+
+2011-06-13 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: remove 'test_prefer_config_shell' from the environment
+ Since commit `v1.11-910-g7df1a9b', the once user-overridable
+ variable `$test_prefer_config_shell' has become an internal
+ detail, and the test scripts now complain and bail out if it is
+ set in the environment.
+ * tests/Makefile.am (AM_TESTS_ENVIRONMENT): Unset the variable
+ `test_prefer_config_shell' if it is set in the environment.
+
+2011-06-13 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: autogenerate list of wrapped tests for `lib/' shell scripts
+ * tests/gen-config-shell-tests: New script, generates distributed
+ makefile snippet `tests/config-shell-tests.am' to list all tests
+ that use the `get_shell_script' function, with names mangled to
+ use suffix `-w.shtst', in ...
+ * tests/Makefile.am (config_shell_tests): ... this macro, whose
+ definition has been consequently removed from Makefile.am.
+ (EXTRA_DIST): Distribute the new script.
+ ($(srcdir)/config-shell-tests.am): Generate using the new script.
+ (include): Include the `config-shell-tests.am' fragment.
+ * bootstrap: Invoke `tests/gen-config-shell-tests' to generate
+ `tests/config-shell-tests.am'.
+ * tests/.gitignore: Ignore `config-shell-tests.am'.
+ * tests/gen-parallel-tests: Fixlet in heading comments.
+
+2011-06-13 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: test mdate-sh with /bin/sh too
+ * tests/mdate5.test: Fetch the `mdate-sh' script using the
+ `get_shell_script' function, and run it directly instead of
+ using `$SHELL'.
+ * tests/mdate6.test: Likewise. Since we are at it, make checks
+ on the `mdate-sh' output stricter, remove now unneeded calls to
+ aclocal and automake and creation/extension of `configure.in',
+ `Makefile.am' and `textutils.tex' files, and add a trailing `:'
+ command.
+ * tests/Makefile.am (config_shell_tests): Add `mdate5-w.shtst'
+ and `mdate6-w.shtst'.
+
+2011-06-13 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: extend tests on `--add-missing' and `--copy' a bit
+ * tests/add-missing.test: Fix typo in heading comments. Try with
+ another testcase that install many (but not all) the auxiliary
+ scripts at once, and uses non-standard (but valid and documented)
+ setups (e.g., defining YACC in Makefile.am instead of calling
+ AC_PROG_YACC from configure.in).
+ * tests/copy.test: Reference `add-missing.test' in heading
+ comments. Try few more test scenarios.
+
+2011-06-16 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: few fixlets and improvements
+ * tests/cond31.test ($required): Remove `cc', it's not really
+ needed.
+ * tests/confh.test: Call autoheader too. The lack of this call
+ wasn't causing spurious failures because the automatic remake
+ rules were somehow invoking it on our behalf (at make time).
+ * tests/fn99subdir.test: Use $subdirname throughout, instead of
+ ${subdirname}, for consistency with the rest of the testsuite.
+ Avoid an unnecessary subshell, which could also cause spurious
+ passes, being guarded by a trailing `|| Exit 1', which neutralize
+ the `errexit' flag. Remove an unnecessary `|| Exit 1' guard.
+ * tests/insh2.test: Rewrite to avoid hackish Makefile.in munging,
+ and to also run configure and make.
+
+2011-06-13 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: don't hard-code test name in txinfo21.test
+ * tests/txinfo21.test: Use `$me' instead of hard-coding the
+ current testcase name "txinfo21". Add a trailing `:' command
+ since we are at it.
+
+2011-06-10 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: new test dedicated to `--add-missing' and `--copy'
+ * tests/add-missing.test: New test.
+ * tests/Makefile.am (TESTS): Update.
+ Suggested by Peter Rosin.
+
+2011-06-08 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: use 'fatal_' and 'framework_failure_' for hard errors
+ * tests/defs (require_xsi): Use `fatal_', not `framework_failure',
+ to report an invalid usage.
+ * tests/remake-gnulib-remove-header.test: Prefer using `fatal_'
+ with a proper error message over a direct call to `Exit 99'.
+ * tests/pr8365-remake-timing.test: Likewise.
+ * tests/cygnus-imply-foreign.test: Likewise.
+ * tests/missing6.test: Likewise.
+ * tests/cond8.test: Likewise.
+ * tests/cond33.test: Likewise.
+ * tests/python-virtualenv.test: Prefer using `framework_failure_'
+ with a proper error message over a direct call to `Exit 99'.
+ * tests/instspc-tests.sh: Prefer using `framework_failure_' and
+ `fatal_' over direct calls to `Exit 99'.
+ (fatal_): Define this (which is a simplified version of the one
+ in `tests/defs') for early uses (i.e., before `tests/defs'
+ gets sourced).
+ * tests/depmode-tests.sh: Likewise. Also, simplify the
+ 'get_depmodes' function and calls to it accordingly.
+
+2011-06-08 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ self tests: check new 'fatal_' function
+ * tests/self-check-exit.test: Also check the new 'fatal_'
+ function.
+
2011-06-08 Stefano Lattarini <stefano.lattarini@gmail.com>
test defs: new function 'fatal_', for hard errors
other reasons. The best way to fix this inconsistency is to
introduce a new function with a more general error message.
Inspired by a recent similar change to Gnulib's tests/init.sh.
- * tests/defs.in (fatal_): New function.
+ * tests/defs (fatal_): New function.
* tests/README (Section "Writing test cases" subsection "Do"):
Suggest the use of `fatal_', not of `framework_failure_', for
generic hard errors. The latter should be reserved for "real"
long lines.
(am__py_compile): New variable, to reduce code duplication.
+
+2011-06-07 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: `lib/' shell scripts transparently tested also with $SHELL
+ With the previous commit, the user could prefer the use of $SHELL
+ over /bin/sh in some tests checking the Automake-provided shell
+ scripts by manually exporting `test_prefer_config_shell' to "yes"
+ in the environment. With this commit, we ensure that such tests
+ *always* and *transparently* run using both $SHELL and /bin/sh
+ to execute the checked scripts. The `test_prefer_config_shell'
+ variable becomes an internal detail, and is no more meant to be
+ manually defined or overridden.
+ * tests/defs-static.in: Check that `test_prefer_config_shell' is
+ not exported in the environment. Error out if this is the case.
+ * tests/config-shell-tests.sh: New file, driver script to run
+ checks on the shell scripts in `lib/' using the $SHELL determined
+ at configure time instead of the default system shell /bin/sh.
+ * Makefile.am (TESTS_EXTENSIONS): Add `.shtst'.
+ (SHTST_LOG_COMPILER): Define, it calls `config-shell-tests.sh'.
+ (config_shell_tests): Define to a list of tests that wraps other
+ `*.test' tests using `config-shell-tests.sh'.
+ ($(config_shell_tests)): Dummy dependency declaration required
+ in order to have make actually produce expected log files from
+ the `.shtst.log' suffix rule.
+ (EXTRA_DIST): Distribute `config-shell-tests.sh'.
+ (TESTS): Add `$(config_shell_tests)'.
+ * tests/self-check-env-sanitize.test: Update, by checking that
+ `test_prefer_config_shell' isn't exported in the environment.
+
+2011-06-07 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: can use also $SHELL to check shell scripts from `lib/'
+ * tests/ar-lib.test: If the variable `$test_prefer_config_shell'
+ is set to "yes", run the script under test with configure-time
+ determined $SHELL, rather than with /bin/sh.
+ The `$test_prefer_config_shell' variable defaults to empty, but
+ can be overridden at runtime by the user, thus allowing more
+ coverage.
+ * tests/compile.test: Likewise.
+ * tests/compile2.test: Likewise.
+ * tests/compile3.test: Likewise.
+ * tests/compile4.test: Likewise.
+ * tests/compile5.test: Likewise.
+ * tests/compile6.test: Likewise.
+ * tests/instsh2.test: Likewise.
+ * tests/instsh3.test: Likewise.
+ * tests/mkinst3.test: Likewise.
+ * tests/missing.test: Likewise.
+ * tests/missing2.test: Likewise.
+ * tests/missing3.test: Likewise.
+ * tests/missing5.test: Likewise.
+ * tests/defs (get_shell_script): New subroutine, factoring out
+ code common to the tests above.
+ (xsi-lib-shell): If `$test_prefer_config_shell' is set to "yes",
+ check that $SHELL, not /bin/sh, supports XSI constructs, as we
+ expect the test will use $SHELL and not /bin/sh to run the
+ script being tested.
+
+2011-06-07 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests defs: better requirements for XSI shells
+ This change avoids potential spurious failures with tests using
+ the requirement 'xsi-shell' to mean that they want */bin/sh* (not
+ $SHELL) to be XSI-conforming. This idiom used to work before
+ commit `v1.11-874-g1321be7' (as back then the test scripts were
+ unconditionally run with /bin/sh), but has become inconsistent
+ now that the test scripts re-execute themselves with configure
+ determined $SHELL.
+ The described spurious failures have already occurred in practice,
+ for examples on Solaris systems which also had GNU Bash installed.
+ From a suggestion by Peter Rosin. See discussion at:
+ <http://lists.gnu.org/archive/html/automake-patches/2011-06/msg00016.html>
+ * tests/defs (xsi-shell): Now check that $SHELL, rather than the
+ shell currently running the test script, is an XSI shell.
+ (xsi-bin-sh): New requirement, checking that /bin/sh (which can
+ differ from $SHELL) is an XSI shell.
+ (xsi-lib-shell): New requirement, checking that the shell that
+ should be used to test the Automake-provided scripts from `lib/'
+ is an XSI shell. For the moment, this is just an alias for
+ `xsi-bin-sh'.
+ (require_xsi): New subroutine, used to factor out code common to
+ the requirements above.
+ ($xsi_shell_code): New variable, contains shell code supposed to
+ work only with XSI shells. Used by the new subroutine above.
+ * tests/ar-lib.test ($required): Require 'xsi-lib-shell' instead
+ of 'xsi-shell', since the script we test here is run with /bin/sh,
+ not with $SHELL.
+ * tests/compile3.test: Likewise.
+ * tests/compile6.test: Likewise.
+
2011-06-05 Stefano Lattarini <stefano.lattarini@gmail.com>
tests: fix typo-related error in auxdir2.test
its failure went unnoticed, even if it was due to a wrong cause.
Bug introduced in commit v1.11-249-g49ac3de.
+2011-06-05 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ self tests: check that `$me' can be overridden
+ * tests/self-check-me.test: Check that `$me' can be overridden
+ before sourcing ./defs, with or without sourcing ./defs-static
+ beforehand, and that this override is honored. Update heading
+ comments.
+
+2011-06-02 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ self tests: fix another spurious failure
+ Our ad-hoc usage of `tests/defs' in the testsuite's self tests
+ stopped working properly when we made the test scripts re-execute
+ themselves with the configure-time $SHELL. Fix this.
+ Fixes a regression introduced by commit 'v1.11-874-g1321be7'.
+ * tests/self-check-exit.test: Export `AM_TESTS_REEXEC' to "no"
+ before running the self tests. This fixes a spurious failure
+ present only when the test was run by hand.
+
+2011-06-02 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ silent-rules tests: fix spurious failures with Sun Studio C++
+ * tests/silentcxx.test: The C++ compiler from Sun Studio is named
+ `CC'. Account for this in our grepping checks on the make output.
+ Since we are at it, throw in a couple of improvements to comments
+ and formatting.
+ * tests/silent-many-generic.test: The C++ compiler from Sun Studio
+ is named `CC', and this can cause spurious failures in our grepping
+ of the make output. Work around this by using a wrapper script
+ around the C++ compiler (generated on the fly), since filtering the
+ make output proved to be too fragile.
+
+2011-06-02 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: fix spurious failure in backcompat2.test on NetBSD
+ * tests/backcompat2.test: Add trailing `:' in the body of a `for'
+ loop, in case the last command there might have an exit status
+ different 0. This work around `set -e' issues in some BSD shells,
+ e.g., NetBSD /bin/ksh.
+
+2011-06-02 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: fix spurious failure in autohdr3.test
+ * tests/autohder3.test (Makefile.am): Let `test' depend on `all',
+ so that config.h header is truly remade.
+ Remove now useless call to `$MAKE' when non-GNU make is in use.
+ Failure introduced by commit `v1.11-895-g5e62b96'.
+
+2011-06-02 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: fix spurious failure of cond29.test on NetBSD
+ * tests/cond29.test: Skip if we cannot safely limit the maximal
+ size of used virtual memory to 20K. This fixes a spurious failure
+ on NetBSD.
+ Bug introduced in commit `v1.11-885-g908d335'.
+
+2011-06-02 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ maintcheck: fix again few more failures
+ * tests/cond29.test: To please the `sc_tests_Exit_not_exit'
+ maintainer check, avoid using `sh -c "exit 0"' where a simple
+ `sh -c :' will do.
+ * tests/depmod-data.test: Use creative quoting to avoid
+ spuriously triggering the `sc_tests_Exit_not_exit' maintainer
+ check.
+
2011-06-02 Stefano Lattarini <stefano.lattarini@gmail.com>
maintcheck: fix some more failures
check.
* tests/depcomp8b.test: Likewise.
+2011-06-02 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: don't require GNU make where it's not strictly needed
+ * tests/autohdr3.test: Rewrite to be stricter when make is GNU
+ make. Drop the requirement of GNU make.
+ * tests/aclocal5.test: Drop GNU make requirement, it's not truly
+ needed. Add reference to similar tests `remake-subdir*.test'.
+ * remake-subdir-gnu.test, remake-subdir-from-subdir.test,
+ remake-subdir.test, remake-subdir2.test: Add reference to
+ each other, and to related test `aclocal5.test'.
+ * tests/aclocal6.test: Drop GNU make requirement, it's not
+ truly needed.
+ * tests/confh6.test: Likewise.
+ * tests/lex3.test: Likewise.
+ * tests/remake11.test: Likewise.
+ * tests/subdir5.test: Likewise.
+ * tests/subdir8.test: Likewise.
+ * tests/werror2.test: Likewise.
+ * tests/conff.test: Likewise, and ensure verbose printing of
+ captured make output.
+ * tests/lex5.test: Tweak so that GNU make is no more required.
+ * tests/version7.test: Likewise.
+ * tests/maken2.test: Add explicative comment for why this test
+ requires GNU make.
+ * tests/maken4.test: Let it run also with BSD makes supporting
+ the `.MAKE' special target.
+ * tests/output6.test: Use proper m4 quoting in configure.in.
+ Expand make macros with one-character name using `$(x)', not
+ `$x', for portability. Move checks in the makefiles, rather
+ than relying on grepping the output from make. Drop the now
+ unneeded GNU make requirement.
+
2011-05-29 Stefano Lattarini <stefano.lattarini@gmail.com>
remake: behave better with non-GNU make in subdirectories
* tests/remake-subdir-from-subdir.test: Likewise.
* tests/Makefile.am (TESTS): Update.
+2011-06-01 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ lex tests: fix spurious failures with Solaris lex
+ * tests/lex-lib.test (foo.l): Avoid empty "rules section", which
+ can confuse Solaris lex.
+ * tests/lex-libobj.test (foo.l): Likewise.
+
+2011-06-01 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ lex tests: do not force the use of flex unconditionally
+ * tests/defs (lex): Act more similarly to what the `yacc'
+ requirement does, i.e., only force the use of flex if the
+ $LEX variable is left unset by the user.
+ (flex): Use `skip_' to skip the test if flex is not found.
+
+2011-06-01 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ lex tests: avoid possible hang; fix and extend
+ * tests/lex3.test (foo.l:yywrap): Return 1, not 0, to avoid hangs.
+ Bug introduced in commit 'v1.11-871-geb147a1'.
+ (Makefile.am): Do not add `@LEXLIB@' to `$(LDADD)', as we define
+ our own `yywrap' function.
+ * tests/lex.test (tscan.l): In `yywrap', return 1, not 0, for
+ consistency with the default flex implementation.
+ * tests/lex-libobj.test (yywrap.c): Likewise.
+ * tests/lex-subobj-nodep.test (s1.l): Likewise.
+ * tests/lexvpath.test (foo.c): Likewise.
+ * tests/silent-lex-gcc (foo.l): Likewise.
+ * tests/silent-lex-generic (foo.l): Likewise.
+ * tests/silent-many-gcc (foo5.l): Likewise.
+ * tests/silent-many-generic (foo5.l): Likewise.
+ * tests/lex-lib.test (mu.c): Likewise.
+ Update heading comments, to refer to ...
+ * tests/lex-lib-external.test: ... this new test, which checks
+ that we can get use the `yywrap' function from a system-wide
+ library, if that's available.
+
+2011-06-01 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: prefer `skip_' over `echo ...; Exit 77'
+ * tests/self-check-cleanup.test: When the test must be skipped,
+ use `skip_ REASON' instead of `echo REASON; Exit 77'. Also,
+ make the skip message shorter and clearer.
+
+2011-06-01 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: fix spurious failures in self tests
+ Our ad-hoc usage of `tests/defs' in the testsuite's self tests
+ stopped working properly when we made the test scripts re-execute
+ themselves with the configure-time $SHELL. Fix this.
+ Fixes a bug introduced by commit 'v1.11-874-g1321be7'.
+ * tests/defs: Only check that we can find the client test script
+ when we must re-execute it.
+ * tests/self-check-cleanup.test: Export `AM_TESTS_REEXEC' to "no"
+ before running the self tests.
+ * tests/self-check-dir.test: Likewise.
+ * tests/self-check-explicit-skips.test: Likewise.
+ * tests/self-check-me.test: Likewise.
+ * tests/self-check-sanity.test: Likewise.
+ * tests/self-check-reexec.test: New test.
+ * tests/Makefile.am (TESTS): Update.
+
+2011-06-01 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: prefer ulimit over timeout in test on a memory-hogging bug
+ * tests/cond29.test: Use 'ulimit' instead of 'timeout' to ensure
+ that automake does not uses up too much resources. This is really
+ and improvement because the bug tested by this script wasn't just
+ a "it hangs" or "it runs too slow" bug, but rather a memory-hogging
+ bug (due to combinatorial explosion when many Automake conditionals
+ had to be handled) which could easily crash the whole system, which
+ is unacceptable. The requirement of a proper and working 'ulimit'
+ builtin might cause the test to be skipped on more systems, but
+ that shouldn't be a problem since the bug isn't about a portability
+ issue, but is rather an automake internal implementation problem.
+ I've verified that the test as updated by this patch still passes
+ with automake 1.8.5, automake 1.10.2, and obviously the development
+ version of automake, and that it fails with automake 1.7.9.
+
+2011-06-01 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: avoid generating `*-p.test' tests, use a wrapper script
+ The generated `*-p.test' tests had already become just thin layers
+ around the corresponding test scripts. This change makes the final
+ step, converting to the use of a generic wrapper script and thus
+ avoiding the extra test generation (similarly to what is done for
+ the `*.instspc' and `*.depmod' tests).
+ * tests/parallel-tests.sh: New file, driver script to run checks
+ the on the `parallel-tests' semantics by wrapping tests that use
+ the generic "Simple Tests" driver.
+ * tests/gen-parallel-tests: Update, mostly to reflect the new
+ `.ptest' extensions used for tests in $(parallel_tests).
+ * Makefile.am (TESTS_EXTENSIONS): Add `.ptest'.
+ (PTEST_LOG_COMPILER): Define, it calls `parallel-tests.sh'.
+ ($(parallel_tests)): Do not really generate `*-p.test' tests
+ anymore; this is now just a dummy dependency declaration required
+ in order to have make actually produce expected log files from
+ the `.ptest.log' suffix rule.
+ (EXTRA_DIST): Distribute `parallel-tests.sh'.
+ (MAINTAINERCLEANFILES): Don't remove the `$(parallel_tests)', it
+ is not necessary anymore.
+ (generated_tests): Variable definition removed.
+ (TESTS): Update, by listing `$(parallel_tests)' directly instead
+ of `$(generated_tests)'
+ (expected_list_of_tests): Remove `$(generated_tests)'.
+ (maintainer-check-list-of-tests): No need to explicitly depend on
+ `$(expected_list_of_tests)' anymore.
+
2011-05-29 Stefano Lattarini <stefano.lattarini@gmail.com>
automake, aclocal: honour configure-time AUTOCONF and AUTOM4TE
now redundant.
* tests/Makefile.am (TESTS): Update.
-2011-05-22 Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
- Stefano Lattarini <stefano.lattarini@gmail.com>
-
- tests/README: fix example about `make -e' usage
- * tests/README (Section "Writing test cases" subsection "Do"): When
- some variable is never initialized in the Makefile, `-e' is not
- necessary in order to override it. DESTDIR is such a variable: we
- ensure that we do not ever initialize it. And as such, it is quite
- portable to use:
- $ make DESTDIR=/foo/bar install
- and in fact, quite widely used.
- So our example about when `make -e' is required, which references
- the `DESTDIR' variable, is poorly chosen, if not downright wrong.
- Rewrite it to use `prefix' as the overridden variable instead.
-
-2011-05-27 Stefano Lattarini <stefano.lattarini@gmail.com>
-
- testsuite: each test case depends on `defs-static'
- * tests/Makefile.am ($(TEST_LOGS)): Depends on `defs-static' too.
- Simplify comments.
+2011-05-22 Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
+ Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests/README: fix example about `make -e' usage
+ * tests/README (Section "Writing test cases" subsection "Do"): When
+ some variable is never initialized in the Makefile, `-e' is not
+ necessary in order to override it. DESTDIR is such a variable: we
+ ensure that we do not ever initialize it. And as such, it is quite
+ portable to use:
+ $ make DESTDIR=/foo/bar install
+ and in fact, quite widely used.
+ So our example about when `make -e' is required, which references
+ the `DESTDIR' variable, is poorly chosen, if not downright wrong.
+ Rewrite it to use `prefix' as the overridden variable instead.
+
+2011-05-29 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: don't require gfortran if any fortran compiler is enough
+ * tests/silentf90.test ($required): Require `fortran', not
+ `gfortran'.
+ * tests/silentf77.test ($required): Require `fortran77', not
+ `gfortran'.
+ * tests/silent-many-generic.test ($required): Require `fortran'
+ and `fortran77' rather than `gfortran'.
+
+2011-05-27 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: improve `ccnoco*.test', better cross-compiling support
+ * tests/ccnoco3.test (Mycomp): Use the `$CC' chosen by `tests/defs'
+ instead of forcing `gcc' unconditionally. This ensures better
+ coverage in case of cross-compiling, when GCC can be named e.g.,
+ `i586-mingw32msvc-gcc', instead of simply `gcc'.
+ * tests/ccnoco.test: Likewise. Remove redundant checks. Modernize
+ the created `configure.in'. Run tests both in-tree and in VPATH.
+ Export `CC' to the overridden value only once.
+ * tests/ccnoco2.test: Slightly stricter grepping of automake
+ stderr. Add trailing `:' command.
+
+2011-05-27 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: each test case depends on `defs-static'
+ * tests/Makefile.am ($(TEST_LOGS)): Depends on `defs-static' too.
+ Simplify comments.
+
+2011-05-27 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: use $SHELL to run tests which are shell scripts
+ * tests/Makefile.am (TEST_LOG_COMPILER): Define so that the
+ configure-time $SHELL is used to run the tests.
+ * tests/defs: Add code to re-execute by default the running
+ test script with configure-time $SHELL. Updated comments.
+ * configure.ac: Check that `set -e' is working for $SHELL,
+ not for /bin/sh.
+ * tests/defs-static.in: Update comments.
+ * tests/README (Supported shells): Updated.
+ (Getting details from failures): Don't tell that tests are
+ run by /bin/sh by default.
+
+2011-05-26 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: split 'subst2.test' to improve modularity and coverage
+ * tests/subst2.test: Test removed, split into ...
+ * tests/programs-primary-rewritten.test: ... this test ...
+ * tests/subst-no-trailing-empty-line.test: ... and this one ...
+ * tests/extra-programs-empty.test: ... and this one.
+ * tests/Makefile.am (TESTS): Update.
+
+2011-05-26 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: tweak and improve tests on Automake conditionals
+ The "avoid the requirement of a working compiler" we refer about
+ below is obtained by disabling automatic dependency tracking and
+ defining `CC' (and `EXEEXT', `OBJEXT', `CXX', etc., if needed) to
+ dummy values, either directly in the `Makefile.am', or by AC_SUBST
+ in `configure.in'.
+ The "cosmetic changes" we refer about below consists in, e.g.,
+ adding a trailing `:' command to a script, using proper m4 quoting
+ in `configure.in' files, adding commands that offer more debugging
+ output, explicitly declaring phony targets as such in Makefiles,
+ making use of idioms more consistent with those employed in other
+ tests, following the GNU formatting standards more closely, etc.
+ * tests/cond.test: Cosmetic changes.
+ * tests/cond3.test: Likewise.
+ * tests/cond10.test: Likewise.
+ * tests/cond14.test: Likewise.
+ * tests/cond15.test: Likewise.
+ * tests/cond17.test: Likewise.
+ * tests/cond40.test: Likewise.
+ * tests/cond13.test: Likewise, and make grepping of the generated
+ `Makefile.in' slightly stricter.
+ * tests/cond6.test: Extend by also running ./configure, make and
+ "make install". Few cosmetic changes.
+ * tests/cond8.test: Likewise.
+ * tests/cond2.test: Do few cosmetic changes. Make grepping of
+ automake stderr stricter.
+ * tests/cond20.test: Likewise.
+ * tests/cond23.test: Likewise.
+ * tests/cond24.test: Likewise.
+ * tests/cond42.test: Likewise.
+ * tests/cond46.test: Likewise.
+ * tests/cond9.test: Move more checks in the `Makefile.am' instead
+ of grepping make's output.
+ * tests/cond38.test: Likewise.
+ * tests/cond11.test: Likewise. Avoid the requirement of a working
+ C compiler.
+ * tests/cond16.test: Likewise.
+ * tests/cond22.test: Likewise.
+ * tests/cond30.test: Likewise.
+ * tests/cond31.test: Likewise.
+ * tests/cond4.test: Likewise, and drop the now-useless GNU make
+ requirement as well.
+ * tests/cond18.test: Likewise.
+ * tests/cond19.test: Likewise.
+ * tests/cond22.test: Likewise.
+ * tests/cond25.test: Add trailing `:' command.
+ * tests/cond26.test: Likewise.
+ * tests/cond27.test: Likewise.
+ * tests/cond28.test: Likewise.
+ * tests/cond29.test: Use `unindent' function for better formatting.
+ Other minor cosmetic changes. Use a `timeout' program (if it's
+ available) to determine whether the script takes too long, instead
+ of just hoping that the user will notice an abnormally long test
+ execution time.
+ * tests/cond33.test: Extend a bit. Fix heading comments. Few more
+ cosmetic changes.
+ * tests/cond37.test: Ensure verbose printing of captured make
+ output. Minor cosmetic changes.
+ * tests/cond34.test: Likewise. Avoid the requirement of a working
+ C compiler. Make grepping of make output slightly stricter.
+ * tests/cond35.test: Quote literal dots in grep regexp. Add extra
+ debugging output. Minor cosmetic changes.
+ * tests/cond36.test: Likewise.
+ * tests/cond41.test: Do not simply check that aclocal fails, but
+ also grep its stderr for the expected error message.
+ * tests/cond43.test: Likewise (but the program being automake).
+ Few cosmetic changes.
+ * tests/condd.test: Avoid the requirement of a working C compiler.
+ * tests/condman3.test: Extend by using more man pages and more
+ manpage sections.
+ * tests/condman.test: Removed, it's completely superseded by
+ `condman3.test'.
+ * tests/Makefile.am (TESTS): Update.
+
+2011-05-25 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ lex tests: avoid spurious failures when LEXLIB isn't found
+ The AC_PROG_LEX Autoconf macro does not diagnose a failure to find
+ the "lex library" expected to provide a `yywrap' function (function
+ which is required to link most lex-generated programs). On the
+ contrary, when all the link attempts (i.e., with `-ll' and `-lfl')
+ fail, configure declares that no lex library is needed, and simply
+ proceeds with the configuration process -- only for the build to
+ possibly fail later, at make time.
+ This behaviour might be (partly) intended; the Autoconf manual
+ reads:
+ ``You are encouraged to use Flex in your sources, since it is
+ both more pleasant to use than plain Lex and the C source
+ it produces is portable. In order to ensure portability,
+ however, you must either provide a function `yywrap' or, if
+ you don't use it (e.g., your scanner has no `#include'-like
+ feature), simply include a `%noyywrap' statement in the
+ scanner's source.''
+ This AC_PROG_LEX behaviour is causing some spurious failures of the
+ Automake testsuite in environments which lack a proper library
+ providing `yywrap' (this happens for example in Linux->MinGW cross
+ compilations). But at this point is clear that a proper workaround
+ is to simply provide a fall-back implementation of `yywrap' in our
+ lexers.
+ * tests/cond35.test: Provide a dummy `yywrap' function.
+ * tests/lex3.test: Likewise.
+ * tests/lexvpath.test: Likewise.
+ * tests/silent-many-gcc.test: Likewise.
+ * tests/silent-many-generic.test: Likewise.
+ * tests/silent-lex-gcc.test: Likewise, and a dummy `main' too.
+ * tests/silent-lex-generic.test: Likewise.
+ * tests/lex-lib.test: New test.
+ * tests/lex-libobj.test: New test.
+ * tests/lex-nowrap.test: New test.
+ * tests/Makefile.am (TESTS): Update.
+ * THANKS: Update.
+ Thanks to Russ Allbery for the suggestion.
+
+2011-05-25 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: require C++ compiler explicitly in tests needing it
+ The list of the test scripts which needed a C++ compiler but
+ failed to require it explicitly has been found by running:
+ $ tests=`grep -lE 'CXX|\.c(c|\+\+|xx|pp)' *.test | tr '\n' ' '`
+ $ am_explicit_skips=yes CXX=false make check TESTS="$tests"
+ and looking for tests that reported FAIL instead of SKIP.
+ After the present change, all those previously failing tests
+ either pass or get correctly skipped.
+ * tests/subobj9.test: Require `c++', instead of explicitly
+ skipping on configure failure. Add excerpts and/or details from
+ the original bug report that prompted this tests to be written.
+ * tests/silentcxx-gcc.test: Require g++.
+ * tests/silentcxx.test: Require c++.
+ * tests/suffix3.test: Likewise.
+
+2011-05-25 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: don't require g++ where any C++ compiler is enough
+ * tests/specflg10.test ($required): Use `c++', not g++.
+ * tests/silent-many-generic.test: Likewise.
+
+2011-05-25 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: require C compiler explicitly in tests needing it
+ The list of the test scripts which needed a C compiler but
+ failed to require it explicitly has been found by running:
+ $ am_explicit_skips=yes CC=false make check
+ and looking for tests that reported FAIL instead of SKIP.
+ After the present change, all those previously failing tests
+ either pass or get correctly skipped.
+ * tests/aclocal4.test ($required): Add `cc'.
+ * tests/ansi.test: Likewise.
+ * tests/ansi10.test: Likewise.
+ * tests/ansi3.test: Likewise.
+ * tests/ansi3b.test: Likewise.
+ * tests/ansi5.test: Likewise.
+ * tests/ansi6.test: Likewise.
+ * tests/ansi7.test: Likewise.
+ * tests/ansi9.test: Likewise.
+ * tests/backcompat6.test: Likewise.
+ * tests/canon7.test: Likewise.
+ * tests/check5.test: Likewise.
+ * tests/check7.test: Likewise.
+ * tests/check8.test: Likewise.
+ * tests/cond4.test: Likewise.
+ * tests/cond11.test: Likewise.
+ * tests/cond16.test: Likewise.
+ * tests/cond18.test: Likewise.
+ * tests/cond19.test: Likewise.
+ * tests/cond22.test: Likewise.
+ * tests/cond30.test: Likewise.
+ * tests/cond31.test: Likewise.
+ * tests/cond32.test: Likewise.
+ * tests/cond34.test: Likewise.
+ * tests/cond35.test: Likewise.
+ * tests/cond39.test: Likewise.
+ * tests/condd.test: Likewise.
+ * tests/cygnus-dependency-tracking.test: Likewise.
+ * tests/depcomp2.test: Likewise.
+ * tests/depcomp6.test: Likewise.
+ * tests/depcomp7.test: Likewise.
+ * tests/depcomp8a.test: Likewise.
+ * tests/depcomp8b.test: Likewise.
+ * tests/depdist.test: Likewise.
+ * tests/depend2.test: Likewise.
+ * tests/depend5.test: Likewise.
+ * tests/distcleancheck.test: Likewise.
+ * tests/distname.test: Likewise.
+ * tests/exeext.test: Likewise.
+ * tests/exeext4.test: Likewise.
+ * tests/extradep.test: Likewise.
+ * tests/extradep2.test: Likewise.
+ * tests/gnits2.test: Likewise.
+ * tests/gnits3.test: Likewise.
+ * tests/instdir-ltlib.test: Likewise.
+ * tests/instdir-prog.test: Likewise.
+ * tests/instfail.test: Likewise.
+ * tests/instfail-libtool.test: Likewise.
+ * tests/lex3.test: Likewise.
+ * tests/lex5.test: Likewise.
+ * tests/lexvpath.test: Likewise.
+ * tests/lex-subobj-nodep.test: Likewise.
+ * tests/lflags.test: Likewise.
+ * tests/libobj-basic.test: Likewise.
+ * tests/libobj2.test: Likewise.
+ * tests/libobj7.test: Likewise.
+ * tests/libobj10.test: Likewise.
+ * tests/libobj16a.test: Likewise.
+ * tests/libobj16b.test: Likewise.
+ * tests/libobj17.test: Likewise.
+ * tests/libobj19.test: Likewise.
+ * tests/libtool2.test: Likewise.
+ * tests/libtool3.test: Likewise.
+ * tests/libtool7.test: Likewise.
+ * tests/libtool9.test: Likewise.
+ * tests/libtoo10.test: Likewise.
+ * tests/libtoo11.test: Likewise.
+ * tests/ltcond.test: Likewise.
+ * tests/ltcond2.test: Likewise.
+ * tests/ltconv.test: Likewise.
+ * tests/ltinit.test: Likewise.
+ * tests/ltlibsrc.test: Likewise.
+ * tests/ltorder.test: Likewise.
+ * tests/nobase.test: Likewise.
+ * tests/nobase-libtool.test: Likewise.
+ * tests/mmodely.test: Likewise.
+ * tests/parallel-tests5.test: Likewise.
+ * tests/parallel-tests-suffix-prog.test: Likewise.
+ * tests/parallel-tests-ext-driver-prog.test: Likewise.
+ * tests/posixsubst-ldadd.test: Likewise.
+ * tests/posixsubst-libraries.test: Likewise.
+ * tests/posixsubst-ltlibraries.test: Likewise.
+ * tests/posixsubst-programs.test: Likewise.
+ * tests/posixsubst-sources.test: Likewise.
+ * tests/pr87.test: Likewise.
+ * tests/pr204.test: Likewise.
+ * tests/pr224.test: Likewise.
+ * tests/pr300-lib.test: Likewise.
+ * tests/pr300-ltlib.test: Likewise.
+ * tests/pr300-prog.test: Likewise.
+ * tests/pr401.test: Likewise.
+ * tests/pr401b.test: Likewise.
+ * tests/pr401c.test: Likewise.
+ * tests/remake-gnulib-add-acsubst.test: Likewise.
+ * tests/remake-gnulib-add-header.test: Likewise.
+ * tests/regex.test: Likewise.
+ * tests/repeated-options.test: Likewise.
+ * tests/silent.test: Likewise.
+ * tests/silent3.test: Likewise.
+ * tests/silent9.test: Likewise.
+ * tests/silent-lex-generic.test: Likewise.
+ * tests/silent-many-generic.test: Likewise.
+ * tests/silent-yacc-generic.test: Likewise.
+ * tests/specflg7.test: Likewise.
+ * tests/specflg9.test: Likewise.
+ * tests/specflg10.test: Likewise.
+ * tests/stdinc.test: Likewise.
+ * tests/strip.test: Likewise.
+ * tests/strip2.test: Likewise.
+ * tests/strip3.test: Likewise.
+ * tests/subdirbuiltsources.test: Likewise.
+ * tests/subobj3.test: Likewise.
+ * tests/subobj6.test: Likewise.
+ * tests/subobj11a.test: Likewise.
+ * tests/subpkg.test: Likewise.
+ * tests/subst2.test: Likewise.
+ * tests/subst3.test: Likewise.
+ * tests/substref.test: Likewise.
+ * tests/substre2.test: Likewise.
+ * tests/suffix5.test: Likewise.
+ * tests/suffix8.test: Likewise.
+ * tests/suffix10.test: Likewise.
+ * tests/suffix11.test: Likewise.
+ * tests/suffix12.test: Likewise.
+ * tests/suffix13.test: Likewise.
+ * tests/target-cflags.test: Likewise.
+ * tests/transform.test: Likewise.
+ * tests/transform2.test: Likewise.
+ * tests/yacc-basic.test: Likewise.
+ * tests/yacc-d-basic.test: Likewise.
+ * tests/yacc-clean.test: Likewise.
+ * tests/yacc-dist-nobuild.test: Likewise.
+ * tests/yacc-nodist.test: Likewise.
+ * tests/yaccvpath.test: Likewise.
+ * tests/yacc-d-vpath.test: Likewise.
+ * tests/yacc4.test: Likewise.
+ * tests/yacc7.test: Likewise.
+ * tests/yacc8.test: Likewise.
+ * tests/yaccdry.test: Likewise.
+ * tests/yflags-cmdline-override.test: Likewise.
+ * tests/yflags-force-override.test: Likewise.
+ * tests/python-virtualenv.test: Likewise. Also, improve skip
+ messages.
+ * tests/subobj5.test ($required): Add `cc'.
+ (Makefile.am): Set `AUTOMAKE_OPTIONS' to `subdir-objects', and
+ add new checking rules `test-build' and `test-distdir'.
+ Extend the test by building and examining the distdir, the
+ program, and the object files.
+ * tests/postproc.test ($required): Add `cc'.
+ Avoid the explicit `|| Exit $?' after call to configure, which
+ is now either redundant (e.g., when `am_explicit_skips' is
+ unset), or counter-productive (e.g., when `am_explicit_skips'
+ is set to "yes").
+ * tests/pr243.test: Likewise.
+ * tests/pr266.test: Likewise.
+ * tests/pr220.test: Simplify so that it doesn't require a
+ C compiler anymore.
+ * tests/subdir5.test: Likewise.
+ * tests/subdir8.test: Likewise.
+ * tests/lflags.test: Likewise.
+ * tests/yflags.test: Likewise.
+ * tests/yflags-force-conditional.test: Likewise.
+ * tests/lflags2.test: Simplify so that it doesn't require a
+ C++ compiler anymore.
+ * tests/yflags2.test: Likewise.
+ * tests/autohdrdry.test (configure.in): Remove unneeded call
+ to `AC_PROG_CC'.
+ * tests/pr287.test: Likewise.
+ * tests/check6.test: Likewise.
+ * tests/cond21.test: Likewise, plus some cosmetic adjustments.
+ * tests/upc.test: Skip if configure fails with status `77'.
+ * tests/upc3.test: Likewise.
+ * tests/vala4.test: Likewise.
+ * tests/nostdinc.test: Likewise. Also, make grepping checks
+ on `Makefile.in'. Update heading comments.
+ * tests/compile5.test: When we must skip, skip explicitly,
+ and with a meaningful message.
+ * tests/instspc-tests.sh: When running in "test-build" or
+ "test-install" mode (as determined by the value of variable
+ `$instspc_action'), require `cc'. Adjust comments.
+ * tests/depmod-tests.sh: Likewise.
+
+2011-05-24 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: allow user to ask for stricter skip semantics
+ Given how the Automake testsuite is currently structured, if a
+ command in a test script fails with status `77', the global test
+ outcome is considered a SKIP, because the value of `77' for `$?'
+ is passed to the exit trap. Indeed, this happens in practice, as
+ an autoconf-generated configure script can exit with status `77'
+ if it fails to find, e.g., a required compiler.
+ While this behaviour is quite useful for avoiding spurious test
+ failures in the wild, it can also sometimes prevent the Automake
+ developers to easily see and declare the requirements of their
+ tests.
+ This change introduces a new variable `am_explicit_skips', meant to
+ be user-overridable, and which, when set to a "true" value (i.e.,
+ `yes' or `1'), require a direct call to `Exit 77' in order to make
+ the test outcome be considered a SKIP.
+ * tests/defs.in ($am__test_skipped): New variable, initialized
+ to `no'.
+ (Exit): Set `$am__test_skipped' to `yes' if passed an exit status
+ of 77.
+ (trap '...' 0): When `$am_explicit_skips' is set to a "true" value,
+ reset an exit status of `77' to `78' if $am__test_skipped is not
+ set to `yes'.
+ * tests/self-check-exit.test: Adjust: unset `am_explicit_skips'.
+ * tests/self-check-explicit-skips.test: New test.
+ * tests/Makefile.am (TESTS): Update.
+
+2011-05-24 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: user can force skipping of compiler-requiring tests
+ * tests/defs (cc, c++, fortran, fortran77): Skip the test if
+ the relevant compiler is disabled by having the corresponding
+ variable (CC, CXX, FC and F77, respectively) set to "false".
+ (yacc): For consistency, skip the test when the YACC variable
+ is set to "false", not when it's set to "no". Since we are at
+ it, fix the skip message to be shorter and more consistent.
+
+2011-05-23 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: tweak and improve tests on "Simple Tests" driver
+ * tests/check.test: Make grepping of generated Makefile.in
+ slightly stricter. Add trailing `:' command.
+ * tests/check2.test: Ensure verbose printing of captured make
+ output. Make grepping of generated Makefile.in and of make
+ output stricter. Add trailing `:' command.
+ * tests/check3.test: Likewise. Also, fix minor error in a
+ comment.
+ * tests/check4.test: Stricter grepping of make output. Use
+ `using_gmake' subroutine instead of copy&paste from defs.in
+ to determine whether $MAKE is GNU make. Add a trailing `:'
+ command.
+ * tests/check6.test: Add trailing `:' command.
+ * tests/check10.test: Prefer redirection `>' over `>>' when
+ creating new files. Ensure verbose printing of captured make
+ output.
+ * tests/check11.test: Likewise.
+ * tests/check7.test: Likewise. Make grepping of make output
+ stricter. Prefer "make distcheck" over a simple "make check"
+ in a VPATH build. Add trailing `:' command.
+ (a.c): Format following GNU coding standards.
+ (Makefile.am): Explicitly declare target `print-xfail-tests'
+ as PHONY. Distribute $(check_SCRIPTS), for distcheck.
+ * tests/parallel-tests.test: Prefer redirection `>' over `>>'
+ when creating new files.
+ * tests/parallel-tests2.test: Likewise.
+ * tests/parallel-tests9.test: Likewise.
+ * tests/parallel-tests8.test: Likewise. Make grepping of
+ automake stderr stricter.
+ * tests/check8.test: Likewise. Make formatting of created
+ *.c files more consistent with GNU Coding Standards.
+ * tests/check-subst-prog.test (configure.in): Break overly
+ long lines.
+ (pass-prog.c, xfail-prog.c): Format them more consistently
+ with GNU Coding Standards.
+ * tests/parallel-tests3.test: Use a shorter and clearer skip
+ message. Use the `unindent' subroutine to improve readability
+ of indented loop bodies. Simplify the hack used to please
+ maintainer-check.
+ * tests/parallel-tests5.test: Make formatting of created *.c
+ files more consistent with GNU Coding Standards. Use the
+ `unindent' subroutine to improve readability of indented loop
+ bodies.
+ * tests/parallel-tests10.test: Add blank line, for clarity.
+ Prefer trailing `:' over trailing `Exit 0', for consistency
+ with other tests.
+
+2011-05-21 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests/README: small consistency improvement
+ * tests/README (Section "User interface" subsection "Running the
+ tests"): In examples, be more consistent w.r.t. the differences
+ between GNU and non-GNU make.
+
+2011-05-21 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests/README: suggest how to run tests in cross-compile mode
+ * tests/README (Section "User interface" subsection "Running the
+ tests"): Briefly explain how to override 'host_alias' at runtime
+ to force the use of cross-compilers by the testsuite. Give an
+ example.
+
+2011-05-21 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ test defs: better detection of cross-compile mode
+ * configure.ac (AC_CANONICAL_HOST): New, probably not strictly
+ necessary, but useful to complements AC_CANONICAL_BUILD.
+ * tests/defs-static.in ($host_alias, $build_alias): New variables,
+ exported, user-overridable, defaults AC_SUBST'd by configure.
+ * tests/defs (cross_compiling): Consider cross-compilation mode
+ active only if `$host_alias' is not empty *and* it differs from
+ `$build_alias'. This is more faithful to the way configure
+ determines whether it is cross-compiling, but still allows the
+ user to easily force cross-compilation by overriding `host_alias'
+ and `build_alias' at runtime.
+ From a report by Ralf Wildenhues.
2011-05-20 Stefano Lattarini <stefano.lattarini@gmail.com>
and 'ANSI2KNR' in the example; instead, use the more common
and typical `DESTDIR'.
+2011-05-18 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: avoid spurious failures in cross-compile mode
+ * tests/depcomp2.test: Ensure verbose printing of captured stderr
+ from configure.
+ * tests/ansi3.test ($required): Add 'native', as the ansi2knr
+ functionality is not meant to work with a cross-compiler.
+ * tests/ansi3b.test: Likewise.
+ * tests/ansi5.test: Likewise.
+ * tests/ansi6.test: Likewise.
+ * tests/ansi7.test: Likewise.
+ * tests/ansi8.test: Likewise.
+ * tests/ansi10.test: Likewise.
+ * tests/subobj3.test: Likewise.
+ * tests/check8.test ($required): Add 'native', as this test is
+ not meant to work with a cross-compiler.
+ * tests/gnits2.test: Likewise.
+ * tests/ltconv.test: Likewise.
+ * tests/ltcond2.test: Likewise.
+ * tests/parallel-tests5.test: Likewise.
+ * tests/specflg7.test: Likewise.
+ * tests/specflg8.test: Likewise.
+ * tests/check5.test: Likewise. Also, ensure verbose printing of
+ captured make output, and thrown in few cosmetic and consistency
+ improvements.
+ * tests/ltinit.test: Be laxer in grepping configure output, to
+ avoid spurious failures on systems which lack POSIX dynamic
+ linking (e.g., MinGW), or when cross-compiling for such systems.
+ * tests/ansi9.test: Skip parts of the test that would require a
+ native compiler.
+ * tests/backcompat6.test: Likewise.
+ * tests/depcomp8a.test: Likewise.
+ * tests/pr401.test: Likewise.
+ * tests/pr401b.test: Likewise.
+ * tests/pr401c.test: Likewise.
+ * tests/target-cflags.test: Likewise.
+ * tests/lex3.test: Likewise.
+ * tests/yacc-basic.test: Likewise.
+ * tests/autohdr4.test: Likewise. Also, extend the test to ensure
+ more coverage in cross-compile mode.
+ * tests/canon7.test: Likewise.
+ * tests/check9.test: Removed, split into two new tests ...
+ * tests/check-subst-prog.test: ... this one, which requires a
+ native C compiler ...
+ * tests/check-subst.test: ... and this one, which doesn't require
+ any compiler.
+ * tests/parallel-tests4.test: Removed, split into two new tests ...
+ * tests/parallel-tests-suffix-prog.test: ... this one, which
+ requires a native C compiler ...
+ * tests/parallel-tests-suffix.test: ... and this one, which
+ doesn't require any compiler.
+ * tests/parallel-tests7.test: Removed, split into two new tests ...
+ * tests/parallel-tests-ext-driver-prog.test: ... this one, which
+ requires a native C compiler ...
+ * tests/parallel-tests-ext-driver.test: ... and this one, which
+ doesn't require any compiler.
+ * tests/Makefile.am (TESTS): Update.
+
2011-05-19 Stefano Lattarini <stefano.lattarini@gmail.com>
test defs: rename requirement 'non-cross' -> 'native'
2011-05-07 Stefano Lattarini <stefano.lattarini@gmail.com>
+ tests: fix spurious failure of extradep.test on FreeBSD
+ * tests/extradep.test: When using `ls -t' to determine whether a
+ file has been updated, make sure to use as reference a file whose
+ timestamp is expected to be *strictly* older that that of the file
+ being checked. This is required because at least FreeBSD `ls' do
+ not sort files with the same timestamp in alphabetical order when
+ using the `-t' option.
+ * tests/extradep2.test: Likewise.
+
+2011-05-07 Stefano Lattarini <stefano.lattarini@gmail.com>
+
tests: fix spurious failure of txinfo21.test on FreeBSD
* tests/txinfo21.test: Use the `is_newest' subroutine instead of
the `ls -t' hack to to determine whether a file has been updated.
2011-05-07 Stefano Lattarini <stefano.lattarini@gmail.com>
- tests: fix spurious failure of extradep.test on FreeBSD
- * tests/extradep.test: When using `ls -t' to determine whether a
- file has been updated, make sure to use as reference a file whose
- timestamp is expected to be *strictly* older that that of the file
- being checked. This is required because at least FreeBSD `ls' do
- not sort files with the same timestamp in alphabetical order when
- using the `-t' option.
- * tests/extradep2.test: Likewise.
+ tests: fix syntax error in 'silentcxx-gcc.test'
+ * tests/silentcxx-gcc.test (for config_args in ...): Remove extra
+ trailing backslash from the list of looped-on items.
+
+2011-05-07 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ depmod tests: fix bug in depmodes list extraction
+ * tests/depmod-tests.sh (get_depmodes): Use `echo $all_depmodes'
+ instead of `... | tr "$nl" "$sp"' to strip extra whitespaces and
+ newlines from `$all_depmodes'. The latter idiom with `tr' is
+ wrong since it gets run before `./defs' is sourced, and thus `$nl'
+ and `$sp' are undefined.
+ Bug revealad by FreeBSD tr(1) implementation, which doesn't accept
+ empty strings as arguments.
2011-05-04 Stefano Lattarini <stefano.lattarini@gmail.com>
2011-05-04 Stefano Lattarini <stefano.lattarini@gmail.com>
+ tests: add excerpts from few original bug reports, for clarity.
+ * tests/comment6.test: Add excerpts and/or details from the
+ original bug report that prompted this tests to be added and/or
+ extended.
+ * tests/dejagnu4.test: Likewise.
+ * tests/installdir.test: Likewise.
+
+2011-05-04 Stefano Lattarini <stefano.lattarini@gmail.com>
+
tests defs: allow requirements for compilers (mostly dummy)
Most of the new requirements that are now accepted in `$required'
as consequence of this patch are still dummy. They are planned
implementation, looking for a generic `lex' program, will follow
in the future.
+2011-05-04 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: 'silent-many-gcc.test' improved and made more robust
+ * tests/silent-many-gcc.test: Also force "fast" gcc depmode
+ for C++ compilations. Add sanity checks verifying that the
+ cache variables we force are really used by configure. Fix
+ typo in comments.
+
+2011-05-04 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: drop useless requirements of 'g++'
+ * tests/lflags2.test ($required): Remove 'g++', as any working
+ C++ compiler should be enough, and ./configure will cause the
+ test to skip if no working C++ compiler is found.
+ * tests/yflags2.test: Likewise.
+ * tests/subobj9.test: Likewise.
+ * tests/silentcxx.test: Likewise. Also, do not force depmodes
+ that could cause non-GNU C++ compilers to fail.
+ * tests/silentcxx-gcc.test: New test, like `silentcxx.test',
+ but forcing "fast" gcc depmode (and thus requiring the GNU C++
+ compiler).
+ * tests/specflg10.test: Add proper "fixme" comment telling that
+ we should make this test work with a generic C++ compiler.
+ * tests/Makefile.am (TESTS): Update.
+
+2011-05-03 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests defs: one more environment sanitization (corner case)
+ * tests/defs (am__using_gmake): Initialize. The `using_gmake'
+ subroutine was using this variable for caching, but wasn't
+ initializing it, which could cause problems in the (admittedly
+ very unlikely) case in which it was pre-existent in the
+ environment.
+
2011-05-03 Stefano Lattarini <stefano.lattarini@gmail.com>
tests: add forgotten test scripts to $(TESTS)
2011-05-03 Stefano Lattarini <stefano.lattarini@gmail.com>
+ maintcheck: consistency of list of test scripts
+ * CheckListOfTests.am: New file.
+ (maintainer-check-list-of-tests): New target,
+ check for consistency between list of tests defined in the
+ including Makefile and list of tests on the filesystem.
+ (clean-maintcheck-testslist-tmp): New rule, to clean up the
+ temporary files that might be left around by the rules associated
+ with the previous target.
+ (clean-local): Depend on it.
+ * lib/Automake/tests/Makefile.am: Include `CheckListOfTests.am'.
+ (maintainer-check): Added dependency from the new target
+ `maintainer-check-list-of-tests'.
+ * tests/Makefile.am: Likewise. Also ...
+ (checked_test_extensions): Define to `.test'.
+ (expected_test_list): Define properly.
+ * Makefile.am (maintainer-check-list-of-test): New target,
+ calling recursively into `tests/' and `lib/Automake/tests/',
+ using ...
+ (TEST_SUBDIRS): ... this new variable.
+
+2011-05-03 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: drop useless requirements of gcc (plus testsuite tweakings)
+ * tests/libobj14.test ($required): Remove 'gcc', as we never run
+ the C compiler.
+ * tests/targetclash.test: Likewise.
+ * tests/ansi6.test ($required): Remove 'gcc', as any working
+ C compiler should be enough.
+ * tests/ansi7.test: Likewise.
+ * tests/ansi9.test: Likewise.
+ * tests/ansi10.test: Likewise.
+ * tests/lex5.test: Likewise.
+ * tests/lexvpath.test: Likewise.
+ * tests/mmodely.test: Likewise.
+ * tests/pr204.test: Likewise.
+ * tests/pr300-lib.test: Likewise.
+ * tests/pr300-prog.test: Likewise.
+ * tests/primary3.test: Likewise.
+ * tests/specflg7.test: Likewise.
+ * tests/specflg8.test: Likewise.
+ * tests/subdir5.test: Likewise.
+ * tests/subdir8.test: Likewise.
+ * tests/subobj6.test: Likewise.
+ * tests/subst3.test: Likewise.
+ * tests/substre2.test: Likewise.
+ * tests/yacc6.test: Likewise.
+ * tests/yacc8.test: Likewise.
+ * tests/depcomp2.test: Likewise. Also, avoid clobbering user-set
+ CFLAGS.
+ * tests/lex3.test: Likewise.
+ * tests/ansi3.test: Likewise. Also, avoid 'CC=gcc' in configure.
+ * tests/ansi3b.test: Likewise.
+ * tests/ansi5.test: Likewise.
+ * tests/autohdr4.test ($required): Remove 'gcc', as any working
+ C compiler should be enough.
+ Also, do not reject slow dependency extractors (which we might
+ be forced to use now that $CC is not necessarily gcc anymore).
+ * tests/cond16.test ($required): Remove 'gcc', as any working
+ C compiler should be enough.
+ Since we are at it, throw in few minor tweakings (mostly cosmetic,
+ stylistic, or consistency-related).
+ * tests/cond18.test: Likewise.
+ * tests/cond35.test: Likewise.
+ * tests/gnits2.test: Likewise.
+ * tests/libtool3.test: Likewise.
+ * tests/libtool7.test: Likewise.
+ * tests/libtool9.test: Likewise.
+ * tests/ltcond.test: Likewise.
+ * tests/ltcond2.test: Likewise.
+ * tests/ltconv.test: Likewise.
+ * tests/ltlibsrc.test: Likewise.
+ * tests/nobase.test: Likewise.
+ * tests/nobase-libtool.test: Likewise.
+ * tests/pr220.test: Likewise.
+ * tests/pr224.test: Likewise.
+ * tests/pr300-ltlib.test: Likewise.
+ * tests/pr401.test: Likewise.
+ * tests/pr401b.test: Likewise.
+ * tests/pr401c.test: Likewise.
+ * tests/subpkg.test: Likewise.
+ * tests/target-cflags.test: Likewise.
+ * tests/transform.test: Likewise.
+ * tests/yacc4.test: Likewise.
+ * tests/cond19.test: Likewise. Also, avoid clobbering user-set
+ CFLAGS.
+ * tests/cond4.test: Likewise.
+ * tests/depend2.test: Likewise.
+ * tests/pr87.test: Likewise.
+ * tests/subobj3.test: Likewise.
+ * tests/substref.test: Likewise.
+
+2011-05-03 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ test defs: increase coverage w.r.t. GNU make
+ * tests/defs (GNUmake): Instead of just skipping the tests
+ requiring GNU make if $MAKE is not GNU make, try to look for
+ it and, if found, redefine $MAKE accordingly. This will help
+ to transparently increase coverage on non-GNU systems which
+ have GNU make available in PATH.
+
+2011-05-03 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: improve few inter-tests references
+ * tests/acloca22.test: Improve and extend the heading comments.
+ Add reference to related tests 'remake-deleted-m4-file.test' and
+ 'remake-renamed-m4-macro-and-file.test'. Since we are at it,
+ add trailing `:' command, and few blank lines for readability.
+ * tests/remake-renamed-m4-macro-and-file.test: In the heading
+ comments, add reference to the related test 'acloca22.test'.
+
+2011-05-03 Stefano Lattarini <stefano.lattarini@gmail.com>
+
tests: fix spurious failure in 'color2.test'
* tests/color2.test: If $MAKE contains command-line arguments (as
in e.g., "make -j2"), expect's directive "spawn $env(MAKE)" fails
spuriously, because it tries to run "$MAKE" as a single command.
Fix this with proper uses of the TCL `eval' builtin.
+2011-05-03 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: few more checks on automatic remake support
+ * tests/remake-all-1.test: New test, check that the "all" target
+ triggers rebuilt of outdated Makefiles.
+ * tests/remake-all-2.test: Likewise, but for when the makefiles
+ are not named `Makefile'.
+ * tests/Makefile.am (TESTS): Update.
+
+2011-05-03 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: check some remake features with non-GNU make too
+ * tests/acloca14.test ($required): Drop "GNUmake". This test
+ works as-is with non-GNU make implementations.
+ * tests/remake1a.test: Likewise.
+ * tests/aclocal4.test ($required): Drop "GNUmake".
+ Modify the test to have it work also with non-GNU make.
+ * tests/remake5.test: Likewise.
+ * tests/remake8a.test: Likewise.
+ * tests/remake8b.test: Likewise.
+ * tests/remake9a.test: Likewise.
+ * tests/remake9b.test: Likewise.
+ * tests/remake9c.test: Likewise.
+ * tests/remake9d.test: Likewise.
+ * tests/remake10a.test: Likewise.
+ * tests/remake10b.test: Likewise.
+ * tests/remake10c.test: Likewise.
+ * tests/remake12.test: ($required): Drop "GNUmake".
+ Adapt the test to make it work also with non-GNU make (if
+ it supports an "include" directive).
+
+2011-05-01 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: extend tests on dependency tracking with VPATH
+ * tests/depmod-tests.sh: New script, fulfilling a threefold role:
+ 1. it is called to generate a Makefile.am snippet, containing the
+ definition of proper lists of tests;
+ 2. it is called to set up a directory containing some common data
+ files and autotools-generated files used by the aforementioned
+ tests (this is done for speed reasons only); and
+ 3. it is called to properly run those tests, one at a time.
+ * tests/depcomp9.test: Delete, it's obsolete now.
+ * tests/depcomp10.test: Likewise.
+ * tests/Makefile.am ($(srcdir)/depmod-tests.am): Include this
+ snippet, which defines ...
+ (depmode_tests): ... this new macro, containing the list of the
+ newly generated `*.depmod' tests.
+ (TESTS_EXTENSIONS): Add `.depmod'.
+ (DEPMOD_LOG_COMPILER): Define. It calls `depmod-tests.sh', so that
+ the "depmode tests" will be executed by passing that driver script
+ a proper parameter.
+ ($(depmod_tests)): New dependency declaration (dummy, but required
+ in order to have make actually produce expected log files from the
+ `.depmod.log' suffix rule).
+ (TESTS): Add $(depmod_tests).
+ (EXTRA_DIST): Distribute depmod-tests.sh.
+ Other minor cosmetic changes and reorderings.
+ * bootstrap: Generate depmod-tests.am.
+ * tests/.gitignore: Updated.
+
+2011-05-01 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: avoid extra test generation (for `instspc' tests)
+ We don't really need to generate tons and tons of wrapper test
+ scripts (presently, ~ 90) just to run what is basically a single
+ test (specifically, `instspc-tests.sh') with different parameters.
+ In fact, the possibility of running a single driver script with
+ multiple files/configurations/parameters is one of the major
+ selling points of the 'parallel-tests' testsuite driver of
+ automake. So why not use ourselves more extensively the best
+ features we provide?
+ * tests/instspc-tests.sh: Update heading comments. Expect
+ to be executed, not sourced. Update handling of command-line
+ arguments. Adjust to reflect the renaming of "instspc tests"
+ from `instspc-*.test' to `*.instspc'.
+ * tests/instspc-data.test: Update heading comments. Execute
+ driver script `instspc-tests.sh', instead of sourcing it.
+ Give more informative messages in case of internal errors.
+ * tests/Makefile.am (TESTS_EXTENSIONS): Define explicitly (to
+ `.test' and `.instspc').
+ (TESTS): Add `$(instspc_tests)'.
+ (generated_tests): Remove `$(instspc_tests)'.
+ (INSTSPC_LOG_COMPILER): Define. Calls `instspc-tests.sh', so
+ that the "instspc tests" will be now executed by passing that
+ driver script a proper parameter.
+ ($(instspc_tests)): Remove, we don't need anymore to generate
+ this tests.
+ ($(instspc_tests:.test=.log)): Remove, substituted by ...
+ ($(instspc_tests:.instspc=.log)): ... this.
+ ($(instspc_tests)): New dependency declaration (dummy, but
+ required in order to have make actually produce expected log
+ files from the `.instspc.log' suffix rule).
+ (MAINTAINERCLEANFILES, generated_tests): Don't extend with
+ $(instspc_tests) anymore.
+ Update comments.
+
+2011-05-01 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ build: improve the definition of the list of testcases
+ * tests/Makefile.am (handwritten_tests): New variable.
+ (generated_tests): Likewise.
+ (TESTS): Redefine as the union of the above.
+ (EXTRA_DIST): Extend using $(handwritten_tests) and
+ $(generated_tests) rather than $(TESTS).
+ * tests/gen-parallel-tests: Update accordingly, and
+ make more robust.
+
+2011-05-01 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests skips: shorter and clearer messages
+ * tests/defs: Use shorter messages when giving reasons for test
+ skipping; it turns out these shorter messages are also clearer.
+ If more info might be useful, send them to the log file only.
+
2011-05-01 Stefano Lattarini <stefano.lattarini@gmail.com>
tests: fix self check spurious failure with older bash versions
and 3.2.39) fail to correctly remove the temporary directory in
the exit trap.
+2011-05-01 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: add sanity check to 'self-check-cleanup.test'
+ * tests/self-check-cleanup.test: Check that the "hacked" file
+ `defs-static' used by the test differs from the "vanilla" one
+ in builddir. This also offers a little more debugging output.
+
+2011-05-01 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: display reasons for skips to the console
+ * tests/Makefile.am (AM_TESTS_ENVIRONMENT): Open file descriptor
+ `9' to the original stderr; define `stderr_fileno_' to `9', and
+ export it.
+ * tests/self-check-report.test: Prevent new spurious failures by
+ removing from the environment any definition of `stderr_fileno_'.
+
+2011-05-01 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: use `skip_' for skipping of tests
+ * tests/defs: Use the `skip_' subroutine for test skipping. Also
+ give sometimes more detailed messages about the reasons for the
+ skipping.
+ * tests/canon7.test: Likewise.
+ * tests/color.test: Likewise.
+ * tests/color2.test: Likewise.
+ * tests/compile2.test: Likewise.
+ * tests/dejagnu7.test: Likewise.
+ * tests/depcomp6.test: Likewise.
+ * tests/depcomp9.test: Likewise.
+ * tests/depcomp10.test: Likewise.
+ * tests/distlinks.test: Likewise.
+ * tests/distlinksbrk.test: Likewise.
+ * tests/fn99.test: Likewise.
+ * tests/fn99subdir.test: Likewise.
+ * tests/forcemiss2.test: Likewise.
+ * tests/fort5.test: Likewise.
+ * tests/gettext3.test: Likewise.
+ * tests/install2.test: Likewise.
+ * tests/instfail-info.test: Likewise.
+ * tests/instfail-java.test: Likewise.
+ * tests/instfail-libtool.test: Likewise.
+ * tests/instfail.test: Likewise.
+ * tests/instmany-mans.test: Likewise.
+ * tests/instmany-python.test: Likewise.
+ * tests/instmany.test: Likewise.
+ * tests/instsh3.test: Likewise.
+ * tests/ltinit.test: Likewise.
+ * tests/makej2.test: Likewise.
+ * tests/mdate6.test: Likewise.
+ * tests/mkinst3.test: Likewise.
+ * tests/parallel-tests3.test: Likewise.
+ * tests/parallel-tests-reset-term.test: Likewise.
+ * tests/parallel-tests-unreadable-log.test: Likewise,
+ * tests/python-virtualenv.test: Likewise.
+ * tests/remake-gnulib-remove-header.test: Likewise.
+ * tests/subobj9.test: Likewise.
+ * tests/symlink2.test: Likewise.
+ * tests/tar.test: Likewise.
+ * tests/tar2.test: Likewise.
+ * tests/txinfo26.test: Likewise.
+ * tests/vala2.test: Likewise.
+ * tests/vala3.test: Likewise.
+ * tests/vala5.test: Likewise.
+ * tests/vtexi4.test: Likewise.
+ * tests/instdir-texi.test: Likewise.
+ * tests/txinfo21.test: Likewise.
+
2011-04-29 Stefano Lattarini <stefano.lattarini@gmail.com>
testsuite: more environment sanitization
tests: fix typo in python5b.test
* tests/python5b.test: Remove extra `:' from $PATH redefinition.
+2011-04-12 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ coverage: test for automake bug#8485 (known regression)
+ * tests/yacc-dist-nobuild-subdir.test: New test.
+ * tests/Makefile.am (TESTS, XFAIL_TESTS): Update.
+
2011-04-14 Stefano Lattarini <stefano.lattarini@gmail.com>
tests: remove redundant settings of `errexit' shell flag
2011-04-12 Stefano Lattarini <stefano.lattarini@gmail.com>
- coverage: test for automake bug#8485 (known regression)
- * tests/yacc-dist-nobuild-subdir.test: New test.
- * tests/Makefile.am (TESTS, XFAIL_TESTS): Update.
-
-2011-04-12 Stefano Lattarini <stefano.lattarini@gmail.com>
-
test defs: define default $distdir (help reducing duplication)
* tests/defs ($distdir): New variable, might be used in
testcases checking distribution-related features.
projects / platform / upstream / automake.git / blobdiff