Some tests in the Automake testsuite already aims only at verifying
the performance, rather than the correctness, of some operations.
Still, they are somewhat shoehorned and forced into the PASS/FAIL
framework (say, with the 'ulimit' shell builtin used to verify some
operation doesn't take up too much time or memory), but that is
conceptually a stretch, and has already caused problems in practice
(see automake bug#11512 for an example).
So we start moving the "performance tests" out of the testsuite proper,
and make them run only "on demand" (when the user exports the variable
'AM_TESTSUITE_PERF' to "yes"). Ideally, we should provide those tests
with a custom runner/driver that measures and displays the relevant
performance information, but doing that correctly and with the right
APIs is definitely more difficult, so we leave it for a later step
(an hope we'll take such a step eventually).
* t/cond29.sh: Move ...
* t/perf/cond.sh: ... here, and adjust.
* t/testsuite-recheck-speed.sh: Move ...
* t/perf/testsuite-recheck.sh: ... here.
* t/testsuite-summary-speed.sh: Move ...
* t/perf/testsuite-summary.sh: ... here.
* t/list-of-tests.mk (perf_TESTS): New variable, listing the tests in
the 't/perf' directory.
(handwritten_TESTS): Adjust.
* defs: Skip any tests in the 't/perf/' subdirectory unless the
'AM_TESTSUITE_PERF' variable is set to "yes" or "y".
* .gitignore: Update.
Signed-off-by: Stefano Lattarini <stefano.lattarini@gmail.com>
/t/*.trs
/t/pm/*.log
/t/pm/*.trs
+/t/perf/*.log
+/t/perf/*.trs
cscope.files
cscope.in.out
cscope.out
## required by them. ##
## ----------------------------------------------------------- ##
+# Performance tests must be enabled explicitly.
+case $argv0 in
+ */perf/*)
+ case $AM_TESTSUITE_PERF in
+ [yY]|[yY]es|1) ;;
+ *) skip_ "performance tests not explicitly enabled" ;;
+ esac
+ ;;
+esac
+
# Look for (and maybe set up) required tools and/or system features; skip
# the current test if they are not found.
for tool in : $required
t/pm/Version3.pl \
t/pm/Wrap.pl
+perf_TESTS = \
+t/perf/cond.sh \
+t/perf/testsuite-recheck.sh \
+t/perf/testsuite-summary.sh
+
# The order here is mostly alphabetical, with the deliberate exception
# that tests having a high runtime (especially TAP tests that run various
# checks sequentially) are listed early; this improves performance on
handwritten_TESTS = \
t/get-sysconf.sh \
$(perl_TESTS) \
+$(perf_TESTS) \
t/instspc.tap \
t/aclocal.sh \
t/aclocal3.sh \
t/cond26.sh \
t/cond27.sh \
t/cond28.sh \
-t/cond29.sh \
t/cond30.sh \
t/cond31.sh \
t/cond32.sh \
t/testsuite-summary-count.sh \
t/testsuite-summary-count-many.sh \
t/testsuite-summary-reference-log.sh \
-t/testsuite-recheck-speed.sh \
-t/testsuite-summary-speed.sh \
t/test-driver-acsubst.sh \
t/test-driver-cond.sh \
t/test-driver-custom-no-extra-driver.sh \
# to define a01_DEPENDENCIES, a02_DEPENDENCIES, a03_OBJECTS,
# a04_OBJECTS, and to rewrite bin_PROGRAM), eating all memory, swap,
# or cpu time it can found.
-# Limit memory available to spawned processes to ~ 150 MB. Considering
-# that 70 MB is enough for x86_64 Fedora 16, that gives tools and
-# libraries plenty of room to grow. Suggested by Jim Meyering in
-# automake bug#10374.
-(ulimit -v 1; sh -c ":") && skip_ "no adequate 'ulimit' builtin found"
-(ulimit -v 150000; sh -c ":") || skip_ "no adequate 'ulimit' builtin found"
-ulimit -v 150000
for i in \
01 02 03 04 05 06 07 08 09 10 \