perf: beginning of a performance testsuite
authorStefano Lattarini <stefano.lattarini@gmail.com>
Tue, 29 May 2012 09:58:02 +0000 (11:58 +0200)
committerStefano Lattarini <stefano.lattarini@gmail.com>
Tue, 29 May 2012 10:04:54 +0000 (12:04 +0200)
commitff022f46d098098bc17100332e2c96ebef715dff
treeb32b43823c0412ff4d889a697d40205db68b0b0d
parente6184b2ce04c2cb2aab55377b0fbd615a99d62c8
perf: beginning of a performance testsuite

Some tests in the Automake testsuite already aims only at verifying
the performance, rather than the correctness, of some operations.
Still, they are somewhat shoehorned and forced into the PASS/FAIL
framework (say, with the 'ulimit' shell builtin used to verify some
operation doesn't take up too much time or memory), but that is
conceptually a stretch, and has already caused problems in practice
(see automake bug#11512 for an example).

So we start moving the "performance tests" out of the testsuite proper,
and make them run only "on demand" (when the user exports the variable
'AM_TESTSUITE_PERF' to "yes").  Ideally, we should provide those tests
with a custom runner/driver that measures and displays the relevant
performance information, but doing that correctly and with the right
APIs is definitely more difficult, so we leave it for a later step
(an hope we'll take such a step eventually).

* t/cond29.sh: Move ...
* t/perf/cond.sh: ... here, and adjust.
* t/testsuite-recheck-speed.sh: Move ...
* t/perf/testsuite-recheck.sh: ... here.
* t/testsuite-summary-speed.sh: Move ...
* t/perf/testsuite-summary.sh: ... here.
* t/list-of-tests.mk (perf_TESTS): New variable, listing the tests in
the 't/perf' directory.
(handwritten_TESTS): Adjust.
* defs: Skip any tests in the 't/perf/' subdirectory unless the
'AM_TESTSUITE_PERF' variable is set to "yes" or "y".
* .gitignore: Update.

Signed-off-by: Stefano Lattarini <stefano.lattarini@gmail.com>
.gitignore
defs
t/list-of-tests.mk
t/perf/cond.sh [moved from t/cond29.sh with 80% similarity]
t/perf/testsuite-recheck.sh [moved from t/testsuite-recheck-speed.sh with 100% similarity]
t/perf/testsuite-summary.sh [moved from t/testsuite-summary-speed.sh with 100% similarity]