From f4a04f21eededc6e0c4f621926fc7ab796680de2 Mon Sep 17 00:00:00 2001 From: Stefano Lattarini Date: Fri, 22 Jun 2012 23:06:55 +0200 Subject: [PATCH] readme: better separation of entries in 't/README' * t/README (Writing test cases): Prepend different entries with a '*' character acting like a bullet in a list. This make different entries better separated, visually-wise. Minor related reformatting. Signed-off-by: Stefano Lattarini --- t/README | 77 +++++++++++++++++++++++++++++++++++++--------------------------- 1 file changed, 45 insertions(+), 32 deletions(-) diff --git a/t/README b/t/README index b666d6b..829b97a 100644 --- a/t/README +++ b/t/README @@ -137,39 +137,41 @@ Reporting failures Writing test cases ================== - If you plan to fix a bug, write the test case first. This way you'll +* If you plan to fix a bug, write the test case first. This way you'll make sure the test catches the bug, and that it succeeds once you have fixed the bug. - Add a copyright/license paragraph. +* Add a copyright/license paragraph. - Explain what the test does. +* Explain what the test does. - Cite the PR number (if any), and the original reporter (if any), so +* Cite the PR number (if any), and the original reporter (if any), so we can find or ask for information if needed. - If a test checks examples or idioms given in the documentation, make - sure the documentation reference them appropriately in comments, as in: +* If a test checks examples or idioms given in the documentation, make + sure the documentation reference them appropriately in comments, as + with: + @c Keep in sync with autodist-config-headers.sh @example ... @end example - Use "required=..." for required tools. Do not explicitly require +* Use "required=..." for required tools. Do not explicitly require tools which can be taken for granted because they're listed in the GNU Coding Standards (for example, 'gzip'). - Include ./defs in every test script (see existing tests for examples +* Include ./defs in every test script (see existing tests for examples of how to do this). - Use the 'skip_' function to skip tests, with a meaningful message if +* Use the 'skip_' function to skip tests, with a meaningful message if possible. Where convenient, use the 'warn_' function to print generic warnings, the 'fail_' function for test failures, and the 'fatal_' function for hard errors. In case a hard error is due to a failed set-up of a test scenario, you can use the 'framework_fail_' function instead. - For those tests checking the Automake-provided test harnesses that are +* For those tests checking the Automake-provided test harnesses that are expected to work also when the 'serial-tests' Automake option is used (thus causing the serial testsuite harness to be used in the generated Makefile), place a line containing "try-with-serial-tests" somewhere @@ -180,7 +182,7 @@ Writing test cases set the shell variable 'am_serial_tests' to "yes" before including ./defs. - Some tests in the Automake testsuite are auto-generated; those tests +* Some tests in the Automake testsuite are auto-generated; those tests might have custom extensions, but their basename (that is, with such extension stripped) is expected to end with "-w" string, optionally followed by decimal digits. For example, the name of a valid @@ -189,7 +191,7 @@ Writing test cases to be confused with auto-generated tests; for example, 'u-v-w.sh' or 'option-w0.tap' are *not* valid name for hand-written tests. - ./defs brings in some commonly required files, and sets a skeleton +* ./defs brings in some commonly required files, and sets a skeleton configure.ac. If possible, append to this file. In some cases you'll have to overwrite it, but this should be the exception. Note that configure.ac registers Makefile.in but do not output anything by @@ -198,55 +200,57 @@ Writing test cases test directory (which is a rare occurrence), set the 'am_create_testdir' shell variable to "empty" before sourcing ./defs. - By default, the testcases are run with the errexit shell flag on, +* By default, the testcases are run with the errexit shell flag on, to make it easier to catch failures you might not have thought of. If this is undesirable in some testcase, you can use "set +e" to disable the errexit flag (but please do so only if you have a very good reason). - End the test script with a ":" or "Exit 0". Otherwise, when somebody +* End the test script with a ":" or "Exit 0". Otherwise, when somebody changes the test by adding a failing command after the last command, the test will spuriously fail because $? is nonzero at the end. Note that this is relevant even if the errexit shell flag is on, in case the test contains commands like "grep ... Makefile.in && Exit 1" (and there are indeed a lot of such tests). - Use $ACLOCAL, $AUTOMAKE, $AUTOCONF, $AUTOUPDATE, $AUTOHEADER, +* Use $ACLOCAL, $AUTOMAKE, $AUTOCONF, $AUTOUPDATE, $AUTOHEADER, $PERL, $MAKE, $EGREP, and $FGREP, instead of the corresponding commands. - Use $sleep when you have to make sure that some file is newer +* Use $sleep when you have to make sure that some file is newer than another. - Use cat or grep or similar commands to display (part of) files that +* Use cat or grep or similar commands to display (part of) files that may be interesting for debugging, so that when a user send a verbose output we don't have to ask him for more details. Display stderr output on the stderr file descriptor. If some redirected command is likely to fail, display its output even in the failure case, before exiting. - Use 'Exit' rather than 'exit' to abort for leave early from a test +* Use 'Exit' rather than 'exit' to abort for leave early from a test case. - Use '$PATH_SEPARATOR', not hard-coded ':', as the separator of +* Use '$PATH_SEPARATOR', not hard-coded ':', as the separator of PATH's entries. - It's more important to make sure that a feature works, than make +* It's more important to make sure that a feature works, than make sure that Automake's output looks correct. It might look correct and still fail to work. In other words, prefer running 'make' over grepping Makefile.in (or do both). - If you run $ACLOCAL, $AUTOMAKE or $AUTOCONF several times in the +* If you run $ACLOCAL, $AUTOMAKE or $AUTOCONF several times in the same test and change configure.ac by the meantime, do + rm -rf autom4te*.cache + before the following runs. On fast machines the new configure.ac could otherwise have the same timestamp as the old autom4te.cache. - Use filenames with two consecutive spaces when testing that some +* Use filenames with two consecutive spaces when testing that some code preserves filenames with spaces. This will catch errors like `echo $filename | ...`. - Make sure your test script can be used to faithfully check an +* Make sure your test script can be used to faithfully check an installed version of automake (as with "make installcheck"). For example, if you need to copy or grep an automake-provided script, do not assume that they can be found in the '$top_srcdir/lib' @@ -254,23 +258,27 @@ Writing test cases such "$am_...dir" variables can be found in the 'defs-static.in' file. - When writing input for lex, include the following in the definitions +* When writing input for lex, include the following in the definitions section: + %{ #define YY_NO_UNISTD_H 1 %} + to accommodate non-ANSI systems, since GNU flex generates code that includes unistd.h otherwise. Also add: + int isatty (int fd) { return 0; } + to the definitions section if the generated code is to be compiled by a C++ compiler, for similar reasons (i.e., the isatty(3) function from that same unistd.h header would be required otherwise). - Before commit: make sure the test is executable, add the tests to +* Before commit: make sure the test is executable, add the tests to TESTS in Makefile.am, add it to XFAIL_TESTS in addition if needed, write a ChangeLog entry, send the diff to . - In test scripts, prefer using POSIX constructs over their old +* In test scripts, prefer using POSIX constructs over their old Bourne-only equivalents: - use $(...), not `...`, for command substitution; @@ -280,23 +288,28 @@ Writing test cases paraphrases like "if CMD; then :; else ...". - prefer use of ${param%pattern} and ${param#pattern} parameter expansions over processing by 'sed' or 'expr'. - - Note however that, when writing Makefile recipes or shell code in a + +* Note however that, when writing Makefile recipes or shell code in a configure.ac, you should still use `...` instead, because the Autoconf generated configure scripts do not ensure they will find a truly POSIX shell (even though they will prefer and use it *if* it's found). - Do not test an Automake error with "$AUTOMAKE && Exit 1", or in three +* Do not test an Automake error with "$AUTOMAKE && Exit 1", or in three years we'll discover that this test failed for some other bogus reason. This happened many times. Better use something like + AUTOMAKE_fails grep 'expected diagnostic' stderr - (Note this doesn't prevent the test from failing for another reason, - but at least it makes sure the original error is still here). - Do not override Makefile variables using make arguments, as in e.g.: + Note this doesn't prevent the test from failing for another reason, + but at least it makes sure the original error is still here. + +* Do not override Makefile variables using make arguments, as in e.g.: + $MAKE prefix=/opt install + This is not portable for recursive targets (targets that call a sub-make may not pass "prefix=/opt" along). Use the following instead: + prefix=/opt $MAKE -e install -- 2.7.4