diff options
Diffstat (limited to 't/README')
-rw-r--r-- | t/README | 299 |
1 files changed, 299 insertions, 0 deletions
diff --git a/t/README b/t/README new file mode 100644 index 000000000..33a653c71 --- /dev/null +++ b/t/README @@ -0,0 +1,299 @@ + The Automake test suite + + +User interface +============== + + +Running the tests +----------------- + + To run all tests: + + make -k check + + You can use '-jN' for faster completion (it even helps on a + uniprocessor system, due to unavoidable sleep delays, as + noted below). + + To rerun only failed tests: + + make -k recheck + + To run only tests that are newer than their last results: + + make -k check RECHECK_LOGS= + + To run only selected tests: + + make -k check TESTS="foo.test bar.test" (GNU make) + env TESTS="foo.test bar.test" make -e -k check (non-GNU make) + + To run the tests in cross-compilation mode, you should first configure + the automake source tree to a cross-compilation setup. For example, to + run with a Linux-to-MinGW cross compiler, you will need something like + this: + + ./configure --host i586-mingw32msvc --build i686-pc-linux-gnu + + To avoid possible spurious error, you really have to *explicitly* specify + '--build' in addition to '--host'; the 'lib/config.guess' script can help + determine the correct value to pass to '--build'. + Then you can just run the testsuite in the usual way, and the test cases + using a compiler should automatically use a cross-compilation setup. + + +Interpretation +-------------- + + Successes: + PASS - success + XFAIL - expected failure + + Failures: + FAIL - failure + XPASS - unexpected success + + Other: + SKIP - skipped tests (third party tools not available) + ERROR - some unexpected error condition + + +Getting details from failures +----------------------------- + + By default, verbose output of a test 'foo.test' or 'foo.tap' is + retained in the log file 'foo.log'. A summary log is created in + the file 'test-suite.log'. + + You can limit the set of files using the TESTS variable, and enable + detailed test output at the end of the test run with the VERBOSE + variable: + + env VERBOSE=x TESTS='first.test second.test ...' make -e check + + You can also run the tests by hand, as explained in the next subsection. + + +About the tests +--------------- + + There are two kinds of tests in the Automake testsuite (both implemented + as shell scripts). The scripts with the '.test' suffix are "simple" + tests, their outcome completely determined by their exit status. Those + with the '.tap' suffix use the TAP protocol. If you want to run a test + by hand, you can do so directly if it is a simple test: + + ./nogzip.test + + (it will be verbose by default), while if it is a TAP test you can pass + it to your preferred TAP runner, as in e.g.: + + prove --verbose --merge ./add-missing.tap + + The tests can also be run directly in a VPATH build, as with: + + /path/to/srcdir/tests/nogzip.test + prove --verbose --merge /path/to/srcdir/tests/add-missing.tap + + +Supported shells +---------------- + + By default, the tests are run by the $SHELL detected at configure + time. They also take care to re-execute themselves with that shell, + unless told not to. So, to run the tests with a different shell, say + '/path/to/another/sh', the user must use: + + AM_TESTS_REEXEC=no /path/to/another/sh ./foo.test + AM_TESTS_REEXEC=no prove -v -e /path/to/another/sh ./bar.tap + + to run a test directly, and: + + make check LOG_COMPILER=/path/to/sh (GNU make) + LOG_COMPILER=/path/to/sh make -e check (non-GNU make) + + to run the test(s) through the makefile test driver. + + The test scripts are written with portability in mind, so that they + should run with any decent Bourne-compatible shell. + + However, some care must be used with Zsh, since, when not directly + started in Bourne-compatibility mode, it has some incompatibilities + in the handling of $0 which conflict with our usage. Our testsuite + can automatically work around these incompatibilities when a version + 4.3 or later of Zsh is used, but unfortunately not when an older + version of Zsh is used. Thus, if you want to run a test script, say + foo.test, with Zsh 4.2, you *can't* simply do "zsh foo.test", but + you *must* resort to: + + AM_TESTS_REEXEC=no zsh -o no_function_argzero foo.test + + Note that this problem does not occur if Zsh is executed through + a symlink with a basename of 'sh', since in that case Zsh starts + in Bourne compatibility mode. So you should be perfectly safe + when /bin/sh is Zsh, even a it's version < 4.3. + + +Reporting failures +------------------ + + Send verbose output, i.e., the contents of test-suite.log, of failing + tests to <bug-automake@gnu.org>, along with the usual version numbers + (which Automake, which Autoconf, which operating system, which make + version, which shell, etc.) + + + +Writing test cases +================== + + +Do +-- + + If you plan to fix a bug, write the test case first. This way you'll + make sure the test catches the bug, and that it succeeds once you have + fixed the bug. + + Add a copyright/license paragraph. + + Explain what the test does. + + Cite the PR number (if any), and the original reporter (if any), so + we can find or ask for information if needed. + + If a test checks examples or idioms given in the documentation, make + sure the documentation reference them appropriately in comments, as in: + @c Keep in sync with autodist-config-headers.test. + @example + ... + @end example + + Use "required=..." for required tools. Do not explicitly require + tools which can be taken for granted because they're listed in the + GNU Coding Standards (for example, 'gzip'). + + Include ./defs in every test script (see existing tests for examples + of how to do this). + + Use the 'skip_' function to skip tests, with a meaningful message if + possible. Where convenient, use the 'warn_' function to print generic + warnings, the 'fail_' function for test failures, and the 'fatal_' + function for hard errors. In case a hard error is due to a failed + set-up of a test scenario, you can use the 'framework_fail_' function + instead. + + For tests that use the 'parallel-tests' Automake option, set the shell + variable 'am_parallel_tests' to "yes" before including ./defs. For + tests that are *not* meant to work with the 'parallel-tests' Automake + option (these should be very very few), set the shell variable + 'am_parallel_tests' to "no" before including ./defs. + + Some tests in the Automake testsuite are auto-generated; those tests + might have custom extensions, but their basename (that is, with such + extension stripped) is expected to end with "-w" string, optionally + followed by decimal digits. For example, the name of a valid + auto-generated test can be 'color-w.test' or 'tap-signal-w09.tap'. + Please don't name hand-written tests in a way that could cause them + to be confused with auto-generated tests; for example, 'u-v-w.test' + or 'option-w0.tap' are *not* valid name for hand-written tests. + + ./defs brings in some commonly required files, and sets a skeleton + configure.ac. If possible, append to this file. In some cases + you'll have to overwrite it, but this should be the exception. Note + that configure.ac registers Makefile.in but do not output anything by + default. If you need ./configure to create Makefile, append AC_OUTPUT + to configure.ac. In case you don't want ./defs to pre-populate your + test directory (which is a rare occurrence), set the 'am_create_testdir' + shell variable to "empty" before sourcing ./defs. + + By default, the testcases are run with the errexit shell flag on, + to make it easier to catch failures you might not have thought of. + If this is undesirable in some testcase, you can use "set +e" to + disable the errexit flag (but please do so only if you have a very + good reason). + + End the test script with a ":" or "Exit 0". Otherwise, when somebody + changes the test by adding a failing command after the last command, + the test will spuriously fail because $? is nonzero at the end. Note + that this is relevant even if the errexit shell flag is on, in case + the test contains commands like "grep ... Makefile.in && Exit 1" (and + there are indeed a lot of such tests). + + Use $ACLOCAL, $AUTOMAKE, $AUTOCONF, $AUTOUPDATE, $AUTOHEADER, + $PERL, $MAKE, $EGREP, and $FGREP, instead of the corresponding + commands. + + Use $sleep when you have to make sure that some file is newer + than another. + + Use cat or grep or similar commands to display (part of) files that + may be interesting for debugging, so that when a user send a verbose + output we don't have to ask him for more details. Display stderr + output on the stderr file descriptor. If some redirected command is + likely to fail, display its output even in the failure case, before + exiting. + + Use 'Exit' rather than 'exit' to abort for leave early from a test + case. + + Use '$PATH_SEPARATOR', not hard-coded ':', as the separator of + PATH's entries. + + It's more important to make sure that a feature works, than make + sure that Automake's output looks correct. It might look correct + and still fail to work. In other words, prefer running 'make' over + grepping Makefile.in (or do both). + + If you run $ACLOCAL, $AUTOMAKE or $AUTOCONF several times in the + same test and change configure.ac by the meantime, do + rm -rf autom4te*.cache + before the following runs. On fast machines the new configure.ac + could otherwise have the same timestamp as the old autom4te.cache. + + Use filenames with two consecutive spaces when testing that some + code preserves filenames with spaces. This will catch errors like + `echo $filename | ...`. + + Make sure your test script can be used to faithfully check an + installed version of automake (as with "make installcheck"). For + example, if you need to copy or grep an automake-provided script, + do not assume that they can be found in the '$top_srcdir/lib' + directory, but use '$am_scriptdir' instead. The complete list of + such "$am_...dir" variables can be found in tests/defs-static.in. + + When writing input for lex, include the following in the definitions + section: + %{ + #define YY_NO_UNISTD_H 1 + %} + to accomodate non-ANSI systems, since GNU flex generates code that + includes unistd.h otherwise. Also add: + %option never-interactive + to the definitions section if the generated code is to be compiled + by a C++ compiler, for similar reasons (i.e., the isatty(3) function + from that same unistd.h header would be required otherwise). + + Before commit: make sure the test is executable, add the tests to + TESTS in Makefile.am, add it to XFAIL_TESTS in addition if needed, + write a ChangeLog entry, send the diff to <automake-patches@gnu.org>. + + +Do not +------ + + Do not test an Automake error with "$AUTOMAKE && Exit 1", or in three + years we'll discover that this test failed for some other bogus reason. + This happened many times. Better use something like + AUTOMAKE_fails + grep 'expected diagnostic' stderr + (Note this doesn't prevent the test from failing for another reason, + but at least it makes sure the original error is still here). + + Do not override Makefile variables using make arguments, as in e.g.: + $MAKE prefix=/opt install + This is not portable for recursive targets (targets that call a + sub-make may not pass "prefix=/opt" along). Use the following + instead: + prefix=/opt $MAKE -e install |