diff options
Diffstat (limited to 't/README')
| -rw-r--r-- | t/README | 66 |
1 files changed, 35 insertions, 31 deletions
@@ -1,4 +1,4 @@ - The Automake test suite + The Automake-NG test suite User interface @@ -8,10 +8,16 @@ User interface Running the tests ----------------- - To run all tests: + To run (almost) all tests: make -k check + With the above, few tests (which are very time-consuming) will be skipped. + If you want to run them too, for extra coverage, you can use either of: + + make -k check-expensive + make -k check RUN_EXPENSIVE_TESTS=yes + By default, verbose output of a test 't/foo.sh' or 't/foo.tap' is retained in the log file 't/foo.log'. Also, a summary log is created in the file 'test-suite.log' (in the top-level directory). @@ -27,17 +33,16 @@ Running the tests To run only tests that are newer than their last results: - make -k check RECHECK_LOGS= + make -k check AM_LAZY_CHECK=yes To run only selected tests: - make -k check TESTS="t/foo.sh t/bar.tap" (GNU make) - env TESTS="t/foo.sh t/bar.tap" make -e -k check (non-GNU make) + make -k check TESTS="t/foo.sh t/bar.tap" To run the tests in cross-compilation mode, you should first configure - the automake source tree to a cross-compilation setup. For example, to - run with a Linux-to-MinGW cross compiler, you will need something like - this: + the Automake-NG source tree to a cross-compilation setup. For example, + to run with a Linux-to-MinGW cross compiler, you will need something + like this: ./configure --host i586-mingw32msvc --build i686-pc-linux-gnu @@ -67,10 +72,10 @@ Interpretation About the tests --------------- - There are two kinds of tests in the Automake testsuite (both implemented - as shell scripts). The scripts with the '.sh' suffix are "simple" - tests, their outcome completely determined by their exit status. Those - with the '.tap' suffix use the TAP protocol. + There are two kinds of tests in the Automake-NG testsuite (both + implemented as shell scripts). The scripts with the '.sh' suffix + are "simple" tests, their outcome completely determined by their + exit status. Those with the '.tap' suffix use the TAP protocol. If you want to run a test by hand, you should be able to do so using the 'runtest' script provided in the Automake distribution: @@ -91,8 +96,7 @@ Supported shells '/bin/my-sh': # Running through the makefile test driver. - make check AM_TEST_RUNNER_SHELL=/bin/my-sh (GNU make) - AM_TEST_RUNNER_SHELL=/bin/my-sh make -e check (non-GNU make) + make check AM_TEST_RUNNER_SHELL=/bin/my-sh # Run a test directly from the command line. AM_TEST_RUNNER_SHELL=/bin/my-sh ./runtest t/foo.sh @@ -108,8 +112,8 @@ Reporting failures ------------------ Send verbose output, i.e., the contents of test-suite.log, of failing - tests to <bug-automake@gnu.org>, along with the usual version numbers - (which Automake, which Autoconf, which operating system, which make + tests to <automake-ng@gnu.org>, along with the usual version numbers + (which Automake-NG, which Autoconf, which operating system, which make version, which shell, etc.) @@ -164,14 +168,14 @@ Writing test cases set the shell variable 'am_serial_tests' to "yes" before including test-init.sh. -* Some tests in the Automake testsuite are auto-generated; those tests - might have custom extensions, but their basename (that is, with such - extension stripped) is expected to end with "-w" string, optionally - followed by decimal digits. For example, the name of a valid - auto-generated test can be 'color-w.sh' or 'tap-signal-w09.tap'. +* Some tests in the Automake-NG testsuite are auto-generated; those + tests might have custom extensions, but their basename (that is, + with such extension stripped) is expected to end with "-w" string, + optionally followed by decimal digits. For example, the name of a + valid auto-generated test can be 'color-w.sh' or 'tap-signal-w09.tap'. Please don't name hand-written tests in a way that could cause them - to be confused with auto-generated tests; for example, 'u-v-w.sh' - or 'option-w0.tap' are *not* valid name for hand-written tests. + to be confused with auto-generated tests; for example, 'u-v-w.sh' or + 'option-w0.tap' are *not* valid name for hand-written tests. * test-init.sh brings in some commonly required files, and sets a skeleton configure.ac. If possible, append to this file. In some cases you'll @@ -231,8 +235,8 @@ Writing test cases * Use '$PATH_SEPARATOR', not hard-coded ':', as the separator of PATH's entries. -* It's more important to make sure that a feature works, than make - sure that Automake's output looks correct. It might look correct +* It's more important to make sure that a feature works, than make sure + that the output of Automake-NG looks correct. It might look correct and still fail to work. In other words, prefer running 'make' over grepping Makefile.in (or do both). @@ -249,8 +253,8 @@ Writing test cases `echo $filename | ...`. * Make sure your test script can be used to faithfully check an - installed version of automake (as with "make installcheck"). For - example, if you need to copy or grep an automake-provided script, + installed version of Automake-NG (as with "make installcheck"). For + example, if you need to copy or grep an Automake-NG provided script, do not assume that they can be found in the '$top_srcdir/lib' directory, but use '$am_scriptdir' instead. The complete list of such "$am_...dir" variables can be found in the 't/ax/test-defs.in' @@ -272,10 +276,6 @@ Writing test cases by a C++ compiler, for similar reasons (i.e., the isatty(3) function from that same unistd.h header would be required otherwise). -* Add any new test to handwritten_TESTS in 't/list-of-tests.mk', and - to XFAIL_TESTS in addition if needed (that is, if the test is expected - to fail). - * In test scripts, prefer using POSIX constructs over their old Bourne-only equivalents: @@ -296,6 +296,10 @@ Writing test cases years we'll discover that this test failed for some other bogus reason. This happened many times. Better use something like +* Do not test an Automake-NG error with "$AUTOMAKE && Exit 1", or in + three years we'll discover that this test failed for some other bogus + reason. This happened many times. Better use something like + AUTOMAKE_fails grep 'expected diagnostic' stderr |