summaryrefslogtreecommitdiff
path: root/lib/test-driver
Commit message (Collapse)AuthorAgeFilesLines
* maint: Update copyright years to 2017.Mathieu Lirzin2017-03-021-1/+1
| | | | This update has been made with 'make update-copyright'.
* maint: port time-stamp-time-zone to strict POSIXPaul Eggert2016-01-121-2/+2
| | | | | Set time-stamp-time-zone to "UTC0", not to "UTC", as POSIX defines TZ="UTC0" not TZ="UTC".
* maint: update copyright years to 2015 (branch 'micro')Stefano Lattarini2015-01-051-1/+1
| | | | Signed-off-by: Stefano Lattarini <stefano.lattarini@gmail.com>
* maint: update copyright yearsStefano Lattarini2014-04-211-1/+1
| | | | | | We've been in 2014 already for few months now... Signed-off-by: Stefano Lattarini <stefano.lattarini@gmail.com>
* testsuite harness: report test exit status in log fileStefano Lattarini2013-12-241-2/+11
| | | | | | | | | | | | | | | | | | The exit status of a test should be reported in the test logs, so that one can see at a glance whether the test has succeeded or failed, without having to look also into the corresponding .trs file. This fixes automake bug#11814. * lib/test-driver: Also report the test script exit status in the test log (as the last line). * t/check-exit-status-reported.sh: Test this new behaviour. * t/list-of-tests.mk: Add the new test. * t/ax/test-lib.sh( am_exit_trap): No longer log the test exit status; this has been made redundant by the change to 'test-driver'. While at it, fix an imperfect quoting. Signed-off-by: Stefano Lattarini <stefano.lattarini@gmail.com>
* test harness: improve catching of usage errors in script 'test-driver'Stefano Lattarini2013-07-211-4/+16
| | | | | | | | | | | | | Fixes automake bug#14840. * lib/test-driver: Catch and report usage errors where the caller has forgotten to specify one of the mandatory options (--test-name, --log-file, --trs-file) or has not passed any non-option argument. Also, be sure to work correctly even when no '--' special argument is passed to separate option from non-options arguments. * THANKS: Update. Signed-off-by: Stefano Lattarini <stefano.lattarini@gmail.com>
* maint: update copyright year for 2013 (in branch maint)Stefano Lattarini2012-12-311-1/+1
| | | | Signed-off-by: Stefano Lattarini <stefano.lattarini@gmail.com>
* Merge branch 'maint'Stefano Lattarini2012-06-301-3/+3
|\ | | | | | | | | | | | | | | * maint: parallel-tests: "recheck" behaves better in case of compilation failures scripts: quote 'like this', not `like this' Signed-off-by: Stefano Lattarini <stefano.lattarini@gmail.com>
| * scripts: quote 'like this', not `like this'Stefano Lattarini2012-06-271-4/+4
| | | | | | | | | | | | * lib/test-driver.sh: Here. Signed-off-by: Stefano Lattarini <stefano.lattarini@gmail.com>
* | check: the parallel-tests driver is now the defaultStefano Lattarini2012-05-181-2/+2
|/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The old serial testsuite driver is still supported, and can be activated by the 'serial-tests' option (introduced by commit 'v1.11-1989-gdeb7773' of 2012-02-20, "automake: new option 'serial-tests'"). * lib/Automake/Options.pm: Now, the 'parallel-tests' option is a no-op (but still explicitly recognized, for backward compatibility), while the 'serial-tests' option is recognized and registered. * automake.in: Adjust to ensure that the parallel test harness is used by default; mostly, this amounts at using "! option('serial-tests')" where "option('parallel-tests')" was used before. * NEWS, doc/automake.texi: Update. * configure.ac (AM_INIT_AUTOMAKE): Drop 'parallel-tests' option, is redundant now. * defs: In the generated AM_INIT_AUTOMAKE call, use the 'serial-tests' when the serial testsuite harness is to be used, and no option otherwise. * lib/am/check.am: Use (static) conditional '%SERIAL_TESTS%' instead of conditional '%PARALLEL_TESTS%'. * lib/test-driver: Adjust heading comments. * t/ax/extract-testsuite-summary.pl: Likewise. * t/check-exported-srcdir.sh: Likewise. * t/check-subst.sh: Likewise. * t/java-compile-run-nested.sh: Likewise. * t/parallel-tests-dry-run-1.sh: Likewise. * t/parallel-tests-dry-run-2.sh: Likewise. * t/parallel-tests-interrupt.sh: Likewise. * t/parallel-tests-many.sh: Likewise. * t/test-metadata-global-log.sh: Likewise. * t/tests-environment-backcompat.sh: Likewise. * m4/init.m4 (AM_INIT_AUTOMAKE): Adjust a comment. * t/check8.sh: Likewise. * t/check-tests-in-builddir.sh: Likewise. * t/java-compile-run-flat.sh: Adjust a botched comment. * t/ax/tap-summary-aux.sh: Remove explicit use of the 'parallel-tests' option. * t/ax/testsuite-summary-checks.sh: Likewise. * t/aclocal-path-precedence.sh: Likewise. * t/dist-aux-many-subdirs.sh: Likewise. * t/parallel-tests-no-color-in-log.sh: Likewise. * t/parallel-tests-reset-term.sh: Likewise. * t/parallel-tests-subdir.sh: Likewise. * t/check-no-test-driver.sh: Fix setup to adjust to the changed semantics. Update heading comments. * t/parallel-tests-driver-install.sh: Adjust to the new semantic. * t/test-driver-is-distributed.sh: Simplified according to the new semantics. Signed-off-by: Stefano Lattarini <stefano.lattarini@gmail.com>
* maint: run "make update-copyright"Stefano Lattarini2012-02-161-1/+1
|
* parallel-tests: fix help screen for test driver scriptsStefano Lattarini2011-08-171-2/+2
| | | | | | | | * lib/tap-driver.pl ($USAGE): The `--trs-file' option is mandatory too. * lib/tap-driver.sh (print_usage): Likewise. * lib/test-driver (print_usage): Likewise. ($scriptversion): Update.
* check: add small "synchronization" comment to `test-driver'Stefano Lattarini2011-08-051-0/+1
| | | | | | * lib/test-driver: Add comment to the code initializing ANSI color escapes, telling to keep it in sync with the similar initialization in `lib/am/check.am'.
* check: small cleanups and refactorings in test harness and driversStefano Lattarini2011-08-051-17/+11
| | | | | | | | | | | | | | | | | | | | | * lib/tap-driver (yn): New subroutine, converts a boolean value to a "yes" or "no" string. (write_test_results): Use it, and related reformatting. * lib/test-driver (trap): Also remove the `.trs' file on signals, for extra safety. (fatal): Remove this function, it's never used. ($logfile, $trsfile): Renamed ... ($log_file, $trs_file): ... to these, for clarity, and in order to be more consistent with the `tap-driver' script. Improve a couple of comments. * automake.in (handle_tests): Don't define anymore the now-obsolete make macro `$(TEST_LOGS_TMP)', nor add it to the list of files to be removed upon "make mostlyclean". * lib/am/check.am ($(TEST_SUITE_LOG)): New shell function `f_ok', tells whether a path refers to an existing, regular, readable file. Use it throughout. (recheck): Be safer w.r.t. make implementation that run recipes with `errexit' shell flag active.
* test harness: be aware of more metadata, simplify test driversStefano Lattarini2011-08-051-17/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | * lib/am/check.am ($(TEST_SUITE_LOG)): When producing the global test log, take into account the new metadata field `:global-test-result:, to write nicely formatted RST section titles, instead of leaving this chore to the individual test drivers. (am_rst_section): Re-introduce this variable, as removed in commit `v1.11-870-ga27c9c4'. * lib/test-driver, lib/tap-driver: Write the global test result as metadata in the `.trs' file, not as (part of) a formatted RST subsection title in the `.log' file. Related simplifications; in particular, get rid of temporary files usage. * doc/automake.texi (Log files generation and test results recording): Document the new metadata. * tests/check12.test: Update. * tests/parallel-tests-harderror.test: Likewise. * tests/parallel-tests-interrupt.test: Likewise. * tests/parallel-tests-log-override-2.test: Likewise. * tests/parallel-tests-log-override-recheck.test: Likewise. * tests/parallel-tests-unreadable.test: Likewise. * tests/tap-global-result.test: Likewise. * tests/test-metadata-results.test: Likewise. * tests/test-log.test: Likewise, and another minor unrelated fix. * tests/test-metadata-global-result.test: New test. * tests/Makefile.am (TESTS): Update.
* test harness: use new `.trs' files to hold test metadataStefano Lattarini2011-08-021-19/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | With this change, the test harness will keep test metadata in dedicated `.trs' files, instead of having them embedded into the `.log' files. This allows for easier forward-compatibility and extension of test metadata, and for more flexibility in the format of the `.log' files. Note that this change makes the `:end-metadata:' field obsolete. * doc/automake.texi (Parallel Test Harness, Log files generation and test results recording): Document the new APIs and behaviour; some related minor rewordings and fixlets. * NEWS: Update. * automake.in (handle_tests): When bringing in the content of `check2.am', substitute %BASE% with the basename of the `.log' file being created by a rule. Add the generated `.trs' files to the list of files to be cleaned by "make mostlyclean". * lib/am/check.am (am__test_driver_flags): Rename ... (am__common_driver_flags): ... to this, and remove the flags `--test-name' and `--log-file' from it: they are now define in the proper rules in `check2.am'. (am__TEST_BASES): New internal variable, holding the names of the tests, with any registered extension removed. (am__stealth_MAKE): New internal variable, can be used instead of $(MAKE) in recipes requiring a recursive call to make, but which are not intended to be executed by "make -n". (.log.trs): New suffix rule, to recover from deletion of `.trs' files. ($(TEST_SUITE_LOG)): Almost completely rewritten to follow the new API of "test logs in `.log' files, test metadata in `.trs' files". It goes to some length to work correctly in face of unreadable or missing `.log' and `.trs' files, and to error out with proper error messages when this is not possible. [%?PARALLEL_TESTS%] (check-TESTS): Also remove relevant "stale" `.trs' files (in addition to `.log files) before remaking the $(TEST_SUITE_LOG). (recheck, recheck-html): Look for the `:recheck:' field in the `.trs' files, not in the `.log' files. * lib/am/check2.am (?GENERIC?%EXT%.log, ?!GENERIC?%OBJ%): Adjust the call to the test driver, in particularly passing the new option `--trs-file'. [%am__EXEEXT%] (?GENERIC?%EXT%$(EXEEXT).log): Likewise. * lib/tap-driver ($USAGE): Adjust the help screen. (Getopt::Long::GetOptions): Handle the `--trs-file' option, through the use of ... ($trs_file): ... this new global variable. (finish): Write metadata for the test run to `$trs_file' rather then to `$log_file', through the use of ... (write_test_results): ... this new function. * lib/test-driver (print_usage): Update the help screen. (Option parsing): Handle the `--trs-file' option, through the use of ... ($resfile): ... this new global variable. (Main code): Write metadata for the test run to `$trsfile' rather than to `$logfile'. Minor related adjustments to comments. * tests/.gitignore: Ignore `*.trs' files. * tests/parallel-tests-unreadable-log.test: Moved ... * tests/parallel-tests-unreadable.test: ... to this, and extended to also check the semantics for unreadable `.trs' files. * tests/test-driver-end-metadata.test: Deleted as obsolete. * tests/test-driver-metadata-no-leading-space.test: Likewise. * tests/test-driver-global-log.test: Renamed ... * tests/test-metadata-global-log.test: ... to this, and modified as to verify the new APIs and semantics. * tests/test-driver-recheck: Renamed ... * tests/test-metadata-recheck.test: ... to this, and modified likewise. * tests/parallel-tests-once.test: New test. * tests/parallel-tests-make-n.test: Likewise. * test-metadata-results.test: Likewise. * test-missing.test: Likewise. * test-missing2.test: Likewise. * test-trs-basic.test: Likewise. * test-trs-recover.test: Likewise. * test-trs-recover2.test: Likewise. * tests/Makefile.am (TESTS): Update.
* test harness: allow more metadata in log filesStefano Lattarini2011-08-021-9/+17
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This change reworks and improves the parallel test harness to use more specialized reStructuredText fields in the log files (instead of relying on specially-placed of "magic lines" and more indirect semantical formatting); the new fields are the following: - ":recheck:": tell whether the associated test will have to be re-run by "make recheck"; - ":copy-in-global-log:": tell whether the content of the log file should be copied in the "global log" `test-suite.log'; - ":end-metadata:", which inhibits the scanning of the rest of the log file (for what concerns test metadata). Also, the special `:test-result:' value "END" has been removed, superseded by the new `:end-metadata:' field. * doc/automake.texi (Log files generation and test results recording): Document the new API and semantics. Remove or fix some obsolete comments. * lib/am/check.am ($(TEST_SUITE_LOG), recheck, recheck-html): Adjust comments and code. * lib/tap-driver (finish): Adjust, with the help of ... (must_recheck, copy_in_global_log): ... these new functions. * lib/test-driver (Main code): Adjust, with the help of ... ($recheck, $gcopy): ... these new variables. * tests/trivial-test-driver: Update to obey the new APIs. * tests/test-driver-recheck.test: Likewise. * tests/test-driver-global-log.test: Likewise. * tests/tap-passthrough.test: Relax the test, avoiding to check what is written in `test-suite.log'; such check has been moved ... * tests/tap-global-log.test: ... in this new test, and extended. * tests/test-driver-metadata-no-leading-space.test: New test. * tests/test-driver-end-test-results.test: Removed, it checked the old APIs; superseded by ... * tests/test-driver-end-metadata.test: ... this new test. * tests/tap-log.test: Improve syncing with ... * tests/test-log.test: ... this new test. * tests/parallel-tests.test: Remove some duplication w.r.t. this last new test. Updated heading comments. * tests/Makefile.am (TESTS): Update.
* parallel-tests: new recognized test result 'ERROR'Stefano Lattarini2011-07-211-2/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * lib/am/check.am ($(TEST_SUITE_LOG)): Recognize a new test result `ERROR'. Use it when encountering unreadable test logs (previously a simple `FAIL' was used in this situations). * lib/test-driver: Set the global test result to `ERROR' when the test exit status is 99. When doing colorized output, color `ERROR' results in magenta. * doc/automake.texi (Log files generation and test results recording): Update by also listing `ERROR' among the list of valid `:test-results:' arguments. * NEWS: Update. * tests/trivial-test-driver: Update. * tests/parallel-tests.test: Likewise. * tests/parallel-tests-harderror.test: Likewise. * tests/parallel-tests-no-spurious-summary.test: Likewise. * tests/test-driver-global-log.test: Likewise. * tests/test-driver-recheck.test: Likewise. * tests/test-driver-custom-multitest-recheck.test: Likewise. * tests/test-driver-custom-multitest-recheck2.test: Likewise. * tests/test-driver-custom-multitest.test: Likewise. * tests/test-driver-custom-no-html.test: Likewise. * tests/test-driver-end-test-results.test: Likewise. * tests/color.test: Likewise. Also, make stricter, and also test from VPATH. * tests/color2.test: Likewise, and improve syncing with color.test. * tests/parallel-tests-exit-statuses.test: New test. * tests/parallel-tests-console-output.test: Likewise. * tests/Makefile.am (TESTS): Update.
* parallel-tests: make parsing of test results saferStefano Lattarini2011-07-061-2/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The new code for parsing the testsuite-generated `.log' files, as introduced in commit `v1.11-872-gc96b881', considers each `:test-result:' field anywhere in a `.log' file as a declaration of a test result, and accounts for it as such in the testsuite summary. Unfortunately this could easily cause spurious test failures being reported in the testsuite summary. This happened in practice with the Automake's own testsuite; for example: $ make check TESTS='check12-p.test'; echo exit: $? ... PASS: check12-p.test ===================================== 4 of 5 tests failed See tests/test-suite.log Please report to bug-automake@gnu.org ===================================== make[2]: *** [test-suite.log] Error 1 make: *** [check-am] Error 2 exit: 2 This change introduces a new special `:test-result:' "END", that, when seen, prevents the rest of the log file from being parsed. For more information, refer to the thread: <http://lists.gnu.org/archive/html/automake-patches/2011-06/msg00199.html> * lib/am/check.am ($(TEST_SUITE_LOG)): Stop the parsing of a log file as soon as the special ":test-result:END" directive is seen. Related changes and enhancements. * lib/test-driver: Protect the rest of the log after the result lined with a ":test-result:END" directive. * tests/parallel-tests-no-spurious-summary.test: New test. * tests/test-driver-end-test-results.test: Likewise. * tests/Makefile.am (TESTS): Update.
* parallel-tests: allow each test to have multiple resultsStefano Lattarini2011-06-211-2/+14
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | With this change, we improve the code creating the `test-suite.log' global log and the console testsuite summary to make it able to grasp multiple results per test script. This is required in order to introduce the planned support for test protocols, like TAP and SubUnit, which can indeed run multiple tests per test script, each with its individual result. The implementation makes use of a custom reStructuredText field `:test-result:'. Note that no new documentation is added by this change; that is be left for follow-up changes. * lib/check.am ($(TEST_SUITE_LOG)): When processing .log files, recognize a report of a test's result only if it is declared with the custom `:test-result:' reStructuredText field placed at the beginning of a line. Extend and add explanatory comments. (recheck, recheck-html): Add explanatory comments. * lib/test-driver: Write an appropriate reStructuredText field `:test-result:' in the generated log file. Use a reStructuredText transition to better separate the test outcome report from the test script's registered output. Improve comments. * tests/test-driver-custom-xfail-tests.test: Adjust. * tests/parallel-tests7.test: Adjust. * tests/parallel-tests-empty-testlogs.test: New test. * tests/parallel-tests-recheck-override.test: Likewise. * tests/parallel-tests2.test: Extend and keep more in-sync with ... * tests/test-driver-custom-html.test: ... this new related test. * tests/test-driver-custom-no-html.test: New test. * tests/test-driver-custom-multitest.test: Likewise. * tests/test-driver-custom-multitest-recheck.test: Likewise. * tests/test-driver-custom-multitest-recheck2.test: Likewise. * tests/trivial-test-driver: New file, used by the last four tests above. * tests/Makefile.am (TESTS): Update. (EXTRA_DIST): Distribute `trivial-test-driver'. (test-driver-custom-multitest.log): Depend on `trivial-test-driver'. (test-driver-custom-multitest-recheck.log): Likewise. (test-driver-custom-multitest-recheck2.log): Likewise. (test-driver-custom-html.log): Likewise.
* parallel-tests: add auxiliary script 'test-driver', refactorStefano Lattarini2011-06-211-0/+129
This refactoring should cause no API of functionality change, and is meant only to simplify the future implementation of TAP and SubUnit testsuite drivers. More precisely, our roadmap is to move most of the "testsuite driving" features out of the Automake-generated Makefiles, and into external scripts with well-defined interfaces. This will allow the user to define its own personalized testsuite drivers, and will also offer us a framework upon which to implement our new TAP and SubUnit drivers, all in a very unobtrusive way and retaining an high degree of code reuse and backward-compatibility. * lib/test-driver: New auxiliary script. * lib/Makefile.am (dist_SCRIPT_DATA): Add it. * automake.in (handle_tests): Require the new auxiliary script `test-driver', and define a new internal makefile variable `$(am__test_driver)', used to call it. Perform new substitution on `DRIVER' when processing the `check2.am' file. * lib/check.am (am__tty_colors): Define new shell variable `$am__color_tests'. (am__rst_section): Removed, its role taken over by the new `test-driver' script. (am__test_driver_flags): New variable, contains the command line options passed to `test-driver'. (am__check_pre): Do not deal with temporary files and exit traps anymore, as the `test-driver' script takes care of that now. Define shell variable `$am__enable_hard_errors', used by `$(am__test_driver_flags)'. Reorder so that we don't need to save and restore the value of the `TERM' environment variable anymore. Other related adjustments. (am__check_post): Remove, as its role has been completely taken over by the `test-driver' script. * am/check2.am (?GENERIC?%EXT%$(EXEEXT).log, ?GENERIC?%EXT%.log, ?!GENERIC?%OBJ%): Call the test script through the Automake substituted `%DRIVER%', and honor the command-line options in `$(am__test_driver_flags)'. Do not call the obsoleted `$(am__check_post)' anymore. * doc/automake.texi (Auxiliary Programs): Mention the new `test-driver' script. (Optional): Mention `test-driver' in AC_CONFIG_AUX_DIR. Since we are at it, break the list of auxiliary scripts by placing one per line, to simplify potential future additions of new scripts. * tests/check.test: Adjust. * tests/check2.test : Likewise. * tests/check3.test : Likewise. * tests/check4.test : Likewise. * tests/check10.test: Likewise. * tests/color.test: Likewise. * tests/color2.test: Likewise. * tests/comment9.test: Likewise. * tests/dejagnu.test: Likewise. * tests/exeext4.test: Likewise. * tests/maken3.test: Likewise. * tests/maken4.test: Likewise. * tests/parallel-tests-interrupt.test: Likewise. * tests/posixsubst-tests.test: Likewise. * tests/repeated-options.test: Likewise. * tests/check-no-test-driver.test: New test. * tests/parallel-test-driver-install.test: Likewise. * tests/Makefile.am (TESTS): Update. * NEWS: Update.
View Lorry Controller Status