test harness: allow more metadata in log files

This change reworks and improves the parallel test harness to use
more specialized reStructuredText fields in the log files (instead
of relying on specially-placed of "magic lines" and more indirect
semantical formatting); the new fields are the following:
 - ":recheck:": tell whether the associated test will have to be
   re-run by "make recheck";
 - ":copy-in-global-log:": tell whether the content of the log
   file should be copied in the "global log" `test-suite.log';
 - ":end-metadata:", which inhibits the scanning of the rest of
   the log file (for what concerns test metadata).
Also, the special `:test-result:' value "END" has been removed,
superseded by the new `:end-metadata:' field.

* doc/automake.texi (Log files generation and test results
recording): Document the new API and semantics.  Remove or fix
some obsolete comments.
* lib/am/check.am ($(TEST_SUITE_LOG), recheck, recheck-html):
Adjust comments and code.
* lib/tap-driver (finish): Adjust, with the help of ...
(must_recheck, copy_in_global_log): ... these new functions.
* lib/test-driver (Main code): Adjust, with the help of ...
($recheck, $gcopy): ... these new variables.
* tests/trivial-test-driver: Update to obey the new APIs.
* tests/test-driver-recheck.test: Likewise.
* tests/test-driver-global-log.test: Likewise.
* tests/tap-passthrough.test: Relax the test, avoiding to check
what is written in `test-suite.log'; such check has been moved ...
* tests/tap-global-log.test: ... in this new test, and extended.
* tests/test-driver-metadata-no-leading-space.test: New test.
* tests/test-driver-end-test-results.test: Removed, it checked
the old APIs; superseded by ...
* tests/test-driver-end-metadata.test: ... this new test.
* tests/tap-log.test: Improve syncing with ...
* tests/test-log.test: ... this new test.
* tests/parallel-tests.test: Remove some duplication w.r.t. this
last new test.  Updated heading comments.
* tests/Makefile.am (TESTS): Update.

1 files changed, 140 insertions, 0 deletions

diff --git a/tests/tap-global-log.test b/tests/tap-global-log.test
new file mode 100755
index 000000000..da5aeb8bd
--- /dev/null
+++ b/tests/tap-global-log.test
@@ -0,0 +1,140 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+#  - which log files get copied in the global log?
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+  || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > ok.test << 'END'
+1..5
+ok 1
+ok 2
+ok 3
+not seen in global log
+ok 4
+ok 5
+END
+
+cat > top << 'END'
+1..6
+ok 1
+ok 2
+ok 3
+END
+
+cat > bot << 'END'
+ok 5
+ok 6
+END
+
+cat top - bot > skip.test << 'END'
+ok # SKIP
+::skip::
+END
+
+cat top - bot > todo.test << 'END'
+not ok # TODO
+::todo::
+END
+
+cat top - bot > fail.test << 'END'
+not ok
+::fail::
+END
+
+cat top - bot > xpass.test << 'END'
+ok # TODO
+::xpass::
+END
+
+cat top - bot > bail.test << 'END'
+::bail::
+Bail out!
+END
+
+cat top - bot > error.test << 'END'
+::error::
+1..7
+END
+
+# Created with "dd if=/dev/urandom count=1 | base64-encode"
+cat > hodgepodge <<'END'
+1+0 records in
+1+0 records out
+512 bytes (512 B) copied, 0.000241092 s, 2.1 MB/s
+gdivnV4VhL4DOzhE3zULJuun3PwqqQqMdATVcZbIQkNgyRvNBoqqHMBQEs7QsjDbp2nK+Szz
+EcelGyvXmHrW7yImaom6Yrg95k31VCmp/pGDRnTDwDrcOPJiv9jDReEmTAQkPuqLO+mFNly+
+DDHM9fNbzGumstsQ3wq3DOXz1pCV3JXwhjeaHSboeEbmr55bX0XHLSKaecctA0rXDXEyZWZ/
+ODlawSrAXzw0H7a+xBwjnNXZ3zYiwk3x+WQrPqNjb+qXiLLTxAKzx2/KnaFhxkPlte5jPRNB
+FciDolL+H/10LsCdSzLOrGnY2zH6vL2JMZfxjnb73zWFcdWWE01LTD7wpN5O1MP3+N47lcVe
+lWbkD04cJvhwxLElYSO24B743GG5EyGYt9SeZRE6xbgwq3fVOS8KqjwGxwi4adSBTtw0CV8W
+S/6n8dck1vBvjA+qpk0zMSYSqc3+jzW9UiGTmTEIwfw80p/lGNsfjcNBJ86nFkWUnHmrsi8k
+Dv57sK70mTg239g08f5Uvdga/5UreMBSgB0hUj5sbq57r7B1fsVr0Kag468la8zKy3ZEZ0gX
+++sbaJ9WGHhnKvjooeH+4Y6HwAFsdINde++FlCmp4EuNKKEEuXbSKLaOTy3+6pJ2DYdvRCL+
+frZwxH4hcrw8qh+8IakB02viewZS/qT57v4=
+END
+
+exec 5>misc.test
+echo 'ok # SKIP' >&5
+cat hodgepodge >&5
+echo 'not ok # TODO' >&5
+echo 'ok' >&5
+exec 5>&-
+
+cat > skipall.test << 'END'
+1..0 # SKIP all
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+END
+echo TESTS = *.test >> Makefile.am
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+# We don't care about the exit status of "make check" here.
+$MAKE check || :
+cat test-suite.log
+
+grep 'ok\.test|not seen' test-suite.log && Exit 1
+
+for s in skip todo fail xpass bail error; do
+  $FGREP "::$s::" test-suite.log
+done
+
+grep '^1\.\.0 # SKIP all$' test-suite.log
+
+case `cat test-suite.log` in
+  *"`cat hodgepodge`"*) ;;
+  *) Exit 1;;
+esac
+
+: