parallel-tests: allow each test to have multiple results

With this change, we improve the code creating the `test-suite.log'
global log and the console testsuite summary to make it	able to
grasp multiple results per test script.  This is required in order
to introduce the planned support for test protocols, like TAP and
SubUnit, which can indeed run multiple tests per test script, each
with its individual result.

The implementation makes use of a custom reStructuredText field
`:test-result:'.

Note that no new documentation is added by this change; that is
be left for follow-up changes.

* lib/check.am ($(TEST_SUITE_LOG)): When processing .log files,
recognize a report of a test's result only if it is declared with
the custom `:test-result:' reStructuredText field placed at the
beginning of a line.  Extend and add explanatory comments.
(recheck, recheck-html): Add explanatory comments.
* lib/test-driver: Write an appropriate reStructuredText field
`:test-result:' in the generated log file.  Use a reStructuredText
transition to better separate the test outcome report from the
test script's registered output.  Improve comments.
* tests/test-driver-custom-xfail-tests.test: Adjust.
* tests/parallel-tests7.test: Adjust.
* tests/parallel-tests-empty-testlogs.test: New test.
* tests/parallel-tests-recheck-override.test: Likewise.
* tests/parallel-tests2.test: Extend and keep more in-sync with ...
* tests/test-driver-custom-html.test: ... this new related test.
* tests/test-driver-custom-no-html.test: New test.
* tests/test-driver-custom-multitest.test: Likewise.
* tests/test-driver-custom-multitest-recheck.test: Likewise.
* tests/test-driver-custom-multitest-recheck2.test: Likewise.
* tests/trivial-test-driver: New file, used by the last four tests
above.
* tests/Makefile.am (TESTS): Update.
(EXTRA_DIST): Distribute `trivial-test-driver'.
(test-driver-custom-multitest.log): Depend on `trivial-test-driver'.
(test-driver-custom-multitest-recheck.log): Likewise.
(test-driver-custom-multitest-recheck2.log): Likewise.
(test-driver-custom-html.log): Likewise.

1 files changed, 28 insertions, 15 deletions

diff --git a/tests/parallel-tests2.test b/tests/parallel-tests2.test
index 8fe5d3027..ab390f862 100755
--- a/tests/parallel-tests2.test
+++ b/tests/parallel-tests2.test
@@ -15,8 +15,9 @@
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 # Check parallel-tests features:
-# - check-html
-# - recheck-html
+#  - check-html
+#  - recheck-html
+# Keep this in sync with sister test `test-driver-custom-html.test'.
 
 parallel_tests=yes
 required=rst2html
@@ -35,22 +36,25 @@ bla:
 CLEANFILES = bla
 END
 
-cat >>foo.test <<'END'
+cat > foo.test <<'END'
 #! /bin/sh
 echo "this is $0"
 test -f bla || exit 1
 exit 0
 END
-cat >>bar.test <<'END'
+
+cat > bar.test <<'END'
 #! /bin/sh
 echo "this is $0"
 exit 99
 END
-cat >>baz.test <<'END'
+
+cat > baz.test <<'END'
 #! /bin/sh
 echo "this is $0"
 exit 1
 END
+
 chmod a+x foo.test bar.test baz.test
 
 $ACLOCAL
@@ -58,30 +62,39 @@ $AUTOCONF
 $AUTOMAKE -a
 
 ./configure
-$MAKE check-html >stdout && { cat stdout; Exit 1; }
-cat stdout
+
+$MAKE check-html && Exit 1
 test -f mylog.html
+# check-html should cause check_SCRIPTS to be created.
+test -f bla
+
+# "make clean" should remove HTML files.
+$MAKE clean
+test ! -f mylog.html
+test ! -f bla
 
 # Always create the HTML output, even if there were no failures.
 rm -f mylog.html
-env TESTS=foo.test $MAKE -e check-html >stdout || { cat stdout; Exit 1; }
-cat stdout
+env TESTS=foo.test $MAKE -e check-html
 test -f mylog.html
 
-# Create HTML output also with recheck-html
+# Create summarizing HTML output also with recheck-html.
 rm -f mylog.html
-env TESTS=foo.test $MAKE -e recheck-html >stdout || { cat stdout; Exit 1; }
-cat stdout
+env TESTS=foo.test $MAKE -e recheck-html
 test -f mylog.html
 
-# check-html and recheck-html should cause check_SCRIPTS to be created,
-# and recheck-html should rerun no tests if check has not been run.
+# check-html should cause check_SCRIPTS to be created.
 $MAKE clean
-env TESTS=foo.test $MAKE -e check-html
+env TEST_LOGS=foo.log $MAKE -e check-html
 test -f bla
+test -f foo.log
+test -f mylog.html
+# recheck-html should cause check_SCRIPTS to be created, and should rerun
+# no tests if it appears that check has not been run.
 $MAKE clean
 env TESTS=foo.test $MAKE -e recheck-html
 test -f bla
 test ! -f foo.log
 test -f mylog.html
+
 :