diff options
author | Stefano Lattarini <stefano.lattarini@gmail.com> | 2012-04-20 21:04:36 +0200 |
---|---|---|
committer | Stefano Lattarini <stefano.lattarini@gmail.com> | 2012-04-23 10:01:05 +0200 |
commit | 53b45c7187a1031f9c07cbd93e6e0a8c90e579bd (patch) | |
tree | 61c280b9bf0dd89c82aefe33d3935ed91edc1190 | |
parent | e4412726eb36809bd21b3bb308b2f791e4154e43 (diff) | |
download | automake-53b45c7187a1031f9c07cbd93e6e0a8c90e579bd.tar.gz |
[ng] parallel-tests: simplify automake-time preprocessing
Our new pattern rules, introduced in recent commit 'v1.11b-124-gaeaba3f',
are smart enough to allow us to ditch most of our cumbersome automake-time
analysis and rewriting of the $(TESTS) variable when the parallel-tests
driver is used. This also makes other related simplifications possible.
This change causes a little regression in that we won't be able anymore to
diagnose at automake time test cases that start with the '$(srcdir)' or
'$(top_srcdir)' component, but that's acceptable, since a later commit will
make the former case actually work, and the second case isn't important
enough to worry to much about.
This change also causes a small behavioural change in automake, in that it
will also need the 'LOG_DRIVER' variable to be defined in each Makefile.am
defining the 'TESTS' variable; and if it's not defined, automake will
define it itself, and require the presence of the 'test-driver' auxiliary
script in the process (or bing it in if the "--add-missing" option is in
use). See the changes to 'NG-NEWS' for more details, and for workarounds.
* automake.in (handle_tests): Don't walk nor rewrite $(TESTS) when the
'parallel-tests' option is active. Other related minor simplifications.
* t/exeext4.sh: Define '$am_parallel_tests' to "no", so that the test
won't be run with the 'parallel-tests' option enabled (as the semantics
in that case have been changed in a way that would make this test fail).
* t/check7.sh: When '$am_parallel_tests' is "yes", skip checks that
don't work anymore with the 'parallel-tests' option is enabled.
* t/test-driver-custom-no-extra-driver.sh: Adjust to avoid spurious
failures to to the "LOG_DRIVER must be defined somehow" semantic change.
* t/tap-common-setup.sh: Likewise.
* t/tap-recheck.sh: Likewise.
* t/tap-basic.sh: Likewise.
* t/tap-diagnostic-custom.sh: Likewise.
* t/tap-more.sh: Likewise.
* t/tap-recheck.sh: Likewise.
* t/tap-signal.tap: Likewise.
* t/test-driver-custom-multitest.sh: Likewise.
* t/test-driver-custom-multitest-recheck.sh: Likewise.
* t/test-driver-custom-multitest-recheck2.sh: Likewise.
* t/test-driver-custom-xfail-tests.sh: Likewise.
* t/test-driver-fail.sh: Likewise.
* t/test-driver-strip-vpath.sh: Likewise.
* t/test-metadata-global-log.sh: Likewise.
* t/test-metadata-global-results.sh: Likewise, and enhanced a little since
we are at it.
* t/test-metadata-results.sh: Likewise.
* t/tap-bad-prog.tap: Likewise.
* t/parallel-tests8.sh: Move the now-failing checks verifying that automake
diagnoses test cases that start with the '$(srcdir)' or '$(top_srcdir)'...
* t/parallel-tests-srcdir-in-test-name.sh: ... this new xfailing test.
* t/spy-pattern-rules.sh: New "spy" test, verify our new assumptions on the
precedence of pattern rules truly hold.
* t/parallel-tests-longest-stem.sh: New test.
* t/list-of-tests.mk: Update.
* doc/automake.texi: Document that $(TESTS) is rewritten for $(EXEEXT)
appending only when the 'parallel-tests' option is *not* in use.
Add a "FIXME" comment about the fact that automake does not diagnose test
cases that start with the '$(srcdir)' or '$(top_srcdir)' anymore.
* NG-NEWS: Update.
Signed-off-by: Stefano Lattarini <stefano.lattarini@gmail.com>
-rw-r--r-- | NG-NEWS | 22 | ||||
-rw-r--r-- | automake.in | 108 | ||||
-rw-r--r-- | doc/automake.texi | 6 | ||||
-rwxr-xr-x | t/check7.sh | 9 | ||||
-rwxr-xr-x | t/exeext4.sh | 1 | ||||
-rw-r--r-- | t/list-of-tests.mk | 4 | ||||
-rwxr-xr-x | t/parallel-tests-longest-stem.sh | 66 | ||||
-rwxr-xr-x | t/parallel-tests-srcdir-in-test-name.sh | 41 | ||||
-rwxr-xr-x | t/parallel-tests8.sh | 18 | ||||
-rwxr-xr-x | t/spy-pattern-rules.sh | 54 | ||||
-rwxr-xr-x | t/tap-bad-prog.tap | 13 | ||||
-rwxr-xr-x | t/tap-basic.sh | 2 | ||||
-rwxr-xr-x | t/tap-common-setup.sh | 8 | ||||
-rwxr-xr-x | t/tap-diagnostic-custom.sh | 1 | ||||
-rwxr-xr-x | t/tap-more.sh | 3 | ||||
-rwxr-xr-x | t/tap-recheck.sh | 3 | ||||
-rwxr-xr-x | t/tap-signal.tap | 2 | ||||
-rwxr-xr-x | t/test-driver-custom-multitest-recheck.sh | 2 | ||||
-rwxr-xr-x | t/test-driver-custom-multitest-recheck2.sh | 2 | ||||
-rwxr-xr-x | t/test-driver-custom-multitest.sh | 2 | ||||
-rwxr-xr-x | t/test-driver-custom-no-extra-driver.sh | 5 | ||||
-rwxr-xr-x | t/test-driver-custom-xfail-tests.sh | 2 | ||||
-rwxr-xr-x | t/test-driver-fail.sh | 8 | ||||
-rwxr-xr-x | t/test-driver-strip-vpath.sh | 11 | ||||
-rwxr-xr-x | t/test-metadata-global-log.sh | 1 | ||||
-rwxr-xr-x | t/test-metadata-global-result.sh | 45 | ||||
-rwxr-xr-x | t/test-metadata-results.sh | 25 |
27 files changed, 334 insertions, 130 deletions
@@ -71,6 +71,28 @@ Parallel testsuite harness variables anymore. These have always been meant for internal use only anyway. +* The $(TESTS) and $(XFAIL_TESTS) variables are not anymore rewritten for + $(EXEEXT) appending. The use of compiled programs in $(TESTS) still + works as before though, even on systems where $(EXEEXT) is non-empty, + thanks to a careful use of pattern rules. + +* The new implementation of the parallel-tests harness tries to defer + as much as possible processing on the $(TESTS) variable at make, + rather than doing it at automake time. This as some positive effects + (FIXME: describe and quantify them), but also entails a minor annoyance. + When automake runs, it cannot know anymore whether $(TESTS) contains + or not any entry with no recognized test suffix; as a consequence, it + always need the '$(LOG_DRIVER)' variable to be defined. If it's not, + automake will define it to a proper invocation of the 'test-driver' + auxiliary script, and require that script to be present in the + config-aux directory (or bring it in if "--add-missing" is in use). + If you know you package won't make use of suffix-less tests (thus, in + particular, no use of compiled programs in $(TESTS)), you can avoid + requiring the 'test-driver' script by adding this line to your + configure.ac: + AC_SUBST([LOG_DRIVER], ['unused but required by automake']) + + Miscellaneous ============= diff --git a/automake.in b/automake.in index 64ec53d16..6b9cb9454 100644 --- a/automake.in +++ b/automake.in @@ -4781,27 +4781,12 @@ sub handle_tests PARALLEL_TESTS => !! option 'parallel-tests', CHECK_DEPS => $check_deps); - # Tests that are known programs should have $(EXEEXT) appended. - # For matching purposes, we need to adjust XFAIL_TESTS as well. - append_exeext { exists $known_programs{$_[0]} } 'TESTS'; - append_exeext { exists $known_programs{$_[0]} } 'XFAIL_TESTS' - if (var ('XFAIL_TESTS')); - if (my $parallel_tests = option 'parallel-tests') { define_variable ('TEST_SUITE_LOG', 'test-suite.log', INTERNAL); my $suff = '.test'; - my $at_exeext = ''; - my $handle_exeext = exists $configure_vars{'EXEEXT'}; - if ($handle_exeext) - { - $at_exeext = subst ('EXEEXT'); - $suff = $at_exeext . ' ' . $suff; - } - if (! var 'TEST_EXTENSIONS') - { - define_variable ('TEST_EXTENSIONS', $suff, INTERNAL); - } + define_variable ('TEST_EXTENSIONS', '.test', INTERNAL) + if (! var 'TEST_EXTENSIONS'); my $var = var 'TEST_EXTENSIONS'; # Currently, we are not able to deal with conditional contents # in TEST_EXTENSIONS. @@ -4818,68 +4803,47 @@ sub handle_tests "invalid test extensions: @invalid_test_suffixes"; } @test_suffixes = grep { is_valid_test_extension $_ } @test_suffixes; - if ($handle_exeext) - { - unshift (@test_suffixes, $at_exeext) - unless $test_suffixes[0] eq $at_exeext; - } - unshift (@test_suffixes, ''); - - transform_variable_recursively - ('TESTS', 'TEST_LOGS', 'am__testlogs', 1, INTERNAL, - sub { - my ($subvar, $val, $cond, $full_cond) = @_; - my $obj = $val; - return $obj - if $val =~ /^\@.*\@$/; - $obj =~ s/\$\(EXEEXT\)$//o; - - if ($val =~ /(\$\((top_)?srcdir\))\//o) - { - msg ('error', $subvar->rdef ($cond)->location, - "parallel-tests: using '$1' in TESTS is currently broken: '$val'"); - } - - foreach my $test_suffix (@test_suffixes) - { - next - if $test_suffix eq $at_exeext || $test_suffix eq ''; - return substr ($obj, 0, length ($obj) - length ($test_suffix)) . '.log' - if substr ($obj, - length ($test_suffix)) eq $test_suffix; - } - handle_per_suffix_test (''); - return "$obj.log"; - }); - - my $nhelper=1; - my $prev = 'TESTS'; - my $post = ''; - my $last_suffix = $test_suffixes[$#test_suffixes]; - my $cur = ''; + # It is *imperative* that the empty 'suffix goes last. + # Otherwise, a declaration like "TESTS = all.test" would cause GNU + # make to mistakenly try to build the 'all.log' and 'all.trs' files + # from a non-existent 'all' program (because the Makefile contains + # an explicit 'all' target) rather than from the 'all.test' script, + # thus causing all sort of mishaps and confusion. + push @test_suffixes, ''; + + define_variable ('am__test_logs1', + '$(patsubst %,%.log,$(strip $(TESTS)))', + INTERNAL); + my $nhelper = 1; foreach my $test_suffix (@test_suffixes) { - if ($test_suffix eq $last_suffix) - { - $cur = 'TEST_LOGS'; - } - else - { - $cur = 'am__test_logs' . $nhelper; - } - define_variable ($cur, - '$(' . $prev . ':' . $test_suffix . $post . '=.log)', INTERNAL); - $post = '.log'; - $prev = $cur; - $nhelper++; - if ($test_suffix ne $at_exeext && $test_suffix ne '') - { - handle_per_suffix_test ($test_suffix); - } + handle_per_suffix_test ($test_suffix); + my @cooked_test_suffixes = ($test_suffix); + push @cooked_test_suffixes, ($test_suffix . '$(EXEEXT)') + if exists $configure_vars{'EXEEXT'}; + foreach my $cooked_test_suffix (@cooked_test_suffixes) + { + my $old_var = 'am__test_logs' . $nhelper++; + my $cur_var = 'am__test_logs' . $nhelper; + define_variable ( + $cur_var, + "\$($old_var:$cooked_test_suffix.log=.log)", + INTERNAL); + } } + define_variable ('TEST_LOGS', "\$(am__test_logs$nhelper)", INTERNAL); $clean_files{'$(TEST_LOGS)'} = MOSTLY_CLEAN; $clean_files{'$(TEST_LOGS:.log=.trs)'} = MOSTLY_CLEAN; $clean_files{'$(TEST_SUITE_LOG)'} = MOSTLY_CLEAN; } + else + { + # Tests that are known programs should have $(EXEEXT) appended. + # For matching purposes, we need to adjust XFAIL_TESTS as well. + append_exeext { exists $known_programs{$_[0]} } 'TESTS'; + append_exeext { exists $known_programs{$_[0]} } 'XFAIL_TESTS' + if (var ('XFAIL_TESTS')); + } } } diff --git a/doc/automake.texi b/doc/automake.texi index 2661af537..d1e076547 100644 --- a/doc/automake.texi +++ b/doc/automake.texi @@ -6938,8 +6938,9 @@ bin_PROGRAMS = liver$(EXEEXT) The targets Automake generates are likewise given the @samp{$(EXEEXT)} extension. -The variables @code{TESTS} and @code{XFAIL_TESTS} (@pxref{Simple Tests}) -are also rewritten if they contain filenames that have been declared as +When the @option{parallel-tests} option is in use, the variables +@code{TESTS} and @code{XFAIL_TESTS} (@pxref{Simple Tests}) are also +rewritten if they contain filenames that have been declared as programs in the same @file{Makefile}. (This is mostly useful when some programs from @code{check_PROGRAMS} are listed in @code{TESTS}.) @@ -9202,6 +9203,7 @@ the inference rules generated by @command{automake} can apply. For literal test names, @command{automake} can generate per-target rules to avoid this limitation. +@c FIXME: Automake does not warn about these anymore! Please note that it is currently not possible to use @code{$(srcdir)/} or @code{$(top_srcdir)/} in the @code{TESTS} variable. This technical limitation is necessary to avoid generating test logs in the source tree. diff --git a/t/check7.sh b/t/check7.sh index 1003b5507..962cbe082 100755 --- a/t/check7.sh +++ b/t/check7.sh @@ -59,9 +59,12 @@ $AUTOMAKE -a ./configure $MAKE check -$MAKE EXEEXT=.bin print-xfail-tests >stdout || { cat stdout; Exit 1; } -cat stdout -$FGREP 'BEG: a.bin b c.bin d.bin :END' stdout + +if test x"$am_parallel_tests" != x"yes"; then + $MAKE EXEEXT=.bin print-xfail-tests >stdout || { cat stdout; Exit 1; } + cat stdout + $FGREP 'BEG: a.bin b c.bin d.bin :END' stdout +fi $MAKE distcheck diff --git a/t/exeext4.sh b/t/exeext4.sh index 298879dee..53c37dfe6 100755 --- a/t/exeext4.sh +++ b/t/exeext4.sh @@ -17,6 +17,7 @@ # Make sure $(EXEEXT) is appended to programs and to tests that are # programs, but not to @substitutions@. +am_parallel_tests=no . ./defs || Exit 1 cat >> configure.ac << 'END' diff --git a/t/list-of-tests.mk b/t/list-of-tests.mk index 436e401b9..7d12d0f89 100644 --- a/t/list-of-tests.mk +++ b/t/list-of-tests.mk @@ -40,6 +40,7 @@ t/instdir-cond2.sh \ t/interp3.sh \ t/java-nobase.sh \ t/objext-pr10128.sh \ +t/parallel-tests-srcdir-in-test-name.sh \ t/parallel-tests-many.sh \ t/pr8365-remake-timing.sh \ t/remake-am-pr10111.sh \ @@ -738,6 +739,8 @@ t/parallel-tests8.sh \ t/parallel-tests9.sh \ t/parallel-tests10.sh \ t/parallel-tests-no-repeat.sh \ +t/parallel-tests-longest-stem.sh \ +t/parallel-tests-srcdir-in-test-name.sh \ t/parallel-tests-recheck-depends-on-all.sh \ t/parallel-tests-exeext.sh \ t/parallel-tests-suffix.sh \ @@ -981,6 +984,7 @@ t/spell3.sh \ t/spelling.sh \ t/spy-autovars.sh \ t/spy-double-colon.sh \ +t/spy-pattern-rules.sh \ t/spy-phony.sh \ t/spy-rm.tap \ t/spy-vpath-rewrite.sh \ diff --git a/t/parallel-tests-longest-stem.sh b/t/parallel-tests-longest-stem.sh new file mode 100755 index 000000000..242d78fd6 --- /dev/null +++ b/t/parallel-tests-longest-stem.sh @@ -0,0 +1,66 @@ +#! /bin/sh +# Copyright (C) 2012 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. + +# The parallel-tests driver must prefer tests with an extension to +# extension-less tests. This is required to allow the user to have +# a, say, 'all.test' test case even in the face of the 'all' target. + +am_parallel_tests=yes +. ./defs || Exit 1 + +echo AC_OUTPUT >> configure.ac + +cat > foo <<'END' +#!/bin/sh +echo "foo without suffix run" >&2 +exit 99 +END + +cat > foo.test <<'END' +#!/bin/sh +echo "$0 has been run" +END +chmod a+x foo.test + +cp foo.test all.test +cp foo.test dist.test +cp foo.test install.test +cp foo.test bad-target.test + + +cat > Makefile.am << 'END' +bad-target: + @echo $@ has been run >&2; exit 1 +install-data-local: + @echo $@ has been run >&2; exit 1 +TESTS = foo.test all.test install.test dist.test bad-target.test +EXTRA_DIST = oops-this-does-not-exist +END + +$ACLOCAL +$AUTOCONF +$AUTOMAKE -a + +./configure --prefix="`pwd`/inst" + +$MAKE check +ls -l # For debugging. +test ! -d inst +for t in foo all install dist bad-target; do + grep "$t\.test has been run" $t.log +done + +: diff --git a/t/parallel-tests-srcdir-in-test-name.sh b/t/parallel-tests-srcdir-in-test-name.sh new file mode 100755 index 000000000..01042a535 --- /dev/null +++ b/t/parallel-tests-srcdir-in-test-name.sh @@ -0,0 +1,41 @@ +#! /bin/sh +# Copyright (C) 2009-2012 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. + +# Check parallel-tests features: +# - listing $(srcdir)/ or $(top_srcdir)/ in TESTS doesn't work ATM, +# and is thus diagnosed. + +# TODO: this test should also ensure that the 'make' implementation +# properly adheres to rules in all cases. See the Autoconf +# manual for the ugliness in this area, when VPATH comes into +# play. :-/ + +am_parallel_tests=yes +. ./defs || Exit 1 + +echo AC_OUTPUT >> configure.ac + +cat > Makefile.am << 'END' +TESTS = $(srcdir)/bar.test $(top_srcdir)/baz.test +END + +$ACLOCAL +$AUTOCONF +AUTOMAKE_fails -a +grep '$(srcdir).*TESTS.*bar\.test' stderr +grep '$(top_srcdir).*TESTS.*baz\.test' stderr + +: diff --git a/t/parallel-tests8.sh b/t/parallel-tests8.sh index 69efc5124..a36cea76f 100755 --- a/t/parallel-tests8.sh +++ b/t/parallel-tests8.sh @@ -14,15 +14,7 @@ # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. -# Check parallel-tests features: -# - generated distributed tests. -# - listing $(srcdir)/ or $(top_srcdir)/ in TESTS doesn't work ATM, -# and is thus diagnosed. - -# TODO: this test should also ensure that the 'make' implementation -# properly adheres to rules in all cases. See the Autoconf -# manual for the ugliness in this area, when VPATH comes into -# play. :-/ +# Check parallel-tests features: generated distributed tests. am_parallel_tests=yes . ./defs || Exit 1 @@ -33,8 +25,6 @@ END cat > Makefile.am << 'END' TESTS = foo.test -## the next line will cause automake to error out: -TESTS += $(srcdir)/bar.test $(top_srcdir)/baz.test .in.test: cp $< $@ && chmod +x $@ check_SCRIPTS = $(TESTS) @@ -50,12 +40,6 @@ END $ACLOCAL $AUTOCONF -AUTOMAKE_fails -a -grep '$(srcdir).*TESTS.*bar\.test' stderr -grep '$(top_srcdir).*TESTS.*baz\.test' stderr - -sed '/^TESTS +=.*srcdir/d' < Makefile.am > t -mv -f t Makefile.am $AUTOMAKE -a ./configure diff --git a/t/spy-pattern-rules.sh b/t/spy-pattern-rules.sh new file mode 100755 index 000000000..b2bb25d96 --- /dev/null +++ b/t/spy-pattern-rules.sh @@ -0,0 +1,54 @@ +#! /bin/sh +# Copyright (C) 2012 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. + +# Check that, if we have two pattern rules from which the same file (or +# set of files) can be built, and both are applicable, the first one wins. +# This is used at least in our 'parallel-tests' support. + +am_create_testdir=empty +. ./defs || Exit 1 + +cat > Makefile <<'END' +default: + +%.foo: % + cp $< $@ +%.foo: %.x + cp $< $@ + +%.bar: %.x + cp $< $@ +%.bar: % + +%.mu %.fu: %.1 + cp $< $*.mu && cp $< $*.fu +%.mu %.fu: %.2 + cp $< $*.mu && cp $< $*.fu +END + +echo one > all +echo two > all.x +$MAKE all.foo all.bar +diff all all.foo +diff all.x all.bar + +echo one > x.1 +echo two > x.2 +$MAKE x.mu +diff x.mu x.1 +diff x.fu x.1 + +: diff --git a/t/tap-bad-prog.tap b/t/tap-bad-prog.tap index 580e7d31c..5eb1c56b8 100755 --- a/t/tap-bad-prog.tap +++ b/t/tap-bad-prog.tap @@ -30,8 +30,9 @@ AC_OUTPUT END cat > Makefile.am << 'END' -TEST_LOG_DRIVER = $(srcdir)/tap-driver -TESTS = none.test noread.test noexec.test +LOG_DRIVER = $(srcdir)/tap-driver +TEST_LOG_DRIVER = $(LOG_DRIVER) +TESTS = none.test noread noexec.test none.test: END @@ -41,8 +42,8 @@ echo 1..1 echo ok 1 END -cp noexec.test noread.test -chmod a-r noread.test +cp noexec.test noread +chmod a-r noread $ACLOCAL $AUTOCONF @@ -72,10 +73,10 @@ else fi desc="non-readable test is reported" -if test -r noread.test; then +if test -r noread; then skip_ -r "any file is readable" "$desc" else - command_ok_ "$desc" -- grep '^ERROR: noread\.test' stdout + command_ok_ "$desc" -- grep '^ERROR: noread' stdout fi # Check that no spurious test result is reported. This is lower-priority diff --git a/t/tap-basic.sh b/t/tap-basic.sh index 96578e59d..417c24817 100755 --- a/t/tap-basic.sh +++ b/t/tap-basic.sh @@ -47,6 +47,8 @@ ok.test: echo 'ok 3 # SKIP' >>$@-t cat $@-t ;: For debugging. mv -f $@-t $@ + +LOG_DRIVER = false # Dummy but required. END cat > success.test << 'END' diff --git a/t/tap-common-setup.sh b/t/tap-common-setup.sh index b14099a5f..cd0a14d62 100755 --- a/t/tap-common-setup.sh +++ b/t/tap-common-setup.sh @@ -19,7 +19,12 @@ am_parallel_tests=yes . ./defs || Exit 1 -cat >> configure.ac << END +cat >> configure.ac << 'END' +# FIXME: must define this otherwise automake will require the presence +# FIXME: of the 'test-driver' script. This issue should be documented +# FIXME: in the manual ... +AC_SUBST([LOG_DRIVER], + ['$(error LOG_DRIVER should be never used) false']) AC_OUTPUT END @@ -32,6 +37,7 @@ END $ACLOCAL $AUTOCONF $AUTOMAKE +test ! -f test-driver ./configure diff --git a/t/tap-diagnostic-custom.sh b/t/tap-diagnostic-custom.sh index d3f28a577..bb189fd03 100755 --- a/t/tap-diagnostic-custom.sh +++ b/t/tap-diagnostic-custom.sh @@ -32,6 +32,7 @@ my_log_driver = $(srcdir)/tap-driver my_log_compiler = cat TEST_EXTENSIONS = TESTS = +LOG_DRIVER = $(error LOG_DRIVER is dummy, required, never extended) END : > later.mk diff --git a/t/tap-more.sh b/t/tap-more.sh index 00abbf28f..14181b646 100755 --- a/t/tap-more.sh +++ b/t/tap-more.sh @@ -29,7 +29,8 @@ am_parallel_tests=yes fetch_tap_driver -cat >> configure.ac <<END +cat >> configure.ac << 'END' +AC_SUBST([LOG_DRIVER], ["dummy but required"]) AC_SUBST([AM_TEST_LOG_DRIVER_FLAGS], ['--comments']) AC_OUTPUT END diff --git a/t/tap-recheck.sh b/t/tap-recheck.sh index 4b120a529..a2af0676c 100755 --- a/t/tap-recheck.sh +++ b/t/tap-recheck.sh @@ -22,11 +22,12 @@ am_parallel_tests=yes fetch_tap_driver -cat >> configure.ac <<END +cat >> configure.ac << 'END' AC_OUTPUT END cat > Makefile.am << 'END' +LOG_DRIVER = $(error LOG_DRIVER should never be expanded) false TEST_LOG_DRIVER = $(srcdir)/tap-driver TESTS = a.test b.test c.test d.test END diff --git a/t/tap-signal.tap b/t/tap-signal.tap index ba17bd402..867b762cf 100755 --- a/t/tap-signal.tap +++ b/t/tap-signal.tap @@ -67,7 +67,7 @@ chmod a+x *.test $ACLOCAL $AUTOCONF -$AUTOMAKE +$AUTOMAKE -a ./configure diff --git a/t/test-driver-custom-multitest-recheck.sh b/t/test-driver-custom-multitest-recheck.sh index 9b4f5dfa3..af3d8518a 100755 --- a/t/test-driver-custom-multitest-recheck.sh +++ b/t/test-driver-custom-multitest-recheck.sh @@ -82,7 +82,7 @@ chmod a+x *.test $ACLOCAL $AUTOCONF -$AUTOMAKE +$AUTOMAKE -a do_recheck () { diff --git a/t/test-driver-custom-multitest-recheck2.sh b/t/test-driver-custom-multitest-recheck2.sh index cf97135e3..eef1043fa 100755 --- a/t/test-driver-custom-multitest-recheck2.sh +++ b/t/test-driver-custom-multitest-recheck2.sh @@ -70,7 +70,7 @@ chmod a+x *.test $ACLOCAL $AUTOCONF -$AUTOMAKE +$AUTOMAKE -a for vpath in : false; do if $vpath; then diff --git a/t/test-driver-custom-multitest.sh b/t/test-driver-custom-multitest.sh index d5e2cf56b..caf6a079a 100755 --- a/t/test-driver-custom-multitest.sh +++ b/t/test-driver-custom-multitest.sh @@ -30,6 +30,8 @@ AC_OUTPUT END cat > Makefile.am << 'END' +LOG_DRIVER = dummy but required + TEST_EXTENSIONS = .t T_LOG_DRIVER = $(SHELL) $(srcdir)/trivial-test-driver diff --git a/t/test-driver-custom-no-extra-driver.sh b/t/test-driver-custom-no-extra-driver.sh index 31000aff1..5f66f6dde 100755 --- a/t/test-driver-custom-no-extra-driver.sh +++ b/t/test-driver-custom-no-extra-driver.sh @@ -24,7 +24,7 @@ am_parallel_tests=yes cat >> configure.ac <<'END' AC_PROG_CC AC_CONFIG_FILES([sub1/Makefile sub2/Makefile]) -AC_SUBST([X_LOG_DRIVER], [none]) +AC_SUBST([LOG_DRIVER], ['unused but required by automake']) AC_OUTPUT END @@ -32,7 +32,6 @@ mkdir sub1 sub2 cat > Makefile.am <<'END' SUBDIRS = sub1 sub2 -LOG_DRIVER = TEST_LOG_DRIVER = : TESTS = foo bar.test END @@ -41,12 +40,12 @@ cat > sub1/Makefile.am <<'END' TEST_EXTENSIONS = .x .sh .pl SH_LOG_DRIVER = dummy1 PL_LOG_DRIVER = dummy2 +X_LOG_DRIVER = dummy3 TESTS = a.pl b.sh c.x END cat > sub2/Makefile.am <<'END' TEST_EXTENSIONS = .bar -LOG_DRIVER = x BAR_LOG_DRIVER = y TESTS = 1 2.bar 3.test 4.t 5.tt $(check_PROGRAMS) check_PROGRAMS = p1 p2$(EXEEXT) p3.bar p4.suf diff --git a/t/test-driver-custom-xfail-tests.sh b/t/test-driver-custom-xfail-tests.sh index 745772fb3..d883b73e3 100755 --- a/t/test-driver-custom-xfail-tests.sh +++ b/t/test-driver-custom-xfail-tests.sh @@ -136,7 +136,7 @@ chmod a+x td $ACLOCAL $AUTOCONF -$AUTOMAKE +$AUTOMAKE --add-missing ./configure diff --git a/t/test-driver-fail.sh b/t/test-driver-fail.sh index 0eb226800..f1c3df80f 100755 --- a/t/test-driver-fail.sh +++ b/t/test-driver-fail.sh @@ -30,14 +30,16 @@ AC_OUTPUT END cat > Makefile.am <<'END' -TEST_LOG_DRIVER = ./oops -TESTS = foo.test +TEST_EXTENSIONS = +LOG_DRIVER = ./oops +TESTS = foo END -cat > foo.test <<'END' +cat > foo <<'END' #! /bin/sh exit 0 END +chmod a+x foo $ACLOCAL $AUTOCONF diff --git a/t/test-driver-strip-vpath.sh b/t/test-driver-strip-vpath.sh index 967018854..f7adb674c 100755 --- a/t/test-driver-strip-vpath.sh +++ b/t/test-driver-strip-vpath.sh @@ -39,9 +39,10 @@ cat > Makefile.am << 'END' # The $(empty) are for eliciting VPATH rewrites on make implementations # that support it (e.g., Solaris make), to improve coverage. empty = -TESTS = $(empty) foo.test src/bar.test ./src/baz.test $(empty) +TESTS = $(empty) foo.test src/bar.test ./src/baz $(empty) $(TESTS): -TEST_LOG_DRIVER = $(srcdir)/checkstrip-driver +LOG_DRIVER = $(srcdir)/checkstrip-driver +TEST_LOG_DRIVER = $(LOG_DRIVER) EXTRA_DIST = checkstrip-driver END @@ -61,8 +62,10 @@ while test $# -gt 0; do done echo "test name: $test_name" # For debugging. case $test_name in - foo.test|./foo.test|src/ba[rz].test|./src/ba[rz].test);; - *) exit 1;; + foo.test|./foo.test) ;; + src/bar.test|./src/bar.test) ;; + src/baz|./src/baz) ;; + *) exit 1 ;; esac echo dummy > "$log_file" echo dummy > "$trs_file" diff --git a/t/test-metadata-global-log.sh b/t/test-metadata-global-log.sh index b10f481d2..5d02e7793 100755 --- a/t/test-metadata-global-log.sh +++ b/t/test-metadata-global-log.sh @@ -24,6 +24,7 @@ am_parallel_tests=yes . ./defs || Exit 1 cat >> configure.ac << 'END' +AC_SUBST([LOG_DRIVER], ['ignored but required by autoamke, sigh!']) AC_OUTPUT END diff --git a/t/test-metadata-global-result.sh b/t/test-metadata-global-result.sh index b2474449c..9c0fd481c 100755 --- a/t/test-metadata-global-result.sh +++ b/t/test-metadata-global-result.sh @@ -29,7 +29,8 @@ cat > Makefile.am << 'END' TEST_EXTENSIONS = .test .x TEST_LOG_DRIVER = ./dummy-driver X_LOG_DRIVER = ./dummy-driver -TESTS = foo.test zar-doz.test +LOG_DRIVER = ./dummy-driver +TESTS = foo.test zar-doz.test mu END cat > dummy-driver <<'END' @@ -76,6 +77,7 @@ $AUTOMAKE : Basic checks. echo :global-test-result: PASS > foo.test +echo :global-test-result: PASS > mu echo :global-test-result: ERROR > zar-doz.x $MAKE check @@ -87,6 +89,11 @@ PASS: foo END have_result <<END +PASS: mu +======== +END + +have_result <<END ERROR: zar-doz ============== END @@ -95,6 +102,7 @@ END echo :global-test-result: PASS/SKIP > foo.test echo :global-test-result: ALMOST PASSED > zar-doz.x +echo :global-test-result: HU? > mu $MAKE check cat test-suite.log @@ -109,6 +117,11 @@ ALMOST PASSED: zar-doz ====================== END +have_result <<END +HU?: mu +======= +END + : Fields ':test-result:' does not interfere with the global test result. cat > foo.test << 'END' @@ -123,6 +136,13 @@ cat > zar-doz.x << 'END' :test-result: XFAIL END +cat > mu << 'END' +:test-result: PASS +:global-test-result: XFAIL +:test-result: PASS +:test-result: PASS +END + $MAKE check && Exit 1 cat test-suite.log @@ -136,6 +156,11 @@ FAIL: zar-doz ============= END +have_result <<END +XFAIL: mu +========= +END + : What happens when ':global-test-result:' is absent. cat > foo.test << 'END' @@ -143,6 +168,7 @@ cat > foo.test << 'END' :test-result: ERROR END : > zar-doz.x +: > mu $MAKE check && Exit 1 cat test-suite.log @@ -157,10 +183,16 @@ RUN: zar-doz ============ END +have_result <<END +RUN: mu +======= +END + # Leading and trailing whitespace gets eaten/normalized. echo ":global-test-result:SKIP${tab} ${tab}${tab}" > foo.test echo ":global-test-result:${tab} ${tab}XFAIL ${tab} " > zar-doz.x +echo ":global-test-result: ${tab} PASS${tab} ${tab}${tab}" > mu $MAKE check cat test-suite.log @@ -175,10 +207,16 @@ XFAIL: zar-doz ============== END +have_result <<END +PASS: mu +======== +END + # Whitespaces before and after ':global-test-result:' are handled OK. echo " $tab:global-test-result:PASS" > foo.test echo "${tab}${tab}:global-test-result:${tab} ${tab}SKIP" > zar-doz.x +echo " :global-test-result:${tab}SKIP" > mu $MAKE check cat test-suite.log @@ -193,4 +231,9 @@ SKIP: zar-doz ============= END +have_result <<END +SKIP: mu +======== +END + : diff --git a/t/test-metadata-results.sh b/t/test-metadata-results.sh index 5978f7f25..1cafbb562 100755 --- a/t/test-metadata-results.sh +++ b/t/test-metadata-results.sh @@ -25,8 +25,9 @@ AC_OUTPUT END cat > Makefile.am << 'END' -TEST_LOG_DRIVER = ./dummy-driver -TESTS = foo.test bar.test +LOG_DRIVER = ./dummy-driver +TEST_LOG_DRIVER = ${LOG_DRIVER} +TESTS = foo.test bar END cat > dummy-driver <<'END' @@ -88,7 +89,7 @@ $AUTOMAKE # "old-style" directives with format "RESULT: test-name" are now ignored. : > foo.test -echo blah blah blah > bar.test +echo blah blah blah > bar mk_check count_test_results total=0 pass=0 fail=0 xpass=0 xfail=0 skip=0 error=0 @@ -96,7 +97,7 @@ cat > foo.test <<END :test-global-result: PASS :test-result: FAIL END -cat > bar.test <<END +cat > bar <<END :test-result: SKIP :test-global-result: ERROR END @@ -108,7 +109,7 @@ FAIL: foo.test :test-result: PASS :test-global-result: XPASS END -echo ERROR: bar.test > bar.test +echo ERROR: bar > bar mk_check count_test_results total=1 pass=1 fail=0 xpass=0 xfail=0 skip=0 error=0 @@ -116,7 +117,7 @@ cat > foo.test <<END :test-global-result: SKIP :test-result: FAIL END -cat > bar.test <<END +cat > bar <<END :test-global-result: PASS END mk_check && Exit 1 @@ -127,7 +128,7 @@ cat > foo.test <<END :test-result: PASS :test-result: SKIP END -cat > bar.test <<END +cat > bar <<END :test-result: SKIP :test-result: PASS :test-result: SKIP @@ -147,18 +148,18 @@ cat > foo.test <<END :test-result: XPASS :test-result: ERROR END -: > bar.test +: > bar mk_check && Exit 1 count_test_results total=6 pass=1 fail=1 xpass=1 xfail=1 skip=1 error=1 -cp foo.test bar.test +cp foo.test bar mk_check && Exit 1 count_test_results total=12 pass=2 fail=2 xpass=2 xfail=2 skip=2 error=2 # Check that we are liberal w.r.t. whitespace use. : > foo.test -: > bar.test +: > bar for RESULT in PASS FAIL XPASS XFAIL SKIP ERROR; do sed -e 's/^ *//' -e 's/|//g' >> foo.test <<END |:test-result:$RESULT| @@ -166,10 +167,10 @@ for RESULT in PASS FAIL XPASS XFAIL SKIP ERROR; do |:test-result:$RESULT $tab| |:test-result:$tab$tab $RESULT$tab $tab | END - echo " $tab $tab$tab :test-result: $RESULT" >> bar.test + echo " $tab $tab$tab :test-result: $RESULT" >> bar done cat foo.test # For debugging. -cat bar.test # Likewise. +cat bar # Likewise. mk_check && Exit 1 count_test_results total=30 pass=5 fail=5 xpass=5 xfail=5 skip=5 error=5 |