tests: better idiom to override make macro defs on the cmdline

We now do so with a new wrapper shell function 'run_make()', which
relies on the use of AM_MAKEFLAGS, avoiding the use of the '-e' make
option.  The use of that option (that causes the environment variables
to take precedence over the macro definitions in the Makefile) has
proved to be quite brittle in the past, causing annoying and sometimes
problematic spurious failures.  This has worsened in some still
unpublished work going on in development branches.  It's time to solve
the issue once and for all.

Note that we do not convert all uses of $MAKE in the testsuite right
away; we might do so in follow-up changes, with leisure, to avoid a
"patch bomb" effect (this commit is already too much of a bomb itself).
What we do in this commit is to get rid of all "$MAKE -e" invocations.

We admit that the implementation and feature-set of 'run_make()' are
far from perfect, but good enough for our current purposes.  We'll
improve 'run_make()' if and when the need arises.

* syntax-checks.mk (sc_tests_no_make_e): New syntax check, guard against
the use of "$MAKE -e".
(syntax_check_rules): Add it.
(sc_tests_overriding_macros_on_cmdline): Adjust.
(lint): New, alias for 'maintainer-check', for lazy typists.  Idea
backported from the 'maint' branch (Automake 1.13a).
* t/ax/am-test-lib.sh (run_make): New function.  Run $MAKE with the
given command-line arguments, handling command-line override of variable
definitions in a smart way (using AM_MAKEFLAGS if a non-GNU make
implementation is detected to be in use).
(useless_vpath_rebuild): Adjust to use 'run_make', to avoid a spurious
maintainer check failure.
(yl_distcheck): Use 'run_make' rather than bare '$MAKE'.
(single_quote, append_single_quoted, is_valid_varname): New auxiliary
function, used, directly or indirectly, by it.
* Many tests: Adjust to avoid the use of "$MAKE -e", and prefer the
use of 'run_make' in few other contexts as well, where it makes sense.
Other minor fixlets while at it.

Signed-off-by: Stefano Lattarini <stefano.lattarini@gmail.com>

1 files changed, 9 insertions, 9 deletions

diff --git a/t/test-driver-custom-multitest-recheck2.sh b/t/test-driver-custom-multitest-recheck2.sh
index c6c0d893a..0c435de2b 100644
--- a/t/test-driver-custom-multitest-recheck2.sh
+++ b/t/test-driver-custom-multitest-recheck2.sh
@@ -83,7 +83,7 @@ for vpath in : false; do
   $srcdir/configure
 
   : Run the tests for the first time.
-  $MAKE check >stdout && { cat stdout; exit 1; }
+  run_make check >stdout && { cat stdout; exit 1; }
   cat stdout
   # All the test scripts should have run.
   test -f a.run
@@ -95,7 +95,7 @@ for vpath in : false; do
 
   : An empty '$(TESTS)' or '$(TEST_LOGS)' means that no test should be run.
   for var in TESTS TEST_LOGS; do
-    env "$var=" $MAKE -e recheck >stdout || { cat stdout; exit 1; }
+    run_make "$var=" recheck >stdout || { cat stdout; exit 1; }
     cat stdout
     count_test_results total=0 pass=0 fail=0 xpass=0 xfail=0 skip=0 error=0
     test ! -e a.run
@@ -106,7 +106,7 @@ for vpath in : false; do
 
   : a.test was successful the first time, no need to re-run it.
   using_gmake || $sleep # Required by BSD make.
-  env TESTS=a.test $MAKE -e recheck >stdout \
+  run_make TESTS=a.test recheck >stdout \
     || { cat stdout; exit 1; }
   cat stdout
   count_test_results total=0 pass=0 fail=0 xpass=0 xfail=0 skip=0 error=0
@@ -117,7 +117,7 @@ for vpath in : false; do
   : b.test failed, it should be re-run.  And make it pass this time.
   using_gmake || $sleep # Required by BSD make.
   echo OK > b.ok
-  TEST_LOGS=b.log $MAKE -e recheck >stdout \
+  run_make TEST_LOGS=b.log recheck >stdout \
     || { cat stdout; exit 1; }
   cat stdout
   test ! -e a.run
@@ -129,7 +129,7 @@ for vpath in : false; do
 
   : No need to re-run a.test or b.test anymore.
   using_gmake || $sleep # Required by BSD make.
-  TEST_LOGS=b.log $MAKE -e recheck >stdout \
+  run_make TEST_LOGS=b.log recheck >stdout \
     || { cat stdout; exit 1; }
   cat stdout
   count_test_results total=0 pass=0 fail=0 xpass=0 xfail=0 skip=0 error=0
@@ -137,7 +137,7 @@ for vpath in : false; do
   test ! -e b.run
   test ! -e c.run
   using_gmake || $sleep # Required by BSD make.
-  TESTS='a.test b.test' $MAKE -e recheck >stdout \
+  run_make TESTS='a.test b.test' recheck >stdout \
     || { cat stdout; exit 1; }
   cat stdout
   count_test_results total=0 pass=0 fail=0 xpass=0 xfail=0 skip=0 error=0
@@ -149,7 +149,7 @@ for vpath in : false; do
   : as it contained an XPASS.  And this time, make it fail with
   : an hard error.
   echo dummy > c.err
-  env TEST_LOGS='a.log c.log' $MAKE -e recheck >stdout \
+  run_make TEST_LOGS='a.log c.log' recheck >stdout \
     && { cat stdout; exit 1; }
   cat stdout
   count_test_results total=1 pass=0 fail=0 xpass=0 xfail=0 skip=0 error=1
@@ -164,7 +164,7 @@ for vpath in : false; do
   # Use 'echo', not ':'; see comments above for why.
   using_gmake || $sleep # Required by BSD make.
   echo dummy > c.ok
-  env TESTS='c.test a.test' $MAKE -e recheck >stdout \
+  run_make TESTS='c.test a.test' recheck >stdout \
     || { cat stdout; exit 1; }
   cat stdout
   count_test_results total=1 pass=1 fail=0 xpass=0 xfail=0 skip=0 error=0
@@ -177,7 +177,7 @@ for vpath in : false; do
   : Nothing should be rerun anymore, as all tests have been eventually
   : successful.
   using_gmake || $sleep # Required by BSD make.
-  $MAKE recheck >stdout || { cat stdout; exit 1; }
+  run_make recheck >stdout || { cat stdout; exit 1; }
   cat stdout
   count_test_results total=0 pass=0 fail=0 xpass=0 xfail=0 skip=0 error=0
   test ! -e a.run