Split out CPPDEFINES handling in Append methods

Rather than having lots of special-case code for CPPDEFINES in
four separate routines, add a new _add_cppdefines function to
handle it, paying attention to append/prepend, unique/duplicating,
and keep-original/replace-original. The existing special case handing
was then removed from Append and AppendUnique (it was never present in
Prepend and PrependUnique anyway - see #3876, but these now get it due
to a call to the new function).

Tuple handling is now consistent with list handling: a single tuple is
treated as macro names to add, not as a name=value pair. A tuple or list
has to be a member of a containing tuple or list to get the macro=value
treatment. This *may* affect some existing usage.  macro=value tuples
without a value can now be entered either in (macro,) form or (macro,
None) form.

Internally, whenever append/prepend is done, existing contents are
forced to a deque, which allows efficient adding at either end without
resorting to the tricks the Prepend functions currently do (they still
do these tricks, but only in non-CPPDEFINES cases).  As a result, values
from a dict are not stored as a dict, which has some effect on ordering:
values will be *consistently* ordered, but the ones from the dict are
no longer necessarily sorted.

In SCons/Defaults.py, processDefines no longer sorts a dict it is
passed, since Python now preserves dict order.  This does not affect
the E2E test for CPPDEFINES - since those all call an Append routine,
CPPDEFINES will always be a deque, and so processDefines never sees a
dict in that case. It could well affect real-life usage - if setup of
CPPDEFINES was such that it used to contain a dict with multiple entries,
the order might change (sorting would have presented the keys from that
dict in alphabetical order).  This would lead to a one-time rebuild for
actions that change (after that it will remain consistent).

In the E2E test CPPDEFINES/append.py some bits were reformatted, and the
handler routine now accounts for the type being a deque - since the test
does a text comparison of internally produced output, it failed if the
word "deque" appeared.  Some new test cases were added to also exercise
strings with spaces embedded in them.

Changes were made to the expected output of the E2E test. These reflect
changes in the way data is now stored in CPPDEFINES, and in some cases
in order. Most of these do not change the meaning (i.e. "result" changes,
but "final" output is the same). These are the exceptions:

- "appending a dict to a list-of-2lists", AppendUnique case: order now
  preserved as entered (previously took the order of the appended dict)

- "appending a string to a dict", Append case: not stored as a dict,
  so ordering is as originally entered.

- "appending a dict to a dict", Append case: no longer merge into a
  dict, so this is now an actual append rather than a merge of dicts which
  caused the uniquing effect even without calling AppendUnique (arguably
  the old way was incorrect).

A new test/CPPDEFINES/prepend.py is added to test Prepend* cases.

append.py and prepend.py are structured to fetch the SConstruct from a
fixture file.

append.py got an added test in the main text matrix, a string of the
macro=value form. The same 5x5 maxtrix is used in the new prepend.py
test as well ("expected" values for these had to be added as well).

Cosmetically, append and prepend now print their test summary so
strings have quotation marks - the "orig" lines in the expected output
was adjusted.  This change looks like:

-   orig = FOO, append = FOO
+   orig = 'FOO', append = 'FOO'

The other tests in test/CPPDEFINES got copyright updating and
reformatting, but otherwise do not change.

Documentation updated to clarify behavior.

Fixes #4254
Fixes #3876

Signed-off-by: Mats Wichmann <mats@linux.com>

1 files changed, 15 insertions, 4 deletions

diff --git a/CHANGES.txt b/CHANGES.txt
index 5a3651ad8..41a966b5c 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -133,6 +133,17 @@ RELEASE  VERSION/DATE TO BE FILLED IN LATER
       Also added "unique" keyword, to control whether a library is added
       or not if it is already in the $LIBS construction var in the
       configure context. (issue #2768).
+    - Special-case handling for CPPDEFINES in the Append/Prepend routines
+      split out into its own function to simplify the remaining code and
+      fix problems. The e2e test test/CPPDEFINES/append.py is expanded
+      to cover missed cases, and AppendUnique no longer mismatches with
+      what Append does (#3876).  Inconsistent handling of tuples to specify
+      macro=value outputs is also cleaned up (#4254). The special handling
+      now also works for Prepend/PrependUnique, and a corresponding test
+      test/CPPDEFINES/prepend.py was added to verify the behavior.
+      SCons used to sort keys set or appended via a dict type, in order to
+      assure order of commandline flags did not change across runs. This
+      behavior has been dropped since Python now assures consistent dict order.
 
 
 RELEASE 4.4.0 -  Sat, 30 Jul 2022 14:08:29 -0700
@@ -295,16 +306,16 @@ RELEASE 4.4.0 -  Sat, 30 Jul 2022 14:08:29 -0700
     - Ninja:Added user configurable setting of ninja depfile format via NINJA_DEPFILE_PARSE_FORMAT.
       Now setting NINJA_DEPFILE_PARSE_FORMAT to [msvc,gcc,clang] can force the ninja expected
       format. Compiler tools will also configure the variable automatically.
-    - Ninja: Made ninja tool force the ninja file as the only target. 
+    - Ninja: Made ninja tool force the ninja file as the only target.
     - Ninja: Improved the default targets setup and made sure there is always a default target for
       the ninja file, which excludes targets that start and stop the daemon.
     - Ninja: Update ninja tool so targets passed to SCons are propagated to ninja when scons
       automatically executes ninja.
-    - Small refactor of scons daemons using a shared StateInfo class for communication 
+    - Small refactor of scons daemons using a shared StateInfo class for communication
       between the scons interactive thread and the http server thread. Added error handling
       for scons interactive failing to startup.
     - Ninja: Updated ninja scons daemon scripts to output errors to stderr as well as the daemon log.
-    - Ninja: Fix typo in ninja scons daemon startup which causes ConnectionRefusedError to not retry 
+    - Ninja: Fix typo in ninja scons daemon startup which causes ConnectionRefusedError to not retry
     - Added SHELL_ENV_GENERATORS construction variable. This variable should be set to a list
       (or an iterable) which contains functions to be called in order
       when constructing the execution environment (Generally this is the shell environment
@@ -546,7 +557,7 @@ RELEASE 4.2.0 - Sat, 31 Jul 2021 18:12:46 -0700
     - Fix Issue #3906 - `IMPLICIT_COMMAND_DEPENDENCIES` was not properly disabled when
       set to any string value (For example ['none','false','no','off'])
       Also previously 'All' wouldn't have the desired affect.
-      
+
   From Ivan Kravets:
     - Provide a custom argument escape function for `TempFileMunge` using a new
       `TEMPFILEARGESCFUNC` variable. Useful if you need to apply extra operations on