Split out CPPDEFINES handling in Append methods

Rather than having lots of special-case code for CPPDEFINES in
four separate routines, add a new _add_cppdefines function to
handle it, paying attention to append/prepend, unique/duplicating,
and keep-original/replace-original. The existing special case handing
was then removed from Append and AppendUnique (it was never present in
Prepend and PrependUnique anyway - see #3876, but these now get it due
to a call to the new function).

Tuple handling is now consistent with list handling: a single tuple is
treated as macro names to add, not as a name=value pair. A tuple or list
has to be a member of a containing tuple or list to get the macro=value
treatment. This *may* affect some existing usage.  macro=value tuples
without a value can now be entered either in (macro,) form or (macro,
None) form.

Internally, whenever append/prepend is done, existing contents are
forced to a deque, which allows efficient adding at either end without
resorting to the tricks the Prepend functions currently do (they still
do these tricks, but only in non-CPPDEFINES cases).  As a result, values
from a dict are not stored as a dict, which has some effect on ordering:
values will be *consistently* ordered, but the ones from the dict are
no longer necessarily sorted.

In SCons/Defaults.py, processDefines no longer sorts a dict it is
passed, since Python now preserves dict order.  This does not affect
the E2E test for CPPDEFINES - since those all call an Append routine,
CPPDEFINES will always be a deque, and so processDefines never sees a
dict in that case. It could well affect real-life usage - if setup of
CPPDEFINES was such that it used to contain a dict with multiple entries,
the order might change (sorting would have presented the keys from that
dict in alphabetical order).  This would lead to a one-time rebuild for
actions that change (after that it will remain consistent).

In the E2E test CPPDEFINES/append.py some bits were reformatted, and the
handler routine now accounts for the type being a deque - since the test
does a text comparison of internally produced output, it failed if the
word "deque" appeared.  Some new test cases were added to also exercise
strings with spaces embedded in them.

Changes were made to the expected output of the E2E test. These reflect
changes in the way data is now stored in CPPDEFINES, and in some cases
in order. Most of these do not change the meaning (i.e. "result" changes,
but "final" output is the same). These are the exceptions:

- "appending a dict to a list-of-2lists", AppendUnique case: order now
  preserved as entered (previously took the order of the appended dict)

- "appending a string to a dict", Append case: not stored as a dict,
  so ordering is as originally entered.

- "appending a dict to a dict", Append case: no longer merge into a
  dict, so this is now an actual append rather than a merge of dicts which
  caused the uniquing effect even without calling AppendUnique (arguably
  the old way was incorrect).

A new test/CPPDEFINES/prepend.py is added to test Prepend* cases.

append.py and prepend.py are structured to fetch the SConstruct from a
fixture file.

append.py got an added test in the main text matrix, a string of the
macro=value form. The same 5x5 maxtrix is used in the new prepend.py
test as well ("expected" values for these had to be added as well).

Cosmetically, append and prepend now print their test summary so
strings have quotation marks - the "orig" lines in the expected output
was adjusted.  This change looks like:

-   orig = FOO, append = FOO
+   orig = 'FOO', append = 'FOO'

The other tests in test/CPPDEFINES got copyright updating and
reformatting, but otherwise do not change.

Documentation updated to clarify behavior.

Fixes #4254
Fixes #3876

Signed-off-by: Mats Wichmann <mats@linux.com>

5 files changed, 469 insertions, 298 deletions

diff --git a/SCons/Defaults.py b/SCons/Defaults.py
index 40c3e4a52..da90260ba 100644
--- a/SCons/Defaults.py
+++ b/SCons/Defaults.py
@@ -36,6 +36,7 @@ import shutil
 import stat
 import sys
 import time
+from collections import deque
 
 import SCons.Action
 import SCons.Builder
@@ -46,7 +47,7 @@ import SCons.PathList
 import SCons.Scanner.Dir
 import SCons.Subst
 import SCons.Tool
-import SCons.Util
+from SCons.Util import is_List, is_String, is_Tuple, is_Dict, flatten
 
 # A placeholder for a default Environment (for fetching source files
 # from source code management systems and the like).  This must be
@@ -166,7 +167,7 @@ def get_paths_str(dest) -> str:
     def quote(arg):
         return f'"{arg}"'
 
-    if SCons.Util.is_List(dest):
+    if is_List(dest):
         elem_strs = [quote(d) for d in dest]
         return f'[{", ".join(elem_strs)}]'
     else:
@@ -202,11 +203,11 @@ def chmod_func(dest, mode) -> None:
     """
     from string import digits
     SCons.Node.FS.invalidate_node_memos(dest)
-    if not SCons.Util.is_List(dest):
+    if not is_List(dest):
         dest = [dest]
-    if SCons.Util.is_String(mode) and 0 not in [i in digits for i in mode]:
+    if is_String(mode) and 0 not in [i in digits for i in mode]:
         mode = int(mode, 8)
-    if not SCons.Util.is_String(mode):
+    if not is_String(mode):
         for element in dest:
             os.chmod(str(element), mode)
     else:
@@ -244,7 +245,7 @@ def chmod_func(dest, mode) -> None:
 
 def chmod_strfunc(dest, mode) -> str:
     """strfunction for the Chmod action function."""
-    if not SCons.Util.is_String(mode):
+    if not is_String(mode):
         return f'Chmod({get_paths_str(dest)}, {mode:#o})'
     else:
         return f'Chmod({get_paths_str(dest)}, "{mode}")'
@@ -272,10 +273,10 @@ def copy_func(dest, src, symlinks=True) -> int:
     """
 
     dest = str(dest)
-    src = [str(n) for n in src] if SCons.Util.is_List(src) else str(src)
+    src = [str(n) for n in src] if is_List(src) else str(src)
 
     SCons.Node.FS.invalidate_node_memos(dest)
-    if SCons.Util.is_List(src):
+    if is_List(src):
         # this fails only if dest exists and is not a dir
         try:
             os.makedirs(dest, exist_ok=True)
@@ -322,7 +323,7 @@ def delete_func(dest, must_exist=False) -> None:
     unless *must_exist* evaluates false (the default).
     """
     SCons.Node.FS.invalidate_node_memos(dest)
-    if not SCons.Util.is_List(dest):
+    if not is_List(dest):
         dest = [dest]
     for entry in dest:
         entry = str(entry)
@@ -348,7 +349,7 @@ Delete = ActionFactory(delete_func, delete_strfunc)
 def mkdir_func(dest) -> None:
     """Implementation of the Mkdir action function."""
     SCons.Node.FS.invalidate_node_memos(dest)
-    if not SCons.Util.is_List(dest):
+    if not is_List(dest):
         dest = [dest]
     for entry in dest:
         os.makedirs(str(entry), exist_ok=True)
@@ -372,7 +373,7 @@ Move = ActionFactory(
 def touch_func(dest) -> None:
     """Implementation of the Touch action function."""
     SCons.Node.FS.invalidate_node_memos(dest)
-    if not SCons.Util.is_List(dest):
+    if not is_List(dest):
         dest = [dest]
     for file in dest:
         file = str(file)
@@ -433,7 +434,7 @@ def _concat_ixes(prefix, items_iter, suffix, env):
     prefix = str(env.subst(prefix, SCons.Subst.SUBST_RAW))
     suffix = str(env.subst(suffix, SCons.Subst.SUBST_RAW))
 
-    for x in SCons.Util.flatten(items_iter):
+    for x in flatten(items_iter):
         if isinstance(x, SCons.Node.FS.File):
             result.append(x)
             continue
@@ -479,8 +480,8 @@ def _stripixes(prefix, itms, suffix, stripprefixes, stripsuffixes, env, c=None):
         else:
             c = _concat_ixes
 
-    stripprefixes = list(map(env.subst, SCons.Util.flatten(stripprefixes)))
-    stripsuffixes = list(map(env.subst, SCons.Util.flatten(stripsuffixes)))
+    stripprefixes = list(map(env.subst, flatten(stripprefixes)))
+    stripsuffixes = list(map(env.subst, flatten(stripsuffixes)))
 
     stripped = []
     for l in SCons.PathList.PathList(itms).subst_path(env, None, None):
@@ -488,7 +489,7 @@ def _stripixes(prefix, itms, suffix, stripprefixes, stripsuffixes, env, c=None):
             stripped.append(l)
             continue
 
-        if not SCons.Util.is_String(l):
+        if not is_String(l):
             l = str(l)
 
         for stripprefix in stripprefixes:
@@ -511,49 +512,49 @@ def _stripixes(prefix, itms, suffix, stripprefixes, stripsuffixes, env, c=None):
 
 
 def processDefines(defs):
-    """process defines, resolving strings, lists, dictionaries, into a list of
-    strings
+    """Return list of strings for preprocessor defines from *defs*.
+
+    Resolves all the different forms CPPDEFINES can be assembled in.
+    Any prefix/suffix is handled elsewhere (usually :func:`_concat_ixes`).
     """
-    if SCons.Util.is_List(defs):
-        l = []
-        for d in defs:
-            if d is None:
+    dlist = []
+    if is_List(defs):
+        for define in defs:
+            if define is None:
                 continue
-            elif SCons.Util.is_List(d) or isinstance(d, tuple):
-                if len(d) >= 2:
-                    l.append(str(d[0]) + '=' + str(d[1]))
+            elif is_List(define) or is_Tuple(define):
+                if len(define) >= 2 and define[1] is not None:
+                    # TODO: do we need to quote define[1] if it contains space?
+                    dlist.append(str(define[0]) + '=' + str(define[1]))
                 else:
-                    l.append(str(d[0]))
-            elif SCons.Util.is_Dict(d):
-                for macro, value in d.items():
+                    dlist.append(str(define[0]))
+            elif is_Dict(define):
+                for macro, value in define.items():
                     if value is not None:
-                        l.append(str(macro) + '=' + str(value))
+                        # TODO: do we need to quote value if it contains space?
+                        dlist.append(str(macro) + '=' + str(value))
                     else:
-                        l.append(str(macro))
-            elif SCons.Util.is_String(d):
-                l.append(str(d))
+                        dlist.append(str(macro))
+            elif is_String(define):
+                dlist.append(str(define))
             else:
-                raise SCons.Errors.UserError("DEFINE %s is not a list, dict, string or None." % repr(d))
-    elif SCons.Util.is_Dict(defs):
-        # The items in a dictionary are stored in random order, but
-        # if the order of the command-line options changes from
-        # invocation to invocation, then the signature of the command
-        # line will change and we'll get random unnecessary rebuilds.
-        # Consequently, we have to sort the keys to ensure a
-        # consistent order...
-        l = []
-        for k, v in sorted(defs.items()):
-            if v is None:
-                l.append(str(k))
+                raise SCons.Errors.UserError(
+                    f"DEFINE {d!r} is not a list, dict, string or None."
+                )
+    elif is_Dict(defs):
+        for macro, value in defs.items():
+            if value is None:
+                dlist.append(str(macro))
             else:
-                l.append(str(k) + '=' + str(v))
+                dlist.append(str(macro) + '=' + str(value))
     else:
-        l = [str(defs)]
-    return l
+        dlist.append(str(defs))
+
+    return dlist
 
 
 def _defines(prefix, defs, suffix, env, target=None, source=None, c=_concat_ixes):
-    """A wrapper around _concat_ixes that turns a list or string
+    """A wrapper around :func:`_concat_ixes` that turns a list or string
     into a list of C preprocessor command-line definitions.
     """
 
diff --git a/SCons/Defaults.xml b/SCons/Defaults.xml
index 27b088294..7b37475a2 100644
--- a/SCons/Defaults.xml
+++ b/SCons/Defaults.xml
@@ -92,68 +92,111 @@ to each definition in &cv-link-CPPDEFINES;.
 <summary>
 <para>
 A platform independent specification of C preprocessor macro definitions.
-The definitions will be added to command lines
+The definitions are added to command lines
 through the automatically-generated
-&cv-link-_CPPDEFFLAGS; &consvar; (see above),
+&cv-link-_CPPDEFFLAGS; &consvar;,
 which is constructed according to
-the type of value of &cv-CPPDEFINES;:
+the contents of &cv-CPPDEFINES;:
 </para>
 
+<itemizedlist>
+<listitem>
 <para>
 If &cv-CPPDEFINES; is a string,
 the values of the
 &cv-link-CPPDEFPREFIX; and &cv-link-CPPDEFSUFFIX; &consvars;
-will be respectively prepended and appended to 
-each definition in &cv-link-CPPDEFINES;.
+are respectively prepended and appended to
+each definition in &cv-CPPDEFINES;,
+split on whitespace.
 </para>
 
 <example_commands>
-# Will add -Dxyz to POSIX compiler command lines,
+# Adds -Dxyz to POSIX compiler command lines,
 # and /Dxyz to Microsoft Visual C++ command lines.
 env = Environment(CPPDEFINES='xyz')
 </example_commands>
 
+</listitem>
+<listitem>
 <para>
 If &cv-CPPDEFINES; is a list,
 the values of the
 &cv-CPPDEFPREFIX; and &cv-CPPDEFSUFFIX; &consvars;
-will be respectively prepended and appended to 
+are respectively prepended and appended to
 each element in the list.
-If any element is a list or tuple,
-then the first item is the name being
-defined and the second item is its value:
+If any element is a tuple (or list)
+then the first item of the tuple is the macro name
+and the second is the macro definition.
+If the definition is not omitted or <literal>None</literal>,
+the name and definition are combined into a single
+<literal>name=definition</literal> item
+before the preending/appending.
 </para>
 
 <example_commands>
-# Will add -DB=2 -DA to POSIX compiler command lines,
+# Adds -DB=2 -DA to POSIX compiler command lines,
 # and /DB=2 /DA to Microsoft Visual C++ command lines.
 env = Environment(CPPDEFINES=[('B', 2), 'A'])
 </example_commands>
 
+</listitem>
+<listitem>
 <para>
 If &cv-CPPDEFINES; is a dictionary,
 the values of the
 &cv-CPPDEFPREFIX; and &cv-CPPDEFSUFFIX; &consvars;
-will be respectively prepended and appended to
-each item from the dictionary.
-The key of each dictionary item
-is a name being defined
-to the dictionary item's corresponding value;
-if the value is
-<literal>None</literal>,
-then the name is defined without an explicit value.
-Note that the resulting flags are sorted by keyword
-to ensure that the order of the options on the
-command line is consistent each time
-&scons;
-is run.
+are respectively prepended and appended to
+each key from the dictionary.
+If the value for a key is not <literal>None</literal>,
+then the key (macro name) and the value
+(macros definition) are combined into a single
+<literal>name=definition</literal> item
+before the prepending/appending.
 </para>
 
 <example_commands>
-# Will add -DA -DB=2 to POSIX compiler command lines,
-# and /DA /DB=2 to Microsoft Visual C++ command lines.
+# Adds -DA -DB=2 to POSIX compiler command lines,
+# or /DA /DB=2 to Microsoft Visual C++ command lines.
 env = Environment(CPPDEFINES={'B':2, 'A':None})
 </example_commands>
+</listitem>
+</itemizedlist>
+
+<para>
+Depending on how contents are added to &cv-CPPDEFINES;,
+it may be transformed into a compound type,
+for example a list containing strings, tuples and/or dictionaries.
+&SCons; can correctly expand such a compound type.
+</para>
+
+<para>
+Note that &SCons; may call the compiler via a shell.
+If a macro definition contains characters such as spaces that
+have meaning to the shell, or is intended to be a string value,
+you may need to use the shell's quoting syntax to avoid
+interpretation by the shell before the preprocessor sees it.
+Function-like macros are not supported via this mechanism
+(and some compilers do not even implement that functionality
+via the command lines).
+When quoting, note that
+one set of quote characters are used to define a &Python; string,
+then quotes embedded inside that would be consumed by the shell
+unless escaped.  These examples may help illustrate:
+</para>
+
+<example_commands>
+env = Environment(CPPDEFINES=['USE_ALT_HEADER=\\"foo_alt.h\\"'])
+env = Environment(CPPDEFINES=[('USE_ALT_HEADER', '\\"foo_alt.h\\"')])
+</example_commands>
+
+<para>
+:<emphasis>Changed in version  4.5</emphasis>:
+&SCons; no longer sorts &cv-CPPDEFINES; values entered
+in dictionary form.  &Python; now preserves dictionary
+keys in the order they are entered, so it is no longer
+necessary to sort them to ensure a stable command line.
+</para>
+
 </summary>
 </cvar>
 
diff --git a/SCons/Environment.py b/SCons/Environment.py
index 7212c89ea..bc69f05cd 100644
--- a/SCons/Environment.py
+++ b/SCons/Environment.py
@@ -35,7 +35,7 @@ import os
 import sys
 import re
 import shlex
-from collections import UserDict
+from collections import UserDict, deque
 
 import SCons.Action
 import SCons.Builder
@@ -193,6 +193,162 @@ def _delete_duplicates(l, keep_last):
     return result
 
 
+def _add_cppdefines(
+    env_dict: dict,
+    val,  # add annotation?
+    prepend: bool = False,
+    unique: bool = False,
+    delete_existing: bool = False,
+) -> None:
+    """Adds to CPPDEFINES, using the rules for C preprocessor macros.
+
+    Split out from regular construction variable handling because these
+    entries can express either a macro with a replacement list or one
+    without.  A macro with replacement list can be supplied three ways:
+    as a combined string ``name=value``; as a tuple contained in
+    a sequence type ``[("name", value)]``; or as a dictionary entry
+    ``{"name": value}``.  Appending/prepending can be unconditional
+    (duplicates allowed) or uniquing (no dupes).
+
+    Note if a replacement list is supplied, "unique" requires a full
+    match - both the name and the replacement must be equal.
+
+    Args:
+        env_dict: the dictionary containing the ``CPPDEFINES`` to be modified.
+        val: the value to add, can be string, sequence or dict
+        prepend: whether to put *val* in front or back.
+        unique: whether to add *val* if it already exists.
+        delete_existing: if *unique* is true, add *val* after removing previous.
+    """
+
+    def _add_define(item, defines: deque, prepend: bool = False) -> None:
+        """Convenience function to prepend/append a single value.
+
+        Sole purpose is to shorten code in the outer function.
+        """
+        if prepend:
+            defines.appendleft(item)
+        else:
+            defines.append(item)
+
+
+    def _is_in(item, defines: deque):
+        """Returns match for *item* if found in *defines*.
+
+        Accounts for type differences: ("FOO", "BAR"), ["FOO", "BAR"]
+        "FOO=BAR" and {"FOO": "BAR"} all iffer as far as Python equality
+        comparison is concerned, but are the same for purposes of creating
+        the preprocessor macro.  Since the caller may wish to remove a
+        matched entry, we need to return it - cannot remove *item*
+        itself unless it happened to be an exact (type) match.
+
+        Called from a place we know *defines* is always a deque, and
+        *item* will not be a dict, so don't need do much type checking.
+        If this ends up used more generally, would need to adjust that.
+
+        Note implied assumption that members of a list-valued define
+        will not be dicts - we cannot actually guarantee this, since
+        if the initial add is a list its contents are not converted.
+        """
+        def _macro_conv(v) -> list:
+            """Normalizes a macro to a list for comparisons."""
+            if is_Tuple(v):
+                return list(v)
+            elif is_String(v):
+                return v.split("=")
+            return v
+
+        if item in defines:  # cheap check first
+            return item
+
+        item = _macro_conv(item)
+        for define in defines:
+           if item == _macro_conv(define):
+               return define
+
+        return False
+
+
+    key = 'CPPDEFINES'
+    try:
+        defines = env_dict[key]
+    except KeyError:
+        # This is a new entry, just save it as is. Defer conversion to
+        # deque until someone tries to amend the value, processDefines
+        # can handle all of these fine.
+        if is_String(val):
+            env_dict[key] = val.split()
+        else:
+            env_dict[key] = val
+        return
+
+    # Convert type of existing to deque to simplify processing of addition -
+    # inserting at either end is cheap.
+    if isinstance(defines, deque):
+        # filter deques out to avoid catching in is_List check below
+        pass
+    elif is_String(defines):
+        env_dict[key] = deque(defines.split())
+    elif is_Tuple(defines) or is_List(defines):
+        # a little extra work in case the initial container has dict
+        # item(s) inside it, so those can be matched by _is_in().
+        result = deque()
+        for define in defines:
+            if is_Dict(define):
+                result.extend(define.items())
+            else:
+                result.append(define)
+        env_dict[key] = result
+    elif is_Dict(defines):
+        env_dict[key] = deque(defines.items())
+    else:
+        env_dict[key] = deque(defines)
+    defines = env_dict[key]  # in case we reassigned it after the try block.
+
+    if is_Dict(val):
+        # Unpack the dict while applying to existing
+        for item in val.items():
+            if unique:
+                match = _is_in(item, defines)
+                if match and delete_existing:
+                    defines.remove(match)
+                    _add_define(item, defines, prepend)
+                elif not match:
+                    _add_define(item, defines, prepend)
+            else:
+                _add_define(item, defines, prepend)
+
+    elif is_String(val):
+        if unique:
+            match = _is_in(val, defines)
+            if match and delete_existing:
+                defines.remove(match)
+                _add_define(val, defines, prepend)
+            elif not match:
+                _add_define(val, defines, prepend)
+        else:
+            _add_define(val, defines, prepend)
+
+    elif is_List(val):
+        tmp = []
+        for item in val:
+            if unique:
+                match = _is_in(item, defines)
+                if match and delete_existing:
+                    defines.remove(match)
+                    tmp.append(item)
+                elif not match:
+                    tmp.append(item)
+            else:
+                tmp.append(item)
+
+        if prepend:
+            defines.extendleft(tmp)
+        else:
+            defines.extend(tmp)
+
+    # else:  # are there any other cases? processDefines doesn't think so.
+
 
 # The following is partly based on code in a comment added by Peter
 # Shannon at the following page (there called the "transplant" class):
@@ -837,8 +993,8 @@ class SubstitutionEnvironment:
     def MergeFlags(self, args, unique=True) -> None:
         """Merge flags into construction variables.
 
-        Merges the flags from ``args`` into this construction environent.
-        If ``args`` is not a dict, it is first converted to one with
+        Merges the flags from *args* into this construction environent.
+        If *args* is not a dict, it is first converted to one with
         flags distributed into appropriate construction variables.
         See :meth:`ParseFlags`.
 
@@ -1215,16 +1371,15 @@ class Base(SubstitutionEnvironment):
 
         kw = copy_non_reserved_keywords(kw)
         for key, val in kw.items():
+            if key == 'CPPDEFINES':
+                _add_cppdefines(self._dict, val)
+                continue
+                
             try:
-                if key == 'CPPDEFINES' and is_String(self._dict[key]):
-                    self._dict[key] = [self._dict[key]]
                 orig = self._dict[key]
             except KeyError:
                 # No existing var in the environment, so set to the new value.
-                if key == 'CPPDEFINES' and is_String(val):
-                    self._dict[key] = [val]
-                else:
-                    self._dict[key] = val
+                self._dict[key] = val
                 continue
 
             try:
@@ -1263,19 +1418,8 @@ class Base(SubstitutionEnvironment):
             # things like UserList will incorrectly coerce the
             # original dict to a list (which we don't want).
             if is_List(val):
-                if key == 'CPPDEFINES':
-                    tmp = []
-                    for (k, v) in orig.items():
-                        if v is not None:
-                            tmp.append((k, v))
-                        else:
-                            tmp.append((k,))
-                    orig = tmp
-                    orig += val
-                    self._dict[key] = orig
-                else:
-                    for v in val:
-                        orig[v] = None
+                for v in val:
+                    orig[v] = None
             else:
                 try:
                     update_dict(val)
@@ -1330,6 +1474,9 @@ class Base(SubstitutionEnvironment):
         """
         kw = copy_non_reserved_keywords(kw)
         for key, val in kw.items():
+            if key == 'CPPDEFINES':
+                _add_cppdefines(self._dict, val, unique=True, delete_existing=delete_existing)
+                continue
             if is_List(val):
                 val = _delete_duplicates(val, delete_existing)
             if key not in self._dict or self._dict[key] in ('', None):
@@ -1338,46 +1485,8 @@ class Base(SubstitutionEnvironment):
                 self._dict[key].update(val)
             elif is_List(val):
                 dk = self._dict[key]
-                if key == 'CPPDEFINES':
-                    tmp = []
-                    for i in val:
-                        if is_List(i):
-                            if len(i) >= 2:
-                                tmp.append((i[0], i[1]))
-                            else:
-                                tmp.append((i[0],))
-                        elif is_Tuple(i):
-                            tmp.append(i)
-                        else:
-                            tmp.append((i,))
-                    val = tmp
-                    # Construct a list of (key, value) tuples.
-                    if is_Dict(dk):
-                        tmp = []
-                        for (k, v) in dk.items():
-                            if v is not None:
-                                tmp.append((k, v))
-                            else:
-                                tmp.append((k,))
-                        dk = tmp
-                    elif is_String(dk):
-                        dk = [(dk,)]
-                    else:
-                        tmp = []
-                        for i in dk:
-                            if is_List(i):
-                                if len(i) >= 2:
-                                    tmp.append((i[0], i[1]))
-                                else:
-                                    tmp.append((i[0],))
-                            elif is_Tuple(i):
-                                tmp.append(i)
-                            else:
-                                tmp.append((i,))
-                        dk = tmp
-                else:
-                    if not is_List(dk):
-                        dk = [dk]
+                if not is_List(dk):
+                    dk = [dk]
                 if delete_existing:
                     dk = [x for x in dk if x not in val]
                 else:
@@ -1386,70 +1495,15 @@ class Base(SubstitutionEnvironment):
             else:
                 dk = self._dict[key]
                 if is_List(dk):
-                    if key == 'CPPDEFINES':
-                        tmp = []
-                        for i in dk:
-                            if is_List(i):
-                                if len(i) >= 2:
-                                    tmp.append((i[0], i[1]))
-                                else:
-                                    tmp.append((i[0],))
-                            elif is_Tuple(i):
-                                tmp.append(i)
-                            else:
-                                tmp.append((i,))
-                        dk = tmp
-                        # Construct a list of (key, value) tuples.
-                        if is_Dict(val):
-                            tmp = []
-                            for (k, v) in val.items():
-                                if v is not None:
-                                    tmp.append((k, v))
-                                else:
-                                    tmp.append((k,))
-                            val = tmp
-                        elif is_String(val):
-                            val = [(val,)]
-                        if delete_existing:
-                            dk = list(filter(lambda x, val=val: x not in val, dk))
-                            self._dict[key] = dk + val
-                        else:
-                            dk = [x for x in dk if x not in val]
-                            self._dict[key] = dk + val
+                    # By elimination, val is not a list.  Since dk is a
+                    # list, wrap val in a list first.
+                    if delete_existing:
+                        dk = list(filter(lambda x, val=val: x not in val, dk))
+                        self._dict[key] = dk + [val]
                     else:
-                        # By elimination, val is not a list.  Since dk is a
-                        # list, wrap val in a list first.
-                        if delete_existing:
-                            dk = list(filter(lambda x, val=val: x not in val, dk))
+                        if val not in dk:
                             self._dict[key] = dk + [val]
-                        else:
-                            if val not in dk:
-                                self._dict[key] = dk + [val]
                 else:
-                    if key == 'CPPDEFINES':
-                        if is_String(dk):
-                            dk = [dk]
-                        elif is_Dict(dk):
-                            tmp = []
-                            for (k, v) in dk.items():
-                                if v is not None:
-                                    tmp.append((k, v))
-                                else:
-                                    tmp.append((k,))
-                            dk = tmp
-                        if is_String(val):
-                            if val in dk:
-                                val = []
-                            else:
-                                val = [val]
-                        elif is_Dict(val):
-                            tmp = []
-                            for i,j in val.items():
-                                if j is not None:
-                                    tmp.append((i,j))
-                                else:
-                                    tmp.append(i)
-                            val = tmp
                     if delete_existing:
                         dk = [x for x in dk if x not in val]
                     self._dict[key] = dk + val
@@ -1726,6 +1780,9 @@ class Base(SubstitutionEnvironment):
 
         kw = copy_non_reserved_keywords(kw)
         for key, val in kw.items():
+            if key == 'CPPDEFINES':
+                _add_cppdefines(self._dict, val, prepend=True)
+                continue
             try:
                 orig = self._dict[key]
             except KeyError:
@@ -1815,6 +1872,9 @@ class Base(SubstitutionEnvironment):
         """
         kw = copy_non_reserved_keywords(kw)
         for key, val in kw.items():
+            if key == 'CPPDEFINES':
+                _add_cppdefines(self._dict, val, unique=True, prepend=True, delete_existing=delete_existing)
+                continue
             if is_List(val):
                 val = _delete_duplicates(val, not delete_existing)
             if key not in self._dict or self._dict[key] in ('', None):
diff --git a/SCons/Environment.xml b/SCons/Environment.xml
index 2e06b1e65..3790a225b 100644
--- a/SCons/Environment.xml
+++ b/SCons/Environment.xml
@@ -499,126 +499,191 @@ Multiple targets can be passed in to a single call to
 </arguments>
 <summary>
 <para>
-Intelligently append values to &consvars; in the &consenv;
-named by <varname>env</varname>.
+Appends value(s) intelligently to &consvars; in
+<varname>env</varname>.
 The &consvars; and values to add to them are passed as
 <parameter>key=val</parameter> pairs (&Python; keyword arguments).
 &f-env-Append; is designed to allow adding values
-without normally having to know the data type of an existing &consvar;.
+without having to think about the data type of an existing &consvar;.
 Regular &Python; syntax can also be used to manipulate the &consvar;,
-but for that you must know the type of the &consvar;:
-for example, different &Python; syntax is needed to combine
-a list of values with a single string value, or vice versa.
+but for that you may need to know the types involved,
+for example pure &Python; lets you directly "add" two lists of strings,
+but adding a string to a list or a list to a string requires
+different syntax - things &f-Append; takes care of.
 Some pre-defined &consvars; do have type expectations
-based on how &SCons; will use them,
+based on how &SCons; will use them:
 for example &cv-link-CPPDEFINES; is normally a string or a list of strings,
-but can be a string,
-a list of strings,
-a list of tuples,
-or a dictionary, while &cv-link-LIBEMITTER;
-would expect a callable or list of callables,
-and &cv-link-BUILDERS; would expect a mapping type.
+but can also be a list of tuples or a dictionary;
+while &cv-link-LIBEMITTER;
+is expected to be a callable or list of callables,
+and &cv-link-BUILDERS; is expected to be a dictionary.
 Consult the documentation for the various &consvars; for more details.
 </para>
 
 <para>
-The following descriptions apply to both the append
-and prepend functions, the only difference being
-the insertion point of the added values.
-</para>
-<para>
-If <varname>env</varname>.  does not have a &consvar;
-indicated by <parameter>key</parameter>,
-<parameter>val</parameter>
-is added to the environment under that key as-is.
-</para>
-
-<para>
-<parameter>val</parameter> can be almost any type,
-and &SCons; will combine it with an existing value into an appropriate type,
-but there are a few special cases to be aware of.
-When two strings are combined,
-the result is normally a new string,
-with the caller responsible for supplying any needed separation.
-The exception to this is the &consvar; &cv-link-CPPDEFINES;,
-in which each item will be postprocessed by adding a prefix
-and/or suffix,
-so the contents are treated as a list of strings, that is,
-adding a string will result in a separate string entry,
-not a combined string. For &cv-CPPDEFINES; as well as
-for &cv-link-LIBS;, and the various <literal>*PATH</literal>;
-variables, &SCons; will supply the compiler-specific
-syntax (e.g. adding a <literal>-D</literal> or <literal>/D</literal>
-prefix for &cv-CPPDEFINES;), so this syntax should be omitted when
+The following descriptions apply to both the &f-Append;
+and &f-Prepend; methods, as well as their
+<emphasis role="bold">Unique</emphasis> variants,
+with the differences being the insertion point of the added values
+and whether duplication is allowed.
+</para>
+
+<para>
+<parameter>val</parameter> can be almost any type.
+If <varname>env</varname> does not have a &consvar;
+named <parameter>key</parameter>,
+then <parameter>key</parameter> is simply
+stored with a value of <parameter>val</parameter>.
+Otherwise, <parameter>val</parameter> is
+combinined with the existing value,
+possibly converting into an appropriate type
+which can hold the expanded contents.
+There are a few special cases to be aware of.
+Normally, when two strings are combined,
+the result is a new string containing their concatenation
+(and you are responsible for supplying any needed separation);
+however, the contents of &cv-link-CPPDEFINES; will
+will be postprocessed by adding a prefix and/or suffix
+to each entry when the command line is produced,
+so &SCons; keeps them separate -
+appending a string will result in a separate string entry,
+not a combined string.
+For &cv-CPPDEFINES;. as well as
+&cv-link-LIBS;, and the various <literal>*PATH</literal> variables,
+&SCons; will amend the variable by supplying the compiler-specific
+syntax (e.g. prepending a <literal>-D</literal> or <literal>/D</literal>
+prefix for &cv-CPPDEFINES;), so you should omit this syntax when
 adding values to these variables.
-Example (gcc syntax shown in the expansion of &CPPDEFINES;):
+Examples (gcc syntax shown in the expansion of &CPPDEFINES;):
 </para>
 
 <example_commands>
 env = Environment(CXXFLAGS="-std=c11", CPPDEFINES="RELEASE")
-print("CXXFLAGS={}, CPPDEFINES={}".format(env['CXXFLAGS'], env['CPPDEFINES']))
-# notice including a leading space in CXXFLAGS value
+print(f"CXXFLAGS = {env['CXXFLAGS']}, CPPDEFINES = {env['CPPDEFINES']}")
+# notice including a leading space in CXXFLAGS addition
 env.Append(CXXFLAGS=" -O", CPPDEFINES="EXTRA")
-print("CXXFLAGS={}, CPPDEFINES={}".format(env['CXXFLAGS'], env['CPPDEFINES']))
-print("CPPDEFINES will expand to {}".format(env.subst("$_CPPDEFFLAGS")))
+print(f"CXXFLAGS = {env['CXXFLAGS']}, CPPDEFINES = {env['CPPDEFINES']}")
+print("CPPDEFINES will expand to", env.subst('$_CPPDEFFLAGS'))
 </example_commands>
 
 <screen>
 $ scons -Q
-CXXFLAGS=-std=c11, CPPDEFINES=RELEASE
-CXXFLAGS=-std=c11 -O, CPPDEFINES=['RELEASE', 'EXTRA']
-CPPDEFINES will expand to -DRELEASE -DEXTRA
+CXXFLAGS = -std=c11, CPPDEFINES = RELEASE
+CXXFLAGS = -std=c11 -O, CPPDEFINES = deque(['RELEASE', 'EXTRA'])
+CPPDEFINES will expand to  -DRELEASE -DEXTRA
 scons: `.' is up to date.
 </screen>
 
 <para>
-Because &cv-link-CPPDEFINES; is intended to
-describe C/C++ pre-processor macro definitions,
+Because &cv-link-CPPDEFINES; is intended for command-line
+specification of C/C++ preprocessor macros,
 it accepts additional syntax.
-Preprocessor macros can be valued, or un-valued, as in
-<computeroutput>-DBAR=1</computeroutput> or
-<computeroutput>-DFOO</computeroutput>.
-The macro can be be supplied as a complete string including the value,
-or as a tuple (or list) of macro, value, or as a dictionary.
-Example (again gcc syntax in the expanded defines):
+A command-line preprocessor macro can predefine a name by itself
+(<computeroutput>-DFOO</computeroutput>),
+which gives it an implicit value,
+or be given with a macro definition
+(<computeroutput>-DBAR=1</computeroutput>).
+&SCons; allows you to specify a macro with a definition
+using a <literal>name=value</literal> string,
+or a tuple <literal>(name, value)</literal>
+(which must be supplied inside a sequence type),
+or a dictionary <literal>{name: value}</literal>.
 </para>
 
 <example_commands>
 env = Environment(CPPDEFINES="FOO")
-print("CPPDEFINES={}".format(env['CPPDEFINES']))
+print("CPPDEFINES =", env['CPPDEFINES'])
 env.Append(CPPDEFINES="BAR=1")
-print("CPPDEFINES={}".format(env['CPPDEFINES']))
-env.Append(CPPDEFINES=("OTHER", 2))
-print("CPPDEFINES={}".format(env['CPPDEFINES']))
+print("CPPDEFINES =", env['CPPDEFINES'])
+env.Append(CPPDEFINES=[("OTHER", 2)])
+print("CPPDEFINES =", env['CPPDEFINES'])
 env.Append(CPPDEFINES={"EXTRA": "arg"})
-print("CPPDEFINES={}".format(env['CPPDEFINES']))
-print("CPPDEFINES will expand to {}".format(env.subst("$_CPPDEFFLAGS")))
+print("CPPDEFINES =", env['CPPDEFINES'])
+print("CPPDEFINES will expand to", env.subst('$_CPPDEFFLAGS'))
 </example_commands>
 
 <screen>
 $ scons -Q
-CPPDEFINES=FOO
-CPPDEFINES=['FOO', 'BAR=1']
-CPPDEFINES=['FOO', 'BAR=1', ('OTHER', 2)]
-CPPDEFINES=['FOO', 'BAR=1', ('OTHER', 2), {'EXTRA': 'arg'}]
+CPPDEFINES = FOO
+CPPDEFINES = deque(['FOO', 'BAR=1'])
+CPPDEFINES = deque(['FOO', 'BAR=1', ('OTHER', 2)])
+CPPDEFINES = deque(['FOO', 'BAR=1', ('OTHER', 2), ('EXTRA', 'arg')])
 CPPDEFINES will expand to -DFOO -DBAR=1 -DOTHER=2 -DEXTRA=arg
 scons: `.' is up to date.
 </screen>
 
 <para>
-Adding a string <parameter>val</parameter>
-to a dictonary &consvar; will enter
-<parameter>val</parameter> as the key in the dict,
+Multiple &cv-CPPDEFINES; macros can be supplied in a sequence of tuples,
+or using the dictionary form.
+If a given macro name should not have a definition,
+omit the value from tuple or give it as
+<constant>None</constant>;
+if using the dictionary form,
+specify the value as <constant>None</constant>.
+</para>
+
+<example_commands>
+env = Environment()
+env.Append(CPPDEFINES=[("ONE", 1), ("TWO", )])
+print("CPPDEFINES =", env['CPPDEFINES'])
+env.Append(CPPDEFINES={"THREE": 3, "FOUR": None})
+print("CPPDEFINES =", env['CPPDEFINES'])
+print("CPPDEFINES will expand to", env.subst('$_CPPDEFFLAGS'))
+</example_commands>
+
+<screen>
+$ scons -Q
+CPPDEFINES = [('ONE', 1), ('TWO',)]
+CPPDEFINES = [('ONE', 1), ('TWO',), {'THREE': 3, 'FOUR': None}]
+CPPDEFINES will expand to -DONE=1 -DTWO -DTHREE=3 -DFOUR
+scons: `.' is up to date.
+</screen>
+
+<para>
+<emphasis>Changed in version  4.5</emphasis>:
+clarified that to receive the special handling,
+the tuple form must be supplied <emphasis>inside</emphasis>
+a sequence (e.g. a list); a single tuple will be interpreted
+just like a single list.
+Previously this form was inconsistently interpreted by
+various &SCons; methods.
+</para>
+
+<example_commands>
+env = Environment()
+env.Append(CPPDEFINES=("MACRO1", "MACRO2"))
+print("CPPDEFINES =", env['CPPDEFINES'])
+env.Append(CPPDEFINES=[("MACRO3", "MACRO4")])
+print("CPPDEFINES =", env['CPPDEFINES'])
+print("CPPDEFINES will expand to", env.subst('$_CPPDEFFLAGS'))
+</example_commands>
+
+<screen>
+$ scons -Q
+CPPDEFINES = ('MACRO1', 'MACRO2')
+CPPDEFINES = deque(['MACRO1', 'MACRO2', ('MACRO3', 'MACRO4')])
+CPPDEFINES will expand to -DMACRO1 -DMACRO2 -DMACRO3=MACRO4
+scons: `.' is up to date.
+</screen>
+
+<para>
+See &cv-link-CPPDEFINES; for more details.
+</para>
+
+<para>
+Appending a string <parameter>val</parameter>
+to a dictonary-typed &consvar; enters
+<parameter>val</parameter> as the key in the dictionary,
 and <literal>None</literal> as its value.
-Using a tuple type to supply a key + value only works
-for the special case of &cv-link-CPPDEFINES;
+Using a tuple type to supply a <literal>key, value</literal>
+only works for the special case of &cv-CPPDEFINES;
 described above.
 </para>
 
 <para>
 Although most combinations of types work without
 needing to know the details, some combinations
-do not make sense and a &Python; exception will be raised.
+do not make sense and &Python; raises an exception.
 </para>
 
 <para>
@@ -626,7 +691,7 @@ When using &f-env-Append; to modify &consvars;
 which are path specifications (conventionally,
 the names of such end in <literal>PATH</literal>),
 it is recommended to add the values as a list of strings,
-even if there is only a single string to add.
+even if you are only adding a single string.
 The same goes for adding library names to &cv-LIBS;.
 </para>
 
@@ -696,20 +761,20 @@ See also &f-link-env-PrependENVPath;.
 
 <scons_function name="AppendUnique">
 <arguments signature="env">
-(key=val, [...], delete_existing=False)
+(key=val, [...], [delete_existing=False])
 </arguments>
 <summary>
 <para>
 Append values to &consvars; in the current &consenv;,
 maintaining uniqueness.
-Works like &f-link-env-Append; (see for details),
-except that values already present in the &consvar;
-will not be added again.
+Works like &f-link-env-Append;,
+except that values that would become duplicates
+are not added.
 If <parameter>delete_existing</parameter>
-is <constant>True</constant>,
-the existing matching value is first removed,
-and the requested value is added,
-having the effect of moving such values to the end.
+is set to a true value, then for any duplicate,
+the existing instance of <parameter>val</parameter> is first removed,
+then <parameter>val</parameter> is appended,
+having the effect of moving it to the end.
 </para>
 
 <para>
@@ -2716,22 +2781,22 @@ See also &f-link-env-AppendENVPath;.
 
 <scons_function name="PrependUnique">
 <arguments signature="env">
-(key=val, delete_existing=False, [...])
+(key=val, [...], [delete_existing=False])
 </arguments>
 <summary>
 <para>
 Prepend values to &consvars; in the current &consenv;,
 maintaining uniqueness.
-Works like &f-link-env-Append; (see for details),
+Works like &f-link-env-Append;,
 except that values are added to the front,
-rather than the end, of any existing value of the &consvar;,
-and values already present in the &consvar;
-will not be added again.
+rather than the end, of the &consvar;,
+and values that would become duplicates
+are not added.
 If <parameter>delete_existing</parameter>
-is <constant>True</constant>,
-the existing matching value is first removed,
-and the requested value is inserted,
-having the effect of moving such values to the front.
+is set to a true value, then for any duplicate,
+the existing instance of <parameter>val</parameter> is first removed,
+then <parameter>val</parameter> is inserted,
+having the effect of moving it to the front.
 </para>
 
 <para>
diff --git a/SCons/Util/types.py b/SCons/Util/types.py
index 9aef13ef9..1602055d6 100644
--- a/SCons/Util/types.py
+++ b/SCons/Util/types.py
@@ -12,7 +12,7 @@ import pprint
 import re
 from typing import Optional
 
-from collections import UserDict, UserList, UserString
+from collections import UserDict, UserList, UserString, deque
 from collections.abc import MappingView
 
 # Functions for deciding if things are like various types, mainly to
@@ -23,20 +23,22 @@ from collections.abc import MappingView
 # exception, but handling the exception when it's not the right type is
 # often too slow.
 
-# We are using the following trick to speed up these
-# functions. Default arguments are used to take a snapshot of
-# the global functions and constants used by these functions. This
-# transforms accesses to global variable into local variables
-# accesses (i.e. LOAD_FAST instead of LOAD_GLOBAL).
-# Since checkers dislike this, it's now annotated for pylint to flag
+# A trick is used to speed up these functions. Default arguments are
+# used to take a snapshot of the global functions and constants used
+# by these functions. This transforms accesses to global variables into
+# local variable accesses (i.e. LOAD_FAST instead of LOAD_GLOBAL).
+# Since checkers dislike this, it's now annotated for pylint, to flag
 # (mostly for other readers of this code) we're doing this intentionally.
-# TODO: PY3 check these are still valid choices for all of these funcs.
+# TODO: experts affirm this is still faster, but maybe check if worth it?
 
 DictTypes = (dict, UserDict)
-ListTypes = (list, UserList)
+ListTypes = (list, UserList, deque)
 
-# Handle getting dictionary views.
-SequenceTypes = (list, tuple, UserList, MappingView)
+# With Python 3, there are view types that are sequences. Other interesting
+# sequences are range and bytearray.  What we don't want is strings: while
+# they are iterable sequences, in SCons usage iterating over a string is
+# almost never what we want. So basically iterable-but-not-string:
+SequenceTypes = (list, tuple, deque, UserList, MappingView)
 
 # Note that profiling data shows a speed-up when comparing
 # explicitly with str instead of simply comparing