Cleanup test logging

We were overlogging because we check for an openssl flag early and warn
if it isn't present. That warning creates a default root streamhandler
that emits to stderr causing all our logging to be emitted there.

Fix this by creating a specific logger for this warning (avoids
polluting the root logger) and add an assertion that the root logger's
handler list is empty when we modify it for testing.

Note I'm not sure why this warning is happening now and wasn't before.
Maybe our openssl installations changed or cryptography modified the
flag? This is worth investigating in a followup.

Change-Id: I2a82cd6575e86facb80b28c81418ddfee8a32fa5

2 files changed, 11 insertions, 2 deletions

diff --git a/tests/base.py b/tests/base.py
index 3ab5329da..cdcf63390 100644
--- a/tests/base.py
+++ b/tests/base.py
@@ -4331,6 +4331,11 @@ class BaseTestCase(testtools.TestCase):
         handler.setFormatter(formatter)
 
         logger = logging.getLogger()
+        # It is possible that a stderr log handler is inserted before our
+        # addHandler below. If that happens we will emit all logs to stderr
+        # even when we don't want to. Error here to make it clear there is
+        # a problem as early as possible as it is easy to overlook.
+        self.assertEqual(logger.handlers, [])
         logger.setLevel(logging.DEBUG)
         logger.addHandler(handler)
 
diff --git a/zuul/lib/encryption.py b/zuul/lib/encryption.py
index 79e92e366..ea7e1f3b1 100644
--- a/zuul/lib/encryption.py
+++ b/zuul/lib/encryption.py
@@ -28,8 +28,12 @@ if hasattr(backend, '_rsa_skip_check_key'):
     backend._rsa_skip_check_key = True
 else:
     import logging
-    logging.warning("Cryptography backend lacks _rsa_skip_check_key flag, "
-                    "key loading may be slow")
+    # Use a specific logger here to avoid polluting the root logger
+    # with the default stderr stream handler. This is important in
+    # testing to ensure we don't over log and create noise.
+    logger = logging.getLogger("zuul.rsa_skip_check_warning")
+    logger.warning("Cryptography backend lacks _rsa_skip_check_key flag, "
+                   "key loading may be slow")
 
 
 # https://cryptography.io/en/latest/hazmat/primitives/asymmetric/rsa/#generation