Freeze job variables at start of build

Freze Zuul job variables when starting a build so that jinja
templates can not be used to expose secrets.  The values will be
frozen by running a playbook with set_fact, and that playbook
will run without access to secrets.  After the playbook
completes, the frozen variables are read from and then removed
from the fact cache.  They are then supplied as normal inventory
variables for any trusted playbooks or playbooks with secrets.

The regular un-frozen variables are used for all other untrusted
playbooks.

Extra-vars are now only used to establish precedence among all
Zuul job variables.  They are no longer passed to Ansible with
the "-e" command line option, as that level of precedence could
also be used to obtain secrets.

Much of this work is accomplished by "squashing" all of the Zuul
job, host, group, and extra variables into a flat structure for
each host in the inventory.  This means that much of the variable
precedence is now handled by Zuul, which then gives Ansible
variables as host vars.  The actual inventory files will be much
more verbose now, since each host will have a copy of every "all"
value.  But this allows the freezing process to be much simpler.

When writing the inventory for the setup playbook, we now use the
!unsafe YAML tag which is understood by Ansible to indicate that
it should not perform jinja templating on variables.  This may
help to avoid any mischief with templated variables since they
have not yet been frozen.

Also, be more strict about what characters are allowed in ansible
variable names.  We already checked job variables, but we didn't
verify that secret names/aliases met the ansible variable
requirements.  A check is added for that (and a unit test that
relied on the erroneous behavior is updated).

Story: 2008664
Story: 2008682
Change-Id: I04d8b822fda6628e87a4a57dc368f20d84ae5ea9

1 files changed, 2 insertions, 1 deletions

diff --git a/tests/unit/test_gitlab_driver.py b/tests/unit/test_gitlab_driver.py
index 7c8edfc98..790ed1180 100644
--- a/tests/unit/test_gitlab_driver.py
+++ b/tests/unit/test_gitlab_driver.py
@@ -19,6 +19,7 @@ import yaml
 import socket
 
 import zuul.rpcclient
+from zuul.lib import strings
 
 from tests.base import random_sha1, simple_layout
 from tests.base import ZuulTestCase, ZuulWebFixture
@@ -106,7 +107,7 @@ class TestGitlabDriver(ZuulTestCase):
         self.assertEqual('master', zuulvars['branch'])
         self.assertEquals('https://gitlab/org/project/merge_requests/1',
                           zuulvars['items'][0]['change_url'])
-        self.assertEqual(zuulvars["message"], description)
+        self.assertEqual(zuulvars["message"], strings.b64encode(description))
         self.assertEqual(2, len(self.history))
         self.assertEqual(2, len(A.notes))
         self.assertEqual(