Create virtualenvs in series to avoid cache race

virtualenv 20.0.24 creates ~/.local/share/virtualenv with the
seed packages needed for making virtualenvs per-python version.
Creating empty virtualenvs is quick, so run those in sequence
to avoid race possibilities. Then, we can still run the
installs into the virtualenvs in parallel.

We also fix a bug in the console stream functional jobs and install pip
with the use of ensure-pip. This is necessary because the virtualenv
fix runs the stream functional jobs and the update to the stream
functional jobs relies on working docker images.

(cherry picked from commit 1f7c6fef9c394928bad28acb32c1180d0006844e)

Change-Id: I3dec251d19dd7b3807848a54e6a20a8e89d30a4e

3 files changed, 14 insertions, 5 deletions

diff --git a/Dockerfile b/Dockerfile
index e1a30d5b3..378a76eb9 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -30,8 +30,10 @@ RUN cd /tmp/src/web && yarn install -d && yarn build
 RUN assemble
 
 # The wheel install method doesn't run the setup hooks as the source based
-# installations do so we have to call zuul-manage-ansible here.
-RUN /output/install-from-bindep && zuul-manage-ansible
+# installations do so we have to call zuul-manage-ansible here. Remove
+# /root/.local/share/virtualenv after because it adds wheels into /root
+# that we don't need after the install step so are a waste of space.
+RUN /output/install-from-bindep && zuul-manage-ansible && rm -rf /root/.local/share/virtualenv
 
 RUN mkdir /tmp/openshift-install \
   && curl -L $OPENSHIFT_URL -o /tmp/openshift-install/openshift-client.tgz \
diff --git a/playbooks/zuul-stream/pre.yaml b/playbooks/zuul-stream/pre.yaml
index 371542307..543d7ab19 100644
--- a/playbooks/zuul-stream/pre.yaml
+++ b/playbooks/zuul-stream/pre.yaml
@@ -1,6 +1,8 @@
 - hosts: controller
   roles:
 
+    - role: ensure-pip
+
     - role: bindep
       bindep_profile: test
       bindep_dir: src/opendev.org/zuul/zuul
diff --git a/zuul/lib/ansible.py b/zuul/lib/ansible.py
index df0ed0213..f4b8fdd94 100644
--- a/zuul/lib/ansible.py
+++ b/zuul/lib/ansible.py
@@ -49,8 +49,6 @@ class ManagedAnsible:
         self.install_root = self._ansible_roots[-1]
 
     def ensure_ansible(self, upgrade=False):
-        self._ensure_venv()
-
         self.log.info('Installing ansible %s, requirements: %s, '
                       'extra packages: %s',
                       self.version, self._requirements, self.extra_packages)
@@ -76,7 +74,7 @@ class ManagedAnsible:
                                              p.stderr.decode()))
         self.log.debug('Successfully installed packages %s', requirements)
 
-    def _ensure_venv(self):
+    def ensure_venv(self):
         if self.python_path:
             self.log.debug(
                 'Virtual environment %s already existing', self.venv_path)
@@ -192,6 +190,13 @@ class AnsibleManager:
         self.default_version = default_version
 
     def install(self, upgrade=False):
+        # virtualenv sets up a shared directory of pip seed packages per
+        # python version. If we run virtualenv in parallel we can have one
+        # create the dirs but not be finished filling them with content, a
+        # second notice the dir is there so it just goes on with what it's
+        # done, and thus races leaving us with virtualenvs minus pip.
+        for a in self._supported_versions.values():
+            a.ensure_venv()
         # Note: With higher number of threads pip seems to have some race
         # leading to occasional failures during setup of all ansible
         # environments. Thus we limit the number of workers to reduce the risk