Fix zuul-web layout update on full/tenant-reconfigure

When an operator issues a full-reconfigure or tenant-reconfigure
command to the scheduler, the branch cache is cleared.  It will
be fully populated again during the subsequent full-reconfiguration,
or eventually populated more slowly over time after a
tenant-reconfiguration.  But during the time the branch cache is
clear, zuul-web will be unable to update layouts since it is incapable
of populating the branch cache itself.

This produces a short time period of errors during a full-reconfig
or a potentially long time period of errors after a tenant-reconfig.

To correct this, we now detect whether a layout update in zuul-web
is incomplete due to a branch cache error, and retry the update
later.  In the scheduler, we only clear the projects from the branch
cache that are affected by the tenants we are reloading (still all the
projects for a full-reconfigure).  This limits the time during which
zuul-web will be unable to update the layout to only the time that
the scheduler spends actually performing a reconfiguration.

(Note that in general, the error here is not because zuul-web is
loading the layout for the tenant that zuul-scheduler is reconfiguring,
but rather it is loading a layout for a tenant which has projects
in common with the tenant that is being reconfigured.)

Change-Id: I6794da4d2316f7df6ab302c74b3efb5df4ce461a

1 files changed, 28 insertions, 4 deletions

diff --git a/zuul/connection/__init__.py b/zuul/connection/__init__.py
index 03562102f..fb43fad4d 100644
--- a/zuul/connection/__init__.py
+++ b/zuul/connection/__init__.py
@@ -21,6 +21,10 @@ from zuul.lib.logutil import get_annotated_logger
 from zuul.model import Project
 
 
+class ReadOnlyBranchCacheError(RuntimeError):
+    pass
+
+
 class BaseConnection(object, metaclass=abc.ABCMeta):
     """Base class for connections.
 
@@ -241,15 +245,35 @@ class ZKBranchCacheMixin:
             # Handle the case where tenant validation doesn't use the cache
             branches = None
 
-        if branches is not None:
+        if branches:
             return sorted(branches)
 
         if self.read_only:
+            if branches is None:
+                # A scheduler hasn't attempted to fetch them yet
+                raise ReadOnlyBranchCacheError(
+                    "Will not fetch project branches as read-only is set")
+            # A scheduler has previously attempted a fetch, but got
+            # the empty list due to an error; we can't retry since
+            # we're read-only
             raise RuntimeError(
                 "Will not fetch project branches as read-only is set")
 
         # We need to perform a query
-        branches = self._fetchProjectBranches(project, exclude_unprotected)
+        try:
+            branches = self._fetchProjectBranches(project, exclude_unprotected)
+        except Exception:
+            # We weren't able to get the branches.  We need to tell
+            # future schedulers to try again but tell zuul-web that we
+            # tried and failed.  Set the branches to the empty list to
+            # indicate that we have performed a fetch and retrieved no
+            # data.  Any time we encounter the empty list in the
+            # cache, we will try again (since it is not reasonable to
+            # have a project with no branches).
+            if self._branch_cache:
+                self._branch_cache.setProjectBranches(
+                    project.name, exclude_unprotected, [])
+            raise
         self.log.info("Got branches for %s" % project.name)
 
         if self._branch_cache:
@@ -315,14 +339,14 @@ class ZKBranchCacheMixin:
             # again.
             event.branch_protected = True
 
-    def clearBranchCache(self):
+    def clearBranchCache(self, projects=None):
         """Clear the branch cache
 
         In case the branch cache gets out of sync with the source,
         this method can be called to clear it and force querying the
         source the next time the cache is used.
         """
-        self._branch_cache.clear()
+        self._branch_cache.clear(projects)
 
 
 class ZKChangeCacheMixin: