Prevent files cache ltime from going backward

This fixes a race condition when two schedulers request files for the
same repository at roughly the same time.

The files cache ltime is generated before creating the cat job. We also
cannot guarantee the order in which the job results are processed.

Since we include the files cache ltime in the min. ltimes as part
of the layout state we need to make sure that the timestamp of the cache
doesn't go backward.

This should fix the following exception we occassionally see in
zuul-web:

    WARNING zuul.ConfigLoader: Zuul encountered an error while accessing the repo org/project.
    The error was:

      Configuration files missing from cache. Check Zuul scheduler logs for more information.

Change-Id: I9fe2ed0b7a95dd12bb978273673e905d06bee72d

1 files changed, 6 insertions, 0 deletions

diff --git a/zuul/configloader.py b/zuul/configloader.py
index 7f8346382..1e7e010cc 100644
--- a/zuul/configloader.py
+++ b/zuul/configloader.py
@@ -2221,6 +2221,12 @@ class TenantParser(object):
                 job.source_context.branch)
             with self.unparsed_config_cache.writeLock(
                     job.source_context.project_canonical_name):
+                # Prevent files cache ltime from going backward
+                if files_cache.ltime >= job.ltime:
+                    self.log.info(
+                        "Discarding job %s result since the files cache was "
+                        "updated in the meantime", job)
+                    continue
                 # Since the cat job returns all required config files
                 # for ALL tenants the project is a part of, we can
                 # clear the whole cache and then populate it with the