Rely on the unparsed config cache in reconfigurations

When performing a tenant reconfiguration, we check the zk-based
file cache for every project-branch to see if we need to refresh
our unparsed config cache from the contents in ZK.  However, we
know which project-branches have changed.  We can assume that if
the unparsed branch cache is valid for for a project-branch ltime
(it won't be for projects that trigger tenant reconfig), then we
can use its contents without checking the ltime of the file cache
in ZK.  This will typically save a few minutes worth of ZK traffic
during tenant reconfigurations on large tenants.

In order to make the background layout update similarly quick, we
need to prove the same min_ltimes to that process which were used
by the last reconfig.  To do this, we store the min_ltimes in ZK
for each reconfig (as sharded data).

Change-Id: Ia0a524053a110e8e48f709d219651e2ad9c8513d

1 files changed, 32 insertions, 0 deletions

diff --git a/tests/unit/test_zk.py b/tests/unit/test_zk.py
index f3b664da2..c8b8c74a8 100644
--- a/tests/unit/test_zk.py
+++ b/tests/unit/test_zk.py
@@ -1,4 +1,5 @@
 # Copyright 2019 Red Hat, Inc.
+# Copyright 2022 Acme Gating, LLC
 #
 # Licensed under the Apache License, Version 2.0 (the "License"); you may
 # not use this file except in compliance with the License. You may obtain
@@ -12,6 +13,7 @@
 # License for the specific language governing permissions and limitations
 # under the License.
 
+from collections import defaultdict
 import json
 import queue
 import threading
@@ -1304,6 +1306,36 @@ class TestLayoutStore(ZooKeeperBaseTestCase):
 
         self.assertGreater(state_two, state_one)
 
+    def test_cleanup(self):
+        store = LayoutStateStore(self.zk_client, lambda: None)
+        min_ltimes = defaultdict(lambda x: -1)
+        min_ltimes['foo'] = 1
+        state_one = LayoutState("tenant", "hostname", 1, uuid.uuid4().hex,
+                                {}, ltime=1)
+        state_two = LayoutState("tenant", "hostname", 2, uuid.uuid4().hex,
+                                {}, ltime=2)
+        store.setMinLtimes(state_one, min_ltimes)
+        store.setMinLtimes(state_two, min_ltimes)
+        store['tenant'] = state_one
+        # Run with the default delay of 5 minutes; nothing should be deleted.
+        store.cleanup()
+        self.assertEqual(store.get('tenant'), state_one)
+        self.assertIsNotNone(
+            self.zk_client.client.exists(
+                f'/zuul/layout-data/{state_one.uuid}'))
+        self.assertIsNotNone(
+            self.zk_client.client.exists(
+                f'/zuul/layout-data/{state_two.uuid}'))
+        # Run again with immediate deletion
+        store.cleanup(delay=0)
+        self.assertEqual(store.get('tenant'), state_one)
+        self.assertIsNotNone(
+            self.zk_client.client.exists(
+                f'/zuul/layout-data/{state_one.uuid}'))
+        self.assertIsNone(
+            self.zk_client.client.exists(
+                f'/zuul/layout-data/{state_two.uuid}'))
+
 
 class TestSystemConfigCache(ZooKeeperBaseTestCase):