From 7f2986fa52c03a8303023e1a169d54fcff630c59 Mon Sep 17 00:00:00 2001 From: "James E. Blair" Date: Wed, 15 Sep 2021 16:29:20 -0700 Subject: Add a zookeeper map to developer docs Add a description of the ZK paths that we use and why, in order to aid developers. Change-Id: Ia656782872a0bf09ee6598029c5b00ecbac906e6 --- doc/source/_static/custom.css | 3 + doc/source/reference/developer/index.rst | 1 + doc/source/reference/developer/zookeeper.rst | 347 +++++++++++++++++++++++++++ 3 files changed, 351 insertions(+) create mode 100644 doc/source/reference/developer/zookeeper.rst (limited to 'doc') diff --git a/doc/source/_static/custom.css b/doc/source/_static/custom.css index b49c2c032..f208eef84 100644 --- a/doc/source/_static/custom.css +++ b/doc/source/_static/custom.css @@ -4,3 +4,6 @@ div.sphinxsidebarwrapper p.logo { text-align: left; } +dl.zuul.path { + margin-bottom: 1em; +} diff --git a/doc/source/reference/developer/index.rst b/doc/source/reference/developer/index.rst index 3c137fb39..30a302f0a 100644 --- a/doc/source/reference/developer/index.rst +++ b/doc/source/reference/developer/index.rst @@ -18,4 +18,5 @@ Zuul, though advanced users may find it interesting. ansible javascript specs/index + zookeeper releasenotes diff --git a/doc/source/reference/developer/zookeeper.rst b/doc/source/reference/developer/zookeeper.rst new file mode 100644 index 000000000..89b96af00 --- /dev/null +++ b/doc/source/reference/developer/zookeeper.rst @@ -0,0 +1,347 @@ +Zookeeper Map +============= + +This is a reference for object layout in Zookeeper. + +.. path:: zuul + + All ephemeral data stored here. Remove the entire tree to "reset" + the system. + +.. path:: zuul/cache/connection/ + + The connection cache root. Each connection has a dedicated space + for its caches. Two types of caches are currently implemented: + change and branch. + +.. path:: zuul/cache/connection//branches + + The connection branch cache root. Contains the cache itself and a + lock. + +.. path:: zuul/cache/connection//branches/data + :type: BranchCacheZKObject (sharded) + + The connection branch cache data. This is a single sharded JSON blob. + +.. path:: zuul/cache/connection//branches/lock + :type: RWLock + + The connection branch cache read/write lock. + +.. path:: zuul/cache/connection//cache + + The connection change cache. Each node under this node is an entry + in the change cache. The node ID is a sha256 of the cache key, the + contents are the JSON serialization of the cache entry metadata. + One of the included items is the `data_uuid` which is used to + retrieve the actual change data. + + When a cache entry is updated, a new data node is created without + deleting the old data node. They are eventually garbage collected. + +.. path:: zuul/cache/connection//data + + Data for the change cache. These nodes are identified by a UUID + referenced from the cache entries. + + These are sharded JSON blobs of the change data. + +.. path:: zuul/cleanup + + This node holds locks for the cleanup routines to make sure that + only one scheduler runs them at a time. + + .. path:: build_requests + .. path:: connection + .. path:: general + .. path:: merge_requests + .. path:: node_request + .. path:: sempahores + +.. path:: zuul/components + + The component registry. Each Zuul process registers itself under + the appropriate node in this hierarchy so the system has a holistic + view of what's running. The name of the node is based on the + hostname but is a sequence node in order to handle multiple + processes. The nodes are ephemeral so an outage is automatically + detected. + + The contents of each node contain information about the running + process and may be updated periodically. + + .. path:: executor + .. path:: fingergw + .. path:: merger + .. path:: scheduler + .. path:: web + +.. path:: zuul/config/cache + + The unparsed config cache. This contains the contents of every + Zuul config file returned by the mergers for use in configuration. + Organized by repo canonical name, branch, and filename. The files + themeselves are sharded. + +.. path:: zuul/config/lock + + Locks for the unparsed config cache. + +.. path:: zuul/events/connection//events + :type: ConnectionEventQueue + + The connection event queue root. Each connection has an event + queue where incoming events are recorded before being moved to the + tenant event queue. + +.. path:: zuul/events/connection//events/queue + + The actual event queue. Entries in the queue reference separate + data nodes. These are sequence nodes to maintain the event order. + +.. path:: zuul/events/connection//events/data + + Event data nodes referenced by queue items. These are sharded. + +.. path:: zuul/events/connection//events/election + + An election to determine which scheduler processes the event queue + and moves events to the tenant event queues. + + Drivers may have additional elections as well. For example, Gerrit + has an election for the watcher and poller. + +.. path:: zuul/events/tenant/ + + Tenant-specific event queues. Each queue described below has a + data and queue subnode. + +.. path:: zuul/events/tenant//management + + The tenant-specific management event queue. + +.. path:: zuul/events/tenant//trigger + + The tenant-specific trigger event queue. + +.. path:: zuul/events/tenant//pipelines + + Holds a set of queues for each pipeline. + +.. path:: zuul/events/tenant//pipelines//management + + The pipeline management event queue. + +.. path:: zuul/events/tenant//pipelines//result + + The pipeline result event queue. + +.. path:: zuul/events/tenant//pipelines//trigger + + The pipeline trigger event queue. + +.. path:: zuul/executor/unzoned + :type: JobRequestQueue + + The unzoned executor build request queue. The generic description + of a job request queue follows: + + .. path:: requests/ + + Requests are added by UUID. Consumers watch the entire tree and + order the requests by znode creation time. + + .. path:: locks/ + :type: Lock + + A consumer will create a lock under this node before processing + a request. The znode containing the lock and the requent znode + have the same UUID. This is a side-channel lock so that the + lock can be held while the request itself is deleted. + + .. path:: params/ + + Parameters can be quite large, so they are kept in a separate + znode and only read when needed, and may be removed during + request processing to save space in ZooKeeper. The data may be + sharded. + + .. path:: result-data/ + + When a job is complete, the results of the merge are written + here. The results may be quite large, so they are sharded. + + .. path:: results/ + + Since writing sharded data is not atomic, once the results are + written to ``result-data``, a small znode is written here to + indicate the results are ready to read. The submitter can watch + this znode to be notified that it is ready. + + .. path:: waiters/ + :ephemeral: + + A submitter who requires the results of the job creates an + ephemeral node here to indicate their interest in the results. + This is used by the cleanup routines to ensure that they don't + prematurely delete the result data. Used for merge jobs + +.. path:: zuul/executor/zones/ + + A zone-specific executor build request queue. The contents are the + same as above. + +.. path:: zuul/layout/ + + The layout state for the tenant. Contains the cache and time data + needed for a component to determine if its in-memory layout is out + of date and update it if so. + +.. path:: zuul/locks + + Holds various types of locks so that multiple components can coordinate. + +.. path:: zuul/locks/events + + Locks related to tenant event queues. + +.. path:: zuul/locks/events/trigger/ + :type: Lock + + The scheduler locks the trigger event queue for each tenant before + processing it. This lock is only needed when processing and + removing items from the queue; no lock is required to add items. + +.. path:: zuul/locks/events/management/ + :type: Lock + + The scheduler locks the management event queue for each tenant + before processing it. This lock is only needed when processing and + removing items from the queue; no lock is required to add items. + +.. path:: zuul/locks/pipeline + + Locks related to pipelines. + +.. path:: zuul/locks/pipeline// + :type: Lock + + The scheduler obtains a lock before processing each pipeline. + +.. path:: zuul/locks/tenant + + Tenant configuration locks. + +.. path:: zuul/locks/tenant/ + :type: RWLock + + A write lock is obtained at this location before creating a new + tenant layout and storing its metadata in ZooKeeper. Components + which later determine that they need to update their tenant + configuration to match the state in ZooKeeper will obtain a read + lock at this location to ensure the state isn't mutated again while + the components are updating their layout to match. + +.. path:: zuul/ltime + + An empty node which serves to coordinate logical timestamps across + the cluster. Components may update this znode which will cause the + latest ZooKeeper transaction ID to appear in the zstat for this + znode. This is known as the `ltime` and can be used to communicate + that any subsequent transactions have occurred after this `ltime`. + This is frequently used for cache validation. Any cache which was + updated after a specified `ltime` may be determined to be + sufficiently up-to-date for use without invalidation. + +.. path:: zuul/merger + :type: JobRequestQueue + + A JobRequestQueue for mergers. See :path:`zuul/executor/unzoned`. + +.. path:: zuul/nodepool + :type: NodepoolEventElection + + An election to decide which scheduler will monitor nodepool + requests and generate node completion events as they are completed. + +.. path:: zuul/results/management + + Stores results from management events (such as an enqueue event). + +.. path:: zuul/scheduler/timer-election + :type: SessionAwareElection + + An election to decide which scheduler will generate events for + timer pipeline triggers. + +.. path:: zuul/scheduler/stats-election + :type: SchedulerStatsElection + + An election to decide which scheduler will report system-wide stats + (such as total node requests). + +.. path:: zuul/semaphores// + :type: SemaphoreHandler + + Represents a semaphore. Information about which builds hold the + semaphore is stored in the znode data. + +.. path:: zuul/system + :type: SystemConfigCache + + System-wide configuration data. + + .. path:: conf + + The serialized version of the unparsed abide configuration as + well as system attributes (such as the tenant list). + + .. path:: conf-lock + :type: WriteLock + + A lock to be acquired before updating :path:`zuul/system/conf` + +.. path:: zuul/tenant/ + + Tenant-specific information here. + +.. path:: zuul/tenant//pipeline/ + + Pipeline state. + +.. path:: zuul/tenant//pipeline//queue + + Holds queue objects. + +.. path:: zuul/tenant//pipeline//item/ + + Items belong to queues, but are held in their own hierarchy since + they may shift to differrent queues during reconfiguration. + +.. path:: zuul/tenant//pipeline//item//buildset/ + + There will only be one buildset under the buildset/ node. If we + reset it, we will get a new uuid and delete the old one. Any + external references to it will be automatically invalidated. + +.. path:: zuul/tenant//pipeline//item//buildset//repo_state + + The global repo state for the buildset is kept in its own node + since it can be large, and is also common for all jobs in this + buildset. + +.. path:: zuul/tenant//pipeline//item//buildset//job/ + + The frozen job. + +.. path:: zuul/tenant//pipeline//item//buildset//job//build/ + + Information about this build of the job. Similar to buildset, + there should only be one entry, and using the UUID automatically + invalidates any references. + +.. path:: zuul/tenant//pipeline//item//buildset//job//build//parameters + + Parameters for the build; these can be large so they're in their + own znode and will be read only if needed. -- cgit v1.2.1