Merge "Add global semaphore support"

6 files changed, 70 insertions, 136 deletions

diff --git a/doc/source/config/job.rst b/doc/source/config/job.rst
index 541a1b7df..87806727f 100644
--- a/doc/source/config/job.rst
+++ b/doc/source/config/job.rst
@@ -219,14 +219,14 @@ Here is an example of two job definitions:
 
    .. attr:: semaphores
 
-      The name of a :ref:`semaphore` (or list of them) which should be
-      acquired and released when the job begins and ends.  If the
-      semaphore is at maximum capacity, then Zuul will wait until it
-      can be acquired before starting the job. The format is either a
-      string, a dictionary, or a list of either of those in the case
-      of multiple semaphores. If it's a string it references a
-      semaphore using the default value for
-      :attr:`job.semaphores.resources-first`.
+      The name of a :ref:`semaphore` (or list of them) or
+      :ref:`global_semaphore` which should be acquired and released
+      when the job begins and ends.  If the semaphore is at maximum
+      capacity, then Zuul will wait until it can be acquired before
+      starting the job. The format is either a string, a dictionary,
+      or a list of either of those in the case of multiple
+      semaphores. If it's a string it references a semaphore using the
+      default value for :attr:`job.semaphores.resources-first`.
 
       If multiple semaphores are requested, the job will not start
       until all have been acquired, and Zuul will wait until all are
diff --git a/doc/source/config/semaphore.rst b/doc/source/config/semaphore.rst
index 74440ae79..91bfc9a4d 100644
--- a/doc/source/config/semaphore.rst
+++ b/doc/source/config/semaphore.rst
@@ -15,6 +15,10 @@ project as long as the value is the same.  This is to aid in branch
 maintenance, so that creating a new branch based on an existing branch
 will not immediately produce a configuration error.
 
+Zuul also supports global semaphores (see :ref:`global_semaphore`)
+which may only be created by the Zuul administrator, but can be used
+to coordinate resources across multiple tenants.
+
 Semaphores are never subject to dynamic reconfiguration.  If the value
 of a semaphore is changed, it will take effect only when the change
 where it is updated is merged.  However, Zuul will attempt to validate
diff --git a/doc/source/developer/specs/global-semaphores.rst b/doc/source/developer/specs/global-semaphores.rst
deleted file mode 100644
index 107fdf620..000000000
--- a/doc/source/developer/specs/global-semaphores.rst
+++ /dev/null
@@ -1,127 +0,0 @@
-Global Semaphores
-=================
-
-.. warning:: This is not authoritative documentation.  These features
-   are not currently available in Zuul.  They may change significantly
-   before final implementation, or may never be fully completed.
-
-Semaphores are useful for limiting access to resources, but their
-implementation as a per-tenant configuration construct may be limiting
-if they are used for real-world resources that span tenants.
-
-This is a proposal to address that by adding global semaphores.
-
-Background
-----------
-
-Semaphores may be used for a variety of purposes.  One of these is to
-limit access to constrained resources.  Doing so allows Zuul to avoid
-requesting nodes and scheduling jobs until these resources are
-available.  This makes the overall system more efficient as jobs don't
-need to wait for resources during their run phase (where they may be
-idling test nodes which could otherwise be put to better use).
-
-A concrete example of this is software licenses.  If a job requires
-software which uses a license server to ensure that the number of
-in-use seats does not exceed the available seats, a semaphore with a
-max value equal to the number of available seats can be used to help
-Zuul avoid starting jobs which would otherwise need to wait for a
-license.
-
-If only one Zuul tenant uses this piece of software, the existing
-implementation of semaphores in Zuul is satisfactory.  But if the
-software licenses are shared across Zuul tenants, then a Zuul
-semaphore can't be used in this way since semaphores are per-tenant
-constructs.
-
-The general solution to sharing Zuul configuration objects across
-tenants is to define them in a single git repo and include that git
-repo in multiple tenants.  That works as expected for Jobs, Project
-Templates, etc.  But semaphores have a definition as well as a
-run-time state (whether they are aquired and by whom).  Including a
-semaphore in multiple tenants essentially makes copies of that
-semaphore, each with its own distinct set of holders.
-
-Proposed Change
----------------
-
-A new global semaphore configuration would be added to the tenant
-configuration file.  Note this is the global configuration file (where
-tenants are defined); not in-repo configuration where semaphores are
-currently defined.
-
-The definition would be identical to the current in-repo semaphore
-configuration.  In order to grant access to only certain tenants, each
-tenant will also need to specify whether that semaphore should be
-available to the tenant.  This scheme is similar to the way that
-authorization rules are defined in this file and then attached to
-tenants.
-
-For example:
-
-.. code-block:: yaml
-
-   - semaphore:
-       name: licensed-software
-       max: 8
-
-   - tenant:
-       name: example-tenant
-       semaphores:
-         - licensed-software
-       source:
-         gerrit:
-            config-projects:
-              ...
-
-The existing in-repo semaphores will remain as they are today -- they
-will not be deprecated (they are still very useful on their own for
-most other use cases).
-
-If an in-repo semaphore is defined with the same name as a global
-semaphore, that will become a configuration error.  The global
-semaphore will take precedence.
-
-Implementation
---------------
-
-The user-visible configuration is described above.
-
-Current semaphores are stored in the ZooKeeper path
-``/zuul/semaphores/<tenant>/<semaphore>``.  Global semaphores will use
-a similar scheme without the tenant name:
-``/zuul/global-semaphores/<semaphore>``.
-
-Locking, releasing, and leak cleanup will all behave similarly to the
-current per-tenant semaphores.  On release, a per-tenant semaphore
-broadcasts a PipelineSemaphoreReleaseEvent to all pipelines in order
-to trigger a pipeline run and start any jobs which may be waiting on
-the semaphore.  A global semaphore will do the same, but for every
-pipeline of every tenant which includes the semaphore.
-
-Alternatives
-------------
-
-We could add a field to the in-repo definitions of semaphores which
-indicates that the semaphore should be global.  As this has the
-ability to affect other tenants, we would need to restrict this to
-config-projects only.  However, that still opens the possibility of
-one tenant affecting another via the contents of a config-project.
-Some method of allowing the administrator to control this via the
-tenant config file would still likely be necessary.  As long as that's
-the case, it seems simpler to just define the semaphores there too.
-
-We could outsource this to Nodepool.  In fact, having nodepool manage
-resources like this seems like a natural fit.  However, the current
-Nodepool implementation doesn't permit more than one provider to
-satisfy a node request, so a hypothetical semaphore provider wouldn't
-be able to be combined with a provider of actual test nodes.
-Addressing this is in-scope and a worthwhile change for Nodepool, but
-it is potentially a large and complex change.  Additionally, the idea
-of waiting for a semaphore before submitting requests for real
-resources adds a new dimension to even that idea -- Nodepool would
-need to know whether to run the semaphore provider first or last
-depending on the desired resource aquisition order.  Meanwhile, since
-Zuul does have the concept of semaphores already and they almost fit
-this use case, this seems like a reasonable change to make in Zuul
-regardless of any potential Nodepool changes.
diff --git a/doc/source/developer/specs/index.rst b/doc/source/developer/specs/index.rst
index 0f64b35c5..d96df0c26 100644
--- a/doc/source/developer/specs/index.rst
+++ b/doc/source/developer/specs/index.rst
@@ -23,4 +23,3 @@ documentation instead.
    enhanced-regional-executors
    tenant-resource-quota
    community-matrix
-   global-semaphores
diff --git a/doc/source/developer/zookeeper.rst b/doc/source/developer/zookeeper.rst
index c14047ad6..ee77f815f 100644
--- a/doc/source/developer/zookeeper.rst
+++ b/doc/source/developer/zookeeper.rst
@@ -414,6 +414,13 @@ This is a reference for object layout in Zookeeper.
    An election to decide which scheduler will report system-wide stats
    (such as total node requests).
 
+.. path:: zuul/global-semaphores/<semaphore>
+   :type: SemaphoreHandler
+
+   Represents a global semaphore (shared by multiple tenants).
+   Information about which builds hold the semaphore is stored in the
+   znode data.
+
 .. path:: zuul/semaphores/<tenant>/<semaphore>
    :type: SemaphoreHandler
 
diff --git a/doc/source/tenants.rst b/doc/source/tenants.rst
index a82031366..c2ed38df3 100644
--- a/doc/source/tenants.rst
+++ b/doc/source/tenants.rst
@@ -384,6 +384,57 @@ configuration. Some examples of tenant definitions are:
          to add finer filtering to admin rules, for example filtering by the ``iss``
          claim (generally equal to the issuer ID).
 
+   .. attr:: semaphores
+
+      A list of names of :attr:`global-semaphore` objects to allow
+      jobs in this tenant to access.
+
+.. _global_semaphore:
+
+Global Semaphore
+----------------
+
+Semaphores are normally defined in in-repo configuration (see
+:ref:`semaphore`), however to support use-cases where semaphores are
+used to represent constrained global resources that may be used by
+multiple Zuul tenants, semaphores may be defined within the main
+tenant configuration file.
+
+In order for a job to use a global semaphore, the semaphore must first
+be defined in the tenant configuration file with
+:attr:`global-semaphore` and then added to each tenant which should
+have access to it with :attr:`tenant.semaphores`.  Once that is done,
+Zuul jobs may use that semaphore in the same way they would use a
+normal tenant-scoped semaphore.
+
+If any tenant which is granted access to a global semaphore also has a
+tenant-scoped semaphore defined with the same name, that definition
+will be treated as a configuration error and subsequently ignored in
+favor of the global semaphore.
+
+An example definition looks similar to the normal semaphore object:
+
+.. code-block:: yaml
+
+   - global-semaphore:
+       name: global-semaphore-foo
+       max: 5
+
+.. attr:: global-semaphore
+
+   The following attributes are available:
+
+   .. attr:: name
+      :required:
+
+      The name of the semaphore, referenced by jobs.
+
+   .. attr:: max
+      :default: 1
+
+      The maximum number of running jobs which can use this semaphore.
+
+
 .. _admin_rule_definition:
 
 Access Rule