Add global semaphore support

This adds support for global semaphores which can be used by multiple tenants.
This supports the use case where they represent real-world resources which
operate independentyl of Zuul tenants.

This implements and removes the spec describing the feature.  One change from
the spec is that the configuration object in the tenant config file is
"global-semaphore" rather than "semaphore".  This makes it easier to distinguish
them in documentation (facilitating easier cross-references and deep links),
and may also make it easier for users to understand that they have distinct
behavoirs.

Change-Id: I5f2225a700d8f9bef0399189017f23b3f4caad17

6 files changed, 70 insertions, 136 deletions

diff --git a/doc/source/config/job.rst b/doc/source/config/job.rst
index 541a1b7df..87806727f 100644
--- a/doc/source/config/job.rst
+++ b/doc/source/config/job.rst
@@ -219,14 +219,14 @@ Here is an example of two job definitions:
 
    .. attr:: semaphores
 
-      The name of a :ref:`semaphore` (or list of them) which should be
-      acquired and released when the job begins and ends.  If the
-      semaphore is at maximum capacity, then Zuul will wait until it
-      can be acquired before starting the job. The format is either a
-      string, a dictionary, or a list of either of those in the case
-      of multiple semaphores. If it's a string it references a
-      semaphore using the default value for
-      :attr:`job.semaphores.resources-first`.
+      The name of a :ref:`semaphore` (or list of them) or
+      :ref:`global_semaphore` which should be acquired and released
+      when the job begins and ends.  If the semaphore is at maximum
+      capacity, then Zuul will wait until it can be acquired before
+      starting the job. The format is either a string, a dictionary,
+      or a list of either of those in the case of multiple
+      semaphores. If it's a string it references a semaphore using the
+      default value for :attr:`job.semaphores.resources-first`.
 
       If multiple semaphores are requested, the job will not start
       until all have been acquired, and Zuul will wait until all are
diff --git a/doc/source/config/semaphore.rst b/doc/source/config/semaphore.rst
index 74440ae79..91bfc9a4d 100644
--- a/doc/source/config/semaphore.rst
+++ b/doc/source/config/semaphore.rst
@@ -15,6 +15,10 @@ project as long as the value is the same.  This is to aid in branch
 maintenance, so that creating a new branch based on an existing branch
 will not immediately produce a configuration error.
 
+Zuul also supports global semaphores (see :ref:`global_semaphore`)
+which may only be created by the Zuul administrator, but can be used
+to coordinate resources across multiple tenants.
+
 Semaphores are never subject to dynamic reconfiguration.  If the value
 of a semaphore is changed, it will take effect only when the change
 where it is updated is merged.  However, Zuul will attempt to validate
diff --git a/doc/source/developer/specs/global-semaphores.rst b/doc/source/developer/specs/global-semaphores.rst
deleted file mode 100644
index 107fdf620..000000000
--- a/doc/source/developer/specs/global-semaphores.rst
+++ /dev/null
@@ -1,127 +0,0 @@
-Global Semaphores
-=================
-
-.. warning:: This is not authoritative documentation.  These features
-   are not currently available in Zuul.  They may change significantly
-   before final implementation, or may never be fully completed.
-
-Semaphores are useful for limiting access to resources, but their
-implementation as a per-tenant configuration construct may be limiting
-if they are used for real-world resources that span tenants.
-
-This is a proposal to address that by adding global semaphores.
-
-Background
-----------
-
-Semaphores may be used for a variety of purposes.  One of these is to
-limit access to constrained resources.  Doing so allows Zuul to avoid
-requesting nodes and scheduling jobs until these resources are
-available.  This makes the overall system more efficient as jobs don't
-need to wait for resources during their run phase (where they may be
-idling test nodes which could otherwise be put to better use).
-
-A concrete example of this is software licenses.  If a job requires
-software which uses a license server to ensure that the number of
-in-use seats does not exceed the available seats, a semaphore with a
-max value equal to the number of available seats can be used to help
-Zuul avoid starting jobs which would otherwise need to wait for a
-license.
-
-If only one Zuul tenant uses this piece of software, the existing
-implementation of semaphores in Zuul is satisfactory.  But if the
-software licenses are shared across Zuul tenants, then a Zuul
-semaphore can't be used in this way since semaphores are per-tenant
-constructs.
-
-The general solution to sharing Zuul configuration objects across
-tenants is to define them in a single git repo and include that git
-repo in multiple tenants.  That works as expected for Jobs, Project
-Templates, etc.  But semaphores have a definition as well as a
-run-time state (whether they are aquired and by whom).  Including a
-semaphore in multiple tenants essentially makes copies of that
-semaphore, each with its own distinct set of holders.
-
-Proposed Change
----------------
-
-A new global semaphore configuration would be added to the tenant
-configuration file.  Note this is the global configuration file (where
-tenants are defined); not in-repo configuration where semaphores are
-currently defined.
-
-The definition would be identical to the current in-repo semaphore
-configuration.  In order to grant access to only certain tenants, each
-tenant will also need to specify whether that semaphore should be
-available to the tenant.  This scheme is similar to the way that
-authorization rules are defined in this file and then attached to
-tenants.
-
-For example:
-
-.. code-block:: yaml
-
-   - semaphore:
-       name: licensed-software
-       max: 8
-
-   - tenant:
-       name: example-tenant
-       semaphores:
-         - licensed-software
-       source:
-         gerrit:
-            config-projects:
-              ...
-
-The existing in-repo semaphores will remain as they are today -- they
-will not be deprecated (they are still very useful on their own for
-most other use cases).
-
-If an in-repo semaphore is defined with the same name as a global
-semaphore, that will become a configuration error.  The global
-semaphore will take precedence.
-
-Implementation
---------------
-
-The user-visible configuration is described above.
-
-Current semaphores are stored in the ZooKeeper path
-``/zuul/semaphores/<tenant>/<semaphore>``.  Global semaphores will use
-a similar scheme without the tenant name:
-``/zuul/global-semaphores/<semaphore>``.
-
-Locking, releasing, and leak cleanup will all behave similarly to the
-current per-tenant semaphores.  On release, a per-tenant semaphore
-broadcasts a PipelineSemaphoreReleaseEvent to all pipelines in order
-to trigger a pipeline run and start any jobs which may be waiting on
-the semaphore.  A global semaphore will do the same, but for every
-pipeline of every tenant which includes the semaphore.
-
-Alternatives
-------------
-
-We could add a field to the in-repo definitions of semaphores which
-indicates that the semaphore should be global.  As this has the
-ability to affect other tenants, we would need to restrict this to
-config-projects only.  However, that still opens the possibility of
-one tenant affecting another via the contents of a config-project.
-Some method of allowing the administrator to control this via the
-tenant config file would still likely be necessary.  As long as that's
-the case, it seems simpler to just define the semaphores there too.
-
-We could outsource this to Nodepool.  In fact, having nodepool manage
-resources like this seems like a natural fit.  However, the current
-Nodepool implementation doesn't permit more than one provider to
-satisfy a node request, so a hypothetical semaphore provider wouldn't
-be able to be combined with a provider of actual test nodes.
-Addressing this is in-scope and a worthwhile change for Nodepool, but
-it is potentially a large and complex change.  Additionally, the idea
-of waiting for a semaphore before submitting requests for real
-resources adds a new dimension to even that idea -- Nodepool would
-need to know whether to run the semaphore provider first or last
-depending on the desired resource aquisition order.  Meanwhile, since
-Zuul does have the concept of semaphores already and they almost fit
-this use case, this seems like a reasonable change to make in Zuul
-regardless of any potential Nodepool changes.
diff --git a/doc/source/developer/specs/index.rst b/doc/source/developer/specs/index.rst
index 0f64b35c5..d96df0c26 100644
--- a/doc/source/developer/specs/index.rst
+++ b/doc/source/developer/specs/index.rst
@@ -23,4 +23,3 @@ documentation instead.
    enhanced-regional-executors
    tenant-resource-quota
    community-matrix
-   global-semaphores
diff --git a/doc/source/developer/zookeeper.rst b/doc/source/developer/zookeeper.rst
index c14047ad6..ee77f815f 100644
--- a/doc/source/developer/zookeeper.rst
+++ b/doc/source/developer/zookeeper.rst
@@ -414,6 +414,13 @@ This is a reference for object layout in Zookeeper.
    An election to decide which scheduler will report system-wide stats
    (such as total node requests).
 
+.. path:: zuul/global-semaphores/<semaphore>
+   :type: SemaphoreHandler
+
+   Represents a global semaphore (shared by multiple tenants).
+   Information about which builds hold the semaphore is stored in the
+   znode data.
+
 .. path:: zuul/semaphores/<tenant>/<semaphore>
    :type: SemaphoreHandler
 
diff --git a/doc/source/tenants.rst b/doc/source/tenants.rst
index a82031366..c2ed38df3 100644
--- a/doc/source/tenants.rst
+++ b/doc/source/tenants.rst
@@ -384,6 +384,57 @@ configuration. Some examples of tenant definitions are:
          to add finer filtering to admin rules, for example filtering by the ``iss``
          claim (generally equal to the issuer ID).
 
+   .. attr:: semaphores
+
+      A list of names of :attr:`global-semaphore` objects to allow
+      jobs in this tenant to access.
+
+.. _global_semaphore:
+
+Global Semaphore
+----------------
+
+Semaphores are normally defined in in-repo configuration (see
+:ref:`semaphore`), however to support use-cases where semaphores are
+used to represent constrained global resources that may be used by
+multiple Zuul tenants, semaphores may be defined within the main
+tenant configuration file.
+
+In order for a job to use a global semaphore, the semaphore must first
+be defined in the tenant configuration file with
+:attr:`global-semaphore` and then added to each tenant which should
+have access to it with :attr:`tenant.semaphores`.  Once that is done,
+Zuul jobs may use that semaphore in the same way they would use a
+normal tenant-scoped semaphore.
+
+If any tenant which is granted access to a global semaphore also has a
+tenant-scoped semaphore defined with the same name, that definition
+will be treated as a configuration error and subsequently ignored in
+favor of the global semaphore.
+
+An example definition looks similar to the normal semaphore object:
+
+.. code-block:: yaml
+
+   - global-semaphore:
+       name: global-semaphore-foo
+       max: 5
+
+.. attr:: global-semaphore
+
+   The following attributes are available:
+
+   .. attr:: name
+      :required:
+
+      The name of the semaphore, referenced by jobs.
+
+   .. attr:: max
+      :default: 1
+
+      The maximum number of running jobs which can use this semaphore.
+
+
 .. _admin_rule_definition:
 
 Access Rule