diff options
author | sheaf <sam.derbyshire@gmail.com> | 2022-09-07 14:22:34 +0200 |
---|---|---|
committer | Marge Bot <ben+marge-bot@smart-cactus.org> | 2023-04-20 18:33:34 -0400 |
commit | 5c8731244bc13a3d813d2a4d53b3188b28dc8355 (patch) | |
tree | 49561b32034c2f4199479290975b2e624c8ffab1 /docs | |
parent | 7012ec2facc632fe4966916f797e4d1f612d7318 (diff) | |
download | haskell-5c8731244bc13a3d813d2a4d53b3188b28dc8355.tar.gz |
Implement -jsem: parallelism controlled by semaphores
See https://github.com/ghc-proposals/ghc-proposals/pull/540/ for a
complete description for the motivation for this feature.
The `-jsem` option allows a build tool to pass a semaphore to GHC which
GHC can use in order to control how much parallelism it requests.
GHC itself acts as a client in the GHC jobserver protocol.
```
GHC Jobserver Protocol
~~~~~~~~~~~~~~~~~~~~~~
This proposal introduces the GHC Jobserver Protocol. This protocol allows
a server to dynamically invoke many instances of a client process,
while restricting all of those instances to use no more than <n> capabilities.
This is achieved by coordination over a system semaphore (either a POSIX
semaphore [6]_ in the case of Linux and Darwin, or a Win32 semaphore [7]_
in the case of Windows platforms).
There are two kinds of participants in the GHC Jobserver protocol:
- The *jobserver* creates a system semaphore with a certain number of
available tokens.
Each time the jobserver wants to spawn a new jobclient subprocess, it **must**
first acquire a single token from the semaphore, before spawning
the subprocess. This token **must** be released once the subprocess terminates.
Once work is finished, the jobserver **must** destroy the semaphore it created.
- A *jobclient* is a subprocess spawned by the jobserver or another jobclient.
Each jobclient starts with one available token (its *implicit token*,
which was acquired by the parent which spawned it), and can request more
tokens through the Jobserver Protocol by waiting on the semaphore.
Each time a jobclient wants to spawn a new jobclient subprocess, it **must**
pass on a single token to the child jobclient. This token can either be the
jobclient's implicit token, or another token which the jobclient acquired
from the semaphore.
Each jobclient **must** release exactly as many tokens as it has acquired from
the semaphore (this does not include the implicit tokens).
```
Build tools such as cabal act as jobservers in the protocol and are
responsibile for correctly creating, cleaning up and managing the
semaphore.
Adds a new submodule (semaphore-compat) for managing and interacting
with semaphores in a cross-platform way.
Fixes #19349
Diffstat (limited to 'docs')
-rw-r--r-- | docs/users_guide/9.8.1-notes.rst | 7 | ||||
-rw-r--r-- | docs/users_guide/using.rst | 54 |
2 files changed, 61 insertions, 0 deletions
diff --git a/docs/users_guide/9.8.1-notes.rst b/docs/users_guide/9.8.1-notes.rst index a624eb2705..0f0848765f 100644 --- a/docs/users_guide/9.8.1-notes.rst +++ b/docs/users_guide/9.8.1-notes.rst @@ -87,6 +87,13 @@ Compiler deriving instance TypeError (Text "Boo") => Bar Baz +- GHC Proposal `#540 https://github.com/ghc-proposals/ghc-proposals/blob/master/proposals/0540-jsem.rst`_ has been implemented. + This adds the `-jsem`:ghc-flag: flag, which instructs GHC to act as a jobserver client. + This enables multiple GHC processes running at once to share system resources + with each other, communicating via the system semaphore specified by + the flag argument. + + GHCi ~~~~ diff --git a/docs/users_guide/using.rst b/docs/users_guide/using.rst index 388704388c..787b6a0503 100644 --- a/docs/users_guide/using.rst +++ b/docs/users_guide/using.rst @@ -751,6 +751,60 @@ search path (see :ref:`search-path`). number of processors. Note that compilation of a module may not begin until its dependencies have been built. + +GHC Jobserver Protocol +~~~~~~~~~~~~~~~~~~~~~~ + +The GHC Jobserver Protocol was specified in `GHC proposal #540 <https://github.com/ghc-proposals/ghc-proposals/blob/master/proposals/0540-jsem.rst>`__. + +This protocol allows +a server to dynamically invoke many instances of a client process, +while restricting all of those instances to use no more than <n> capabilities. +This is achieved by coordination over a system semaphore (either a POSIX +semaphore in the case of Linux and Darwin, or a Win32 semaphore +in the case of Windows platforms). + +There are two kinds of participants in the GHC Jobserver protocol: + +- The *jobserver* creates a system semaphore with a certain number of + available tokens. + + Each time the jobserver wants to spawn a new jobclient subprocess, it **must** + first acquire a single token from the semaphore, before spawning + the subprocess. This token **must** be released once the subprocess terminates. + + Once work is finished, the jobserver **must** destroy the semaphore it created. + +- A *jobclient* is a subprocess spawned by the jobserver or another jobclient. + + Each jobclient starts with one available token (its *implicit token*, + which was acquired by the parent which spawned it), and can request more + tokens through the Jobserver Protocol by waiting on the semaphore. + + Each time a jobclient wants to spawn a new jobclient subprocess, it **must** + pass on a single token to the child jobclient. This token can either be the + jobclient's implicit token, or another token which the jobclient acquired + from the semaphore. + + Each jobclient **must** release exactly as many tokens as it has acquired from + the semaphore (this does not include the implicit tokens). + + GHC itself acts as a jobclient which can be enabled by using the flag ``-jsem``. + +.. ghc-flag:: -jsem + :shortdesc: When compiling with :ghc-flag:`--make`, coordinate with + other processes through the semaphore ⟨sem⟩ to compile + modules in parallel. + :type: dynamic + :category: misc + + Perform compilation in parallel when possible, coordinating with other + processes through the semaphore ⟨sem⟩ (specified as a string). + Error if the semaphore doesn't exist. + + Use of ``-jsem`` will override use of :ghc-flag:``-j[⟨n⟩]``, + and vice-versa. + .. _multi-home-units: Multiple Home Units |