docs: move doc/ to docs/

The docs/ directory is special in GitHub, since it can be used to serve GitHub
Pages from, so there's a benefit to switching to it in order to expose it
directly as a website.

Updated references to it from the documentations themselves, from the
CONTRIBUTING.md file and from Meson build files.

1 files changed, 456 insertions, 0 deletions

diff --git a/docs/CGROUP_DELEGATION.md b/docs/CGROUP_DELEGATION.md
new file mode 100644
index 0000000000..63d9d41b10
--- /dev/null
+++ b/docs/CGROUP_DELEGATION.md
@@ -0,0 +1,456 @@
+# Control Group APIs and Delegation
+
+*Intended audience: hackers working on userspace subsystems that require direct
+cgroup access, such as container managers and similar.*
+
+So you are wondering about resource management with systemd, you know Linux
+control groups (cgroups) a bit and are trying to integrate your software with
+what systemd has to offer there. Here's a bit of documentation about the
+concepts and interfaces involved with this.
+
+What's described here has been part of systemd and documented since v205
+times. However, it has been updated and improved substantially, even
+though the concepts stayed mostly the same. This is an attempt to provide more
+comprehensive up-to-date information about all this, particular in light of the
+poor implementations of the components interfacing with systemd of current
+container managers.
+
+Before you read on, please make sure you read the low-level [kernel
+documentation about
+cgroupsv2](https://www.kernel.org/doc/Documentation/cgroup-v2.txt). This
+documentation then adds in the higher-level view from systemd.
+
+This document augments the existing documentation we already have:
+
+* [The New Control Group Interfaces](https://www.freedesktop.org/wiki/Software/systemd/ControlGroupInterface/)
+* [Writing VM and Container Managers](https://www.freedesktop.org/wiki/Software/systemd/writing-vm-managers/)
+
+These wiki documents are not as up to date as they should be, currently, but
+the basic concepts still fully apply. You should read them too, if you do something
+with cgroups and systemd, in particular as they shine more light on the various
+D-Bus APIs provided. (That said, sooner or later we should probably fold that
+wiki documentation into this very document, too.)
+
+## Two Key Design Rules
+
+Much of the philosophy behind these concepts is based on a couple of basic
+design ideas of cgroupsv2 (which we however try to adapt as far as we can to
+cgroupsv1 too). Specifically two cgroupsv2 rules are the most relevant:
+
+1. The **no-processes-in-inner-nodes** rule: this means that it's not permitted
+to have processes directly attached to a cgroup that also has child cgroups and
+vice versa. A cgroup is either an inner node or a leaf node of the tree, and if
+it's an inner node it may not contain processes directly, and if it's a leaf
+node then it may not have child cgroups. (Note that there are some minor
+exceptions to this rule, though. E.g. the root cgroup is special and allows
+both processes and children — which is used in particular to maintain kernel
+threads.)
+
+2. The **single-writer** rule: this means that each cgroup only has a single
+writer, i.e. a single process managing it. It's OK if different cgroups have
+different processes managing them. However, only a single process should own a
+specific cgroup, and when it does that ownership is exclusive, and nothing else
+should manipulate it at the same time. This rule ensures that various pieces of
+software don't step on each other's toes constantly.
+
+These two rules have various effects. For example, one corollary of this is: if
+your container manager creates and manages cgroups in the system's root cgroup
+you violate rule #2, as the root cgroup is managed by systemd and hence off
+limits to everybody else.
+
+Note that rule #1 is generally enforced by the kernel if cgroupsv2 is used: as
+soon as you add a process to a cgroup it is ensured the rule is not
+violated. On cgroupsv1 this rule didn't exist, and hence isn't enforced, even
+though it's a good thing to follow it then too. Rule #2 is not enforced on
+either cgroupsv1 nor cgroupsv2 (this is UNIX after all, in the general case
+root can do anything, modulo SELinux and friends), but if you ignore it you'll
+be in constant pain as various pieces of software will fight over cgroup
+ownership.
+
+Note that cgroupsv1 is currently the most deployed implementation, even though
+it's semantically broken in many ways, and in many cases doesn't actually do
+what people think it does. cgroupsv2 is where things are going, and most new
+kernel features in this area are only added to cgroupsv2, and not cgroupsv1
+anymore. For example cgroupsv2 provides proper cgroup-empty notifications, has
+support for all kinds of per-cgroup BPF magic, supports secure delegation of
+cgroup trees to less privileged processes and so on, which all are not
+available on cgroupsv1.
+
+## Three Different Tree Setups 🌳
+
+systemd supports three different modes how cgroups are set up. Specifically:
+
+1. **Unified** — this is the simplest mode, and exposes a pure cgroupsv2
+logic. In this mode `/sys/fs/cgroup` is the only mounted cgroup API file system
+and all available controllers are exclusively exposed through it.
+
+2. **Legacy** — this is the traditional cgroupsv1 mode. In this mode the
+various controllers each get their own cgroup file system mounted to
+`/sys/fs/cgroup/<controller>/`. On top of that systemd manages its own cgroup
+hierarchy for managing purposes as `/sys/fs/cgroup/systemd/`.
+
+3. **Hybrid** — this is a hybrid between the unified and legacy mode. It's set
+up mostly like legacy, except that there's also an additional hierarchy
+`/sys/fs/cgroup/unified/` that contains the cgroupsv2 hierarchy. In this mode
+compatibility with cgroupsv1 is retained while some cgroupsv2 features are
+available too. This mode is a stopgap. Don't bother with this too much unless
+you have too much free time.
+
+To say this clearly, legacy and hybrid modes have no future. If you develop
+software today and don't focus on the unified mode, then you are writing
+software for yesterday, not tomorrow. They are primarily supported for
+compatibility reasons and will not receive new features. Sorry.
+
+Superficially, in legacy and hybrid modes it might appear that the parallel
+cgroup hierarchies for each controller are orthogonal from each other. In
+systemd they are not: the hierarchies of all controllers are always kept in
+sync (at least mostly: sub-trees might be suppressed in certain hierarchies if
+no controller usage is required for them). The fact that systemd keeps these
+hierarchies in sync means that the legacy and hybrid hierarchies are
+conceptually very close to the unified hierarchy. In particular this allows us
+to talk of one specific cgroup and actually mean the same cgroup in all
+available controller hierarchies. E.g. if we talk about the cgroup `/foo/bar/`
+then we actually mean `/sys/fs/cgroup/cpu/foo/bar/` as well as
+`/sys/fs/cgroup/memory/foo/bar/`, `/sys/fs/cgroup/pids/foo/bar/`, and so on.
+Note that in cgroupsv2 the controller hierarchies aren't orthogonal, hence
+thinking about them as orthogonal won't help you in the long run anyway.
+
+If you wonder how to detect which of these three modes is currently used, use
+`statfs()` on `/sys/fs/cgroup/`. If it reports `CGROUP2_SUPER_MAGIC` in its
+`.f_type` field, then you are in unified mode. If it reports `TMPFS_MAGIC` then
+you are either in legacy or hybrid mode. To distuingish these two cases, run
+`statfs()` again on `/sys/fs/cgroup/unified/`. If that succeeds and reports
+`CGROUP2_SUPER_MAGIC` you are in hybrid mode, otherwise not.
+
+## systemd's Unit Types
+
+The low-level kernel cgroups feature is exposed in systemd in three different
+"unit" types. Specifically:
+
+1. 💼 The `.service` unit type. This unit type is for units encapsulating
+   processes systemd itself starts. Units of these types have cgroups that are
+   the leaves of the cgroup tree the systemd instance manages (though possibly
+   they might contain a sub-tree of their own managed by something else, made
+   possible by the concept of delegation, see below). Service units are usually
+   instantiated based on a unit file on disk that describes the command line to
+   invoke and other properties of the service. However, service units may also
+   be declared and started programmatically at runtime through a D-Bus API
+   (which is called *transient* services).
+
+2. 👓 The `.scope` unit type. This is very similar to `.service`. The main
+   difference: the processes the units of this type encapsulate are forked off
+   by some unrelated manager process, and that manager asked systemd to expose
+   them as a unit. Unlike services, scopes can only be declared and started
+   programmatically, i.e. are always transient. That's because they encapsulate
+   processes forked off by something else, i.e. existing runtime objects, and
+   hence cannot really be defined fully in 'offline' concepts such as unit
+   files.
+
+3. 🔪 The `.slice` unit type. Units of this type do not directly contain any
+   processes. Units of this type are the inner nodes of part of the cgroup tree
+   the systemd instance manages. Much like services, slices can be defined
+   either on disk with unit files or programmatically as transient units.
+
+Slices expose the trunk and branches of a tree, and scopes and services are
+attached to those branches as leaves. The idea is that scopes and services can
+be moved around though, i.e. assigned to a different slice if needed.
+
+The naming of slice units directly maps to the cgroup tree path. This is not
+the case for service and scope units however. A slice named `foo-bar-baz.slice`
+maps to a cgroup `/foo.slice/foo-bar.slice/foo-bar-baz.slice/`. A service
+`quux.service` which is attached to the slice `foo-bar-baz.slice` maps to the
+cgroup `/foo.slice/foo-bar.slice/foo-bar-baz.slice/quux.service/`.
+
+By default systemd sets up four slice units:
+
+1. `-.slice` is the root slice. i.e. the parent of everything else. On the host
+   system it maps directly to the top-level directory of cgroupsv2.
+
+2. `system.slice` is where system services are by default placed, unless
+   configured otherwise.
+
+3. `user.slice` is where user sessions are placed. Each user gets a slice of
+   its own below that.
+
+4. `machines.slice` is where VMs and containers are supposed to be
+   placed. `systemd-nspawn` makes use of this by default, and you're very welcome
+   to place your containers and VMs there too if you hack on managers for those.
+
+Users may define any amount of additional slices they like though, the four
+above are just the defaults.
+
+## Delegation
+
+Container managers and suchlike often want to control cgroups directly using
+the raw kernel APIs. That's entirely fine and supported, as long as proper
+*delegation* is followed. Delegation is a concept we inherited from cgroupsv2,
+but we expose it on cgroupsv1 too. Delegation means that some parts of the
+cgroup tree may be managed by different managers than others. As long as it is
+clear which manager manages which part of the tree each one can do within its
+sub-graph of the tree whatever it wants.
+
+Only sub-trees can be delegated (though whoever decides to request a sub-tree
+can delegate sub-sub-trees further to somebody else if they like). Delegation
+takes place at a specific cgroup: in systemd there's a `Delegate=` property you
+can set for a service or scope unit. If you do, it's the cut-off point for
+systemd's cgroup management: the unit itself is managed by systemd, i.e. all
+its attributes are managed exclusively by systemd, however your program may
+create/remove sub-cgroups inside it freely, and those then become exclusive
+property of your program, systemd won't touch them — all attributes of *those*
+sub-cgroups can be manipulated freely and exclusively by your program.
+
+By turning on the `Delegate=` property for a scope or service you get a few
+guarantees:
+
+1. systemd won't fiddle with your sub-tree of the cgroup tree anymore. It won't
+   change attributes of any cgroups below it, nor will it create or remove any
+   cgroups thereunder, nor migrate processes across the boundaries of that
+   sub-tree as it deems useful anymore.
+
+2. If your service makes use of the `User=` functionality, then the sub-tree
+   will be `chown()`ed to the indicated user so that it can correctly create
+   cgroups below it. Note however that systemd will do that only in the unified
+   hierarchy (in unified and hybrid mode) as well as on systemd's own private
+   hierarchy (in legacy and hybrid mode). It won't pass ownership of the legacy
+   controller hierarchies. Delegation to less privileges processes is not safe
+   in cgroupsv1 (as a limitation of the kernel), hence systemd won't facilitate
+   access to it.
+
+3. Any BPF IP filter programs systemd installs will be installed with
+   `BPF_F_ALLOW_MULTI` so that your program can install additional ones.
+
+In unit files the `Delegate=` property is superficially exposed as
+boolean. However, since v236 it optionally takes a list of controller names
+instead. If so, delegation is requested for listed controllers
+specifically. Note hat this only encodes a request. Depending on various
+parameters it might happen that your service actually will get fewer
+controllers delegated (for example, because the controller is not available on
+the current kernel or was turned off) or more.  If no list is specified
+(i.e. the property simply set to `yes`) then all available controllers are
+delegated.
+
+Let's stress one thing: delegation is available on scope and service units
+only. It's expressly not available on slice units. Why? Because slice units are
+our *inner* nodes of the cgroup trees and we freely attach service and scopes
+to them. If we'd allow delegation on slice units then this would mean that
+both systemd and your own manager would create/delete cgroups below the slice
+unit and that conflicts with the single-writer rule.
+
+So, if you want to do your own raw cgroups kernel level access, then allocate a
+scope unit, or a service unit (or just use the service unit you already have
+for your service code), and turn on delegation for it.
+
+## Three Scenarios
+
+Let's say you write a container manager, and you wonder what to do regarding
+cgroups for it, as you want your manager to be able to run on systemd systems.
+
+You basically have three options:
+
+1. 😊 The *integration-is-good* option. For this, you register each container
+   you have either as a systemd service (i.e. let systemd invoke the executor
+   binary for you) or a systemd scope (i.e. your manager executes the binary
+   directly, but then tells systemd about it. In this mode the administrator
+   can use the usual systemd resource management and reporting commands
+   individually on those containers. By turning on `Delegate=` for these scopes
+   or services you make it possible to run cgroup-enabled programs in your
+   containers, for example a nested systemd instance. This option has two
+   sub-options:
+
+   a. You transiently register the service or scope by directly contacting
+      systemd via D-Bus. In this case systemd will just manage the unit for you
+      and nothing else.
+
+   b. Instead you register the service or scope through `systemd-machined`
+      (also via D-Bus). This mini-daemon is basically just a proxy for the same
+      operations as in a. The main benefit of this: this way you let the system
+      know that what you are registering is a container, and this opens up
+      certain additional integration points. For example, `journalctl -M` can
+      then be used to directly look into any container's journal logs (should
+      the container run systemd inside), or `systemctl -M` can be used to
+      directly invoke systemd operations inside the containers. Moreover tools
+      like "ps" can then show you to which container a process belongs (`ps -eo
+      pid,comm,machine`), and even gnome-system-monitor supports it.
+
+2. 🙁 The *i-like-islands* option. If all you care about is your own cgroup tree,
+   and you want to have to do as little as possible with systemd and no
+   interest in integration with the rest of the system, then this is a valid
+   option. For this all you have to do is turn on `Delegate=` for your main
+   manager daemon. Then figure out the cgroup systemd placed your daemon in:
+   you can now freely create sub-cgroups beneath it. Don't forget the
+   *no-processes-in-inner-nodes* rule however: you have to move your main
+   daemon process out of that cgroup (and into a sub-cgroup) before you can
+   start further processes in any of your sub-cgroups.
+
+3. 🙁 The *i-like-continents* option. In this option you'd leave your manager
+   daemon where it is, and would not turn on delegation on its unit. However,
+   as first thing you register a new scope unit with systemd, and that scope
+   unit would have `Delegate=` turned on, and then you place all your
+   containers underneath it. From systemd's PoV there'd be two units: your
+   manager service and the big scope that contains all your containers in one.
+
+BTW: if for whatever reason you say "I hate D-Bus, I'll never call any D-Bus
+API, kthxbye", then options #1 and #3 are not available, as they generally
+involve talking to systemd from your program code, via D-Bus. You still have
+option #2 in that case however, as you can simply set `Delegate=` in your
+service's unit file and you are done and have your own sub-tree. In fact, #2 is
+the one option that allows you to completely ignore systemd's existence: you
+can entirely generically follow the single rule that you just use the cgroup
+you are started in, and everything below it, whatever that might be. That said,
+maybe if you dislike D-Bus and systemd that much, the better approach might be
+to work on that, and widen your horizon a bit. You are welcome.
+
+## Controller Support
+
+systemd supports a number of controllers (but not all). Specifically, supported
+are:
+
+* on cgroupsv1: `cpu`, `cpuacct`, `blkio`, `memory`, `devices`, `pids`
+* on cgroupsv2: `cpu`, `io`, `memory`, `pids`
+
+It is our intention to natively support all cgroupsv2 controllers as they are
+added to the kernel. However, regarding cgroupsv1: at this point we will not
+add support for any other controllers anymore. This means systemd currently
+does not and will never manage the following controllers on cgroupsv1:
+`freezer`, `cpuset`, `net_cls`, `perf_event`, `net_prio`, `hugetlb`. Why not?
+Depending on the case, either their API semantics or implementations aren't
+really usable, or it's very clear they have no future on cgroupsv2, and we
+won't add new code for stuff that clearly has no future.
+
+Effectively this means that all those mentioned cgroupsv1 controllers are up
+for grabs: systemd won't manage them, and hence won't delegate them to your
+code (however, systemd will still mount their hierarchies, simply because it
+mounts all controller hierarchies it finds available in the kernel). If you
+decide to use them, then that's fine, but systemd won't help you with it (but
+also not interfere with it). To be nice to other tenants it might be wise to
+replicate the cgroup hierarchies of the other controllers in them too however,
+but of course that's between you and those other tenants, and systemd won't
+care. Replicating the cgroup hierarchies in those unsupported controllers would
+mean replicating the full cgroup paths in them, and hence the prefixing
+`.slice` components too, otherwise the hierarchies will start being orthogonal
+after all, and that's not really desirable. On more thing: systemd will clean
+up after you in the hierarchies it manages: if your daemon goes down, its
+cgroups will be removed too. You basically get the guarantee that you start
+with a pristine cgroup sub-tree for your service or scope whenever it is
+started. This is not the case however in the hierarchies systemd doesn't
+manage. This means that your programs should be ready to deal with left-over
+cgroups in them — from previous runs, and be extra careful with them as they
+might still carry settings that might not be valid anymore.
+
+Note a particular asymmetry here: if your systemd version doesn't support a
+specific controller on cgroupsv1 you can still make use of it for delegation,
+by directly fiddling with its hierarchy and replicating the cgroup tree there
+as necessary (as suggested above). However, on cgroupsv2 this is different:
+separately mounted hierarchies are not available, and delegation has always to
+happen through systemd itself. This means: when you update your kernel and it
+adds a new, so far unseen controller, and you want to use it for delegation,
+then you also need to update systemd to a version that groks it.
+
+## systemd as Container Payload
+
+systemd can happily run as a container payload's PID 1. Note that systemd
+unconditionally needs write access to the cgroup tree however, hence you need
+to delegate a sub-tree to it. Note that there's nothing too special you have to
+do beyond that: just invoke systemd as PID 1 inside the root of the delegated
+cgroup sub-tree, and it will figure out the rest: it will determine the cgroup
+it is running in and take possession of it. It won't interfere with any cgroup
+outside of the sub-tree it was invoked in. Use of `CLONE_NEWCGROUP` is hence
+optional (but of course wise).
+
+Note one particular asymmetry here though: systemd will try to take possession
+of the root cgroup you pass to it *in* *full*, i.e. it will not only
+create/remove child cgroups below it, it will also attempt to manage the
+attributes of it. OTOH as mentioned above, when delegating a cgroup tree to
+somebody else it only passes the rights to create/remove sub-cgroups, but will
+insist on managing the delegated cgroup tree's top-level attributes. Or in
+other words: systemd is *greedy* when accepting delegated cgroup trees and also
+*greedy* when delegating them to others: it insists on managing attributes on
+the specific cgroup in both cases. A container manager that is itself a payload
+of a host systemd which wants to run a systemd as its own container payload
+instead hence needs to insert an extra level in the hierarchy in between, so
+that the systemd on the host and the one in the container won't fight for the
+attributes. That said, you likely should do that anyway, due to the
+no-processes-in-inner-cgroups rule, see below.
+
+When systemd runs as container payload it will make use of all hierarchies it
+has write access to. For legacy mode you need to make at least
+`/sys/fs/cgroup/systemd/` available, all other hierarchies are optional. For
+hybrid mode you need to add `/sys/fs/cgroup/unified/`. Finally, for fully
+unified you (of course, I guess) need to provide only `/sys/fs/cgroup/` itself.
+
+## Some Dos
+
+1. ⚡ If you go for implementation option 1a or 1b (as in the list above), then
+   each of your containers will have its own systemd-managed unit and hence
+   cgroup with possibly further sub-cgroups below. Typically the first process
+   running in that unit will be some kind of executor program, which will in
+   turn fork off the payload processes of the container. In this case don't
+   forget that there are two levels of delegation involved: first, systemd
+   delegates a group sub-tree to your executor. And then your executor should
+   delegate a sub-tree further down to the container payload. Oh, and because
+   of the no-process-in-inner-nodes rule, your executor needs to migrate itself
+   to a sub-cgroup of the cgroup it got delegated, too. Most likely you hence
+   want a two-pronged approach: below the cgroup you got started in, you want
+   one cgroup maybe called `supervisor/` where your manager runs in and then
+   for each container a sibling cgroup of that maybe called `payload-xyz/`.
+
+2. ⚡ Don't forget that the cgroups you create have to have names that are
+   suitable as UNIX file names, and that they live in the same namespace as the
+   various kernel attribute files. Hence, when you want to allow the user
+   arbitrary naming, you might need to escape some of the names (for example,
+   you really don't want to create a cgroup named `tasks`, just because the
+   user created a container by that name, because `tasks` after all is a magic
+   attribute in cgroupsv1, and your `mkdir()` will hence fail with `EEXIST`. In
+   systemd we do escaping by prefixing names that might collide with a kernel
+   attribute name with an underscore. You might want to do the same, but this
+   is really up to you how you do it. Just do it, and be careful.
+
+## Some Don'ts
+
+1. 🚫 Never create your own cgroups below arbitrary cgroups systemd manages, i.e
+   cgroups you haven't set `Delegate=` in. Specifically: 🔥 don't create your
+   own cgroups below the root cgroup 🔥. That's owned by systemd, and you will
+   step on systemd's toes if you ignore that, and systemd will step on
+   yours. Get your own delegated sub-tree, you may create as many cgroups there
+   as you like. Seriously, if you create cgroups directly in the cgroup root,
+   then all you do is ask for trouble.
+
+2. 🚫 Don't attempt to set `Delegate=` in slice units, and in particular not in
+   `-.slice`. It's not supported, and will generate an error.
+
+3. 🚫 Never *write* to any of the attributes of a cgroup systemd created for
+   you. It's systemd's private property. You are welcome to manipulate the
+   attributes of cgroups you created in your own delegated sub-tree, but the
+   cgroup tree of systemd itself is out of limits for you. It's fine to *read*
+   from any attribute you like however. That's totally OK and welcome.
+
+4. 🚫 When not using `CLONE_NEWCGROUP` when delegating a sub-tree to a
+   container payload running systemd, then don't get the idea that you can bind
+   mount only a sub-tree of the host's cgroup tree into the container. Part of
+   the cgroup API is that `/proc/$PID/cgroup` reports the cgroup path of every
+   process, and hence any path below `/sys/fs/cgroup/` needs to match what
+   `/proc/$PID/cgroup` of the payload processes reports. What you can do safely
+   however, is mount the upper parts of the cgroup tree read-only (or even
+   replace the middle bits with an intermediary `tmpfs` — but be careful not to
+   break the `statfs()` detection logic discussed above), as long as the path
+   to the delegated sub-tree remains accessible as-is.
+
+5. ⚡ Currently, the algorithm for mapping between slice/scope/service unit
+   naming and their cgroup paths is not considered public API of systemd, and
+   may change in future versions. This means: it's best to avoid implementing a
+   local logic of translating cgroup paths to slice/scope/service names in your
+   program, or vice versa — it's likely going to break sooner or later. Use the
+   appropriate D-Bus API calls for that instead, so that systemd translates
+   this for you. (Specifically: each Unit object has a `ControlGroup` property
+   to get the cgroup for a unit. The method `GetUnitByControlGroup()` may be
+   used to get the unit for a cgroup.)
+
+6. ⚡ Think twice before delegating cgroupsv1 controllers to less privileged
+   containers. It's not safe, you basically allow your containers to freeze the
+   system with that and worse. Delegation is a strongpoint of cgroupsv2 though,
+   and there it's safe to treat delegation boundaries as privilege boundaries.
+
+And that's it for now. If you have further questions, refer to the systemd
+mailing list.
+
+— Berlin, 2018-04-20