mkosi: Switch to use mkosi presets with prebuilt initrds

Instead of building the initrds for the mkosi images with dracut,
let's switch to using mkosi presets to build the initrd with mkosi
as well.

This commit splits up our single image build into three separate
mkosi presets:

1. The "base" preset. This image contains systemd and all its runtime
dependencies. The sole purpose of this image is to serve as a base image
for the initrd and the final image. It's also responsible for building
systemd from source with the build script. The results are installed into
the base image. Note that we install the systemd and udev packages into this
image as well to prevent package managers from overriding the systemd we built
from source with the distro packaged systemd if it's pulled in as a dependency
by another package from the initrd or final profiles.
2. The "initrd" preset. This image provides the initrd. It's trivial and does
nothing more than packaging the base image up as a zstd compressed initramfs and
adds /init and /etc/initrd-release symlinks to the image.
3. The "final" preset. This image builds on top of the base image and adds
a kernel and extra packages that are useful for testing and debugging.

We also split out the optional kernel build into a separate set of config files
that are only included if a kernel to build is actually provided.

Note that this commit doesn't really change anything about how mkosi is used.
The commands remain the same, except that mkosi will now build all the presets
in order. "mkosi summary" will show the summary of all the presets. "mkosi qemu,
boot, shell" will always boot the final preset. With "-f", all presets will be
built and the final one is booted. "-i" makes a cache of each preset.

The only thing to keep in mind is that specifying config via the mkosi CLI will
apply to each of the presets. e.g. any extra packages added with "-p" will be
installed in both the initrd and the final image. To apply local configuration
to a single preset, create a file 00-local.conf in
mkosi.presets/<profile>/mkosi.conf.d and put all the preset specific configuration
in there.

15 files changed, 554 insertions, 0 deletions

diff --git a/mkosi.presets/00-base/mkosi.build b/mkosi.presets/00-base/mkosi.build
new file mode 100755
index 0000000000..e19251157d
--- /dev/null
+++ b/mkosi.presets/00-base/mkosi.build
@@ -0,0 +1,164 @@
+#!/bin/sh
+# SPDX-License-Identifier: LGPL-2.1-or-later
+set -e
+
+# This is a build script for OS image generation using mkosi (https://github.com/systemd/mkosi).
+# Simply invoke "mkosi" in the project directory to build an OS image.
+
+# If mkosi.builddir/ exists mkosi will set $BUILDDIR to it, let's then use it
+# as out-of-tree build dir. Otherwise, let's make up our own builddir.
+[ -z "$BUILDDIR" ] && BUILDDIR="$PWD"/build
+
+# Let's make sure we're using stuff from the build directory first if available there.
+PATH="$BUILDDIR:$PATH"
+export PATH
+
+# The bpftool script shipped by Ubuntu tries to find the actual program to run via querying `uname -r` and
+# using the current kernel version. This obviously doesn't work in containers. As a workaround, we override
+# the ubuntu script with a symlink to the first bpftool program we can find.
+for bpftool in /usr/lib/linux-tools/*/bpftool; do
+    [ -x "$bpftool" ] || continue
+    ln -sf "$bpftool" "$BUILDDIR"/bpftool
+    break
+done
+
+# CentOS Stream 8 includes bpftool 4.18.0 which is lower than what we need. However, they've backported the
+# specific feature we need ("gen skeleton") to this version, so we replace bpftool with a script that reports
+# version 5.6.0 to satisfy meson which makes bpf work on CentOS Stream 8 as well.
+if [ "$(grep '^ID=' /etc/os-release)" = "ID=\"centos\"" ] && [ "$(grep '^VERSION=' /etc/os-release)" = "VERSION=\"8\"" ]; then
+    cat >"$BUILDDIR"/bpftool <<EOF
+#!/bin/sh
+if [ "\$1" = --version ]; then
+    echo 5.6.0
+else
+    exec /usr/sbin/bpftool \$@
+fi
+EOF
+    chmod +x "$BUILDDIR"/bpftool
+fi
+
+if [ ! -f "$BUILDDIR"/build.ninja ] ; then
+    sysvinit_path=$(realpath /etc/init.d)
+
+    init_path=$(realpath /sbin/init 2>/dev/null)
+    if [ -z "$init_path" ] ; then
+        rootprefix=""
+    else
+        rootprefix=${init_path%/lib/systemd/systemd}
+        rootprefix=/${rootprefix#/}
+    fi
+
+    # On debian-like systems the library directory is not /usr/lib64 but /usr/lib/<arch-triplet>/.
+    # It is important to use the right one especially for cryptsetup plugins, otherwise they will be
+    # installed in the wrong directory and not be found by cryptsetup. Assume native build.
+    if grep -q -e "ID=debian" -e "ID_LIKE=debian" /etc/os-release && command -v dpkg 2>/dev/null; then
+        LIBDIR="-Drootlibdir=/usr/lib/$(dpkg-architecture -qDEB_HOST_MULTIARCH)"
+        PAMDIR="-Dpamlibdir=/usr/lib/$(dpkg-architecture -qDEB_HOST_MULTIARCH)/security"
+    fi
+
+    # Cannot quote $LIBDIR and $PAMDIR, because they may be empty, and meson will fail.
+    # shellcheck disable=SC2086
+    meson setup "$BUILDDIR" \
+            ${LIBDIR:-} \
+            ${PAMDIR:-} \
+            -D "sysvinit-path=$sysvinit_path" \
+            -D "rootprefix=$rootprefix" \
+            -D default-timeout-sec=${DEFAULT_TIMEOUT_SEC:-10} \
+            -D default-user-timeout-sec=${DEFAULT_TIMEOUT_SEC:-10} \
+            -D man=false \
+            -D translations=false \
+            -D version-tag="${VERSION_TAG}" \
+            -D mode=developer \
+            -D b_sanitize="${SANITIZERS:-none}" \
+            -D install-tests=true \
+            -D tests=unsafe \
+            -D slow-tests=true \
+            -D utmp=true \
+            -D hibernate=true \
+            -D ldconfig=true \
+            -D resolve=true \
+            -D efi=true \
+            -D tpm=true \
+            -D environment-d=true \
+            -D binfmt=true \
+            -D repart=true \
+            -D sysupdate=true \
+            -D coredump=true \
+            -D pstore=true \
+            -D oomd=true \
+            -D logind=true \
+            -D hostnamed=true \
+            -D localed=true \
+            -D machined=true \
+            -D portabled=true \
+            -D sysext=true \
+            -D userdb=true \
+            -D homed=true \
+            -D networkd=true \
+            -D timedated=true \
+            -D timesyncd=true \
+            -D remote=true \
+            -D nss-myhostname=true \
+            -D nss-mymachines=true \
+            -D nss-resolve=true \
+            -D nss-systemd=true \
+            -D firstboot=true \
+            -D randomseed=true \
+            -D backlight=true \
+            -D vconsole=true \
+            -D quotacheck=true \
+            -D sysusers=true \
+            -D tmpfiles=true \
+            -D importd=true \
+            -D hwdb=true \
+            -D rfkill=true \
+            -D xdg-autostart=true \
+            -D translations=true \
+            -D polkit=true \
+            -D acl=true \
+            -D audit=true \
+            -D blkid=true \
+            -D fdisk=true \
+            -D kmod=true  \
+            -D pam=true \
+            -D pwquality=true \
+            -D microhttpd=true \
+            -D libcryptsetup=true \
+            -D libcurl=true \
+            -D idn=true \
+            -D libidn2=true \
+            -D qrencode=true \
+            -D gcrypt=true \
+            -D gnutls=true \
+            -D openssl=true \
+            -D cryptolib=openssl \
+            -D p11kit=true \
+            -D libfido2=true \
+            -D tpm2=true \
+            -D elfutils=true \
+            -D zstd=true \
+            -D xkbcommon=true \
+            -D pcre2=true \
+            -D glib=true \
+            -D dbus=true \
+            -D bootloader=true \
+            -D kernel-install=true \
+            -D analyze=true \
+            -D bpf-framework=true \
+            -D ukify=true
+fi
+
+ninja -C "$BUILDDIR" "$@"
+if [ "$WITH_TESTS" = 1 ] ; then
+    if [ -n "$SANITIZERS" ]; then
+        export ASAN_OPTIONS="$MKOSI_ASAN_OPTIONS"
+        export UBSAN_OPTIONS="$MKOSI_UBSAN_OPTIONS"
+        TIMEOUT_MULTIPLIER=3
+    else
+        TIMEOUT_MULTIPLIER=1
+    fi
+
+    meson test -C "$BUILDDIR" --print-errorlogs --timeout-multiplier=$TIMEOUT_MULTIPLIER
+fi
+
+meson install -C "$BUILDDIR" --quiet --no-rebuild --only-changed
diff --git a/mkosi.presets/00-base/mkosi.conf b/mkosi.presets/00-base/mkosi.conf
new file mode 100644
index 0000000000..a65d427067
--- /dev/null
+++ b/mkosi.presets/00-base/mkosi.conf
@@ -0,0 +1,34 @@
+# SPDX-License-Identifier: LGPL-2.1-or-later
+
+[Output]
+Format=directory
+
+[Content]
+Bootable=no
+BuildSources=../..
+CleanPackageMetadata=no
+Packages=
+        kmod
+        less
+        systemd
+        udev
+
+BuildPackages=
+        acl
+        diffutils
+        gawk
+        binutils
+        clang
+        gettext
+        git
+        gperf
+        grep
+        lld
+        llvm
+        make
+        meson
+        pkgconf
+        rsync
+        sed
+        tar
+        zstd
diff --git a/mkosi.presets/00-base/mkosi.conf.d/10-arch.conf b/mkosi.presets/00-base/mkosi.conf.d/10-arch.conf
new file mode 100644
index 0000000000..473d199718
--- /dev/null
+++ b/mkosi.presets/00-base/mkosi.conf.d/10-arch.conf
@@ -0,0 +1,31 @@
+# SPDX-License-Identifier: LGPL-2.1-or-later
+
+[Match]
+Distribution=arch
+
+[Content]
+Packages=
+        gnutls
+        libbpf
+        libfido2
+        libmicrohttpd
+        libnftnl
+        libpwquality
+        libxkbcommon
+        openssl
+        qrencode
+        systemd-sysvcompat
+        tpm2-tss
+
+BuildPackages=
+        bpf
+        docbook-xsl
+        glib2
+        libxslt
+        linux-api-headers
+        perl
+        python
+        python-jinja
+        python-lxml
+        python-pefile
+        python-pyelftools
diff --git a/mkosi.presets/00-base/mkosi.conf.d/10-centos-fedora.conf b/mkosi.presets/00-base/mkosi.conf.d/10-centos-fedora.conf
new file mode 100644
index 0000000000..090daf60db
--- /dev/null
+++ b/mkosi.presets/00-base/mkosi.conf.d/10-centos-fedora.conf
@@ -0,0 +1,67 @@
+# SPDX-License-Identifier: LGPL-2.1-or-later
+
+[Match]
+Distribution=centos fedora
+
+[Content]
+Packages=
+        audit-libs
+        cryptsetup-libs
+        gnutls
+        libasan
+        libbpf
+        libfido2
+        libgcrypt
+        libmicrohttpd
+        libnftnl
+        libubsan
+        libxcrypt
+        libxkbcommon
+        openssl-libs
+        qrencode-libs
+        tpm2-tss
+        util-linux
+
+BuildPackages=
+        /usr/bin/pkg-config
+        bpftool
+        docbook-xsl
+        findutils
+        libxslt
+        pam-devel
+        perl-interpreter
+        pkgconfig(audit)
+        pkgconfig(blkid)
+        pkgconfig(bzip2)
+        pkgconfig(dbus-1)
+        pkgconfig(fdisk)
+        pkgconfig(glib-2.0)
+        pkgconfig(gnutls)
+        pkgconfig(libacl)
+        pkgconfig(libbpf)
+        pkgconfig(libcap)
+        pkgconfig(libcryptsetup)
+        pkgconfig(libcurl)
+        pkgconfig(libdw)
+        pkgconfig(libfido2)
+        pkgconfig(libidn2)
+        pkgconfig(libkmod)
+        pkgconfig(libmicrohttpd)
+        pkgconfig(libnftnl)
+        pkgconfig(libpcre2-8)
+        pkgconfig(libqrencode)
+        pkgconfig(libseccomp)
+        pkgconfig(libselinux)
+        pkgconfig(libzstd)
+        pkgconfig(mount)
+        pkgconfig(numa)
+        pkgconfig(openssl)
+        pkgconfig(openssl)
+        pkgconfig(p11-kit-1)
+        pkgconfig(pwquality)
+        pkgconfig(tss2-esys)
+        pkgconfig(tss2-mu)
+        pkgconfig(tss2-rc)
+        pkgconfig(valgrind)
+        pkgconfig(xkbcommon)
+        rpm
diff --git a/mkosi.presets/00-base/mkosi.conf.d/10-debian-ubuntu.conf b/mkosi.presets/00-base/mkosi.conf.d/10-debian-ubuntu.conf
new file mode 100644
index 0000000000..f5c3afbef4
--- /dev/null
+++ b/mkosi.presets/00-base/mkosi.conf.d/10-debian-ubuntu.conf
@@ -0,0 +1,66 @@
+# SPDX-License-Identifier: LGPL-2.1-or-later
+
+[Match]
+Distribution=debian ubuntu
+
+[Content]
+Packages=
+        libfdisk1
+        libfido2-1
+        libglib2.0-0
+        libgnutls30
+        libidn2-0
+        libmicrohttpd12
+        libnftnl11
+        libp11-kit0
+        libpam0g
+        libpwquality1
+        libqrencode4
+        libssl3
+        libtss2-dev # Use the -dev package to avoid churn in updating version numbers
+        systemd
+        systemd-sysv
+        tzdata
+
+BuildPackages=
+        docbook-xsl
+        dpkg-dev
+        g++
+        libacl1-dev
+        libaudit-dev
+        libblkid-dev
+        libbpf-dev
+        libbz2-dev
+        libcap-dev
+        libcryptsetup-dev
+        libcurl4-openssl-dev
+        libdbus-1-dev
+        libdw-dev
+        libfdisk-dev
+        libfido2-dev
+        libgcrypt20-dev
+        libglib2.0-dev
+        libgnutls28-dev
+        libidn2-dev
+        libiptc-dev
+        libkmod-dev
+        libmicrohttpd-dev
+        libmount-dev
+        libnftnl-dev
+        libp11-kit-dev
+        libpam0g-dev
+        libpwquality-dev
+        libqrencode-dev
+        libseccomp-dev
+        libsmartcols-dev
+        libssl-dev
+        libxen-dev
+        libxkbcommon-dev
+        libzstd-dev
+        python3
+        python3-jinja2
+        python3-lxml
+        python3-pefile
+        python3-pyelftools
+        python3-pytest
+        xsltproc
diff --git a/mkosi.presets/00-base/mkosi.conf.d/10-debian.conf b/mkosi.presets/00-base/mkosi.conf.d/10-debian.conf
new file mode 100644
index 0000000000..020b02b61c
--- /dev/null
+++ b/mkosi.presets/00-base/mkosi.conf.d/10-debian.conf
@@ -0,0 +1,11 @@
+# SPDX-License-Identifier: LGPL-2.1-or-later
+
+[Match]
+Distribution=debian
+
+[Content]
+Packages=
+        libbpf1
+
+BuildPackages=
+        bpftool
diff --git a/mkosi.presets/00-base/mkosi.conf.d/10-fedora.conf b/mkosi.presets/00-base/mkosi.conf.d/10-fedora.conf
new file mode 100644
index 0000000000..035715979c
--- /dev/null
+++ b/mkosi.presets/00-base/mkosi.conf.d/10-fedora.conf
@@ -0,0 +1,15 @@
+# SPDX-License-Identifier: LGPL-2.1-or-later
+
+[Match]
+Distribution=fedora
+
+[Content]
+BuildPackages=
+        pkgconfig(libgcrypt)
+        pkgconfig(xencontrol)
+        python3
+        python3dist(jinja2)
+        python3dist(lxml)
+        python3dist(pefile)
+        python3dist(pyelftools)
+        python3dist(pytest)
diff --git a/mkosi.presets/00-base/mkosi.conf.d/10-opensuse.conf b/mkosi.presets/00-base/mkosi.conf.d/10-opensuse.conf
new file mode 100644
index 0000000000..b8bce7148e
--- /dev/null
+++ b/mkosi.presets/00-base/mkosi.conf.d/10-opensuse.conf
@@ -0,0 +1,81 @@
+# SPDX-License-Identifier: LGPL-2.1-or-later
+
+[Match]
+Distribution=opensuse
+
+[Content]
+# We install gawk, gzip, grep, xz here explicitly so that the busybox versions don't get installed instead.
+Packages=
+        gawk
+        grep
+        gzip
+        libbpf1
+        libcrypt1
+        libcryptsetup12
+        libdw1
+        libelf1
+        libfido2
+        libgcrypt20
+        libglib-2_0-0
+        libkmod2
+        libmount1
+        libnftnl11
+        libopenssl3
+        libp11-kit0
+        libqrencode4
+        libseccomp2
+        libxkbcommon0
+        libzstd1
+        pam
+        shadow
+        tpm2-0-tss
+        xz
+
+BuildPackages=
+        audit-devel
+        bpftool
+        dbus-1-devel
+        docbook-xsl-stylesheets
+        fdupes
+        gcc-c++
+        glib2-devel
+        glibc-locale
+        intltool
+        libacl-devel
+        libapparmor-devel
+        libblkid-devel
+        libbpf-devel
+        libcap-devel
+        libcryptsetup-devel
+        libcurl-devel
+        libdw-devel
+        libelf-devel
+        libfdisk-devel
+        libfido2-devel
+        libgcrypt-devel
+        libgnutls-devel
+        libkmod-devel
+        libmicrohttpd-devel
+        libmount-devel
+        libnftnl-devel
+        libpwquality-devel
+        libseccomp-devel
+        libselinux-devel
+        libxkbcommon-devel
+        libxslt-tools
+        libzstd-devel
+        openssl-devel
+        pam-devel
+        pciutils-devel
+        python3
+        python3-Jinja2
+        python3-lxml
+        python3-pefile
+        python3-pyelftools
+        python3-pytest
+        qrencode-devel
+        shadow
+        systemd-sysvinit
+        timezone
+        tpm2-0-tss-devel
+        xen-devel
diff --git a/mkosi.presets/00-base/mkosi.conf.d/10-ubuntu.conf b/mkosi.presets/00-base/mkosi.conf.d/10-ubuntu.conf
new file mode 100644
index 0000000000..717809fd03
--- /dev/null
+++ b/mkosi.presets/00-base/mkosi.conf.d/10-ubuntu.conf
@@ -0,0 +1,12 @@
+# SPDX-License-Identifier: LGPL-2.1-or-later
+
+[Match]
+Distribution=ubuntu
+
+[Content]
+Packages=
+        libbpf0
+
+BuildPackages=
+        linux-tools-common
+        linux-tools-generic
diff --git a/mkosi.presets/00-base/mkosi.conf.d/11-centos-8.conf b/mkosi.presets/00-base/mkosi.conf.d/11-centos-8.conf
new file mode 100644
index 0000000000..ef001c92c9
--- /dev/null
+++ b/mkosi.presets/00-base/mkosi.conf.d/11-centos-8.conf
@@ -0,0 +1,16 @@
+# SPDX-License-Identifier: LGPL-2.1-or-later
+
+[Match]
+Distribution=centos
+Release=8
+
+[Content]
+BuildPackages=
+        libgcrypt-devel # CentOS Stream 8 libgcrypt-devel doesn't ship a pkg-config file.
+        platform-python
+        python3.9dist(jinja2)
+        python3.9dist(lxml)
+        python3.9dist(pefile)
+        python3.9dist(pyelftools)
+        python3.9dist(pytest)
+        python39
diff --git a/mkosi.presets/00-base/mkosi.conf.d/11-centos-9.conf b/mkosi.presets/00-base/mkosi.conf.d/11-centos-9.conf
new file mode 100644
index 0000000000..61c1d27f44
--- /dev/null
+++ b/mkosi.presets/00-base/mkosi.conf.d/11-centos-9.conf
@@ -0,0 +1,15 @@
+# SPDX-License-Identifier: LGPL-2.1-or-later
+
+[Match]
+Distribution=centos
+Release=9
+
+[Content]
+BuildPackages=
+        pkgconfig(libgcrypt)
+        platform-python
+        python3dist(jinja2)
+        python3dist(lxml)
+        python3dist(pefile)
+        python3dist(pyelftools)
+        python3dist(pytest)
diff --git a/mkosi.presets/00-base/mkosi.extra/usr/lib/systemd/system-preset/00-mkosi.preset b/mkosi.presets/00-base/mkosi.extra/usr/lib/systemd/system-preset/00-mkosi.preset
new file mode 100644
index 0000000000..64baf3ba6e
--- /dev/null
+++ b/mkosi.presets/00-base/mkosi.extra/usr/lib/systemd/system-preset/00-mkosi.preset
@@ -0,0 +1,19 @@
+# SPDX-License-Identifier: LGPL-2.1-or-later
+
+# mkosi adds its own ssh units via the --ssh switch so disable the default ones.
+disable ssh.service
+disable sshd.service
+
+# These are started manually in integration tests so don't start them by default.
+disable dnsmasq.service
+disable isc-dhcp-server.service
+disable isc-dhcp-server6.service
+
+# Pulled in via dracut-network by kexec-tools on Fedora.
+disable NetworkManager.service
+
+# Make sure dbus-broker is started by default on Debian/Ubuntu.
+enable dbus-broker.service
+
+# systemd-networkd is disabled by default on Fedora so make sure it is enabled.
+enable systemd-networkd.service
diff --git a/mkosi.presets/00-base/mkosi.extra/usr/lib/systemd/system-preset/99-mkosi.preset b/mkosi.presets/00-base/mkosi.extra/usr/lib/systemd/system-preset/99-mkosi.preset
new file mode 100644
index 0000000000..710ee7c6f9
--- /dev/null
+++ b/mkosi.presets/00-base/mkosi.extra/usr/lib/systemd/system-preset/99-mkosi.preset
@@ -0,0 +1,4 @@
+# SPDX-License-Identifier: LGPL-2.1-or-later
+
+# Make sure that services are disabled by default (primarily for Debian/Ubuntu).
+disable *
diff --git a/mkosi.presets/00-base/mkosi.extra/usr/lib/tmpfiles.d/locale.conf b/mkosi.presets/00-base/mkosi.extra/usr/lib/tmpfiles.d/locale.conf
new file mode 100644
index 0000000000..e1a8e8171a
--- /dev/null
+++ b/mkosi.presets/00-base/mkosi.extra/usr/lib/tmpfiles.d/locale.conf
@@ -0,0 +1 @@
+L /etc/default/locale - - - - ../locale.conf
diff --git a/mkosi.presets/00-base/mkosi.prepare b/mkosi.presets/00-base/mkosi.prepare
new file mode 100755
index 0000000000..269ba3e153
--- /dev/null
+++ b/mkosi.presets/00-base/mkosi.prepare
@@ -0,0 +1,18 @@
+#!/bin/sh
+# SPDX-License-Identifier: LGPL-2.1-or-later
+set -e
+
+if [ "$1" = "build" ]; then
+    . /etc/os-release
+
+    if [ "$ID" = "centos" ] && [ "$VERSION" = "8" ]; then
+        alternatives --install /usr/bin/python3 python3 /usr/bin/python3.9 1
+        alternatives --set python3 /usr/bin/python3.9
+    fi
+
+    # Make sure the necessary test users are available in the build image. We do this here because the build
+    # script does not run as root.
+    for id in 1 2 3; do
+        getent group $id >/dev/null || echo "g testgroup$id $id -" | systemd-sysusers -
+    done
+fi