find-esp: don't silently error bootctl install if presumed XBOOTLDR part is stx_dev_major=0 but not btrfs

btrfs_get_block_device_fd() returns -ENOTTY if fstatfs().f_type !=
BTRFS_SUPER_MAGIC

btrfs_get_block_device_fd() is run by verify_fsroot_dir() by
verify_xbootldr() by find_xbootldr_and_warn() if
statx($presumed-XBOOTLDR).stx_dev_major == 0 ("maybe a btrfs device")

Every bootctl verb_install() runs find_xbootldr_and_warn(), by default
with /boot

If your /boot .stx_dev_major=0 but /not/ btrfs, bootctl install/update
quietly exits 1 with no note so as to what exactly failed (debug also
empty, and the strace isn't exactly clear since no syscall actually
failed)

This is the case on ZFS and the Debian filesystem layout: /boot/efi is
the ESP, and everything else under / is ZFS:
  $ sudo env SYSTEMD_LOG_LEVEL=debug bootctl update
  Found cgroup2 on /sys/fs/cgroup/, full unified hierarchy
  Found container virtualization none.
  File system "/boot" is not a FAT EFI System Partition (ESP) file system.
  Using EFI System Partition at /boot/efi.
  Checking whether /boot/efi/EFI/systemd/ contains any files…
  $ echo $?
  1
and funnier still:
  $ sudo bootctl update --graceful
  $ echo $?
  1

Which is great, and also breaks postinst, which runs precisely the
latter, with no feedback at all

By checking for -ENOTTY we accept that the path being investigated
"is not it" if it's on ZFS (and any other filesystem that returns
.stx_dev_major == 0 but isn't btrfs)

1 files changed, 1 insertions, 1 deletions

diff --git a/src/shared/find-esp.c b/src/shared/find-esp.c
index 084dd1c1e2..d474bb41a0 100644
--- a/src/shared/find-esp.c
+++ b/src/shared/find-esp.c
@@ -527,7 +527,7 @@ int find_esp_and_warn(
                                flags | VERIFY_ESP_SEARCHING);
                 if (r >= 0)
                         goto found;
-                if (!IN_SET(r, -ENOENT, -EADDRNOTAVAIL, -ENOTDIR)) /* This one is not it */
+                if (!IN_SET(r, -ENOENT, -EADDRNOTAVAIL, -ENOTDIR, -ENOTTY)) /* This one is not it */
                         return r;
 
                 p = mfree(p);