tests: Rework tests/installed → tests/kola

Previously we made an effort to use the [Fedora Standard Test Interface](https://docs.fedoraproject.org/en-US/ci/standard-test-interface/).
This effort was not very successful; the primary thing that
it really died on is Ansible just didn't support rebooting
very well.  I think that's since gotten better, but even
then, Ansible wasn't the best thing for a test framework
for us anyways.

In the meantime Fedora CoreOS happened emphasizing Ignition
and not "post-hoc reconciliation" models like Ansible over
ssh.

And, [coreos-assembler](https://github.com/coreos/coreos-assembler) happened too.

Furthermore, we really need to test OSTree's interaction
with Ignition as we've invented several special things there.

Then most recently, I've been working on having
cosa/kola support running externally defined tests:
https://github.com/coreos/coreos-assembler/pull/1215

There's a lot of things to clean up after this but at least this
works for me:

```
$ cd /srv/fcos
$ cosa kola run -- --parallel 4 --output-dir tmp/kola -E ~/src/github/ostreedev/ostree/ 'ext.ostree.*'
```

NOTE: This *does not* drop ostree binaries into the target.  See:
https://github.com/coreos/coreos-assembler/pull/1252#issuecomment-600623315

This drops our dependency on Python in the installed tests, and
also fixes a few bugs that came up.

I disabled the `itest-bare-user-root.sh` one because it's
entangled with the shell script infrastructure for the unit tests.

18 files changed, 1042 insertions, 0 deletions

diff --git a/tests/kola/README.md b/tests/kola/README.md
new file mode 100644
index 00000000..19cd5528
--- /dev/null
+++ b/tests/kola/README.md
@@ -0,0 +1 @@
+This uses https://github.com/coreos/coreos-assembler/pull/1252
diff --git a/tests/kola/destructive/data b/tests/kola/destructive/data
new file mode 120000
index 00000000..a96aa0ea
--- /dev/null
+++ b/tests/kola/destructive/data
@@ -0,0 +1 @@
+..
\ No newline at end of file
diff --git a/tests/kola/destructive/itest-bare-root.sh b/tests/kola/destructive/itest-bare-root.sh
new file mode 100755
index 00000000..2834a829
--- /dev/null
+++ b/tests/kola/destructive/itest-bare-root.sh
@@ -0,0 +1,45 @@
+#!/bin/bash
+
+# Tests of the "raw ostree" functionality using the host's ostree repo as uid 0.
+
+set -xeuo pipefail
+
+. ${KOLA_EXT_DATA}/libinsttest.sh
+
+echo "1..2"
+date
+
+require_writable_sysroot
+
+cd /ostree/repo/tmp
+rm co -rf
+rm co-testref -rf
+ostree refs --delete testref
+ostree checkout -H ${host_refspec} co
+victim_symlink=/usr/bin/gtar  # Seems likely to stick around
+# Copy the link to avoid corrupting it
+cp co/${victim_symlink}{,.tmp}
+mv co/${victim_symlink}{.tmp,}
+# Add another xattr to a symlink and a directory, since otherwise this is unusual
+setfattr -n security.biometric -v iris co/${victim_symlink}
+setfattr -n security.crunchy -v withketchup co/usr/bin
+csum=$(ostree commit -b testref --link-checkout-speedup --tree=dir=co)
+ostree fsck
+ostree ls -X testref ${victim_symlink} > ls.txt
+assert_file_has_content ls.txt 'security\.biometric'
+ostree ls -X ${host_refspec} ${victim_symlink} > ls.txt
+assert_not_file_has_content ls.txt 'security\.biometric'
+ostree ls -X testref usr/bin > ls.txt
+assert_file_has_content ls.txt 'security\.crunchy'
+
+ostree checkout -H testref co-testref
+getfattr -n security.biometric co-testref/${victim_symlink} > xattr.txt
+assert_file_has_content xattr.txt 'security\.biometric="iris"'
+getfattr -n security.crunchy co-testref/usr/bin > xattr.txt
+assert_file_has_content xattr.txt 'security\.crunchy="withketchup"'
+
+rm co -rf
+rm co-testref -rf
+
+echo "ok xattrs"
+date
diff --git a/tests/kola/destructive/itest-deploy-selinux.sh b/tests/kola/destructive/itest-deploy-selinux.sh
new file mode 100755
index 00000000..099b5c27
--- /dev/null
+++ b/tests/kola/destructive/itest-deploy-selinux.sh
@@ -0,0 +1,59 @@
+#!/bin/bash
+
+# Verify our /etc merge works with selinux
+
+set -xeuo pipefail
+
+. ${KOLA_EXT_DATA}/libinsttest.sh
+
+require_writable_sysroot
+
+date
+# Create a new deployment
+ostree admin deploy --karg-proc-cmdline ${host_refspec}
+new_deployment_path=/ostree/deploy/${host_osname}/deploy/${host_commit}.1
+
+# Test /etc directory mtime
+if ! test ${new_deployment_path}/etc/NetworkManager -nt /etc/NetworkManager; then
+    ls -al ${new_deployment_path}/etc/NetworkManager /etc/NetworkManager
+    fatal "/etc directory mtime not newer"
+fi
+
+# A set of files that have a variety of security contexts
+for file in fstab passwd exports hostname sysctl.conf yum.repos.d \
+            NetworkManager/dispatcher.d/hook-network-manager; do
+    if ! test -e /etc/${file}; then
+        continue
+    fi
+
+    current=$(cd /etc && ls -Z ${file})
+    new=$(cd ${new_deployment_path}/etc && ls -Z ${file})
+    assert_streq "${current}" "${new}"
+done
+
+# Cleanup
+ostree admin undeploy 0
+
+cd /ostree/repo/tmp
+ostree checkout --fsync=0 -H ${host_commit} test-label
+rm -vf test-label/usr/lib/ostree-boot/vmlinuz*
+rm -vf test-label/usr/lib/ostree-boot/initramfs*
+cd test-label/usr/lib/modules/*
+rm initramfs.img
+echo new initramfs > initramfs.img
+cd -
+ostree commit --link-checkout-speedup --selinux-policy=test-label -b test-label --consume --tree=dir=test-label
+
+ostree admin deploy --karg-proc-cmdline test-label
+
+# This captures all of the boot entries; it'd be slightly annoying
+# to try to figure out the accurate one, so let's just ensure that at least
+# one entry is boot_t.
+# https://bugzilla.redhat.com/show_bug.cgi?id=1536991
+ls -Z /boot/ostree/*/ > bootlsz.txt
+assert_file_has_content_literal bootlsz.txt 'system_u:object_r:boot_t:s0 vmlinuz-'
+assert_file_has_content_literal bootlsz.txt 'system_u:object_r:boot_t:s0 initramfs-'
+
+ostree admin undeploy 0
+ostree refs --delete test-label
+date
diff --git a/tests/kola/destructive/itest-label-selinux.sh b/tests/kola/destructive/itest-label-selinux.sh
new file mode 100755
index 00000000..87fb26f0
--- /dev/null
+++ b/tests/kola/destructive/itest-label-selinux.sh
@@ -0,0 +1,88 @@
+#!/bin/bash
+
+# Test commit --selinux-policy
+
+set -xeuo pipefail
+
+. ${KOLA_EXT_DATA}/libinsttest.sh
+
+date
+cd /ostree/repo/tmp
+rm co -rf
+ostree checkout -H ${host_refspec} co
+testbin=co/usr/bin/foo-a-generic-binary
+assert_not_has_file "${testbin}"
+# Make a test binary that we label as shell_exec_t on disk, but should be
+# reset by --selinux-policy back to bin_t
+echo 'test foo' > ${testbin}
+chcon --reference co/usr/bin/true ${testbin}
+oldcon=$(getfattr --only-values -m security.selinux ${testbin})
+chcon --reference co/usr/bin/bash ${testbin}
+newcon=$(getfattr --only-values -m security.selinux ${testbin})
+assert_not_streq "${oldcon}" "${newcon}"
+ostree commit -b testbranch --link-checkout-speedup \
+       --selinux-policy co --tree=dir=co
+ostree ls -X testbranch /usr/bin/foo-a-generic-binary > ls.txt
+assert_file_has_content ls.txt ${oldcon}
+ostree fsck
+
+ostree refs --delete testbranch
+rm co -rf
+echo "ok commit with sepolicy"
+
+# Now let's check that selinux policy labels can be applied on checkout
+
+rm rootfs -rf
+if ostree checkout -H \
+    --selinux-policy / ${host_refspec} co; then
+  assert_not_reached "checked out with -H and --selinux-policy"
+fi
+# recommit just two binaries into a new branch with selinux labels stripped
+mkdir -p rootfs/usr/bin
+oldcon=$(getfattr --only-values -m security.selinux /usr/bin/bash)
+cp /usr/bin/{true,bash} rootfs/usr/bin
+newcon=$(getfattr --only-values -m security.selinux rootfs/usr/bin/bash)
+assert_not_streq "${oldcon}" "${newcon}"
+echo "ok checkout with sepolicy setup"
+
+ostree commit -b testbranch rootfs
+ostree checkout testbranch --selinux-policy / co
+newcon=$(getfattr --only-values -m security.selinux co/usr/bin/bash)
+assert_streq "${oldcon}" "${newcon}"
+rm co -rf
+echo "ok checkout with sepolicy"
+ostree checkout testbranch --selinux-policy / --subpath /usr/bin co
+newcon=$(getfattr --only-values -m security.selinux co/bash)
+assert_streq "${oldcon}" "${newcon}"
+rm co -rf
+echo "ok checkout with sepolicy and subpath"
+
+# now commit tree with mismatched leading dirs
+mkdir -p rootfs/subdir
+mv rootfs/{usr,subdir}
+ostree commit -b testbranch rootfs
+ostree checkout testbranch --selinux-policy / co
+newcon=$(getfattr --only-values -m security.selinux co/subdir/usr/bin/bash)
+assert_not_streq "${oldcon}" "${newcon}"
+rm co -rf
+ostree checkout testbranch --selinux-policy / \
+  --subpath subdir --selinux-prefix / co
+newcon=$(getfattr --only-values -m security.selinux co/usr/bin/bash)
+assert_streq "${oldcon}" "${newcon}"
+rm co -rf
+echo "ok checkout with sepolicy and selinux-prefix"
+
+# Now check that combining --selinux-policy with --skip-list doesn't blow up
+echo > skip-list.txt << EOF
+/usr/bin/true
+EOF
+ostree checkout testbranch --selinux-policy / --skip-list skip-list.txt \
+  --subpath subdir --selinux-prefix / co
+! test -f co/usr/bin/true
+test -f co/usr/bin/bash
+newcon=$(getfattr --only-values -m security.selinux co/usr/bin/bash)
+assert_streq "${oldcon}" "${newcon}"
+rm co -rf
+ostree refs --delete testbranch
+echo "ok checkout selinux and skip-list"
+date
diff --git a/tests/kola/destructive/staged-deploy.sh b/tests/kola/destructive/staged-deploy.sh
new file mode 100755
index 00000000..47d7c9af
--- /dev/null
+++ b/tests/kola/destructive/staged-deploy.sh
@@ -0,0 +1,118 @@
+#!/bin/bash
+set -xeuo pipefail
+
+. ${KOLA_EXT_DATA}/libinsttest.sh
+
+require_writable_sysroot
+prepare_tmpdir
+
+n=$(nth_boot)
+case "${n}" in
+  1)
+    commit=${host_commit}
+  # Test the deploy --stage functionality; first, we stage a deployment
+  # reboot, and validate that it worked.
+  # for now, until the preset propagates down
+  # Start up path unit
+    systemctl enable --now ostree-finalize-staged.path
+  # Write staged-deploy commit
+    export OSTREE_SYSROOT_DEBUG="test-staged-path"
+    cd /ostree/repo/tmp
+    # https://github.com/ostreedev/ostree/issues/1569
+    ostree checkout -H ${commit} t
+    ostree commit --no-bindings --parent="${commit}" -b staged-deploy -I --consume t
+    newcommit=$(ostree rev-parse staged-deploy)
+    orig_mtime=$(stat -c '%.Y' /sysroot/ostree/deploy)
+    systemctl show -p SubState ostree-finalize-staged.path | grep -q waiting
+    systemctl show -p ActiveState ostree-finalize-staged.service | grep -q inactive
+    systemctl show -p TriggeredBy ostree-finalize-staged.service | grep -q path
+    ostree admin deploy --stage staged-deploy | tee out.txt
+    assert_file_has_content out.txt 'test-staged-path: Not running'
+    systemctl show -p SubState ostree-finalize-staged.path | grep running
+    systemctl show -p ActiveState ostree-finalize-staged.service | grep active
+    new_mtime=$(stat -c '%.Y' /sysroot/ostree/deploy)
+    test "${orig_mtime}" != "${new_mtime}"
+    test -f /run/ostree/staged-deployment
+    ostree refs | grep -E -e '^ostree/' | while read ref; do
+      if test "$(ostree rev-parse ${ref})" = "${newcommit}"; then
+        touch deployment-ref-found
+      fi
+    done
+    test -f deployment-ref-found
+    rm deployment-ref-found
+    if ostree admin pin 0 2>err.txt; then
+      fatal "Pinned staged deployment"
+    fi
+    assert_file_has_content err.txt 'Cannot pin staged deployment'
+    kola_reboot
+    ;;
+  2) 
+    # Check that deploy-staged service worked
+    rpm-ostree status
+    # Assert that the previous boot had a journal entry for it
+    journalctl -b "-1" -u ostree-finalize-staged.service > svc.txt
+    assert_file_has_content svc.txt 'Bootloader updated; bootconfig swap: yes; deployment count change: 1'
+    rm -f svc.txt
+    # And there should not be a staged deployment
+    test '!' -f /run/ostree/staged-deployment
+
+    # Upgrade with staging
+    test '!' -f /run/ostree/staged-deployment
+    ostree admin deploy --stage staged-deploy
+    test -f /run/ostree/staged-deployment
+    origcommit=$(ostree rev-parse staged-deploy)
+    cd /ostree/repo/tmp
+    ostree checkout -H "${origcommit}" t
+    ostree commit --no-bindings --parent="${origcommit}" -b staged-deploy -I --consume t
+    newcommit=$(ostree rev-parse staged-deploy)
+    env OSTREE_EX_STAGE_DEPLOYMENTS=1 ostree admin upgrade >out.txt
+    test -f /run/ostree/staged-deployment
+    # Debating bouncing back out to Ansible for this
+    firstdeploycommit=$(rpm-ostree status |grep 'Commit:' |head -1|sed -e 's,^ *Commit: *,,')
+    assert_streq "${firstdeploycommit}" "${newcommit}"
+    # Cleanup
+    rpm-ostree cleanup -rp
+    echo "ok upgrade with staging"
+
+    # Ensure we can unstage
+    # Write staged-deploy commit, then unstage
+    ostree admin deploy --stage staged-deploy
+    ostree admin status > status.txt
+    assert_file_has_content_literal status.txt '(staged)'
+    test -f /run/ostree/staged-deployment
+    ostree admin undeploy 0
+    ostree admin status > status.txt
+    grep -vqFe '(staged)' status.txt
+    test '!' -f /run/ostree/staged-deployment
+    echo "ok unstage"
+
+   # Staged should be overwritten by non-staged as first
+    commit=$(rpmostree_query_json '.deployments[0].checksum')
+    ostree admin deploy --stage staged-deploy
+    test -f /run/ostree/staged-deployment
+    ostree --repo=/ostree/repo refs --create nonstaged-deploy "${commit}"
+    ostree admin deploy nonstaged-deploy
+    ostree admin status > status.txt
+    grep -vqFe '(staged)' status.txt
+    test '!' -f /run/ostree/staged-deployment
+    ostree admin undeploy 0
+    echo "ok staged overwritten by non-staged"
+
+  # Staged is retained when pushing rollback
+    commit=$(rpmostree_query_json '.deployments[0].checksum')
+    ostree admin deploy --stage staged-deploy
+    test -f /run/ostree/staged-deployment
+    ostree admin deploy --retain-pending --not-as-default nonstaged-deploy
+    test -f /run/ostree/staged-deployment
+    ostree admin status > status.txt
+    assert_file_has_content_literal status.txt '(staged)'
+    ostree admin undeploy 0
+    ostree admin undeploy 1
+    echo "ok staged retained"
+
+    # Cleanup refs
+    ostree refs --delete staged-deploy nonstaged-deploy
+    echo "ok cleanup refs"
+    ;;
+  *) fatal "Unexpected boot count" ;;
+esac
diff --git a/tests/kola/destructive/var-mount.sh b/tests/kola/destructive/var-mount.sh
new file mode 100755
index 00000000..b3fe3d98
--- /dev/null
+++ b/tests/kola/destructive/var-mount.sh
@@ -0,0 +1,23 @@
+#!/bin/bash
+# https://github.com/ostreedev/ostree/issues/1667
+set -xeuo pipefail
+
+. ${KOLA_EXT_DATA}/libinsttest.sh
+
+n=$(nth_boot)
+case "${n}" in
+  1) 
+    require_writable_sysroot
+    # Hack this off for now
+    chattr -i /sysroot
+    cp -a /var /sysroot/myvar
+    touch /sysroot/myvar/somenewfile
+    echo '/sysroot/myvar /var none bind 0 0' >> /etc/fstab
+    kola_reboot
+    ;;
+  2)
+    systemctl status var.mount
+    test -f /var/somenewfile
+    ;;
+  *) fatal "Unexpected boot count $n"
+esac
diff --git a/tests/kola/libinsttest.sh b/tests/kola/libinsttest.sh
new file mode 100644
index 00000000..2552cf78
--- /dev/null
+++ b/tests/kola/libinsttest.sh
@@ -0,0 +1,91 @@
+# Common definitions for installed, privileged tests
+#
+# Copyright (C) 2017 Colin Walters <walters@verbum.org>
+#
+# SPDX-License-Identifier: LGPL-2.0+
+#
+# This library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License as published by the Free Software Foundation; either
+# version 2 of the License, or (at your option) any later version.
+#
+# This library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this library; if not, write to the
+# Free Software Foundation, Inc., 59 Temple Place - Suite 330,
+# Boston, MA 02111-1307, USA.
+
+. ${KOLA_EXT_DATA}/libtest-core.sh
+
+# Copy of bits from tap-test
+test_tmpdir=
+function _tmpdir_cleanup () {
+    if test -z "${TEST_SKIP_CLEANUP:-}" &&
+       test -n "${test_tmpdir}" && test -f ${test_tmpdir}/.testtmp; then
+	      rm "${test_tmpdir}" -rf
+    fi
+}
+prepare_tmpdir() {
+    local tmpdir=${1:-/var/tmp}
+    test_tmpdir=$(mktemp -p ${tmpdir} -d ostree-insttest.XXXXXXXX)
+    touch ${test_tmpdir}/.testtmp
+    cd ${test_tmpdir}
+}
+
+# This is copied from flatpak/flatpak/tests/test-webserver.sh
+run_tmp_webserver() {
+    dir=$1
+
+    port=8000
+    podman create --name ostree-httpd --privileged  -ti --net=host -v "${dir}":/srv --workdir /srv \
+        registry.svc.ci.openshift.org/coreos/fedora:31 python3 -m http.server "${port}"
+    podman generate systemd ostree-httpd > /etc/systemd/system/ostree-httpd.service
+    systemctl daemon-reload
+    systemctl start ostree-httpd.service
+
+    address="http://127.0.0.1:${port}"
+    while ! curl --head "${address}" &>/dev/null; do sleep 1; done
+    echo "${address}" > ${test_tmpdir}/httpd-address
+}
+
+# Yeah this is a hack.  Doing this better requires more first class support
+# for creating derived commits in ostree potentially.  Or barring that,
+# just recommend to people to use `unshare -m` or equivalent and
+# mount -o remount,rw /sysroot in their code.
+require_writable_sysroot() {
+    if ! test -w /sysroot; then
+        mount -o remount,rw /sysroot
+    fi
+}
+
+nth_boot() {
+    journalctl --list-boots | wc -l
+}
+
+kola_reboot() {
+    kill -TERM $$
+    sleep 2m
+    echo "failed to reboot?" 1>&2
+    exit 1
+}
+
+# Determine our origin refspec - we'll use this as a test base
+rpmostree=$(which rpm-ostree 2>/dev/null)
+if test -z "${rpmostree}"; then
+    skip "no rpm-ostree, at some point point this to raw ostree too"
+fi
+
+# We need to be root
+assert_streq $(id -u) 0
+
+rpmostree_query_json() {
+    query=$1
+    rpm-ostree status --json | jq -r "${query}"
+}
+host_refspec=$(rpmostree_query_json '.deployments[0].origin')
+host_commit=$(rpmostree_query_json '.deployments[0].checksum')
+host_osname=$(rpmostree_query_json '.deployments[0].osname')
diff --git a/tests/kola/libtest-core.sh b/tests/kola/libtest-core.sh
new file mode 100644
index 00000000..945d2857
--- /dev/null
+++ b/tests/kola/libtest-core.sh
@@ -0,0 +1,153 @@
+# Core source library for shell script tests; the
+# canonical version lives in:
+#
+#   https://github.com/ostreedev/ostree
+#
+# Known copies are in the following repos:
+#
+# - https://github.com/projectatomic/rpm-ostree
+#
+# Copyright (C) 2017 Colin Walters <walters@verbum.org>
+#
+# SPDX-License-Identifier: LGPL-2.0+
+#
+# This library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License as published by the Free Software Foundation; either
+# version 2 of the License, or (at your option) any later version.
+#
+# This library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this library; if not, write to the
+# Free Software Foundation, Inc., 59 Temple Place - Suite 330,
+# Boston, MA 02111-1307, USA.
+
+fatal() {
+    echo $@ 1>&2; exit 1
+}
+# fatal() is shorter to type, but retain this alias
+assert_not_reached () {
+    fatal "$@"
+}
+
+# Some tests look for specific English strings. Use a UTF-8 version
+# of the C (POSIX) locale if we have one, or fall back to en_US.UTF-8
+# (https://sourceware.org/glibc/wiki/Proposals/C.UTF-8)
+#
+# If we can't find the locale command assume we have support for C.UTF-8
+# (e.g. musl based systems)
+if type -p locale >/dev/null; then
+    export LC_ALL=$(locale -a | grep -iEe '^(C|en_US)\.(UTF-8|utf8)$' | head -n1 || true)
+    if [ -z "${LC_ALL}" ]; then fatal "Can't find suitable UTF-8 locale"; fi
+else
+    export LC_ALL=C.UTF-8
+fi
+# A GNU extension, used whenever LC_ALL is not C
+unset LANGUAGE
+
+# This should really be the default IMO
+export G_DEBUG=fatal-warnings
+
+assert_streq () {
+    test "$1" = "$2" || fatal "$1 != $2"
+}
+
+assert_str_match () {
+    if ! echo "$1" | grep -E -q "$2"; then
+	      fatal "$1 does not match regexp $2"
+    fi
+}
+
+assert_not_streq () {
+    (! test "$1" = "$2") || fatal "$1 == $2"
+}
+
+assert_has_file () {
+    test -f "$1" || fatal "Couldn't find '$1'"
+}
+
+assert_has_dir () {
+    test -d "$1" || fatal "Couldn't find '$1'"
+}
+
+# Dump ls -al + file contents to stderr, then fatal()
+_fatal_print_file() {
+    file="$1"
+    shift
+    ls -al "$file" >&2
+    sed -e 's/^/# /' < "$file" >&2
+    fatal "$@"
+}
+
+assert_not_has_file () {
+    if test -f "$1"; then
+        _fatal_print_file "$1" "File '$1' exists"
+    fi
+}
+
+assert_not_file_has_content () {
+    fpath=$1
+    shift
+    for re in "$@"; do
+        if grep -q -e "$re" "$fpath"; then
+            _fatal_print_file "$fpath" "File '$fpath' matches regexp '$re'"
+        fi
+    done
+}
+
+assert_not_has_dir () {
+    if test -d "$1"; then
+	      fatal "Directory '$1' exists"
+    fi
+}
+
+assert_file_has_content () {
+    fpath=$1
+    shift
+    for re in "$@"; do
+        if ! grep -q -e "$re" "$fpath"; then
+            _fatal_print_file "$fpath" "File '$fpath' doesn't match regexp '$re'"
+        fi
+    done
+}
+
+assert_file_has_content_literal () {
+    fpath=$1; shift
+    for s in "$@"; do
+        if ! grep -q -F -e "$s" "$fpath"; then
+            _fatal_print_file "$fpath" "File '$fpath' doesn't match fixed string list '$s'"
+        fi
+    done
+}
+
+assert_file_has_mode () {
+    mode=$(stat -c '%a' $1)
+    if [ "$mode" != "$2" ]; then
+        fatal "File '$1' has wrong mode: expected $2, but got $mode"
+    fi
+}
+
+assert_symlink_has_content () {
+    if ! test -L "$1"; then
+        fatal "File '$1' is not a symbolic link"
+    fi
+    if ! readlink "$1" | grep -q -e "$2"; then
+        _fatal_print_file "$1" "Symbolic link '$1' doesn't match regexp '$2'"
+    fi
+}
+
+assert_file_empty() {
+    if test -s "$1"; then
+        _fatal_print_file "$1" "File '$1' is not empty"
+    fi
+}
+
+# Use to skip all of these tests
+skip() {
+    echo "1..0 # SKIP" "$@"
+    exit 0
+}
diff --git a/tests/kola/nondestructive/data b/tests/kola/nondestructive/data
new file mode 120000
index 00000000..a96aa0ea
--- /dev/null
+++ b/tests/kola/nondestructive/data
@@ -0,0 +1 @@
+..
\ No newline at end of file
diff --git a/tests/kola/nondestructive/itest-bare-unit.sh b/tests/kola/nondestructive/itest-bare-unit.sh
new file mode 100644
index 00000000..d399d78d
--- /dev/null
+++ b/tests/kola/nondestructive/itest-bare-unit.sh
@@ -0,0 +1,42 @@
+#!/bin/bash
+
+# Run test-basic.sh as root.
+# https://github.com/ostreedev/ostree/pull/1199
+
+set -xeuo pipefail
+
+. ${KOLA_EXT_DATA}/libinsttest.sh
+
+fatal "FIXME - need to also sync over the installed tests"
+
+date
+# These tests sort of bypass the installed-tests spec;
+# fixing that would require installing g-d-t-r, though
+# more ideally we architect things with a "control" container
+# distinct from the host.
+export G_TEST_SRCDIR=$(realpath $dn/../../..)
+
+# Use /var/tmp to hopefully use XFS + O_TMPFILE etc.
+prepare_tmpdir /var/tmp
+trap _tmpdir_cleanup EXIT
+/usr/libexec/installed-tests/libostree/test-basic.sh
+/usr/libexec/installed-tests/libostree/test-basic-c
+date
+
+# Test error message when opening a non-world-readable object
+# https://github.com/ostreedev/ostree/issues/1562
+rm repo files -rf
+chmod a+rx .
+ostree --repo=repo init --mode=bare
+mkdir files
+touch files/unreadable
+chmod 0 files/unreadable
+ostree --repo=repo commit -b testbranch --tree=dir=files
+# We should be able to read as root due to CAP_DAC_OVERRIDE
+ostree --repo=repo cat testbranch /unreadable >/dev/null
+if setpriv --reuid bin --regid bin --clear-groups ostree --repo=repo cat testbranch /unreadable 2>err.txt; then
+    fatal "Listed unreadable object as non-root"
+fi
+assert_file_has_content err.txt "Opening content object.*openat: Permission denied"
+
+date
diff --git a/tests/kola/nondestructive/itest-bare-user-root.sh b/tests/kola/nondestructive/itest-bare-user-root.sh
new file mode 100755
index 00000000..ad3d2dac
--- /dev/null
+++ b/tests/kola/nondestructive/itest-bare-user-root.sh
@@ -0,0 +1,40 @@
+#!/bin/bash
+
+# Tests of the "raw ostree" functionality using the host's ostree repo as uid 0.
+
+set -xeuo pipefail
+
+. ${KOLA_EXT_DATA}/libinsttest.sh
+
+echo "1..1"
+date
+
+prepare_tmpdir
+ostree --repo=repo init --mode=bare-user
+mkdir -p components/{dbus,systemd}/usr/{bin,lib}
+echo dbus binary > components/dbus/usr/bin/dbus-daemon
+chmod a+x components/dbus/usr/bin/dbus-daemon
+echo dbus lib > components/dbus/usr/lib/libdbus.so.1
+echo dbus helper > components/dbus/usr/lib/dbus-daemon-helper
+chmod a+x components/dbus/usr/lib/dbus-daemon-helper
+echo systemd binary > components/systemd/usr/bin/systemd
+chmod a+x components/systemd/usr/bin/systemd
+echo systemd lib > components/systemd/usr/lib/libsystemd.so.1
+
+# Make the gid on dbus 81 like fedora, also ensure no xattrs
+ostree --repo=repo commit --no-xattrs -b component-dbus --owner-uid 0 --owner-gid 81 --tree=dir=components/dbus
+ostree --repo=repo commit --no-xattrs -b component-systemd --owner-uid 0 --owner-gid 0 --tree=dir=components/systemd
+rm rootfs -rf
+for component in dbus systemd; do
+    ostree --repo=repo checkout -U -H component-${component} --union rootfs
+done
+echo 'some rootfs data' > rootfs/usr/lib/cache.txt
+# Commit using the host's selinux policy
+ostree --repo=repo commit --selinux-policy / -b rootfs --link-checkout-speedup --tree=dir=rootfs
+ostree --repo=repo ls rootfs /usr/bin/systemd >ls.txt
+assert_file_has_content ls.txt '^-007.. 0 0 .*/usr/bin/systemd'
+ostree --repo=repo ls -X rootfs /usr/lib/dbus-daemon-helper >ls.txt
+assert_file_has_content ls.txt '^-007.. 0 81 .*security\.selinux.*/usr/lib/dbus-daemon-helper'
+assert_not_file_has_content ls.txt 'user\.ostreemeta'
+echo "ok bare-user link-checkout-speedup with modified xattrs maintains uids"
+date
diff --git a/tests/kola/nondestructive/itest-bareuser-nouserxattrs.sh b/tests/kola/nondestructive/itest-bareuser-nouserxattrs.sh
new file mode 100755
index 00000000..7908d31b
--- /dev/null
+++ b/tests/kola/nondestructive/itest-bareuser-nouserxattrs.sh
@@ -0,0 +1,24 @@
+#!/bin/bash
+
+# Test that initializing a bare-user repo on tmpfs fails
+# Maybe at some point this will be fixed in the kernel
+# but I doubt it'll be soon
+# https://www.spinics.net/lists/linux-mm/msg109775.html
+
+set -xeuo pipefail
+
+. ${KOLA_EXT_DATA}/libinsttest.sh
+
+prepare_tmpdir
+trap _tmpdir_cleanup EXIT
+date
+
+mkdir mnt
+mount -t tmpfs tmpfs mnt
+if ostree --repo=mnt/repo init --mode=bare-user 2>err.txt; then
+    umount mnt
+    assert_not_reached "bare-user on tmpfs worked?"
+fi
+umount mnt
+assert_file_has_content err.txt "Operation not supported"
+date
diff --git a/tests/kola/nondestructive/itest-payload-link.sh b/tests/kola/nondestructive/itest-payload-link.sh
new file mode 100755
index 00000000..6cfe291a
--- /dev/null
+++ b/tests/kola/nondestructive/itest-payload-link.sh
@@ -0,0 +1,140 @@
+#!/bin/bash
+#
+# Copyright (C) 2018 Red Hat, Inc.
+#
+# SPDX-License-Identifier: LGPL-2.0+
+#
+# This library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License as published by the Free Software Foundation; either
+# version 2 of the License, or (at your option) any later version.
+#
+# This library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this library; if not, write to the
+# Free Software Foundation, Inc., 59 Temple Place - Suite 330,
+# Boston, MA 02111-1307, USA.
+
+set -xeuo pipefail
+
+. ${KOLA_EXT_DATA}/libinsttest.sh
+
+echo "1..1"
+date
+
+# Use /var/tmp so we have O_TMPFILE etc.
+prepare_tmpdir /var/tmp
+trap _tmpdir_cleanup EXIT
+# We use this user down below, it needs access too
+setfacl -d -m u:bin:rwX .
+setfacl -m u:bin:rwX .
+ostree --repo=repo init --mode=archive
+echo -e '[archive]\nzlib-level=1\n' >> repo/config
+
+mkdir content
+cd content
+dd if=/dev/urandom of=bigobject bs=4k count=2560
+cp --reflink=auto bigobject bigobject2
+# Different metadata, same content
+chown bin:bin bigobject2
+cd ..
+ostree --repo=repo commit -b dupobjects --consume --selinux-policy=/ --tree=dir=content
+ostree --repo=repo summary -u
+
+run_tmp_webserver $(pwd)/repo
+
+origin=$(cat ${test_tmpdir}/httpd-address)
+
+cleanup() {
+    cd ${test_tmpdir}
+    for mnt in ${mnts:-}; do
+        umount ${mnt} || true
+    done
+    for blkdev in ${blkdevs:-}; do
+        losetup -d ${blkdev} || true
+    done
+}
+trap cleanup EXIT
+
+truncate -s 2G testblk1.img
+blkdev1=$(losetup --find --show $(pwd)/testblk1.img)
+blkdevs="${blkdev1}"
+# This filesystem must support reflinks
+mkfs.xfs -m reflink=1 ${blkdev1}
+mkdir mnt1
+mount ${blkdev1} mnt1
+mnts=mnt1
+
+truncate -s 2G testblk2.img
+blkdev2=$(losetup --find --show $(pwd)/testblk2.img)
+blkdevs="${blkdev1} ${blkdev2}"
+mkfs.xfs -m reflink=1 ${blkdev2}
+mkdir mnt2
+mount ${blkdev2} mnt2
+mnts="mnt1 mnt2"
+
+cd mnt1
+# See above for setfacl rationale
+setfacl -d -m u:bin:rwX .
+setfacl -m u:bin:rwX .
+ls -al .
+runuser -u bin mkdir foo
+runuser -u bin touch foo/bar
+ls -al foo
+
+# Test that reflink is really there (not just --reflink=auto)
+touch a
+cp --reflink a b
+mkdir repo
+ostree --repo=repo init
+ostree config --repo=repo set core.payload-link-threshold 0
+ostree --repo=repo remote add origin --set=gpg-verify=false ${origin}
+ostree --repo=repo pull --disable-static-deltas origin dupobjects
+find repo -type l -name '*.payload-link' >payload-links.txt
+assert_streq "$(wc -l < payload-links.txt)" "1"
+
+cat payload-links.txt | while read i; do
+    payload_checksum=$(basename $(dirname $i))$(basename $i .payload-link)
+    payload_checksum_calculated=$(sha256sum $(readlink -f $i) | cut -d ' ' -f 1)
+    assert_streq "${payload_checksum}" "${payload_checksum_calculated}"
+done
+echo "ok payload link"
+
+ostree --repo=repo checkout dupobjects content
+# And another object which differs just in metadata
+cp --reflink=auto content/bigobject{,3}
+chown operator:0 content/bigobject3
+cat >unpriv-child-repo.sh <<EOF
+#!/bin/bash
+# Check that commit to an user repo that has a parent still works
+set -xeuo pipefail
+ostree --repo=child-repo init --mode=bare-user
+ostree --repo=child-repo remote add origin --set=gpg-verify=false ${origin}
+ostree --repo=child-repo config set core.parent $(pwd)/repo
+ostree --repo=child-repo commit -b test content
+EOF
+chmod a+x unpriv-child-repo.sh
+runuser -u bin ./unpriv-child-repo.sh
+find child-repo -type l -name '*.payload-link' >payload-links.txt
+assert_streq "$(wc -l < payload-links.txt)" "0"
+rm content -rf
+
+echo "ok reflink unprivileged with parent repo"
+
+# We can't reflink across devices though
+cd ../mnt2
+ostree --repo=repo init --mode=archive
+ostree --repo=repo config set core.parent $(cd ../mnt1/repo && pwd)
+ostree --repo=../mnt1/repo checkout dupobjects content
+ostree --repo=repo commit -b dupobjects2 --consume --tree=dir=content
+ostree --repo=repo pull --disable-static-deltas origin dupobjects
+find repo -type l -name '*.payload-link' >payload-links.txt
+assert_streq "$(wc -l < payload-links.txt)" "0"
+
+echo "ok payload link across devices"
+
+date
diff --git a/tests/kola/nondestructive/itest-pull-space.sh b/tests/kola/nondestructive/itest-pull-space.sh
new file mode 100755
index 00000000..97524f67
--- /dev/null
+++ b/tests/kola/nondestructive/itest-pull-space.sh
@@ -0,0 +1,116 @@
+#!/bin/bash
+# Test min-free-space-percent using loopback devices
+
+set -xeuo pipefail
+
+. ${KOLA_EXT_DATA}/libinsttest.sh
+date
+
+prepare_tmpdir
+truncate -s 20MB testblk.img
+blkdev=$(losetup --find --show $(pwd)/testblk.img)
+mkfs.xfs ${blkdev}
+mkdir mnt
+mount ${blkdev} mnt
+
+# first test min-free-space-percent
+ostree --repo=mnt/repo init --mode=bare-user
+echo 'fsync=false' >> mnt/repo/config
+if ostree --repo=mnt/repo pull-local /ostree/repo ${host_commit} 2>err.txt; then
+    fatal "succeeded in doing a pull with no free space"
+fi
+assert_file_has_content err.txt "min-free-space-percent"
+echo "ok min-free-space-percent"
+
+# now test min-free-space-size
+rm -rf mnt/repo
+ostree --repo=mnt/repo init --mode=bare-user
+echo 'fsync=false' >> mnt/repo/config
+echo 'min-free-space-size=10MB' >> mnt/repo/config
+if ostree --repo=mnt/repo pull-local /ostree/repo ${host_commit} 2>err.txt; then
+    fatal "succeeded in doing a pull with no free space"
+fi
+assert_file_has_content err.txt "min-free-space-size"
+echo "ok min-free-space-size (error)"
+
+
+# min-free-space-size success
+ostree --repo=repo init --mode=bare-user
+echo 'fsync=false' >> repo/config
+echo 'min-free-space-size=1MB' >> repo/config
+ostree --repo=repo pull-local /ostree/repo ${host_commit}
+echo "ok min-free-space-size (success)"
+
+# metadata object write should succeed even if free space is lower than
+# min-free-space value
+rm -rf mnt/repo
+ostree --repo=mnt/repo init --mode=bare-user
+echo 'fsync=false' >> mnt/repo/config
+echo 'min-free-space-size=10MB' >> mnt/repo/config
+ostree --repo=mnt/repo pull-local --commit-metadata-only /ostree/repo ${host_commit}
+echo "ok metadata write even when free space is lower than min-free-space value"
+
+rm -rf mnt/repo
+
+# Test min-free-space-size on deltas
+
+#helper function copied from test-delta.sh
+get_assert_one_direntry_matching() {
+    local path=$1
+    local r=$2
+    local child=""
+    local bn
+    for p in ${path}/*; do
+  bn=$(basename $p)
+  if ! echo ${bn} | grep -q "$r"; then
+      continue
+  fi
+  if test -z "${child}"; then
+      child=${bn}
+  else
+      assert_not_reached "Expected only one child matching ${r} in ${path}";
+  fi
+    done
+    if test -z "${child}"; then
+  assert_not_reached "Failed to find child matching ${r}"
+    fi
+    echo ${child}
+}
+
+mkdir mnt/repo1 mnt/repo2
+ostree --repo=mnt/repo1 init --mode=bare-user
+ostree --repo=mnt/repo2 init --mode=bare-user
+mkdir files
+echo "hello" >> files/test.txt
+truncate -s 2MB files/test.txt
+
+host_commit=$(ostree --repo=mnt/repo1 commit -b test -s test --tree=dir=files)
+
+origrev=$(ostree --repo=mnt/repo1 rev-parse test)
+ostree --repo=mnt/repo1 static-delta generate --empty --to=${origrev}
+echo 'fsync=false' >> mnt/repo2/config
+echo 'min-free-space-size=12MB' >> mnt/repo2/config
+
+deltaprefix=$(get_assert_one_direntry_matching mnt/repo1/deltas '.')
+deltadir=$(get_assert_one_direntry_matching mnt/repo1/deltas/${deltaprefix} '.')
+
+# Try to pull delta and trigger error
+if ostree --repo=mnt/repo2 static-delta apply-offline mnt/repo1/deltas/${deltaprefix}/${deltadir} 2>err.txt; then
+    fatal "succeeded in doing a delta pull with no free space"
+fi
+assert_file_has_content err.txt "min-free-space-size"
+echo "OK min-free-space-size delta pull (error)"
+
+# Re-adjust min-free-space-size so that delta pull succeeds
+sed -i s/min-free-space-size=12MB/min-free-space-size=1MB/g mnt/repo2/config
+rm -rf mnt/repo2/deltas
+ostree --repo=mnt/repo2 static-delta apply-offline mnt/repo1/deltas/${deltaprefix}/${deltadir}
+
+echo "OK min-free-space-size delta pull (success)"
+rm -rf files
+
+umount mnt
+losetup -d ${blkdev}
+rm testblk.img
+
+date
diff --git a/tests/kola/nondestructive/itest-pull.sh b/tests/kola/nondestructive/itest-pull.sh
new file mode 100755
index 00000000..770f2444
--- /dev/null
+++ b/tests/kola/nondestructive/itest-pull.sh
@@ -0,0 +1,82 @@
+#!/bin/bash
+
+# Using the host ostree, test HTTP pulls
+
+set -xeuo pipefail
+
+# FIXME: https://github.com/ostreedev/ostree/pull/1548
+exit 0
+
+. ${KOLA_EXT_DATA}/libinsttest.sh
+date
+
+prepare_tmpdir /var/tmp
+trap _tmpdir_cleanup EXIT
+
+# Take the host's ostree, and make it archive
+mkdir repo
+ostree --repo=repo init --mode=archive
+echo -e '[archive]\nzlib-level=1\n' >> repo/config
+host_nonremoteref=$(echo ${host_refspec} | sed 's,[^:]*:,,')
+log_timestamps() {
+    date
+    "$@"
+    date
+}
+log_timestamps ostree --repo=repo pull-local /ostree/repo ${host_commit}
+ostree --repo=repo refs ${host_commit} --create=${host_nonremoteref}
+
+run_tmp_webserver $(pwd)/repo
+# Now test pulling via HTTP (no deltas) to a new bare-user repo
+ostree --repo=bare-repo init --mode=bare-user
+ostree --repo=bare-repo remote add origin --set=gpg-verify=false $(cat ${test_tmpdir}/httpd-address)
+log_timestamps ostree --repo=bare-repo pull --disable-static-deltas origin ${host_nonremoteref}
+
+echo "ok pull"
+
+# fsck marks commits partial
+# https://github.com/ostreedev/ostree/pull/1533
+for d in $(find bare-repo/objects/ -maxdepth 1 -type d); do
+    (find ${d} -name '*.file' || true) | head -20 | xargs rm -f
+done
+date
+if ostree --repo=bare-repo fsck |& tee fsck.txt; then
+    fatal "fsck unexpectedly succeeded"
+fi
+date
+assert_streq $(grep -cE -e 'Marking commit as partial' fsck.txt) "1"
+log_timestamps ostree --repo=bare-repo pull origin ${host_nonremoteref}
+# Don't need a full fsck here
+ostree --repo=bare-repo ls origin:${host_nonremoteref} >/dev/null
+
+rm bare-repo repo -rf
+
+# Try copying the host's repo across a mountpoint for direct
+# imports.
+cd ${test_tmpdir}
+mkdir tmpfs mnt
+mount --bind tmpfs mnt
+cd mnt
+ostree --repo=repo init --mode=bare
+log_timestamps ostree --repo=repo pull-local /ostree/repo ${host_commit}
+log_timestamps ostree --repo=repo fsck
+cd ..
+
+# Also, we shouldn't copy xattrs on metadata objects
+commit_path=objects/${host_commit:0:2}/${host_commit:2}.commit
+ostree --repo=testarchive init --mode=archive
+ostree --repo=testarchive pull-local --commit-metadata-only /ostree/repo ${host_commit}
+setfattr -n user.ostreetesting -v hello testarchive/${commit_path}
+ostree --repo=mnt/testarchive2 init --mode=archive
+ostree --repo=mnt/testarchive2 pull-local --commit-metadata-only testarchive ${host_commit}
+if getfattr -m user.ostreetesting mnt/testarchive2/${commit_path} 2>/dev/null; then
+    fatal "copied metadata xattr"
+fi
+echo "ok no metadata xattr copy"
+
+umount mnt
+
+# Cleanup
+kill -TERM $(cat ${test_tmpdir}/httpd-pid)
+echo "ok"
+date
diff --git a/tests/kola/nondestructive/itest-remotes.sh b/tests/kola/nondestructive/itest-remotes.sh
new file mode 100755
index 00000000..d1fb455b
--- /dev/null
+++ b/tests/kola/nondestructive/itest-remotes.sh
@@ -0,0 +1,17 @@
+#!/bin/bash
+
+# Test that we didn't regress /etc/ostree/remotes.d handling
+
+set -xeuo pipefail
+
+. ${KOLA_EXT_DATA}/libinsttest.sh
+date
+
+prepare_tmpdir
+trap _tmpdir_cleanup EXIT
+
+ostree remote list > remotes.txt
+if ! test -s remotes.txt; then
+    assert_not_reached "no ostree remotes"
+fi
+date
diff --git a/tests/kola/nondestructive/libtest-core.sh b/tests/kola/nondestructive/libtest-core.sh
new file mode 120000
index 00000000..d26203e2
--- /dev/null
+++ b/tests/kola/nondestructive/libtest-core.sh
@@ -0,0 +1 @@
+../libtest-core.sh
\ No newline at end of file