ostree repo commit: Speed up composing trees with `--tree=ref`

Running `ostree commit --tree=ref=a --tree=dir=b` involves reading the
whole of a into an `OstreeMutableTree` before composing `b` on top.  This
is inefficient if a is a complete rootfs and b is just touching one file.
We process O(size of a + size of b) directories rather than
O(number of touched dirs).

This commit makes `ostree commit` more efficient at composing multiple
directories together.  With `ostree_mutable_tree_fill_empty_from_dirtree`
we create a lazy `OstreeMutableTree` which only reads the underlying
information from disk when needed.  We don't need to read all the
subdirectories just to get the checksum of a tree already checked into the
repo.

This provides great speedups when composing a rootfs out of multiple other
rootfs as we do in our build system.  We compose multiple containers
together with:

    ostree commit --tree=ref=base-rootfs --tree=ref=container1 --tree=ref=container2

and it is much faster now.

As a test I ran

    time ostree --repo=... commit --orphan --tree=ref=big-rootfs --tree=dir=modified_etc

Where modified_etc contained a modified sudoers file under /etc.  I used
`strace` to count syscalls and I seperatly took timing measurements.  To
test with a cold cache I ran

    sync && echo 3 | sudo tee /proc/sys/vm/drop_caches

Results:

|                      | Before | After |
| -------------------- | ------ | ----- |
| Time (cold cache)    |   8.1s | 0.12s |
| Time (warm cache)    |   3.7s | 0.08s |
| `openat` calls       |  53589 |   246 |
| `fgetxattr` calls    |  78916 |     0 |

I'm not sure if this will have some negative interaction with the
`_ostree_repo_commit_modifier_apply` which is short-circuited here.  I
think it was disabled for `--tree=ref=` anyway, but I'm not certain.  All
the tests pass anyway.

I originally implemented this in terms of the `OstreeRepoFile` APIs, but
it was *way* less efficient, opening and reading many files unnecessarily.

Closes: #1643
Approved by: cgwalters

1 files changed, 16 insertions, 0 deletions

diff --git a/src/libostree/ostree-mutable-tree.h b/src/libostree/ostree-mutable-tree.h
index 2cb5e9a7..4b7f853e 100644
--- a/src/libostree/ostree-mutable-tree.h
+++ b/src/libostree/ostree-mutable-tree.h
@@ -53,6 +53,11 @@ _OSTREE_PUBLIC
 OstreeMutableTree *ostree_mutable_tree_new (void);
 
 _OSTREE_PUBLIC
+OstreeMutableTree * ostree_mutable_tree_new_from_checksum (OstreeRepo *repo,
+                                                           const char *contents_checksum,
+                                                           const char *metadata_checksum);
+
+_OSTREE_PUBLIC
 void ostree_mutable_tree_set_metadata_checksum (OstreeMutableTree *self,
                                                 const char        *checksum);
 
@@ -101,6 +106,17 @@ gboolean ostree_mutable_tree_walk (OstreeMutableTree   *self,
                                    GError             **error);
 
 _OSTREE_PUBLIC
+gboolean ostree_mutable_tree_fill_empty_from_dirtree (OstreeMutableTree *self,
+                                                      OstreeRepo        *repo,
+                                                      const char        *contents_checksum,
+                                                      const char        *metadata_checksum);
+
+_OSTREE_PUBLIC
+gboolean
+ostree_mutable_tree_check_error (OstreeMutableTree     *self,
+                                 GError               **error);
+
+_OSTREE_PUBLIC
 GHashTable * ostree_mutable_tree_get_subdirs (OstreeMutableTree *self);
 _OSTREE_PUBLIC
 GHashTable * ostree_mutable_tree_get_files (OstreeMutableTree *self);