summaryrefslogtreecommitdiff
path: root/src/libostree/ostree-repo-static-delta-compilation.c
Commit message (Collapse)AuthorAgeFilesLines
* tree-wide: Run clang-formatColin Walters2023-05-021-436/+350
| | | | | This is a one-time tree wide reformatting to ensure consistency going forward.
* tree-wide: Fix various include ordering issuesColin Walters2023-05-011-2/+1
| | | | This fixes the build with `clang-format`.
* lib: assert more invariantsLuca BRUNO2022-11-241-1/+2
| | | | | This converts a few more safety checks into either plain GErrors or hard assertions.
* lib/static-delta: document and check parameters formatLuca BRUNO2022-10-131-3/+7
| | | | | | | | | | | This enhances the logic handling GVariant parameters within `ostree_repo_static_delta_generate()`. Several of those entries are expected to be zero-terminated values, and this implicit assumption has been observed to be an hidden trap in languages where strings and arrays may not carry a terminator value (e.g. Rust). In order to improve the situation, this makes the documentation more explicit and actively tries to catch invalid input parameters.
* otutil: add error handling to variant buildersLuca BRUNO2022-09-081-5/+8
| | | | | | This enhances a bunch of helpers related to GVariant building, in order to properly handle errors and avoid some potential cases of unexpected NULL results.
* Update FSF license notices to use URL instead of addressJoseph Marrero2021-12-071-3/+1
|
* deltas: Fix leak of matchesColin Walters2021-02-021-1/+1
| | | | Found by ASAN.
* lib/deltas: Annotate from checksum as nullableDan Nicholson2020-11-021-3/+3
| | | | | Without this you can't create a scratch delta from GI. While here, switch the deprecated allow-none annotations to nullable.
* lib/deltas: Add inline signature for static-delta superblockFrédéric Danis2020-09-141-6/+91
| | | | | | | | | | | | | | | | | While the commits contained in the single static-delta file are signed so we can check them and operate on trusted data, the superblock isn't signed in any way, so it end up operating on untrusted data to: 1. actually find where the trusted data is, and 2. check whether the update is fit for the current device by looking at the collection id stored in the metadata This commit generates signatures of all static data, and concatenate them to the existing static delta format, i.e. as a GVariant layout `a{sv}ay` where - a{sv}: signatures - ay: existing delta variant Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>
* lib/deltas: convert ostree_repo_static_delta_generate to new styleFrédéric Danis2020-07-031-34/+25
| | | | | | | The "new style" code generally avoids `goto err` because it conflicts with `__attribute__((cleanup))`. This fixes a compiler warning. Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>
* lib/repo: Create repo directories as 0775Jonathan Lebon2020-02-181-1/+1
| | | | | | | | | | | | | For repo structure directories like `objects`, `refs`, etc... we should be more permissive and let the system's `umask` narrow down the permission bits as wanted. This came up in a context where we want to be able to have read/write access on an OSTree repo on NFS from two separate OpenShift apps by using supplemental groups[1] so we don't require SCCs for running as the same UID (supplemental groups are part of the default restricted SCC). [1] https://docs.openshift.com/container-platform/3.11/install_config/persistent_storage/persistent_storage_nfs.html#nfs-supplemental-groups
* ostree_repo_static_delta_generate: Fix leakAlexander Larsson2018-08-131-1/+1
| | | | | | | There is no need to ref the argument of g_variant_builder_add_value Closes: #1701 Approved by: jlebon
* lib: Fix some minor memory leaksJonathan Lebon2018-07-261-2/+1
| | | | | | | | | | | I initially was going to add a `G_DEFINE_AUTOPTR_CLEANUP_FUNC` for `FetchStaticDeltaData`, but it honestly didn't seem worth mucking around ownership everywhere and potentially getting it wrong. Discovered by Coverity. Closes: #1692 Approved by: cgwalters
* lib/deltas: Squash some GCC maybe-uninitialized warningsColin Walters2018-03-221-8/+2
| | | | | | | | These show up in the RPM build, I didn't yet try to figure out why we're not reproducing them outside of that. Closes: #1510 Approved by: jlebon
* Add SPDX-License-Identifier to source filesMarcus Folkesson2018-01-301-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | SPDX License List is a list of (common) open source licenses that can be referred to by a “short identifier”. It has several advantages compared to the common "license header texts" usually found in source files. Some of the advantages: * It is precise; there is no ambiguity due to variations in license header text * It is language neutral * It is easy to machine process * It is concise * It is simple and can be used without much cost in interpreted environments like java Script, etc. * An SPDX license identifier is immutable. * It provides simple guidance for developers who want to make sure the license for their code is respected See http://spdx.org for further reading. Signed-off-by: Marcus Folkesson <marcus.folkesson@gmail.com> Closes: #1439 Approved by: cgwalters
* bin/delta: Fix compilation with relative subdirs --filenameColin Walters2018-01-221-25/+7
| | | | | | | | | | | | | | | Currently we were parsing `opt_filename` twice...I dug through the history a bit and it looks like it may have been an accident from refactoring. What we're fixing here concretely is that using relative subdirectories like `--filename somesubdir/foo` broke because we were incorrectly passing the `somesubdir/` again. Closes: #1423 Closes: #1427 Approved by: jlebon
* deltas: Add an if0'd bit to print bsdiff objectsColin Walters2017-11-011-0/+14
| | | | | | | I used this to find the bsdiff'd objects. Closes: #1314 Approved by: jlebon
* bin/delta-compilation: Add a "progress bar" for bsdiff generationColin Walters2017-11-011-9/+18
| | | | | | | | | | It's the slowest part, let's show admins something. This "update every 10%" code was copied from the fsck command; obviously a better approach would be "progress every N seconds" but doing that somewhat accurately requires making things async; not worth it here yet. Closes: #1314 Approved by: jlebon
* lib: Minor static delta fixesColin Walters2017-10-301-2/+2
| | | | | | | | | | | | | | | | | | First, the manual crosscheck script bitrotted; it got caught up in the "use libtest repo creation wrapper" bit, and also it seems like at some point `pull --require-static-deltas` changed meaning when dealing with `file:///` repos. I have more work to unwind that. Next, I'm seeing a delta failure which looks like a static delta miscompilation with rollsums; change the compiler to print out the source object too, which helped me debug this. And finally in the processing code, fix incorrect error prefixing, which was misleading. Closes: #1311 Approved by: ashcrow
* static deltas: Process each part as soon as its doneAlexander Larsson2017-10-271-133/+180
| | | | | | | | | | | | | Directly when we allocate a new part we finish the old one, writing the compressed data to a temporary file and generating the delta header for it. When all these are done we loop over them and collect the headers, sizes and either copy the tempfile data into the inlined superblock or link the tempfiles to disk with the proper names. Closes: #1309 Approved by: cgwalters
* static-delta-compilation: Move some things aroundAlexander Larsson2017-10-271-43/+43
| | | | | | | We will do some changes later that need these earliers, so move them up. Closes: #1309 Approved by: cgwalters
* static deltas: Use OtVariantBuilder to create deltasAlexander Larsson2017-10-271-47/+76
| | | | | | | | | | | | | This allows us to create the final delta desciptor directly on disk rather than having it all in memory. This is nice because it can become quite large if inlined parts are used. Note however, that we currently generate all the delta parts in memory before adding them to the delta, so we still keep all individual parts in memory. Fixing that is the next step. Closes: #1309 Approved by: cgwalters
* tree-wide: Update to new libglnx fd APIsColin Walters2017-10-111-2/+2
| | | | | | | | | | | | | | | | | | | This ends up a lot better IMO. This commit is *mostly* just `s/glnx_close_fd/glnx_autofd`, but there's also a number of hunks like: ``` - if (self->sysroot_fd != -1) - { - (void) close (self->sysroot_fd); - self->sysroot_fd = -1; - } + glnx_close_fd (&self->sysroot_fd); ``` Update submodule: libglnx Closes: #1259 Approved by: jlebon
* lib/deltas: Move variant read helper into the deltas codeColin Walters2017-10-051-2/+15
| | | | | | | | | What the deltas code is doing is weird/unfortunate. The name `ot_variant_read()` conflicts too much with `ot_variant_read_fd()`. Since nothing else uses it, move it into the deltas code. Closes: #1254 Approved by: jlebon
* tree-wide: Remove Emacs modelinesColin Walters2017-09-211-2/+1
| | | | | | | | | We added a `.dir-locals.el` in commit: 9a77017d87b74c5e2895cdd64ad098018929403f There's no need to have it per-file, with that people might think to add other editors, which is the wrong direction. Closes: #1206 Approved by: jlebon
* libutil: Add a helper for O_TMPFILE + mmap()Colin Walters2017-06-291-18/+2
| | | | | | | | | | | | | I added `glnx_open_anonymous_tmpfile()`, but then later noticed that the usage of this was really to be combined with `mmap()`, and we had two versions of that in the delta code. Add a helper. (Bigger picture...how is this different from glibc's "mmap() of /dev/zero" approach for large chunks? One advantage is the storage can be "swapped" to `/var/tmp`, but still deleted automatically, rather than requiring swap space) Closes: #973 Approved by: jlebon
* tree-wide: Misc porting to newer libglnx APIsColin Walters2017-06-281-12/+2
| | | | | | | | | | | | | - Use the new tmpfile bits - `glnx_try_fallocate` - `glnx_renameat()` Depends: https://github.com/GNOME/libglnx/pull/57 Update submodule: libglnx Closes: #970 Approved by: jlebon
* Port to GLnxTmpfileColin Walters2017-06-271-31/+23
| | | | | | | | | | | | | | | | | | | | | There's lots of mechanically replacing `OtTmpFile` with `GLnxTmpfile`; the biggest changes are in the commit path. Symlink commits are now very clearly separated from regular files. Symlinks are `OtCleanupUnlinkat`, and regular files are `GLnxTmpfile`. The commit codepath separates those as `_ostree_repo_commit_path_final()` and `_ostree_repo_commit_tmpf_final()`. A nice aspect of all of this is that they both *consume* the temporary on success. This avoids an extra spurious `unlink()` call. One of the biggest bits of code motion is in `commit_loose_regfile_object()`, which no longer needs to care about symlinks. For the most parth though it's just removing conditionals. Update submodule: libglnx Closes: #958 Approved by: jlebon
* lib/deltas: Port to more to new code styleColin Walters2017-06-261-172/+112
| | | | | | | Looking at the uses of `ostree_repo_load_file()` here. Closes: #953 Approved by: jlebon
* deltas: Don't put unreadable *from* objects in fallbackColin Walters2017-02-171-5/+1
| | | | | | | | | | | | | | | | | In https://github.com/ostreedev/ostree/pull/634 we introduced a subtle regression - the unreadable object was added to the *new* reachable objects, when it shouldn't have been. Because it was a *from* object, clients already had it. This became more obvious now that I'm working on fixing delta progress - I noticed my deltas were always starting out with 40MB fetched, which turned out to be a non-world-readable initramfs object. This code should simply *skip* the unreadable object, and the delta processing below properly iterates over "new objects", so we'll pick it up from there. Closes: #678 Approved by: giuseppe
* static-delta: Pretend that world unreadable objects are new objectsMario Sanchez Prada2016-12-161-0/+35
| | | | | | | | | | | | | | | | | This will prevent including in the delta the bits to update files that are not world readable, so that we don't run into a permissions problem when applying the deltas from a bare-user repository that has a bare repository set as its parent. This is the case for Endless when updating flatpak runtimes, as the temporary directory created in ~/.local/share/flatpak/system-cache will be of type bare-user with its parent set to /var/lib/flatpak which is a bare repository in EOS, as it's shared with the one at /ostree/repo. https://phabricator.endlessm.com/T14159 Closes: #634 Approved by: cgwalters
* [ASAN] delta compilation: More leak fixesColin Walters2016-11-301-13/+14
| | | | | | | | Now that I remembered to do `env G_SLICE=always-malloc`, lots more leaks become apparent. Nothing major. Closes: #598 Approved by: jlebon
* delta compilation: Fix leakAlexander Larsson2016-11-281-11/+11
| | | | | | | We need to ref-sik the new varian for g_autoptr to work Closes: #597 Approved by: cgwalters
* [UBSAN] deltas: Don't call memset(NULL, NULL, 0) with no xattrsColin Walters2016-11-171-0/+3
| | | | | | | | | | | | This is actually fine in practice, but it triggers this `-fsanitize=undefined` warning I saw in the test suite log: ``` src/libostree/ostree-repo-static-delta-compilation.c:160:10: runtime error: null pointer passed as argument 1, which is declared to never be null ``` Closes: #584 Approved by: jlebon
* deltas: Only keep one file open at a time during compilationColin Walters2016-11-071-66/+45
| | | | | | | | | | | | | | | Otherwise it's possible for us to exhaust available file descriptors or (on 32 bit) run up against mmap limits. In the rollsum case, we didn't need to hold open the "from" object at all. And in the bsdiff case, we weren't even looking at either of the files until we started processing. Also, while we have the patient open, switch to using O_TMPFILE if available. Closes: #567 Approved by: giuseppe
* Define an initializer for GVariant{Builder,Dict}Colin Walters2016-10-271-3/+3
| | | | | | | | So we build warning-free on GLib (< 2.50, >= 2.50). This is a band aid until we hard-require >= 2.50. Closes: #547 Approved by: jlebon
* tree-wide: Remove unused variables detected by CLangColin Walters2016-10-271-8/+0
| | | | | | | | | | CLang finds these, whereas GCC treats having `__attribute__((cleanup))` as a use. This obsoletes https://github.com/ostreedev/ostree/pull/411 Closes: #548 Approved by: jlebon
* delta: Unreference files we've processedColin Walters2016-09-081-4/+10
| | | | | | | | This should help avoid address space exhaustion on 32 bit systems, and in general is obviously going to improve efficiency. Closes: #498 Approved by: alexlarsson
* delta: Add missing `goto out` for failure to mmap()Colin Walters2016-09-081-0/+2
| | | | | | | | This was hit in practice when generating a delta for a flatpak app on ARM it looks like. Closes: #497 Approved by: alexlarsson
* deltas: Use F_DUPFD_CLOEXEC properlyDan Nicholson2016-08-101-1/+1
| | | | | | | | | | You need to supply an argument to F_DUPFD_CLOEXEC or fcntl will return EINVAL. Use 3 as the minimum fd number as is standard. Closes: #454 Closes: #448 Approved by: cgwalters
* deltas: Handle cleanup of fd array properlyDan Nicholson2016-08-101-7/+8
| | | | | | | | | | If there's an early error, part_temp_fds will be NULL and dereferencing the len member will segfault. Closes: #454 Closes: #448 Approved by: cgwalters
* lib: Fix a compiler warning introduced from earlier patchColin Walters2016-08-051-1/+1
| | | | | | | | | I think we'd end up closing stdin...yuck. Need to investigate having this be fatal, but CentOS 7 `gcc-4.8.5` doesn't understand `-Werror=int-conversion`. Closes: #430 Approved by: giuseppe
* deltas: Port compilation to libglnx/fd-relativeColin Walters2016-08-031-31/+73
| | | | | | | | | | This was the last use of libgsystem tmpfile APIs. The change here is a bit uglier than it needs to be because we support creating the delta in place as well as in the repo, so we needed to abstract over locations. Closes: #424 Approved by: giuseppe
* libglnx porting: Port away from gs_file_ensure_directory()Colin Walters2016-05-301-1/+2
| | | | | | | | The init-fs code in particular gets (IMO) a lot cleaner being fd relative. Closes: #312 Approved by: giuseppe
* libostree: Fix bracket-missing warningsKrzesimir Nowak2016-05-131-3/+3
| | | | | | | | | | | | | Apparently I got the bracketing wrong in 862e6ecdcc58f025696b1394adfc0fcf7322df23: src/libostree/ostree-repo.c: In function 'ostree_repo_delete_object': src/libostree/ostree-repo.c:3538:11: warning: missing braces around initializer [-Wmissing-braces] g_auto(GVariantBuilder) builder = {0,}; Closes: #298 Approved by: cgwalters
* libostree: Variant-related leak plugs and fixesKrzesimir Nowak2016-05-121-6/+8
| | | | | | | | | | | | | | | | | | | | | | | This tries to avoid leaking GVariantBuilders and GVariants in some situations. The leaks were usually happening when some error occurred or because of unclear variant ownership situation. The former is mostly about making sure that g_variant_builder_clear is called on builders that didn't finish their variant building process. The latter is surely more work - sometimes the result of g_variant_builder_end() should not be passed directly to a function, but rather stored in a g_autoptr(GVariant), sunk and then passed to a function. IMO, with an advent of g_autoptr, GVariants should be always sunk instead of relying on some receiver function sinking it. This would make an easy-to-follow policy of always sinking your variants. Functions could then assume that the passed variant is already sunk. These leaks are still happenning in commands, but they are less harmful, since that code will not be used by some daemon as a library routine. Closes: #291 Approved by: cgwalters
* lib: Fix a typoKrzesimir Nowak2016-04-191-1/+1
| | | | | Closes: #267 Approved by: cgwalters
* static-delta: Put temp files in /var/tmpAlexander Larsson2016-04-141-1/+1
| | | | | | | We may not have write permissions in the current directory. Closes: #259 Approved by: cgwalters
* libglnx porting: Use glnx_set_error_from_errnoColin Walters2016-03-231-1/+1
| | | | | | | | | :warning: There is a notable spiked pit trap here around `posix_fallocate()` and `errno`. This has bit other projects, see e.g. https://github.com/systemd/systemd/commit/7bb87460e691d30c1a7fd23a1a8240776957e05f Otherwise the port was straightforward.
* libglnx porting: gs_transfer_out_value -> g_steal_pointerColin Walters2016-03-181-4/+8
| | | | It's a bit more verbose but...eh.
View Lorry Controller Status