summaryrefslogtreecommitdiff
path: root/src/libostree/ostree-repo-prune.c
Commit message (Collapse)AuthorAgeFilesLines
* tree-wide: Run clang-formatColin Walters2023-05-021-124/+97
| | | | | This is a one-time tree wide reformatting to ensure consistency going forward.
* Strip trailing whitespace on all C filesDan Nicholson2023-02-071-6/+6
| | | | | | | | My editor started following the configuration in .editorconfig and is applying this rule to many files I'm editing. Let's just get this over with and strip everything. This was done like so: git ls-files | grep '\.[ch]$' | xargs sed -ri 's/\s+$//'
* lib/prune: speed up pruning by retrieving only commitsSaqib Ali2022-06-271-5/+24
| | | | | | After landing the new --commit-only functionality, we still noticed exceedingly long pruning times in large repos. Lets add an optimization that will only retrieve commit objects when --commit-only flag is used.
* prune: Also use object set API in `ostree_repo_prune_from_reachable()`Colin Walters2022-06-081-4/+4
| | | | | I missed the second prune path when working on https://github.com/ostreedev/ostree/pull/2635
* repo: Further optimize `ostree_repo_list_objects_set()`Colin Walters2022-06-081-10/+4
| | | | | | | | | | | | In a prior change we discovered that for bad historical reasons libostree was returning a mapping "object type+checksum" => "metadata" but the "metadata" was redundant and pointless. Optimize the prune API to use a (currently internal) object listing API which returns a set, not a map. This allows `GHashTable` to avoid allocating a separate array for the values, neatly cutting memory usage in half (from ~13MB to ~6MB) on my test case of a dry-run prune of a FCOS build.
* src/ostree: Add --commit-only option to ostree pruneSaqib Ali2022-02-251-27/+56
| | | | | | | | | | | | Recently we have noticed exceedingly long execution times for multiple invocations of ostree prune. This is a result of calculating full reachability on each invocation. The --commit-only flag provides an alternative strategy. It will only traverse and delete commit objects to avoid the more expensive reachability calculations. This allows us to chain multiple --commit-only commands cheaply, and then follow with a more expensive ostree prune invocation at the end to clean up orphaned meta and content objects.
* Update FSF license notices to use URL instead of addressJoseph Marrero2021-12-071-3/+1
|
* lib/prune: Avoid unnecessary object serializationDan Nicholson2021-11-181-12/+9
| | | | | | `repo_prune_internal` was deserializing each object and passing the components to `maybe_prune_loose_object`, which promptly reserialized it.
* repo: Make locking APIs publicColin Walters2021-06-051-4/+4
| | | | | | | | Doing anything even somewhat sophisticated requires this; turns out our own `ostree prune` CLI wants this, e.g. https://github.com/ostreedev/ostree/issues/2337 Closes: https://github.com/ostreedev/ostree/issues/2286
* lib: add minimum version to various symbols based on libostree-released.symFelix Krull2019-05-191-0/+2
| | | | | Closes: #1861 Approved by: cgwalters
* lib/prune: Don't modify dirent->d_name in placeMatthew Leeds2018-06-151-6/+9
| | | | | | | | | | | | | | | | | Currently when I run `ostree prune` it hits a seg fault when the hash_func is used (in this case g_str_hash) from the call stack _ostree_repo_prune_tmp() -> g_hash_table_contains() -> g_hash_table_lookup_node(). So the key, in this case dent->d_name, must be corrupt in some way. glnx_dirfd_iterator_next_dent() uses readdir() to get the dirent struct. And according to the man page for readdir(3), "POSIX.1 explicitly notes that this field should not be used as an lvalue" (in reference to d_name). So this commit avoids modifying d_name in place and copies it instead. This seems to avoid the seg fault. Closes: #1627 Approved by: jlebon
* lib: Add a public helper method for pruning to find all ref'd commitsColin Walters2018-05-241-28/+59
| | | | | | | | | Prep for reworking how we do sysroot cleanup. We're going to start doing more lowlevel pruning work there, and I wanted to avoid duplicating the ref enumeration. Closes: #1566 Approved by: jlebon
* lib: Use `Locking:` term in docsColin Walters2018-05-021-3/+3
| | | | | | | | | This is easier to `git grep` etc. versus ad-hoc English. Although we still have some English for the prepare_transaction/commit which acquire/release in separate phases. Closes: #1572 Approved by: jlebon
* lib/repo: Enable locking by default, but drop external APIColin Walters2018-04-301-6/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The code has been sitting around for a while but since I disabled it by default, I doubt anyone is really using it or relying on it. This patch and turns on locking by default, and also drops the API which was only public in the experimental API builds. Conceptually these are two distinct things, and we may actually want to split up the patches. I don't think this will break anyone, but it's hard to say for sure. It's also going to be hard to find out until we actually release I suspect... But anyone who is broken should be able to add `locking=false` into their repo config. On the flip side Endless has been shipping with this enabled and it is reported to help. The reason to drop the APIs: I'm a bit concerned about the interactions over time between libostree's use of the API and any apps that start using it. For example, if an app specifies a SHARED lock in their code, then later internally we decide to temporarily grab an `EXCLUSIVE`, but the app had a second thread/process that was `EXCLUSIVE` already, and that process was waiting on the first bit of code, then we could deadlock. I can't think of a real world situation where this would happen yet though. We are likely to in the future have say `fsck` take an external lock, `checkout` grab a shared one, etc. Closes: #1555 Approved by: jlebon
* commit: add logic for .payload-linkGiuseppe Scrivano2018-03-071-4/+40
| | | | | | | | | | | | | | | | | | | | | | | | When a new object is added to the repository, create a $PAYLOAD-SHA256.payload-link symlink file as well. The target of the symlink is the checksum of the object that was added the repository. Whenever we add a new object file, in addition to lookup if the file is already present with the same checksum we also check if an object with the same payload is in the repository. If a file with the same payload is already present in the repository, we copy it with `glnx_regfile_copy_bytes` that internally attempts to create a reflink (ioctl (..., FICLONE, ..)) to the target file if the file system supports it. This enables to have objects that share the payload but have a different inode and xattrs. By default the payload-link-threshold value is G_MAXUINT64 that disables the feature. Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com> Closes: #1443 Approved by: cgwalters
* lib/repo: Do account for size with prune --no-pruneColin Walters2018-03-051-7/+10
| | | | | | | | | | | | | | I think this got changed in a refactor. We definitely want to total up the amount of space that *would* be freed even with `--no-prune` AKA `OSTREE_REPO_PRUNE_FLAGS_NO_PRUNE`. It's actually a bit terrifying this is apparently the first test case for the `--no-prune` option... Closes: https://github.com/ostreedev/ostree/issues/1480 Closes: #1483 Approved by: jlebon
* Add SPDX-License-Identifier to source filesMarcus Folkesson2018-01-301-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | SPDX License List is a list of (common) open source licenses that can be referred to by a “short identifier”. It has several advantages compared to the common "license header texts" usually found in source files. Some of the advantages: * It is precise; there is no ambiguity due to variations in license header text * It is language neutral * It is easy to machine process * It is concise * It is simple and can be used without much cost in interpreted environments like java Script, etc. * An SPDX license identifier is immutable. * It provides simple guidance for developers who want to make sure the license for their code is respected See http://spdx.org for further reading. Signed-off-by: Marcus Folkesson <marcus.folkesson@gmail.com> Closes: #1439 Approved by: cgwalters
* lib/repo: Add an API to mark a commit as partialColin Walters2017-12-141-11/+1
| | | | | | | | | | | For the [rpm-ostree jigdo ♲📦](https://github.com/projectatomic/rpm-ostree/issues/1081) work. We're basically doing "pull" via a non-libostree mechanism, and this should be fully supported. As I mentioned earlier we should try to have `ostree-repo-pull.c` only use public APIs; this gets us closer to that. Closes: #1376 Approved by: jlebon
* lib/prune: Take exclusive repository lockDan Nicholson2017-12-051-0/+25
| | | | | | | | | Add exclusive repository locking to all the pruning entry points. This ensures that objects and deltas will not be removed while another process is writing to the repository. Closes: #1343 Approved by: cgwalters
* tree-wide: Remove Emacs modelinesColin Walters2017-09-211-2/+1
| | | | | | | | | We added a `.dir-locals.el` in commit: 9a77017d87b74c5e2895cdd64ad098018929403f There's no need to have it per-file, with that people might think to add other editors, which is the wrong direction. Closes: #1206 Approved by: jlebon
* tree-wide: Use helpers for unlinkat()Colin Walters2017-09-071-9/+3
| | | | | | | | | | | | We have `ot_ensure_unlinked_at()` for the "ignore ENOENT" case, and `glnx_unlinkat()` otherwise. Port all in-tree callers to one or the other as appropriate. Just noticed an unprefixed error in the refs case and decided to do a tree-wide check. Closes: #1142 Approved by: jlebon
* lib/repo-refs: Include remote refs when using collectionsMatthew Leeds2017-08-241-1/+1
| | | | | | | | | | | | | | | | | | | | | | When working with collections it can be useful to see remote refs rather than just local and mirrored ones. This commit changes the "ostree refs -c" output to include remote refs, and includes remote refs with collection IDs in summary file generation as well. The former behavior is consistent with how "ostree refs" works, and the latter behavior is useful in facilitating P2P updates even when mirrors haven't been configured. To accomplish this, OstreeRepoListRefsExtFlags was extended with an EXCLUDE_REMOTES flag. This was done rather than an INCLUDE_REMOTES flag so that existing calls to ostree_repo_list_refs_ext continue to have the same behavior. This flag was added to ostree_repo_list_collection_refs (which is an experimental API break). Also, add unit tests for the "refs -c" and summary file behavior, and update relevant tests. Closes: #1069 Approved by: cgwalters
* codebase: start using GLNX_HASH_TABLE_FOREACH macrosJonathan Lebon2017-06-281-23/+5
| | | | | | | | | | | Use the new macros introduced recently in libglnx to make iterating over hash tables cleaner. This is just a start, it does not migrate the whole tree. Update submodule: libglnx Closes: #971 Approved by: cgwalters
* lib/refs: Add methods for setting/listing collection–refsPhilip Withnall2017-06-261-1/+22
| | | | | | | | | | | | | These are tuples of (collection ID, ref name) which are a globally-unique form of local ref. They use OstreeCollectionRef as an identifier, and hence need to be accessed using new API, as the existing API uses string identifiers and sometimes accepts refspecs. Remote names are not supported as part an OstreeCollectionRef. Signed-off-by: Philip Withnall <withnall@endlessm.com> Closes: #924 Approved by: cgwalters
* lib: Add "open dfd iter handling noent" helper, port tree-wideColin Walters2017-05-161-12/+7
| | | | | | | | | Follow up to a previous patch that addressed a double-close; I realized we already had a helper for doing "open dfd iter, do nothing if we get ENOENT". Raise it to libotuil, and port all consumers. Closes: #863 Approved by: jlebon
* lib/prune: Complete porting to new code styleColin Walters2017-05-111-60/+30
| | | | | | | | Only non-mechanical bit here was creating a local autoptr for a bit where we'd previously done an unref for a struct member. Closes: #847 Approved by: jlebon
* Add support for more selective pruningColin Walters2017-01-191-44/+114
| | | | | | | | | | | | | | | | | | | | | | | | There are use cases for having a single repo with branches with different lifecycles; a simple example of what I was trying to do in CentOS Atomic Host work is have "stable" and "devel" branches, were we want to prune devel, but retain *all* of stable. This patch is split into two parts - first we add a low level "delete all objects not in this set" API, and change the current prune API to use this. Next, we move more logic into the "ostree prune" command. This paves the way for demonstrating how more sophisticated algorithms/logic could be developed outside of the ostree core. Also, the --keep-younger-than logic already lived in the commandline, so it makes sense to keep extending it there. Closes: https://github.com/ostreedev/ostree/issues/604 Closes: #646 Approved by: jlebon
* Fix pruning of partial commitsAlexander Larsson2016-10-241-40/+4
| | | | | | | | | | | | | | | | | | | | If we have a partial commit it is not an error for a dirmeta to be missing (in fact, that is likely), so instead of returning a not-found error from ostree_repo_traverse_commit() we ignore the error and continue. In particular, this means we don't stop early at the first missing dirmeta, which previously caused ostree_repo_prune() to thing the dirmetas after that to be unreached and thus purged. Also, we remove the special casing in ostree_repo_prune() to not report errors for commitpartial, because these should not be reported anymore. This fixes https://github.com/ostreedev/ostree/issues/541 Closes: #542 Approved by: cgwalters
* lib: Add an API to list only "our" objects, fix prune to use itColin Walters2016-08-251-2/+2
| | | | | | | | | | | | When doing a prune, we should not try to delete objects in parent repos, since it'll fail. There is a bigger discussion about the semantics of `parent=` to be had, but this will fix trying to use `ostree prune --repo=/ostree/repo/extensions/rpmostree/pkgcache`. Closes: https://github.com/ostreedev/ostree/issues/467 Closes: #471 Approved by: jlebon
* Add cache_dir_fd to OstreeRepoAlexander Larsson2016-04-141-1/+4
| | | | | | | This will allow us later to easily swap out the cache dir. Closes: #250 Approved by: cgwalters
* core: Add verbose messages for pruningDan Nicholson2016-03-261-0/+7
| | | | | | | | | | When prune fails, it can be really difficult to figure out why. This at least lets you know which objects are being considered. https://bugzilla.gnome.org/show_bug.cgi?id=764006 Closes: #224 Approved by: cgwalters
* prune: Don't fail on partial commitsDan Nicholson2016-03-231-6/+42
| | | | | | | | | If a commit only pull has been done, then the commit object exists in the object store in addition to the commitpartial file. Traversing this partial commit will likely fail, but that's expected. If traverse returns a G_IO_ERROR_NOT_FOUND in this case, continue with pruning. https://bugzilla.gnome.org/show_bug.cgi?id=764091
* libglnx porting: Use glnx_shutil_rm_rf_at()Colin Walters2016-03-231-1/+1
| | | | | | | | | | In some cases (such as `ostree-sysroot-cleanup.c`), the surrounding code would be substantially cleaner if it was also ported to fd-relative, but I'm going to do that in a separate patch. That way these patches are easier to review for mechanical correctness. I used an Emacs keyboard macro as the poor man's [Coccinelle](http://coccinelle.lip6.fr/).
* prune: delete all cached summaries filesGiuseppe Scrivano2016-03-151-0/+65
| | | | Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
* repo: new function ostree_repo_prune_static_deltasGiuseppe Scrivano2015-12-181-41/+74
| | | | | | | Extract existing code from ostree_repo_prune and add an argument COMMIT, that controls which commit purge. If not set, the old behavior is kept. Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
* Use g_autoptr(GVariant) instead of gs_unref_variantMatthew Barnes2015-05-061-1/+1
|
* Use g_autoptr(GPtrArray) instead of gs_unref_ptrarrayMatthew Barnes2015-05-061-1/+1
|
* Use g_autoptr(GHashTable) instead of gs_unref_hashtableMatthew Barnes2015-05-061-2/+2
|
* Use g_autofree instead of gs_freeMatthew Barnes2015-05-061-2/+2
|
* core: Cleanup commitpartial file with fd-relative lookupsColin Walters2015-05-061-3/+9
| | | | | | | | First, this is just a general continuation of the `GFile -> openat` transition. Second, it's preparatory work for fsck to gain awareness of partial commits.
* deltas: Prune deltas when the corresponding "to" commit vanishesColin Walters2015-02-161-0/+44
| | | | We want prune to actually give you back disk space when using deltas.
* repo-pull: Allow pulling only one directoryAnne LoVerso2014-08-201-0/+23
| | | | | | | | | Changes the pull API to allow pulling only a single directory instead of the whole deployment. This option is utilized by the check-diff option in rpm-ostree. Add a new state directory to hold <checksum>.commitpartial files, so we know that we've only downloaded partial state.
* core: Unify object deletion code with pruneColin Walters2014-07-201-20/+8
| | | | | | | | | | The prune API duplicated logic to delete objects, and furthermore the core API to delete an object didn't clean up detached metadata. Fix the duplication by doing the obvious thing: prune should call _delete. https://bugzilla.gnome.org/show_bug.cgi?id=733452
* repo: Tweak traversal APIColin Walters2013-10-091-3/+3
| | | | | It's convenient for bindings if we have a version that doesn't mutate the hash table, because they pass temporary hash tables as input.
* core: Add detached metadata, readd metadata to commitsColin Walters2013-09-091-0/+7
| | | | | | | | | | | | | | | | Previously I thought we'd have to ditch the current commit format to avoid a{sv} due to See https://bugzilla.gnome.org/show_bug.cgi?id=673012 But I realized that we don't really have to care about unpacking/repacking commit objects, so let's just re-expose the existing metadata a{sv} in commits in the API. Also, add support for "detached" metadata that can be updated at any time post-commit. This is specifically designed for GPG signatures. https://bugzilla.gnome.org/show_bug.cgi?id=707379
* Fix warnings about unused variablesTobias Hunger2013-08-301-1/+0
|
* libostree: Extend gtk-doc coverage for refs and prune APIsColin Walters2013-08-191-6/+31
|
* Use { 0, } for structure initialization rather than memset()Colin Walters2013-08-181-3/+1
| | | | | | | It's cleaner, safer, and I had a totally wrong idea stuck in my head about why memset() should be used. https://bugzilla.gnome.org/show_bug.cgi?id=705968
* core: Finish making object path API privateColin Walters2013-07-311-2/+2
| | | | This allows us to more easily change the internals later.
* Install a shared libraryColin Walters2013-07-261-1/+2
| | | | | This required a fair bit of surgery because previously ostree.h included otutil.h, but that's supposed to be a private library.
View Lorry Controller Status