| Commit message (Collapse) | Author | Age | Files | Lines |
|\
| |
| |
| |
| | |
Switch to roaring bitmaps for the loader dependency caches
See merge request BuildStream/buildstream!1128
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
The dependency cache in LoadElement causes the peak RAM consumption of
the loader to be exceedingly large, upwards of 25GB for a test of a
Debian stack. By switching from the old dict cache to a roaring bitmap
based cache, we reduce the cost of the loader cache in RAM terms to
around 5.5GB for the same stack, which makes it plausible for the short
term.
Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>
|
|/
|
|
|
|
|
| |
In order to support use of roaring bitmaps in the loader, we need to
depend on it here.
Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>
|
|\
| |
| |
| |
| | |
Add --remote, -r option to bst build, inline with pull & push
See merge request BuildStream/buildstream!1119
|
|/
|
|
|
|
| |
Providing a remote will limit build's pull/push remote actions to
the given remote specifically, ignoring those defined via user or
project configuration.
|
|\
| |
| |
| |
| | |
Fail when we explictly try to include/exclude non-existent domains in a filter element
See merge request BuildStream/buildstream!1117
|
|/
|
|
|
|
|
|
|
| |
This patch also uncovered the fact that our test_filter_deps_ok() test
has been inaccurate. Thus the element built in this test
(deps-permitted.bst) has been modified so that it build depends on the
input.bst element, as it should.
tests/filter.py: Ensure deps_ok test passes
|
|\
| |
| |
| |
| |
| |
| | |
Improve our filter documentation
Closes #278
See merge request BuildStream/buildstream!1112
|
| | |
|
|/ |
|
|\
| |
| |
| |
| | |
tests/cachekey: Test cache keys are independent of target elements
See merge request BuildStream/buildstream!1123
|
|/ |
|
|\
| |
| |
| |
| | |
Cleanup loader by linking LoadElements sooner
See merge request BuildStream/buildstream!1122
|
| |
| |
| |
| | |
This simplifies the loading
|
|/
|
|
|
|
| |
This removes the need for the 'Dependency' list to then be matched
with the corresponding LoadElement and will allow iterating the
graph more easily
|
|\
| |
| |
| |
| | |
testutils/site.py: Support parsing more exotic git versions
See merge request BuildStream/buildstream!1118
|
|/
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
We use output of `git --version` to determine if we can run some tests
that rely on features from newer git versions. Usually, we expect the
output to be like:
git version 2.17.2
On some platforms, like MacOS, there could be a suffix after the version
string, so that it looks something like:
git version 2.17.2 (Apple Git-113)
This causes things to fail like so:
ValueError: invalid literal for int() with base 10: '2 (Apple Git-113)\n'
Fix logic around `HAVE_OLD_GIT` such that we split the output of
`git --version` without limit on how many times we split. Previously we
used to split only twice so the suffixes like `(Apple Git-113)` are not
part of the parsed version.
|
|\
| |
| |
| |
| | |
tox.ini: Specify minimum version of click-man
See merge request BuildStream/buildstream!1120
|
|/
|
|
|
|
|
|
|
|
|
| |
`click-man` versions < 0.3.0 do not properly support multiple
entrypoints. Since this was added to `tox` after `0.3.0` was released,
`tox` should never be pulling older versions. But, let's add it here for
documentation purposes.
See
https://gitlab.com/BuildStream/buildstream/merge_requests/1107#note_135187046
for some background on this.
|
|\
| |
| |
| |
| |
| |
| | |
filter.py: don't recurse when staging dependencies
Closes #883
See merge request BuildStream/buildstream!1110
|
|/
|
|
|
|
|
| |
Also bump the element's version so cached artifacts would be
invalidated.
Fixes #883
|
|\
| |
| |
| |
| | |
BREAK:remove unconditional 'are you sure?' prompts
See merge request BuildStream/buildstream!1061
|
|/
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This is a breaking change, as it affects behaviour that people might be
relying on. An entry has been added to NEWS.
As proposed on the mailing list, this change removes the unconditional
prompts on:
o: bst workspace reset
o: bst workspace close --remove-dir
If interactive, these commands would always interrupt you with a prompt
like this:
This will remove all your changes, are you sure?
This seems like it may just save someone's work some time. It may also
condition folks to hit 'y' quickly without thinking.
This change also makes the non-interactive behaviour consistent with the
interactive behaviour in the default case. There is also the case of the
prompt configured by 'really-workspace-close-project-inaccessible',
which may be tackled in later work.
This change also removes the new config options to suppress those
prompts, and their associated news entry.
The relevant bit of the mailing list conversation is here:
https://mail.gnome.org/archives/buildstream-list/2018-December/msg00106.html
The issue to make interactive and non-interactive behaviour consistent
is here:
https://gitlab.com/BuildStream/buildstream/issues/744
|
|\
| |
| |
| |
| |
| |
| | |
Make sure testing cache directory exists
Closes #873
See merge request BuildStream/buildstream!1092
|
|/
|
|
| |
Fixes #873
|
|\
| |
| |
| |
| |
| |
| | |
Fix type of error codes in CAS server
Closes #882
See merge request BuildStream/buildstream!1099
|
|/
|
|
| |
Fixes #882.
|
|\
| |
| |
| |
| |
| |
| | |
Fix crash when spawned job completes very fast
Closes #857
See merge request BuildStream/buildstream!1095
|
|/
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Job can complete before we return from `Job.span()` to
`Scheduler._spawn_job()`, so that `_active_jobs` would not yet contain
the job.
This would print a stack on the console and try to run a second time
the job which can have unexpected effects.
In order to reproduce the issue, in
`buildstream/_scheduler/jobs/job.py`, in `Job.spawn`,
add a call to `time.sleep()` right before call to
`asyncio.get_child_watcher()`.
This fixes issue #857.
|
|\
| |
| |
| |
| |
| |
| | |
requirements/requirements.in: update minimum versions
Closes #884
See merge request BuildStream/buildstream!1114
|
| |
| |
| |
| | |
This is needed since 629a6e524, and was lost in the conversion to requirements.in
|
|/
|
|
| |
This is needed since 0f2bc3754
|
|\
| |
| |
| |
| |
| |
| | |
Download buildtrees on demand for bst shell --use-buildtree
Closes #829
See merge request BuildStream/buildstream!1050
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Provide bst shell --use-buildtree the ability to attempt to
acquire missing buildtrees, given respective option, user
pull-buildtree context and remote availability.
_frontend/cli.py: Refactor logic for determining --use-buildtree
option with given opportunity to attempt pulling a non-local
buildtree. Element loaded with artifact_config to allow remote
querying.
_stream.py: With given user option and element state, construct
PullQueue to fetch remote buildtree. Continue or Error without
buildtree if cannot be attained.
tests/integration/build-tree.py: Update to support new usecases
|
|/
|
|
|
| |
use_artifact_config added as an optional default arg, allowing
for loading of given elements artifact remote config.
|
|\
| |
| |
| |
| | |
Fixup refs to 'bst track' and 'bst fetch'
See merge request BuildStream/buildstream!1086
|
| |
| |
| |
| |
| | |
Now that 'bst fetch' is obsolete, change guidance to refer to the
replacement 'bst source fetch' instead.
|
| |
| |
| |
| |
| | |
Now that 'bst track' is obsolete, change guidance to refer to the
replacement 'bst source track' instead.
|
|/ |
|
|\
| |
| |
| |
| |
| |
| | |
Generate man pages using tox & update them
Closes #880 and #881
See merge request BuildStream/buildstream!1107
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
If the `man` directory is empty, then it won't be copied in the source
distribution, and `list_man_pages()` will throw an exception when trying
to list files in a non-existent directory. This prevents us from
installing the BuildStream package when the man pages are not there.
The most common use-case for this is when we want to re-generate the man
pages but want to install the package before re-generating them.
|
| |
| |
| |
| |
| |
| | |
Simplify our docs, by requesting users to run `tox -e man` to update man
pages, instead of manually installing `click-man` and running the
command manually.
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Notable changes:
* New `source` and `artifact` command groups
* Man pages corresponding to obsolete commands, that were marked as
hidden in Click, have now been removed.
Fixes #881.
|
|/
|
|
|
|
| |
Previously, one had to manually install `click-man` package, and
remember to run the correct command. Now, we can simply run `tox -e man`
to update the man pages.
|
|\
| |
| |
| |
| |
| |
| | |
_artifactcache.py: Don't require the quota to be available on disk.
Closes #869 and #733
See merge request BuildStream/buildstream!1106
|
|/
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Instead only rely on the headroom to be enough to protect against
out of space conditions. The headroom can become configurable as
a separate step is required.
The changes to achieve this are:
* Rename ArtifactCache.has_quota_exceeded() to ArtifactCache.full().
* ArtifactCache.full() now also reports True if the available
space on the artifact cache volume is smaller than the headroom.
This ensures jobs get triggered to cleanup the cache when
reaching the end of the disk.
* When loading the artifact quota, it is now only an error if
the quota exceeds the overall disk space, not if it does not
fit in the available space.
It is still a warning if the quota does not fit in the
available space on the artifact cache volume.
* Updated scheduler.py and buildqueue.py for the API rename
* tests: Updated the artifactcache/expiry.py test for its
expectations in this regard.
Added a new test to test an error when quota was specified to
exceed total disk space, and adjusted the existing tests to
expect a warning when the quota does not fit in the available
space.
This fixes issue #733 and #869.
|
|\
| |
| |
| |
| | |
Add new 'scheduler' and 'load-selection' profiling topics
See merge request BuildStream/buildstream!1088
|
| | |
|
| |
| |
| |
| |
| | |
profile_start() and profile_end() calls have been added to
Stream.load_selection()
|
|/
|
|
|
| |
profile_start() and profile_end() calls have been incorporated into
Scheduler.run()
|