| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
| |
Both pulling and pushing INFO messages are now status messages.
Calls to the messaging API through `self.context.message()` have now
been switched to `element.info`.
|
| |
|
|
|
|
|
|
| |
Pulled/Pushed messages will no longer be produced from within
element.py, instead they will be produced during CasCache push() and
pull() appropriately.
Message consistency has also been improved.
|
| |
|
|
|
| |
This removes the timed_activity for an element _push action. This is
unnecessary as the job is already being timed elsewhere.
|
| |
|
|
|
| |
Adds a test to ensure that BuildStream alerts the user of a skipped push
when the remote already has the artifact cached.
|
| |
|
|
|
|
|
|
|
|
| |
The SKIPPED message type is now used to indicate the end of a task which
was successful without having to perform the given task.
This overhauls the use of `Queue.done()` and therefore queues do not
need to provide a processed/skipped return value from `done()`. Instead
this is replaced with the action of raising a `SkipJob` exception from
within `Queue.process()`.
|
| |\
| |
| |
| |
| | |
_frontend/status.py: Completely remove the blessings dependency from BuildStream
See merge request BuildStream/buildstream!809
|
| |/
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This actually improves reliability of the status bar because we
now disable it completely in the case that not all of the terminal
escape sequences are supported on the given terminal.
This replaces the few functions we were using, to move the cursor
up one line, move it to the beginning of the line, and to clear a
line, with low level functions provided by the curses module in
the standard library.
This change makes it easier for downstream distro package maintainers
to package BuildStream, particularly on Fedora.
Asides from changing _frontend/status.py, this commit includes the
following changes:
* _frontend/app.py: Use python isatty() function to determine if
we are connected to a tty, instead of relying
on blessings.
* setup.py: Remove the dependency on blessings.
|
| |\
| |
| |
| |
| | |
Fix artifact config crash (1.2)
See merge request BuildStream/buildstream!805
|
| | |
| |
| |
| |
| | |
Test that we get the expected error when configuring a client-cert
without client-key, or the inverse.
|
| |/
|
|
|
|
|
|
|
|
| |
mal-specified
When configuring a push remote and specifying either the client-cert
or the client-key, then both must be specified. This ensures we
get an informative error instead of a stack trace and BUG.
Fixes issue #625
|
| |\
| |
| |
| |
| | |
Fix override options 1.2
See merge request BuildStream/buildstream!803
|
| | |
| |
| |
| | |
This is a regression test for issue #658
|
| |/
|
|
|
|
|
|
|
| |
This ensures that option expressions are resolved in the project
level overrides before attempting to composite them on the instantiated
elements. Seems this is a regression from introducing the include
directive.
This fixes issue #658
|
| |
|
|
|
|
| |
Somehow I missed this when originally forking the file from the click
library, now noticing that we should have followed what was written
in: https://github.com/pallets/click/blob/master/LICENSE
|
| |\
| |
| |
| |
| | |
Backport of !678 (Add validation of configuration variables) to 1.2 branch.
See merge request BuildStream/buildstream!789
|
| | | |
|
| | |
| |
| |
| | |
Ensure that protected variables are not being redefined by the user.
|
| | |
| |
| |
| |
| | |
And remove then from the defaults as they are dynamically set by
BuildStream.
|
| |/
|
|
| |
Setting "max-jobs" won't be allowed anymore in a following commit.
|
| |\
| |
| |
| |
| | |
Don't delete required artifacts when tracking is enabled
See merge request BuildStream/buildstream!794
|
| | |
| |
| |
| |
| |
| | |
Same test as test_never_delete_required(), except that this test ensures
that we never delete required artifacts when their cache keys are
discovered dynamically during the build.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
* create_element_size()
Now uses a git Repo object instead of a local source, and
returns the repo.
* update_element_size()
Added this function which can now resize the expected output
of an element generated with create_element_size(), useful
to allow testing sized elements with the tracking feature.
|
| | |
| |
| |
| |
| | |
This allows one to modify a file in an existing git repo,
as opposed to adding a new one.
|
| | |
| |
| |
| |
| |
| |
| | |
These tests were not checking that we fail for the expected reasons.
Added `res.assert_task_error(ErrorDomain.ARTIFACT, 'cache-too-full')`
where we expect to fail because the cache is too full.
|
| | |
| |
| |
| |
| |
| |
| | |
This commit renames test_never_delete_dependencies() to
test_never_delete_required(), renders the test more readable by renaming
some elements and reordering some statements and makes the comments more
straight forward and accurate.
|
| |/
|
|
|
|
|
|
|
| |
This was previously append_required_artifacts(), which presumed that
we knew at startup time what the cache keys of all elements in the
loaded pipeline would be.
This fixes unexpected deletions of required artifacts when
dynamic tracking is enabled with `bst build --track-all target.bst`
|
| |\
| |
| |
| |
| | |
Tristan/fix cache exclusivity 1.2
See merge request BuildStream/buildstream!779
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This does a lot of house cleaning, finally bringing cache
cleanup logic to a level of comprehensibility.
Changes in this commit include:
o _artifactcache/artifactcache.py: _cache_size, _cache_quota and
_cache_lower_threshold are now all private variables.
get_approximate_cache_size() is now get_cache_size()
Added get_quota_exceeded() for the purpose of safely checking
if we have exceeded the quota.
set_cache_size() now asserts that the passed size is not None,
it is not acceptable to set a None size cache anymore.
o _artifactcache/cascache.py: No longer set the ArtifactCache
'cache_size' variable violently in the commit() method.
Also the calculate_cache_size() method now unconditionally
calculates the cache size, that is what it's for.
o _scheduler/jobs/cachesizejob.py & _scheduler/jobs/cleanupjob.py:
Now check the success status. Don't try to set the cache size
in the case that the job was terminated.
o _scheduler/jobs/elementjob.py & _scheduler/queues/queue.py:
No longer passing around the cache size from child tasks,
this happens only explicitly, not implicitly for all tasks.
o _scheduler/queues/buildqueue.py & _scheduler/scheduler.py:
Use get_quota_exceeded() accessor
This is a part of #623
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
The artifact cache is anyway a singleton, the Element itself
has an internal handle on the artifact cache because it is
needed very frequently; but an artifact cache is not element
specific and should not be looked up by surrounding code
on a per element basis.
Updated _scheduler/queues/queue.py, _scheduler/queues/buildqueue.py
and _scheduler/jobs/elementjob.py to get the artifact cache directly
from the Platform
|
| | |
| |
| |
| |
| |
| |
| |
| | |
This was previously poking directly at the Platform._instance.
Also, use the name 'artifacts' to hold the artifact cache to
be consistent with other parts of the codebase, instead of
calling it 'cache'.
|
| | |
| |
| |
| |
| |
| |
| |
| | |
This was previously poking directly at the Platform._instance.
Also, use the name 'artifacts' to hold the artifact cache to
be consistent with other parts of the codebase, instead of
calling it 'cache'.
|
| | |
| |
| |
| |
| |
| |
| |
| | |
There is no justification to hold onto this state here.
Instead, just make `Element._assemble()` return the size of the
artifact it cached, and localize handling of that return value in
the BuildQueue implementation where the value is observed.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Previously, the API contract was to expose the estimated_size variable
on the ArtifactCache instance for all to see, however it is only relevant
to the ArtifactCache abstract class code. Subclasses were informed to
update the estimated_size variable in their calculate_cache_size()
implementation.
To untangle this and hide away the estimated size, this commit
does the following:
o Introduces ArtifactCache.compute_cache_size() API for external
callers
o ArtifactCache.compute_cache_size() calls the abstract method
for the CasCache subclass to implement
o ArtifactCache.compute_cache_size() updates the private
estimated_size variable
o All direct callers to ArtifactCache.calculate_cache_size(),
have been updated to use the ArtifactCache.compute_cache_size()
method instead, which takes care of updating anything local
to the ArtifactCache abstract class code (the estimated_size)
|
| | |
| |
| |
| |
| |
| |
| |
| |
| | |
The ArtifactCache._local variable used to exist in order to
use a special hack to allow absolute paths to a remote artifact
cache, this was all for the purpose of testing.
This has all gone away with the introduction of CAS, leaving behind
a stale variable.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Here we have a very private looking _check_cache_size_real() function
which no-one would ever want to call from outside of the _scheduler,
especially given it's `_real()` prefix we should look for another
outward facing API to use.
However this is not private to the scheduler, and is intended to
be called by the `Queue` implementations.
o Renamed this to check_cache_size()
o Moved it to the public API section of the Scheduler object
o Added the missing API documenting comment
o Also added the missing API documenting comment to the private
`_run_cleanup()` callback which runs in response to completion
of the cache size calculation job.
o Also place the cleanup job logs into a cleanup subdirectory,
for better symmetry with the cache_size jobs which now have
their own subdirectory
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
The artifact cache provides the following public methods for
external callers, but was hiding them away as if they are private.
o ArtifactCache.add_artifact_size()
o ArtifactCache.set_cache_size()
Mark these properly public
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This runs after every pull, and does not need the cache exclusively,
only the cleanup job requires the cache exclusively.
Without this, every time a cache_size job is queued, all pull and
build jobs need to complete before cache_size job can run exclusively,
which is not good.
This is a part of #623
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This means there is no cap for shared resource requests.
Together with the previous commit, this causes the cleanup
job and the pull/build jobs to all require unlimited shared
access to the CACHE resource; while only the cleanup job
requires exclusive access to the resource.
This is a part of #623
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
the CACHE
This is what the whole resource.py thing was created for, the
cleanup job must have exclusive access to the cache, while the pull
and build jobs which result in adding new artifacts, must only require
shared access.
This is a part of #623
|
| |/ |
|
| | |
|
| |\
| |
| |
| |
| | |
Updating docs session files before the release
See merge request BuildStream/buildstream!774
|
| | |
| |
| |
| |
| |
| |
| | |
We regenerate these with releases so that packagers dont
need to rebuild them, because rebuilding these sessions requires
external resources and we expect packagers to build the package
in restricted environments without internet access.
|
| |/
|
|
|
| |
This is a workaround for a bug that this regenerated file causes
changes to the git repository while building documentation
|
| |\
| |
| |
| |
| | |
Source fetcher changes 1.2
See merge request BuildStream/buildstream!773
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Plugin.configure()
This cannot test for unaliased URLs, as those can be discovered later
on outside of user provided element configuration; at least we
assert that if an alias was used, we have seen it at load time.
This will cause a BUG to occur for a plugin which marks an aliased
URL (or attempts to translate one) outside of `Plugin.configure()`,
if that URL was not previously seen.
This is a part of #620
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
The Source must now mention whether the marked or translated
URL is "primary" or not. Even when a Source may have multiple
URLs, the auxilliary URLs are derived from the primary one, not
only is this true for git, but it is mandated by our tracking
API which assumes there is a primary URL.
This adjusts the `git` source and the test `fetch_source.py` source
to behave properly and advertize it's primary URL properly.
This is a part of #620
|
| | |
| |
| |
| |
| |
| |
| |
| |
| | |
Keeps track of whether the plugin is currently being configured.
Adjusted Element and Source classes to call _configure() in place
of calling configure() directly.
This is a part of #620
|
| | |
| |
| |
| |
| |
| |
| | |
A download URL must be interpreted by the core at `Plugin.configure()`
time, even if only employed later on.
This is a part of #620
|
