| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
This removes the stateful Pipeline object and leaves behind only a toolbox
of functions for constructing element lists, such as _pipeline.get_selection()
and _pipeline.except_elements(), and some helpers for asserting element states
on lists of elements.
This makes it easier for Stream to manage it's own internal state, so that
Stream can more easily decide to operate without hard requiring a Project
instance be available.
This also adds type annotations to the new version of _pipeline.py.
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
Use Messenger convinence functions instead.
|
| |
|
|
|
|
|
|
|
| |
Instead of having _pipeline.py implement load_artifacts() by calling
_project.py's other implementation of load_artifacts(), instead just
implement _load_artifacts() directly in _stream.py.
This of course removes the load_artifacts() implementations from
_pipeline.py and _project.py.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This is a large breaking change, a summary of the changes are that:
* The Scope type is now private, since Element plugins do not have
the choice to view any other scopes.
* Element.dependencies() API change
Now it accepts a "selection" (sequence) of dependency elements, so
that Element.dependencies() can iterate over a collection of dependencies,
ensuring that we iterate over every element only once even when we
need to iterate over multiple element's dependencies.
The old API is moved to Element._dependencies() and still used internally.
* Element.stage_dependency_artifacts() API change
This gets the same treatment as Element.dependencies(), and the old
API is also preserved as Element._stage_dependency_artifacts(), so
that the CLI can stage things for `bst artifact checkout` and such.
* Element.search() API change
The Scope argument is removed, and the old API is preserved as
Element._search() temporarily, until we can remove this completely.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This prepares the ground for policing the dependencies which are visible
to an Element plugin, such that plugins are only allowed to see the
elements in their Scope.BUILD scope, even if they call Element.dependencies()
on a dependency.
This commit does the following:
* Element.dependencies() is now a user facing frontend which yields
ElementProxy elements instead of Elements.
* Various core codepaths have been updated to call the internal
Element._dependencies() codepath which still returns Elements.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Instead of having an assertion here, lets just have an early return
and make the __resolved_initial_state variable internal private
(with two leading underscores).
We also stop checking for it in _pipeline.py before resolving state.
Some background:
* We only defer _initialize_state() to a later stage because
it can be a resource intensive task which interrogates the
disk or the local CAS, thus we have the Pipeline iterate
over the instantiated elements and resolve them separately
for better user feedback.
* Some "first pass" elements must have their state initialized
earlier, i.e. the "link" and "junction" elements need to be
usable during the load sequence.
|
| |
|
|
| |
An ElementSources object represents the combined sources of an element.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Instead of passing around many details though calling signatures
throughout the loader code, create a single LoadContext object
which holds any overall loading state along with any values which
are constant to a full load process.
Overall this patch does:
* _frontend/app.py: No need to pass Stream.fetch_subprojects() along anymore
* _loader/loadelement.pyx: collect_element_no_deps() no longer takes a task argument
* _loader/loader.py: Now the Loader has a `load_context` member, and no more
`_fetch_subprojects` member or `_context` members
Further, `rewritable` and `ticker` is no longer passed along through all
of the recursing calling signatures, and `ticker` itself is finally removed
because this has been replaced a long time ago with `Task` API from `State`.
* _pipeline.py: The load() function no longer has a `rewritable` parameter
* _project.py: The Project() is responsible for creating the toplevel
LoadContext() if one doesn't exist yet, and this is passed through
to the Loader() (and also passed to the Project() constructor by the
Loader() when instantiating subprojects).
* _stream.py: The `Stream._fetch_subprojects()` is now private and set
on the project when giving the Project to the Stream in `Stream.set_project()`,
also the Stream() sets the `rewritable` state on the `LoadContext` at the
earliest opportunity, as the `Stream()` is the one who decides this detail.
Further, some double underscore private functions are now regular single
underscores, there was no reason for this inconsistency.
* tests/internals/loader.py: Updated for API change
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
`get_consistency` is coarse grained and hard to optimize, in addition
to being un-userfriendly.
This adds a new `is_resolved` that has for default implementation
`get_ref() is not None`, which is true for most sources in BuildStream.
Sources for which this is not true can override the method to give a
more accurate description.
Checking for this before looking whether the source is cached can
reduce the amount of work necessary in some pipeline and opens the
door for more optimizations and the removal of the source state check.
|
| |
|
|
|
|
|
| |
This replaces the _get_consistency method by two methods:
`_has_all_sources_resolved` and `_has_all_sources_cached` which allows
a more fine grained control on what information is needed.
|
| |
|
|
|
|
| |
'_source_cached' is not explicit enough as it doesn't distinguishes
between sources in their respective caches and sources in the global
sourcecache.
|
| |
|
|
|
|
|
|
| |
`get_consistency` doesn't allow being fine grained and asking only for a
specific bit of information.
This introduces methods `is_cached` and `is_resolved` which will be more
flexible for refactoring.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
PipelineSelection is one of the few stringy types that weren't
converted to FastEnum, presumably because we lacked a mechanism for
only allowing a sub-set of options as CLI arguments.
We've re-designed this since, and as part of the UI/UX refactor we'd
like to generally clean this, but that is probably still a while out.
Since that hasn't happened, for now, this adds a feature to the
FastEnumType that allows specifying only a subset of values is allowed
for a specific command, so that we can use the type as a proper
enum.
We also get rid of a number of accidental uses of strings, and move
PipelineSelection to buildstream.types so that we don't have a
significant import overhead for it.
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
| |
As discussed over the mailing list, reformat code using Black. This is a
one-off change to reformat all our codebase. Moving forward, we
shouldn't expect such blanket reformats. Rather, we expect each change
to already comply with the Black formatting style.
|
| |
|
|
|
|
| |
Attempting to open a workspace for the same element without closing it
now raises. This makes this kwarg unnecessary and tests should close
workspaces between attempts to open.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
* tracking not needed in reset
* support workspace opening for already open workspaces
remove existing files to preserve behaviour
Add ignore_workspaces kwarg to element loading via Stream().load
Setting this to true will ignore special handling of sources for open
workspaces and load the sources specified rather than a workspace
source. This avoids having to reload elements when re-opening
workspaces.
|
| | |
|
| | |
|
| |
|
|
|
|
|
| |
Resolving the initial cached state of Elements could
potentially take a while, especially for large projects
or local caches which are on a latent filesystem. We should
report progress.
|
| |
|
|
|
|
| |
If remotes exist, each remote will be checked for the target
artifacts. If an artifact is cached remotely, we make a
record of this.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
Adding the element full name and display key into all element related
messages removes the need to look up the plugintable via a plugin
unique_id just to retrieve the same values for logging and widget
frontend display. Relying on plugintable state is also incompatible
if the frontend will be running in a different process, as it will
exist in multiple states.
The element full name is now displayed instead of the unique_id,
such as in the debugging widget. It is also displayed in place of
'name' (i.e including any junction prepend) to be more informative.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Once an Element's strict cache key is determined, we should
attempt to update the strict cache key of it's reverse
dependencies. The state of a reverse dependency will be
updated once all of its dependencies have strict cache keys
This patch introduces the potential for a RecursionError because
_update_state() can now trigger further _update_state calls (on
reverse dependencies). Therefore, the maximum recursion limit
for our "test_max_recursion_depth" test has been lowered.
If this becomes a problem, we can always consider setting a
larger recursion limit, for now, this change has been tested
with the Debian stack and works as expected.
|
| |
|
|
|
|
|
| |
It's essential to call preflight() when loading/resolving the Elements.
This patch moves the preflight call to new_from_meta, so that it is
called as soon as the Element is created. This avoids the need for
having multiple callsites.
|
| |
|
|
|
|
|
| |
Lists of elements should never contain duplicate elements.
This commit also uses the helper to calculate the list of elements
when pulling missing elements in `bst shell`
|
| |
|
|
|
|
| |
Instead of having methods in Context forward calls on to the Messenger,
have folks call the Messenger directly. Remove the forwarding methods in
Context.
|
| |
|
|
|
|
| |
Treat junction element sources the same as sources of any other element
and always fetch subprojects as needed. Do not ask the user to manually
fetch subprojects.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
An Element becomes ready for runtime and cached when:
1. It has a strong cache key
2. It is cached
3. Its runtime dependencies are ready for runtime and cached
(this ensures the runtimes of runtimes are also cached).
This patch introduces the method attempt_to_notify_reverse_dependencies
which will notify all direct reverse dependencies of an Element once
said Element becomes ready for runtime and cached.
|
| |
|
|
|
|
|
| |
This patch includes setting a _depth to each element
once the pipeline has been sorted. This is necessary
as we need to store elements in the heapq sorted by
their depth.
|
|
|
This was discussed in #1008.
Fixes #1009.
|
