| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This changes how the scheduler works and adapts all the code that needs
adapting in order to be able to run in threads instead of in
subprocesses, which helps with Windows support, and will allow some
simplifications in the main pipeline.
This addresses the following issues:
* Fix #810: All CAS calls are now made in the master process, and thus
share the same connection to the cas server
* Fix #93: We don't start as many child processes anymore, so the risk
of starving the machine are way less
* Fix #911: We now use `forkserver` for starting processes. We also
don't use subprocesses for jobs so we should be starting less
subprocesses
And the following highlevel changes where made:
* cascache.py: Run the CasCacheUsageMonitor in a thread instead of a
subprocess.
* casdprocessmanager.py: Ensure start and stop of the process are thread
safe.
* job.py: Run the child in a thread instead of a process, adapt how we
stop a thread, since we ca't use signals anymore.
* _multiprocessing.py: Not needed anymore, we are not using `fork()`.
* scheduler.py: Run the scheduler with a threadpool, to run the child
jobs in. Also adapt how our signal handling is done, since we are not
receiving signals from our children anymore, and can't kill them the
same way.
* sandbox: Stop using blocking signals to wait on the process, and use
timeouts all the time.
* messenger.py: Use a thread-local context for the handler, to allow for
multiple parameters in the same process.
* _remote.py: Ensure the start of the connection is thread safe
* _signal.py: Allow blocking entering in the signal's context managers
by setting an event. This is to ensure no thread runs long-running
code while we asked the scheduler to pause. This also ensures all the
signal handlers is thread safe.
* source.py: Change check around saving the source's ref. We are now
running in the same process, and thus the ref will already have been
changed.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
This ensures that we can cleanly cleanup processes and threads on
termination of BuildStream.
Plugins should use this helper whenever there is a risk of them being
blocked on a syscall for an indefinite amount of time
* downloadablefilesource.py: Use this new helper to do the actual
download, which would prevent the process from completely blocking if
we have a badly behaving upstream
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This behavior has regressed a while back when introducing the messenger
object in 0026e379 from merge request !1500.
Main behavior change:
- Messages in the master log always appear with the task element's
element name and cache key, even if the element or plugin issuing
the log line is not the primary task element.
- Messages logged in the task specific log, retain the context of the
element names and cache keys which are issuing the log lines.
Changes include:
* _message.py: Added the task element name & key members
* _messenger.py: Log the element key as well if it is provided
* _widget.py: Prefer the task name & key when logging, we fallback
to the element name & key in case messages are being logged outside
of any ongoing task (main process/context)
* job.py: Unconditionally stamp messages with the task name & key
Also removed some unused parameters here, clearing up an XXX comment
* plugin.py: Add new `_message_kwargs` instance property, it is the responsibility
of the core base class to maintain the base keyword arguments which
are to be used as kwargs for Message() instances created on behalf
of the issuing plugin.
Use this method to construct messages in Plugin.__message() and to
pass kwargs along to Messenger.timed_activity().
* element.py: Update the `_message_kwargs` when the cache key is updated
* tests/frontend/logging.py: Fix test to expect the cache key in the logline
* tests/frontend/artifact_log.py: Fix test to expect the cache key in the logline
Fixes #1393
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
As a rule, throughout the codebase we should not be using internal
ProvenanceInformation objects in our APIs, but rather Node objects.
This is because ProvenanceInformation is generated on the fly from
a Node object, and it is needlessly expensive to instantiate one before
it is absolutely needed.
This patch unilaterally fixes the codebase to pass `provenance_node`
Node objects around as arguments rather than `provenance` ProvenanceInformation
objects.
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
| |
When opening up the possibility of loading the same junction twice,
it becomes possible to have multiple elements with the same name
following the current implementation logic which enforces that
you must have a toplevel junction to coalesce junctions and synchronize
their configuration.
Since we no longer coalesce junction names, we need to allow elements
to be reported in the UI with their full depth.
|
| |
|
|
|
| |
This will allow all users to not have to care about whether the option
is expanded or not, making it easier to use variables everywhere
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This is mostly a semantic change which defines how deprecation warnings
are suppressed in a more consistent fashion, by declaring such suppressions
in the plugin origin declarations rather than on the generic element/source
configuration overrides section.
Other side effects of this commit are that the warnings have been enhanced
to include the provenance of whence the deprecated plugins have been used in
the project, and that the custom deprecation message is optional and will
appear in the message detail string rather than in the primary warning text,
which now simply indicates that the plugin being used is deprecated.
Documentation and test cases are updated.
This fixes #1291
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
The BST_MIN_VERSION guards assert that the BuildStream core which
loaded the plugin is compatible with the plugin itself.
This commit adds BST_MIN_VERSION to the base plugin.py with documentation
informing Plugin authors how to set the minimum version, and also adds
the assertions at plugin loading time in pluginfactory.py.
This commit also:
* Adds the BST_MIN_VERSION specification to all current core plugins
* Adds the BST_MIN_VERSION specification to plugins used in test cases
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Plugin format versioning was decided to be removed for local
plugins and any plugins for which we do not load an explicitly
provided plugin. For pip, this will be handled with a standard
distutils/setuptools approach, allowing users to specify pip
style version boundaries in the plugin origins.
This patch refactors plugin loading so that all related code
goes into the private _pluginfactory module, a new small
PluginOrigin type was added to better manipulate loaded
origins.
Test cases have been removed and will be readded in a following
commit, adjusted to new expectations.
|
| |
|
|
|
|
|
|
|
|
|
| |
With BuildStream 2.0, we start with a fresh API, everything that
is present at the time of the 2.0 release can implicitly be considered
to be available since 2.0.
In the future, we will be expressing new API additions using the
min-version semantic versioning style instead of format version too
(so new YAML configurations will be expressed as "since: 2.2" and
so on).
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This does not make sense to keep in the public API surface
documentation. As we are heading towards a release of 2.0, this
represents a "reset" in public API, and older annotations only
serve to clutter the documentation with information that is not
relevant to the reader.
Everything which is public at the time of the 2.0 release can
be considered available "Since: 2.0" implicitly (as this is
going to be the starting point of this new stable API).
It will make sense to start adding these annotations again for any
added API in 2.2 and forward.
|
| |
|
|
|
| |
This is not needed now that we have 'is_resolved' and 'is_cached'.
We can therefore drop all calling places and implementations of it.
|
| |
|
|
|
|
|
| |
As discussed over the mailing list, reformat code using Black. This is a
one-off change to reformat all our codebase. Moving forward, we
shouldn't expect such blanket reformats. Rather, we expect each change
to already comply with the Black formatting style.
|
| |
|
|
|
|
|
|
|
| |
Remove the need for plugins to find and return the factory they came
from. Also take the opportunity to combine source and element pickling
into a single 'plugin' pickling path.
This will make it easier for us to later support pickling plugins from
the 'first_pass_config' of projects.
|
| |
|
|
|
|
|
|
|
|
| |
As a first step, add type hints to variables whose type `mypy` cannot
infer automatically. This is the minimal set of type hints that allow
running `mypy` without any arguments, and having it not fail.
We currently ignore C extensions that mypy can't process directly.
Later, we can look into generating stubs for such modules (potentially
automatically).
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
Adding the element full name and display key into all element related
messages removes the need to look up the plugintable via a plugin
unique_id just to retrieve the same values for logging and widget
frontend display. Relying on plugintable state is also incompatible
if the frontend will be running in a different process, as it will
exist in multiple states.
The element full name is now displayed instead of the unique_id,
such as in the debugging widget. It is also displayed in place of
'name' (i.e including any junction prepend) to be more informative.
|
| |
|
|
|
|
| |
Once project & type are resolved, the full_name can be computed
and cached for efficiency. The accessor for getting the private
member should also be moved to the correct section.
|
| |
|
|
|
|
|
|
|
| |
Pave the way toward supporting the 'spawn' method of creating jobs, by
adding support for pickling ChildJobs. Introduce a new 'jobpickler'
module that provides an entrypoint for this functionality.
This also makes replays of jobs possible, which has made the debugging
of plugins much easier for me.
|
| |
|
|
|
| |
We need to update every place where we were passing a yaml 'dict' to
now pass a 'MappingNode'
|
| | |
|
| |
|
|
|
|
|
|
| |
This replaces the helper method by adding a 'get_provenance' on the node
directly
- Adapt all call sites
- Delay getting provenance wherever possible without major refactor
|
| |
|
|
|
|
|
| |
Now that we have scalar nodes, we can directly give the scalar to the
method.
- Adapt the plugin method to also not access via the key
|
| |
|
|
| |
- adapt all call sites to use the new API
|
| |
|
|
|
|
|
| |
Using 'Node.from_dict({})' can replace new_empty_node, and the rest
is not needed anymore.
- Adapt all call sites
|
| |
|
|
|
|
| |
- Implement __setitem__ on 'MappingNode'
- Implement __setitem__ on 'SequenceNode'
- Adapt all call sites to use the new calling way.
|
| |
|
|
|
|
| |
One difference is that 'MappingNode.items()' does not strip the
provenance from scalars and lists, which ends up not affecting the
code much.
|
| |
|
|
| |
This can now be done more easily with the Node api
|
| |
|
|
|
|
|
| |
The new yaml API is able of getting these access with the same error
messages handling.
Having this helper is therefore not useful anymore.
|
| |
|
|
|
| |
This method is unused and can be easily reimplemented with the new
Node API by looping over nodes.
|
| |
|
|
|
|
|
|
|
|
|
| |
- 'as_bool()' casts a ScalarNode into a boolean, understanding both
'True' and 'False' as truthy-falsy values, as per node_get(type=bool)
behavior
- 'is_none()' allwos checking whether the scalar node contains a 'None'
value. Since 'None' cannot be used when working with booleans, we need
to have a way of checking for 'None' when we actually need the
information of whether the value is unset.
- Adapt all call places to use the new API
|
| |
|
|
|
|
| |
Instead of having methods in Context forward calls on to the Messenger,
have folks call the Messenger directly. Remove the forwarding methods in
Context.
|
| |
|
|
|
| |
Also update documentation in plugin for APIs that do accept Node and
not `dict`.
|
|
|
This was discussed in #1008.
Fixes #1009.
|
