summaryrefslogtreecommitdiff
path: root/src/buildstream/_messenger.py
Commit message (Collapse)AuthorAgeFilesLines
* _messenger.py: Adding convenience methods for issuing info and warn messagesTristan van Berkom2020-12-211-0/+65
| | | | | | Several parts of the core have replicated codepaths for issuing info and warn messages, remove the need for these functions by providing a convenience layer in the Messenger object.
* _messenger.py: Adding (almost) full pep484 type annotations.Tristan van Berkom2020-12-211-72/+104
| | | | | | | | | This omits the type annotation for the message handler callback, as this callback contains a keyword argument and can only be annotated using `Protocol` type, which will only be available in python >= 3.8. Added a FIXME comment so that we can recitify this when dropping support for python 3.7
* _state.py: Full type hintingtristan/refactor-retry-taskTristan van Berkom2020-12-101-1/+1
| | | | | | | | | | | | | | | This improves overall documentation comments on the State object, adds full pep484 type hinting, and renames the Task.set_render_cb() to Task.set_task_changed_callback() to be more consistently named. This also adds missing frontend facing API for the group changed status notifications, even though the frontend does not currently use these, it makes better sense to have them than to remove the entire codepaths and callback lists. This also reorders the classes in this file so that Task and TaskGroup are both defined before State, this helps a bit with undefined references for type hinting information.
* scheduler.py: Use threads instead of processes for jobsBenjamin Schubert2020-12-041-31/+29
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This changes how the scheduler works and adapts all the code that needs adapting in order to be able to run in threads instead of in subprocesses, which helps with Windows support, and will allow some simplifications in the main pipeline. This addresses the following issues: * Fix #810: All CAS calls are now made in the master process, and thus share the same connection to the cas server * Fix #93: We don't start as many child processes anymore, so the risk of starving the machine are way less * Fix #911: We now use `forkserver` for starting processes. We also don't use subprocesses for jobs so we should be starting less subprocesses And the following highlevel changes where made: * cascache.py: Run the CasCacheUsageMonitor in a thread instead of a subprocess. * casdprocessmanager.py: Ensure start and stop of the process are thread safe. * job.py: Run the child in a thread instead of a process, adapt how we stop a thread, since we ca't use signals anymore. * _multiprocessing.py: Not needed anymore, we are not using `fork()`. * scheduler.py: Run the scheduler with a threadpool, to run the child jobs in. Also adapt how our signal handling is done, since we are not receiving signals from our children anymore, and can't kill them the same way. * sandbox: Stop using blocking signals to wait on the process, and use timeouts all the time. * messenger.py: Use a thread-local context for the handler, to allow for multiple parameters in the same process. * _remote.py: Ensure the start of the connection is thread safe * _signal.py: Allow blocking entering in the signal's context managers by setting an event. This is to ensure no thread runs long-running code while we asked the scheduler to pause. This also ensures all the signal handlers is thread safe. * source.py: Change check around saving the source's ref. We are now running in the same process, and thus the ref will already have been changed.
* Restore task element name / element name distinction in UITristan van Berkom2020-10-271-1/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This behavior has regressed a while back when introducing the messenger object in 0026e379 from merge request !1500. Main behavior change: - Messages in the master log always appear with the task element's element name and cache key, even if the element or plugin issuing the log line is not the primary task element. - Messages logged in the task specific log, retain the context of the element names and cache keys which are issuing the log lines. Changes include: * _message.py: Added the task element name & key members * _messenger.py: Log the element key as well if it is provided * _widget.py: Prefer the task name & key when logging, we fallback to the element name & key in case messages are being logged outside of any ongoing task (main process/context) * job.py: Unconditionally stamp messages with the task name & key Also removed some unused parameters here, clearing up an XXX comment * plugin.py: Add new `_message_kwargs` instance property, it is the responsibility of the core base class to maintain the base keyword arguments which are to be used as kwargs for Message() instances created on behalf of the issuing plugin. Use this method to construct messages in Plugin.__message() and to pass kwargs along to Messenger.timed_activity(). * element.py: Update the `_message_kwargs` when the cache key is updated * tests/frontend/logging.py: Fix test to expect the cache key in the logline * tests/frontend/artifact_log.py: Fix test to expect the cache key in the logline Fixes #1393
* _messenger.py: Use kwargs in timed_activity() and simple_task()Tristan van Berkom2020-10-271-20/+19
| | | | | This allows more flexible usage of the APIs, to allow passing all of the Message() constructor arguments through kwargs.
* _state.py: Use separate task identifierJürg Billeter2020-09-101-2/+2
| | | | | | | | `State.add_task()` required the job name to be unique in the session. However, the tuple `(action_name, full_name)` is not guaranteed to be unique. E.g., multiple `ArtifactElement` objects with the same element name may participate in a single session. Use a separate task identifier to fix this.
* _messenger.py: Make `timed_suspendable` public and use it in job.pybschubert/timed-suspendableBenjamin Schubert2020-08-221-30/+30
| | | | This reduces the amount of code duplication
* Completely abolish job pickling.tristan/nuke-pickle-jobberTristan van Berkom2020-06-151-43/+0
|
* Reformat code using BlackChandan Singh2019-11-141-23/+24
| | | | | | | As discussed over the mailing list, reformat code using Black. This is a one-off change to reformat all our codebase. Moving forward, we shouldn't expect such blanket reformats. Rather, we expect each change to already comply with the Black formatting style.
* testing, messenger: make dummy_context picklableAngelos Evripiotis2019-10-291-0/+7
| | | | | | | | | | | Note that unittest.MagicMock is unfortunately not pickable, so if one tries to cross over to a ChildJob then we're in trouble. Sacrifice some convenience and a bit of safety by implementing our own _DummyTask instead of using MagicMock. Don't try to pickle the 'simple_task' context manager, if it's been overridden when creating a test context.
* _messenger.py: Don't display subtask information before 3 secondsJames Ennis2019-09-041-3/+10
| | | | | | | Prior to this patch, subtask information was always displayed, regardless of how long it took. Now, I've implemented a _DISPLAY_LIMIT of 3 seconds. If a task takes longer than 3 seconds, the subtask information will be printed.
* _message.py: Use element_name & element_key instead of unique_idtpollard/messageobjectTom Pollard2019-08-081-20/+19
| | | | | | | | | | | | | Adding the element full name and display key into all element related messages removes the need to look up the plugintable via a plugin unique_id just to retrieve the same values for logging and widget frontend display. Relying on plugintable state is also incompatible if the frontend will be running in a different process, as it will exist in multiple states. The element full name is now displayed instead of the unique_id, such as in the debugging widget. It is also displayed in place of 'name' (i.e including any junction prepend) to be more informative.
* _messenger: Fix complex objects leaking into child jobsJonathan Maw2019-07-261-0/+7
|
* Render progress information for loading and processing elementsJonathan Maw2019-07-261-18/+144
|
* Make ChildJobs and friends picklableAngelos Evripiotis2019-07-241-0/+29
| | | | | | | | | Pave the way toward supporting the 'spawn' method of creating jobs, by adding support for pickling ChildJobs. Introduce a new 'jobpickler' module that provides an entrypoint for this functionality. This also makes replays of jobs possible, which has made the debugging of plugins much easier for me.
* _messenger: clarify silent_nested behaviouraevri/messengerAngelos Evripiotis2019-07-051-6/+11
| | | | | | | | | | | Change the argument name on the 'silence' context manager, to avoid the interpretation that non-nested messages may be silenced, or that it can 'un-silence' (it can't). Be more specific about what 'important' means w.r.t. messages. Require 'actually_silence' to be specified as a keyword, so that callsites are easier to understand.
* _messenger: _message_depth -> _silence_scope_depthAngelos Evripiotis2019-07-051-17/+10
| | | | Replace the _message_depth list with an int, "_silence_scope_depth".
* _messenger: use silence ctx, not *_message_depthAngelos Evripiotis2019-07-051-4/+2
| | | | | Simplify by not calling _{push,pop}_message_depth outside of the silence context manager, let that do the work instead.
* Remove unused Message.depth attributeAngelos Evripiotis2019-07-051-4/+0
|
* _messenger: refactor, pythonic _message_depthAngelos Evripiotis2019-07-051-9/+5
| | | | Use some Python features to make Messenger._message_depth a bit simpler.
* Refactor, use context.messenger directlyAngelos Evripiotis2019-07-051-3/+3
| | | | | | Instead of having methods in Context forward calls on to the Messenger, have folks call the Messenger directly. Remove the forwarding methods in Context.
* Refactor: message handlers take 'is_silenced'Angelos Evripiotis2019-07-051-8/+13
| | | | | | | | Remove the need to pass the Context object to message handlers, by passing what is usually requested from the context instead. This paves the way to sharing less information with some child jobs - they won't need the whole context object, just the messenger.
* _context:extract _messenger.Messenger from ContextAngelos Evripiotis2019-07-051-0/+292
Separate out the responsibility of messaging from the rest of the Context object. This change removes some member variables and some code from member functions, but keeps the existing member functions. In later work we'll want things to use the Messenger directly, so we can remove some member functions too, and de-couple further. Update some comments to refer to the "Messenger API" instead of the "Context API" in the copied code. Fixup the arg documentation to timed_activity() while we're there.