| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
| |
Several parts of the core have replicated codepaths for issuing info and
warn messages, remove the need for these functions by providing a convenience
layer in the Messenger object.
|
| |
|
|
|
|
|
|
|
| |
This omits the type annotation for the message handler callback, as
this callback contains a keyword argument and can only be annotated
using `Protocol` type, which will only be available in python >= 3.8.
Added a FIXME comment so that we can recitify this when dropping
support for python 3.7
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This improves overall documentation comments on the State object,
adds full pep484 type hinting, and renames the Task.set_render_cb()
to Task.set_task_changed_callback() to be more consistently named.
This also adds missing frontend facing API for the group changed
status notifications, even though the frontend does not currently
use these, it makes better sense to have them than to remove the
entire codepaths and callback lists.
This also reorders the classes in this file so that Task and TaskGroup
are both defined before State, this helps a bit with undefined references
for type hinting information.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This changes how the scheduler works and adapts all the code that needs
adapting in order to be able to run in threads instead of in
subprocesses, which helps with Windows support, and will allow some
simplifications in the main pipeline.
This addresses the following issues:
* Fix #810: All CAS calls are now made in the master process, and thus
share the same connection to the cas server
* Fix #93: We don't start as many child processes anymore, so the risk
of starving the machine are way less
* Fix #911: We now use `forkserver` for starting processes. We also
don't use subprocesses for jobs so we should be starting less
subprocesses
And the following highlevel changes where made:
* cascache.py: Run the CasCacheUsageMonitor in a thread instead of a
subprocess.
* casdprocessmanager.py: Ensure start and stop of the process are thread
safe.
* job.py: Run the child in a thread instead of a process, adapt how we
stop a thread, since we ca't use signals anymore.
* _multiprocessing.py: Not needed anymore, we are not using `fork()`.
* scheduler.py: Run the scheduler with a threadpool, to run the child
jobs in. Also adapt how our signal handling is done, since we are not
receiving signals from our children anymore, and can't kill them the
same way.
* sandbox: Stop using blocking signals to wait on the process, and use
timeouts all the time.
* messenger.py: Use a thread-local context for the handler, to allow for
multiple parameters in the same process.
* _remote.py: Ensure the start of the connection is thread safe
* _signal.py: Allow blocking entering in the signal's context managers
by setting an event. This is to ensure no thread runs long-running
code while we asked the scheduler to pause. This also ensures all the
signal handlers is thread safe.
* source.py: Change check around saving the source's ref. We are now
running in the same process, and thus the ref will already have been
changed.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This behavior has regressed a while back when introducing the messenger
object in 0026e379 from merge request !1500.
Main behavior change:
- Messages in the master log always appear with the task element's
element name and cache key, even if the element or plugin issuing
the log line is not the primary task element.
- Messages logged in the task specific log, retain the context of the
element names and cache keys which are issuing the log lines.
Changes include:
* _message.py: Added the task element name & key members
* _messenger.py: Log the element key as well if it is provided
* _widget.py: Prefer the task name & key when logging, we fallback
to the element name & key in case messages are being logged outside
of any ongoing task (main process/context)
* job.py: Unconditionally stamp messages with the task name & key
Also removed some unused parameters here, clearing up an XXX comment
* plugin.py: Add new `_message_kwargs` instance property, it is the responsibility
of the core base class to maintain the base keyword arguments which
are to be used as kwargs for Message() instances created on behalf
of the issuing plugin.
Use this method to construct messages in Plugin.__message() and to
pass kwargs along to Messenger.timed_activity().
* element.py: Update the `_message_kwargs` when the cache key is updated
* tests/frontend/logging.py: Fix test to expect the cache key in the logline
* tests/frontend/artifact_log.py: Fix test to expect the cache key in the logline
Fixes #1393
|
| |
|
|
|
| |
This allows more flexible usage of the APIs, to allow passing all of the
Message() constructor arguments through kwargs.
|
| |
|
|
|
|
|
|
| |
`State.add_task()` required the job name to be unique in the session.
However, the tuple `(action_name, full_name)` is not guaranteed to be
unique. E.g., multiple `ArtifactElement` objects with the same element
name may participate in a single session. Use a separate task identifier
to fix this.
|
| |
|
|
| |
This reduces the amount of code duplication
|
| | |
|
| |
|
|
|
|
|
| |
As discussed over the mailing list, reformat code using Black. This is a
one-off change to reformat all our codebase. Moving forward, we
shouldn't expect such blanket reformats. Rather, we expect each change
to already comply with the Black formatting style.
|
| |
|
|
|
|
|
|
|
|
|
| |
Note that unittest.MagicMock is unfortunately not pickable, so if one
tries to cross over to a ChildJob then we're in trouble.
Sacrifice some convenience and a bit of safety by implementing our own
_DummyTask instead of using MagicMock.
Don't try to pickle the 'simple_task' context manager, if it's been
overridden when creating a test context.
|
| |
|
|
|
|
|
| |
Prior to this patch, subtask information was always displayed,
regardless of how long it took. Now, I've implemented a
_DISPLAY_LIMIT of 3 seconds. If a task takes longer than 3 seconds,
the subtask information will be printed.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
Adding the element full name and display key into all element related
messages removes the need to look up the plugintable via a plugin
unique_id just to retrieve the same values for logging and widget
frontend display. Relying on plugintable state is also incompatible
if the frontend will be running in a different process, as it will
exist in multiple states.
The element full name is now displayed instead of the unique_id,
such as in the debugging widget. It is also displayed in place of
'name' (i.e including any junction prepend) to be more informative.
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
| |
Pave the way toward supporting the 'spawn' method of creating jobs, by
adding support for pickling ChildJobs. Introduce a new 'jobpickler'
module that provides an entrypoint for this functionality.
This also makes replays of jobs possible, which has made the debugging
of plugins much easier for me.
|
| |
|
|
|
|
|
|
|
|
|
| |
Change the argument name on the 'silence' context manager, to avoid the
interpretation that non-nested messages may be silenced, or that it can
'un-silence' (it can't).
Be more specific about what 'important' means w.r.t. messages.
Require 'actually_silence' to be specified as a keyword, so that
callsites are easier to understand.
|
| |
|
|
| |
Replace the _message_depth list with an int, "_silence_scope_depth".
|
| |
|
|
|
| |
Simplify by not calling _{push,pop}_message_depth outside of the silence
context manager, let that do the work instead.
|
| | |
|
| |
|
|
| |
Use some Python features to make Messenger._message_depth a bit simpler.
|
| |
|
|
|
|
| |
Instead of having methods in Context forward calls on to the Messenger,
have folks call the Messenger directly. Remove the forwarding methods in
Context.
|
| |
|
|
|
|
|
|
| |
Remove the need to pass the Context object to message handlers, by
passing what is usually requested from the context instead.
This paves the way to sharing less information with some child jobs -
they won't need the whole context object, just the messenger.
|
|
|
Separate out the responsibility of messaging from the rest of the
Context object.
This change removes some member variables and some code from member
functions, but keeps the existing member functions. In later work we'll
want things to use the Messenger directly, so we can remove some member
functions too, and de-couple further.
Update some comments to refer to the "Messenger API" instead of the
"Context API" in the copied code.
Fixup the arg documentation to timed_activity() while we're there.
|
