| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This changes how the scheduler works and adapts all the code that needs
adapting in order to be able to run in threads instead of in
subprocesses, which helps with Windows support, and will allow some
simplifications in the main pipeline.
This addresses the following issues:
* Fix #810: All CAS calls are now made in the master process, and thus
share the same connection to the cas server
* Fix #93: We don't start as many child processes anymore, so the risk
of starving the machine are way less
* Fix #911: We now use `forkserver` for starting processes. We also
don't use subprocesses for jobs so we should be starting less
subprocesses
And the following highlevel changes where made:
* cascache.py: Run the CasCacheUsageMonitor in a thread instead of a
subprocess.
* casdprocessmanager.py: Ensure start and stop of the process are thread
safe.
* job.py: Run the child in a thread instead of a process, adapt how we
stop a thread, since we ca't use signals anymore.
* _multiprocessing.py: Not needed anymore, we are not using `fork()`.
* scheduler.py: Run the scheduler with a threadpool, to run the child
jobs in. Also adapt how our signal handling is done, since we are not
receiving signals from our children anymore, and can't kill them the
same way.
* sandbox: Stop using blocking signals to wait on the process, and use
timeouts all the time.
* messenger.py: Use a thread-local context for the handler, to allow for
multiple parameters in the same process.
* _remote.py: Ensure the start of the connection is thread safe
* _signal.py: Allow blocking entering in the signal's context managers
by setting an event. This is to ensure no thread runs long-running
code while we asked the scheduler to pause. This also ensures all the
signal handlers is thread safe.
* source.py: Change check around saving the source's ref. We are now
running in the same process, and thus the ref will already have been
changed.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* This modifies the signal terminator so that it can be called from any
thread.
This checks that either:
- The signal handler is already in place
- Or the caller is in the main thread, allowing to set the signal
handler.
This also removes the exact callback that was added instead of removing
the last one, and fixes the `suspend_handler` to do the same.
This is required, as we don't know which interleaving of calls will be
done, and we can't guarantee that the last one is the right one to
remove
|
| | |
|
| |
|
|
| |
Use exit code from the `SystemExit` exception raised by `os.exit()`.
|
| |
|
|
|
|
|
| |
As discussed over the mailing list, reformat code using Black. This is a
one-off change to reformat all our codebase. Moving forward, we
shouldn't expect such blanket reformats. Rather, we expect each change
to already comply with the Black formatting style.
|
| |
|
|
|
|
|
|
|
|
| |
As a first step, add type hints to variables whose type `mypy` cannot
infer automatically. This is the minimal set of type hints that allow
running `mypy` without any arguments, and having it not fail.
We currently ignore C extensions that mypy can't process directly.
Later, we can look into generating stubs for such modules (potentially
automatically).
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
Avoid confusion by not referring to starting another process as
'spawning'. Note that 'spawn' is a process creation method, which is an
alternative to forking.
Say 'create child process' instead of 'fork' where it doesn't harm
understanding. Although we currently only use the 'fork' method for
creating subprocesses, there are reasons for us to support 'spawn' in
the future.
More information on forking and spawning:
https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods
|
|
|
This was discussed in #1008.
Fixes #1009.
|
