| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
|
|
|
|
|
|
| |
* .pylintrc: Disable new `raise-missing-from` check. We might want to
enable that later, but it fails in many places. Let's not merge both
changes here.
* pluginoriginpip.py: Catch the newer thrown exception from
pkg_resources. The previous one still exists, so we should be good
keeping the same compatibility as before
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This changes how the scheduler works and adapts all the code that needs
adapting in order to be able to run in threads instead of in
subprocesses, which helps with Windows support, and will allow some
simplifications in the main pipeline.
This addresses the following issues:
* Fix #810: All CAS calls are now made in the master process, and thus
share the same connection to the cas server
* Fix #93: We don't start as many child processes anymore, so the risk
of starving the machine are way less
* Fix #911: We now use `forkserver` for starting processes. We also
don't use subprocesses for jobs so we should be starting less
subprocesses
And the following highlevel changes where made:
* cascache.py: Run the CasCacheUsageMonitor in a thread instead of a
subprocess.
* casdprocessmanager.py: Ensure start and stop of the process are thread
safe.
* job.py: Run the child in a thread instead of a process, adapt how we
stop a thread, since we ca't use signals anymore.
* _multiprocessing.py: Not needed anymore, we are not using `fork()`.
* scheduler.py: Run the scheduler with a threadpool, to run the child
jobs in. Also adapt how our signal handling is done, since we are not
receiving signals from our children anymore, and can't kill them the
same way.
* sandbox: Stop using blocking signals to wait on the process, and use
timeouts all the time.
* messenger.py: Use a thread-local context for the handler, to allow for
multiple parameters in the same process.
* _remote.py: Ensure the start of the connection is thread safe
* _signal.py: Allow blocking entering in the signal's context managers
by setting an event. This is to ensure no thread runs long-running
code while we asked the scheduler to pause. This also ensures all the
signal handlers is thread safe.
* source.py: Change check around saving the source's ref. We are now
running in the same process, and thus the ref will already have been
changed.
|
|
|
|
|
|
|
|
| |
This updates all dependencies on the project, which is mainly needed
by python3.8 but can be done independentely.
This also disables multiple false positive lint errors and disable a
new check that we don't need.
|
|
|
|
|
|
|
| |
Since we use Black to format our code, disable code-formatting messages
from Pylint as they are already taken care of by Black. This ensures
that we don't get double warnings for the same problem, and that Pylint
and Black don't fight with each other over style.
|
|
|
|
|
|
|
|
|
|
|
|
| |
Add `ujson` to the `extension-pkg-whitelist` (we trust it!). Without
this, `pylint` throws the following (non-fatal) warning:
```
src/buildstream/_cachekey.py:65:14: I1101: Module 'ujson' has no 'dumps'
member, but source is unavailable. Consider adding this module to
extension-pkg-whitelist if you want to perform analysis based on
run-time introspection of living objects. (c-extension-no-member)
```
|
|
|
|
|
|
|
| |
'Enum' has a big performance impact on the running code. Replacing
it with a safe subset of functionality removes lots of this overhead
without removing the benefits of using enums (safe comparisions,
uniqueness)
|
| |
|
|
|
|
|
| |
`url_directory_name` is heavily called from any downloadable source
plugin, and moving it to cython gives a more than 10x speedup
|
|
|
|
|
| |
This makes the 'Node' API public, and available for use directly for
plugins.
|
| |
|
|
|
|
|
|
| |
Types is a simple module that accounts for a few percent of a basic
'show' operation. Having it cythonized allows us to get better
performance without too much wokr
|
|
|
|
|
|
|
|
|
| |
- Create a new _loader/utils.pyx cython module for functions cythonized
in the loader module.
- Move valid_chars_name from loader to utils and cythonize.
This function is called extensively, and easy to extract
|
|
|
|
|
|
|
|
|
|
|
| |
Node used to be a NamedTuple that we used to access by index
for speed reasons. Moving to an extension class allows us to
access attributes by name, making the code easier to read and
less error prone. Moreover, we do gain speed and memory by
doing this move.
Also fix a few places where we would not have an entire `Node`
object but were instead just returning a tuple, missing some entries.
|
|
|
|
|
|
|
| |
Move _variables.py to be a Cython module.
`_internal_expand` is a function that is called a lot in BuildStream.
It is also entirely isolated and easy to cythonize.
|
|
|
|
|
|
|
|
| |
Cyclic imports can be confusing because the order in which we import
dependencies can make the import fail or not. We should not rely on
ordering of imports for our code. This fixes everywhere possible the
imports and silence explicitely some which are not convenient or would
require big refactors
|
|
|
|
| |
This enables the "logging-format-interpolation" pylint error check
|
| |
|
|
|
|
|
|
|
|
| |
A project path is a path relative to a project directory.
A project path can not also refer to the parent directory in the first
path component, or point to symbolic links, fifos, sockets and
block/character devices.
|
|
|
|
| |
This allows code generation with ./setup.py build_grpc
|
|
|
|
|
|
|
| |
This will give us an error if an instance attribute is ever
declared outside of the constructor, which usually constitutes
either a bug or a violation of the coding standard, rendering
code more difficult to read.
|
| |
|
| |
|
|
|
|
| |
classes
|
| |
|
| |
|
| |
|
| |
|
| |
|
| |
|
| |
|
| |
|
|
|
|
| |
catching-non-exception warnings
|
| |
|
| |
|
| |
|
| |
|
| |
|
| |
|
| |
|
| |
|
|
|
|
|
|
| |
NOTE: We have categorised the warning messages into two groups.
Those that are of no use to us in the buildstream project and
those that should be enabled at some point.
|
|
NOTE: pylint was installed using pip: `pip3 install pylint` and
.pylintrc is the pylint configuration file which was generated with
`pylint --generate-rcfile` from a fresh fedora-27 docker image.
|