| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
Avoid confusion by not referring to starting another process as
'spawning'. Note that 'spawn' is a process creation method, which is an
alternative to forking.
Say 'create child process' instead of 'fork' where it doesn't harm
understanding. Although we currently only use the 'fork' method for
creating subprocesses, there are reasons for us to support 'spawn' in
the future.
More information on forking and spawning:
https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods
|
| |\
| |
| |
| |
| | |
Split ChildJob out from Job class
See merge request BuildStream/buildstream!1334
|
| | | |
|
| | | |
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Make it clearer what happens in which process by splitting out a
'ChildJob', which encapsulates the work that happens in the child
process.
This also makes it possible to control what is transferred to the
child process. This is very useful for adding support for the 'spawn'
method of creating child processes as opposed to the 'fork' method.
|
| | |
| |
| |
| | |
This just does the default behaviour, clearer to remove it.
|
| |/ |
|
| |\
| |
| |
| |
| | |
Optimize _yaml.node_get
See merge request BuildStream/buildstream!1370
|
| | |
| |
| |
| |
| | |
This reduces considerably the number of nodes created and thus speeds
up the calls to node_get.
|
| |/
|
|
|
|
|
|
|
|
|
| |
Calls to `isinstance` can be particularily costly. Using type() is
much faster. The only known case where the `isinstance` was useful
is for dictionnaries where we would ask for a 'Mapping' instead.
Disallowing 'Mapping' for expected_type considerably speeds up the
calls to this functions.
Also add into NEWS
|
| |\
| |
| |
| |
| | |
setup.py: Be more restrictive with BST_CYTHON_TRACE values
See merge request BuildStream/buildstream!1368
|
| |/
|
|
|
|
| |
"0" evaluates to 'True' in python, which incorrectly switched on
the BST_CYTHON_TRACE. Forcing an int for this environment variable
allows us to ensure we are correct.
|
| |\
| |
| |
| |
| | |
Setup.py: Disable linetrace by default and only enable when using coverage
See merge request BuildStream/buildstream!1367
|
| |/
|
|
|
|
|
|
|
| |
It turns out that enabling 'linetrace', does have a runtime cost even
if not enabled in distutils.
Therefore disabling it by default. In order to run coverage, we
therefore need to retranspile the .pyx files with ENABLE_CYTHON_TRACE
set.
|
| |\
| |
| |
| |
| |
| |
| | |
Update docs regarding artifact and source caches
Closes #1024
See merge request BuildStream/buildstream!1362
|
| | |
| |
| |
| |
| |
| |
| |
| | |
Now that we have both artifact and source caches the documentation
is updated to reflect that. Some sections headings/links etc. have been
expanded and changed.
Part of #1025
|
| | |
| |
| |
| | |
Part of #1024
|
| |/
|
|
| |
Part of #1024
|
| |\
| |
| |
| |
| | |
Make more parts of the loader iterative
See merge request BuildStream/buildstream!1365
|
| | |
| |
| |
| |
| |
| |
| | |
When multiple top level elements are specified we need to keep track
of whether we have completed the iterative collection process.
Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>
|
| | |
| |
| |
| |
| |
| |
| | |
To reduce stack usage during load, make the LoadElement to MetaElement
conversion be iterative.
Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>
|
| |/
|
|
|
|
|
|
| |
The _sort_dependencies() method does not need to be an instance method,
nor does it need to be recursive. This fixes both of those things which
can get us closer to being able to cythonize the loader.
Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>
|
| |\
| |
| |
| |
| | |
Rewrite `Loader._check_circular_deps()` to be iterative
See merge request BuildStream/buildstream!1364
|
| | |
| |
| |
| |
| |
| |
| |
| | |
In an effort to reduce the places where the stack might be a problem
as we Cythonize things, rewrite the circular dependency checker to
be iterative.
Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>
|
| |/
|
|
|
|
|
|
| |
This did not need to be an instance method, making it static might
improve performance and definitely makes it clear that it's not
actually bound to the loader instances.
Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>
|
| |\
| |
| |
| |
| | |
Introduce Cython for better performances
See merge request BuildStream/buildstream!1350
|
| | |
| |
| |
| |
| | |
This allows for a quicker comparison while keeping a good readability
of the code
|
| | |
| |
| |
| |
| |
| |
| | |
The `SYNTHETIC_COUNTER` is an iterator that is called a lot in
_yaml, one for each synthetic node. Cython is not able to optimize
`itertools.counter` well enough. Providing a custom C function allows
to reduce the amount of python code called in this critical codepath
|
| | |
| |
| |
| |
| | |
This requires the addition of a definition file (.pxd), to list
symbols exported.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| | |
Providing c definitions for functions allows us to not have to
go back to the python interpreter when inside the module. We
therefore gain more performance.
One gotcha is that keyword only arguments are not valid in cpdef
functions.
|
| | |
| |
| |
| |
| |
| |
| |
| | |
We used to have a tuple to keep file information.
This makes it hard to read, accessing attributes by index.
With an extension class FileInfo, we get better readability,
without sacrificing performance
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
`Representer` is the main interface with the `ruamel` library to
parse the yaml files. Rewriting it with Cython introduces significant
performance gains.
Since `Representer` is not a python class anymore, we can't call
`getattr` on it, and therefore have to do a manual switch on the types
of events. While this is harder to read, it is also much more
performant.
Finally, sotp using `yaml.parse`, but call the parser manually, in order
to avoid going in and out of the python code. This part could be made
even better in the future when `ruamel` becomes stable and if they
expose cython definitions, as they are coded in Cython.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| | |
_yaml is a bottleneck in a normal BuildStream run. Typing the
external API allows us to reduce this bottleneck.
Since we type the input variables, we can also remove some asserts
that are checking if the parameters are of the correct type as
Cython will throw TypeError if called incorrectly.
|
| | |
| |
| |
| |
| |
| | |
Internal functions are not used outside the module, and are
therefore easy to refactor and only call from C, leading us to
significant performance gains.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Node used to be a NamedTuple that we used to access by index
for speed reasons. Moving to an extension class allows us to
access attributes by name, making the code easier to read and
less error prone. Moreover, we do gain speed and memory by
doing this move.
Also fix a few places where we would not have an entire `Node`
object but were instead just returning a tuple, missing some entries.
|
| | | |
|
| | | |
|
| | |
| |
| |
| |
| |
| | |
Extension class are faster for access and take less memory than
a normal Python class. Variables is self contained and easy to
cythonize.
|
| | |
| |
| |
| | |
Also type the `PARSE_CACHE` in order to speedup access to it.
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| | |
Move _variables.py to be a Cython module.
`_internal_expand` is a function that is called a lot in BuildStream.
It is also entirely isolated and easy to cythonize.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Cython requires a plugin to allow coverage of cython files, which
was updated in coveragerc. It also means we need to build the
dependencies and install cython for coverage.
Cython requires access to both source and compiled files when
running coverage. We therefore need to install project in develop
mode.
Updated documentation to explain how to run tests without tox but
with coverage
|
| | |
| |
| |
| |
| |
| |
| |
| |
| | |
Using pyproject.toml, defined in PEP518, allows us to have
an isolated build environment, ensuring that Cython can be installed
before calling setup.py in tox. This allows us to use cython helpers
in the setup.py script.
This is a prerequisite for introducing Cython in the codebase
|
| |/
|
|
|
| |
This removes the precomputation of the size, which makes it slightly
faster.
|
| |\
| |
| |
| |
| | |
Make the loading process iterative
See merge request BuildStream/buildstream!1363
|
| | |
| |
| |
| |
| | |
Delegate properly everything concerning a sub-loader to the sub-loader
Stop iterating twice the loop for every dependency
|
| |/
|
|
|
|
|
|
|
|
| |
In order to reduce the cost of recursion in loading a project,
this commit makes a semi-iterative treatment of the _load_file()
pathway. It's only semi-iterative in the sense that whenever we
cross a loader boundary (junctions) we recurse. This should mean
that high-depth projects load more safely.
Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>
|
| |\
| |
| |
| |
| |
| |
| | |
.gitlab-ci.yml: ostree plugin depends on ostree/python3-gobject-base
Closes #1035
See merge request BuildStream/buildstream!1361
|
