summaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
* element.py: Optimize assemble_done()juerg/assemble_doneJürg Billeter2020-01-163-9/+24
| | | | | After a successful build we know that the artifact is cached. Avoid querying buildbox-casd and the filesystem.
* Merge branch 'bschubert/optimize-consistency' into 'master'Benjamin Schubert2020-01-1635-285/+288
|\ | | | | | | | | | | | | Optimize consistency and state handling Closes #1126 See merge request BuildStream/buildstream!1739
| * NEWS: Add news information about Consistency breakagebschubert/optimize-consistencyBenjamin Schubert2020-01-161-0/+21
| |
| * element.py: Stop asserting we are cached after fetch and mark us as suchBenjamin Schubert2020-01-161-9/+4
| | | | | | | | | | | | This assert would only be run in development mode and would actually not mark the source as downloaded. Therefore forcefully mark them as fetched and remove the assert. Our test suite will catch if there is a problem.
| * types.py: Completely remove 'Consistency'Benjamin Schubert2020-01-162-43/+13
| | | | | | | | | | 'Consistency' is not needed anymore, now that we have everything in place to not use it
| * element.py: Optimize _should_fetch conditionBenjamin Schubert2020-01-161-6/+3
| | | | | | | | | | By looking at the flag first, we can avoid expensive checks on whether the element is cached or not.
| * element.py: Compute whether an element is cached only on `is_cached`Benjamin Schubert2020-01-163-23/+20
| | | | | | | | | | | | | | | | | | | | | | This removes the early call to get whether sources are locally cached in `_update_source_state` by delegating it to the call of `is_cached`. Once it is cached, the element is assumed to stay that way for the duration of the pipeline, we can therefore cache the result once it is true. Also remove `Consistency.IS_CACHED`, which is not used anywhere else.
| * source.py: Remove 'get_consistency' completelyBenjamin Schubert2020-01-1622-163/+32
| | | | | | | | | | This is not needed now that we have 'is_resolved' and 'is_cached'. We can therefore drop all calling places and implementations of it.
| * source.py: Remove the reliance on consistency to get whether a source is cachedBenjamin Schubert2020-01-1618-12/+114
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This removes the need to use consistency in Sources, by asking explicitely whether the source is cached or not. This introduces a new public method on source: `is_cached` that needs implementation and that should return whether the source has a local copy or not. - On fetch, also reset whether the source was cached or set if as cached when we know it was. - Validate the cache's source after fetching it This doesn't need to be run in the scheduler's process and can be offloaded to the child, which will allow better multiprocessing
| * source.py: Add a new 'is_resolved' to get whether a source is resolved.Benjamin Schubert2020-01-1612-16/+55
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | `get_consistency` is coarse grained and hard to optimize, in addition to being un-userfriendly. This adds a new `is_resolved` that has for default implementation `get_ref() is not None`, which is true for most sources in BuildStream. Sources for which this is not true can override the method to give a more accurate description. Checking for this before looking whether the source is cached can reduce the amount of work necessary in some pipeline and opens the door for more optimizations and the removal of the source state check.
| * element.py: Remove _get_consistency and introduce explicit methodsBenjamin Schubert2020-01-165-30/+33
| | | | | | | | | | | | | | This replaces the _get_consistency method by two methods: `_has_all_sources_resolved` and `_has_all_sources_cached` which allows a more fine grained control on what information is needed.
| * element.py: Rename '_source_cached' to '_has_all_sources_in_source_cache'Benjamin Schubert2020-01-169-27/+27
| | | | | | | | | | | | '_source_cached' is not explicit enough as it doesn't distinguishes between sources in their respective caches and sources in the global sourcecache.
| * source.py: Introduce methods to query state instead of get_consistencyBenjamin Schubert2020-01-163-9/+19
| | | | | | | | | | | | | | | | `get_consistency` doesn't allow being fine grained and asking only for a specific bit of information. This introduces methods `is_cached` and `is_resolved` which will be more flexible for refactoring.
| * tox.ini: Bump version of bst-plugins-experimental to 0.14.0Benjamin Schubert2020-01-161-2/+2
|/
* Merge branch 'chandan/remove-bst-experimental-unused' into 'master'Chandan Singh2020-01-161-6/+0
|\ | | | | | | | | tests/integration/project.conf: Remove unused external plugins See merge request BuildStream/buildstream!1799
| * tests/integration/project.conf: Remove unused external pluginschandan/remove-bst-experimental-unusedChandan Singh2020-01-151-6/+0
|/ | | | | There are no tests that rely on these plugins anymore, so remove the corresponding declarations from project.conf.
* Merge branch 'tpollard/removeassert' into 'master'Tom Pollard2020-01-151-10/+0
|\ | | | | | | | | | | | | element.py: Remove unused __staged_sources_directory & assert Closes #1249 See merge request BuildStream/buildstream!1798
| * element.py: Remove unused __staged_sources_directory & asserttpollard/removeassertTom Pollard2020-01-151-10/+0
|/
* Merge branch 'tpollard/messagerate' into 'master'Tom Pollard2020-01-154-13/+62
|\ | | | | | | | | Make message & status rendering be tick driven by default See merge request BuildStream/buildstream!1745
| * userconfig.yaml: Add bool for disabling console output throttlingtpollard/messagerateTom Pollard2020-01-154-3/+33
| | | | | | | | Also highlight the change & configuration in NEWS
| * _frontend/app.py: Make message printing tick drivenTom Pollard2020-01-151-12/+31
|/
* Merge branch 'chandan/gitlab-ci-toxenvs' into 'master'Chandan Singh2020-01-141-5/+2
|\ | | | | | | | | .gitlab-ci.yml: Simplify test environment declaration See merge request BuildStream/buildstream!1795
| * .gitlab-ci.yml: Simplify test environment declarationChandan Singh2020-01-141-5/+2
|/ | | | | | | Defining `EXTERNAL_TESTS_COMMAND` as an extra command makes it easy to miss it, especially when adding new environments (as I was recently bitten by it). Instead of that, we can control the environments using `TOXENV` variable.
* Merge branch 'bschubert/optimize-loading-multiple-targets' into 'master'Benjamin Schubert2020-01-142-3/+16
|\ | | | | | | | | loader.py: Optimize sorting of elements when they are multiple targets See merge request BuildStream/buildstream!1794
| * loader.py: Optimize sorting of elements when they are multiple targetsbschubert/optimize-loading-multiple-targetsBenjamin Schubert2020-01-142-3/+16
|/ | | | | | | Currently, with multiple targets, we would go through all the elements in the target and sort them, even though they might already be sorted. This ensures we sort every element only once.
* Merge branch 'bschubert/standardized-tests' into 'master'Benjamin Schubert2020-01-1327-351/+62
|\ | | | | | | | | introduce cross-repo standardized source tests See merge request BuildStream/buildstream!1757
| * tox.ini: Add a external plugins environment test and run it in CIBenjamin Schubert2020-01-134-9/+57
| | | | | | | | | | | | | | | | | | | | | | This runs two versions of the plugins: - The latest stable is not allowed failures and is run on every platform - The master version is allowed failure, and only runs on a single architecture This also adds a new entrypoint to register source tests to run against BuildStream.
| * tox.ini: Move coverage handling in commands_post.Benjamin Schubert2020-01-131-2/+3
| | | | | | | | | | This is not part of the test running itself, but more handling afterwards
| * tox.ini: Remove external plugins testsBenjamin Schubert2020-01-133-91/+3
| | | | | | | | | | These tests don't do anything, as they have no plugins registered at all.
| * tests: Remove some ostree stuffThomas Coldrick2020-01-132-2/+0
| |
| * Tests: Remove bst-plugins-experimental dependencyThomas Coldrick2020-01-1314-99/+2
| | | | | | | | This stops the tests from being interlinked for BuildStream itself.
| * Remove integration tests for moved pluginsThomas Coldrick2020-01-137-150/+0
| |
| * tox.ini: Remove duplicated move of the coverage fileBenjamin Schubert2020-01-131-1/+0
|/
* Merge branch 'juerg/batch-command-logging' into 'master'Jürg Billeter2020-01-135-7/+17
|\ | | | | | | | | _sandboxreapi.py: Improve batch command logging See merge request BuildStream/buildstream!1793
| * tests/integration/workspace.py: Relax assert for buildbox-runjuerg/batch-command-loggingJürg Billeter2020-01-131-2/+2
| | | | | | | | | | When commands are batched, the failure message can't include the command group label with buildbox-run.
| * _sandboxreapi.py: Improve batch command loggingJürg Billeter2020-01-133-5/+9
| | | | | | | | | | | | | | | | This adds a timed activity message to bring batch command logging for REAPI-based sandboxes in line with the default sandbox as far as possible. Thix fixes 3 xfails with buildbox-run.
| * sandbox.py: Add combined_label() method to batch itemsJürg Billeter2020-01-131-0/+6
|/ | | | This will be used to improve batch command logging with REAPI.
* Merge branch 'tpollard/loop_exception' into 'master'traveltissues/mr5Tom Pollard2020-01-102-2/+25
|\ | | | | | | | | | | | | scheduler.py: Handle exceptions that are caught under the event loop Closes #1245 See merge request BuildStream/buildstream!1788
| * scheduler.py: Handle exceptions that are caught under the event looptpollard/loop_exceptionTom Pollard2020-01-102-2/+25
|/ | | | | | The default exception handler of the async event loop bypasses our custom global exception handler. Ensure that isn't the case, as these exceptions are BUGS & should cause bst to exit.
* Merge branch 'tpollard/showjunction' into 'master'Tom Pollard2020-01-094-2/+20
|\ | | | | | | | | | | | | _frontend/widget.py: show_pipeline() Don't show buildable for junction Closes #1205 See merge request BuildStream/buildstream!1764
| * _frontend/widget.py: show_pipeline() Don't show buildable for junctionTom Pollard2020-01-094-2/+20
|/ | | | | Instead output "junction" in magenta for info. Also include test in format/junctions.py
* Merge branch 'tlater/pipeline-enums' into 'master'Tristan Maat2020-01-085-120/+162
|\ | | | | | | | | | | | | Make PipelineSelection a proper enum type Closes #1135 See merge request BuildStream/buildstream!1787
| * _pipeline.py: Refactor if/elif/else usageTristan Maat2020-01-081-16/+15
| |
| * Make PipelineSelection a proper enum typeTristan Maat2020-01-085-110/+153
|/ | | | | | | | | | | | | | | | | | PipelineSelection is one of the few stringy types that weren't converted to FastEnum, presumably because we lacked a mechanism for only allowing a sub-set of options as CLI arguments. We've re-designed this since, and as part of the UI/UX refactor we'd like to generally clean this, but that is probably still a while out. Since that hasn't happened, for now, this adds a feature to the FastEnumType that allows specifying only a subset of values is allowed for a specific command, so that we can use the type as a proper enum. We also get rid of a number of accidental uses of strings, and move PipelineSelection to buildstream.types so that we don't have a significant import overhead for it.
* Merge branch 'tlater/fix-monkeypatch-warning' into 'master'Tristan Maat2020-01-081-1/+1
|\ | | | | | | | | tests/artifactcache/config.py: Cast our tmpdir to a string See merge request BuildStream/buildstream!1790
| * tests/artifactcache/config.py: Cast our tmpdir to a stringtlater/fix-monkeypatch-warningTristan Daniël Maat2020-01-081-1/+1
|/ | | | Monkeypatch is *really* scared of non-str values. Let's calm it down.
* Merge branch 'chandan/no-warn-bwrap' into 'master'Javier Jardón2020-01-081-51/+0
|\ | | | | | | | | setup.py: Remove check for bubblewrap See merge request BuildStream/buildstream!1789
| * setup.py: Remove check for bubblewrapchandan/no-warn-bwrapChandan Singh2020-01-071-51/+0
|/ | | | | | | | | | | | | | Bubblewrap is not a hard requirement for BuildStream now that it supports other sandboxing mechanisms, i.e. buildbox-run and remote execution. Aside from that, this check is already a bit misplaced in `setup.py`. Bubblewrap is a runtime requirement, so warning about it at build time isn't quite right. It makes sense when BuildStream is being installed from source, but not when it is installed from any binary distribution. Note that the corresponding sandbox implementation already does this check at runtime.
* Merge branch 'chandan/tag-1.91.3' into 'master'1.91.3Chandan Singh2020-01-021-0/+18
|\ | | | | | | | | Prepare NEWS for 1.91.3 development snapshot release See merge request BuildStream/buildstream!1784
| * Prepare NEWS for 1.91.3 development snapshot releasechandan/tag-1.91.3Chandan Singh2020-01-021-0/+18
|/
View Lorry Controller Status