summaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
* job.py: Remove the ability to send back child databschubert/merge-parent-child-jobBenjamin Schubert2020-08-293-36/+5
| | | | | We only use this for workspace information that we can now get directly since we run in the same process
* downloadablefilesource.py: Reset the file opener between every testBenjamin Schubert2020-08-292-1/+7
| | | | | | | This is required when we run this in the main process, with the threaded scheduler rework. Otherwise the state is kept between tests
* scheduler.py: Use threads instead of processes for jobsBenjamin Schubert2020-08-2915-448/+315
| | | | | | | This changes how the scheduler works and adapts all the code that needs adapting in order to be able to run in threads instead of in subprocesses, which helps with Windows support, and will allow some simplifications in the main pipeline.
* _signals.py: allow calling signal handler from non-main threadsBenjamin Schubert2020-08-292-8/+12
| | | | | | | | | | | | | | | | | | This modifies the signal terminator so that it can be called from any thread. This checks that either: - The signal handler is already in place - Or the caller is in the main thread, allowing to set the signal handler. This also removes the exact callback that was added instead of removing the last one, and fixes the `suspend_handler` to do the same. This is required, as we don't know which interleaving of calls will be done, and we can't guarantee that the last one is the right one to remove
* utils.py: Don't block on the call's `communicate` callBenjamin Schubert2020-08-291-1/+15
| | | | | | This ensures that, if we were to receive signals or other things while we are on this blocking call, we would be able to process them instead of waiting for the end of the process
* utils.py: Catch correctly if a process is dead when trying to kill itBenjamin Schubert2020-08-291-2/+12
| | | | | | | | Before, we would throw an exception that the process is not dead yet if it happened to die at the exact moment we tried to access it. This ensures we don't throw exceptions in such cases, since the process would be already dead, as we wanted
* Merge branch 'bschubert/remove-urllib-warnings' into 'master'bst-marge-bot2020-08-291-15/+8
|\ | | | | | | | | downloadablefilesource: Remove deprecated methods in python3.8 See merge request BuildStream/buildstream!2046
| * downloadablefilesource: Remove deprecated methods in python3.8bschubert/remove-urllib-warningsBenjamin Schubert2020-08-281-15/+8
|/ | | | | | splituser and such are deprecated since python3.8 We should stop using them
* Merge branch 'qinusty/bb-asset-hub-ci' into 'master'bst-marge-bot2020-08-267-6/+152
|\ | | | | | | | | Add artifact cache testing with buildbarn Asset API implementation See merge request BuildStream/buildstream!2037
| * Update overnight test cache endpointJosh Smith2020-08-261-1/+1
| | | | | | | | | | This new endpoint targets a buildbarn deployment with bb-storage alongside bb-asset-hub for index storage.
| * .gitlab-ci.yml Improve remote-test usage documentationJosh Smith2020-08-261-6/+8
| |
| * Add bb-storage/bb-asset-hub cache test jobJosh Smith2020-08-264-0/+127
| |
| * runcli.py: Configurable index/storage artifact typesJosh Smith2020-08-263-1/+18
|/ | | | | This enables for seperate index/storage artifact servers to be configured by environment variables passed through tox.
* Merge branch 'juerg/source-cache-key' into 'master'bst-marge-bot2020-08-2529-29/+29
|\ | | | | | | | | _elementsources.py: Fix source names in unique key See merge request BuildStream/buildstream!2023
| * _elementsources.py: Fix source names in unique keyjuerg/source-cache-keyJürg Billeter2020-08-2529-29/+29
|/ | | | | | | | | | | | | | The goal was to include the source plugin kind in the element cache key as the unique key of a source may not be unique across different source plugins. This is the source equivalent of the `element-plugin-name` value in the element cache key. However, `Source._get_source_name()` was the wrong method for this as that also includes the key itself, which may not even be set yet. This results in a cache key change. Fixes: 3953bcc6 ("element.py: clobber sources with workspace")
* Merge branch 'bschubert/remove-custom-sched-messages' into 'master'bst-marge-bot2020-08-231-43/+0
|\ | | | | | | | | job.py: Remove 'SUBCLASS_CUSTOM_MESSAGE', it is never used See merge request BuildStream/buildstream!2039
| * job.py: Remove ability of job classes to send custom messagesbschubert/remove-custom-sched-messagesBenjamin Schubert2020-08-231-43/+0
|/ | | | | | | | We previously were sending custom messages from child jobs to parent jobs for example for reporting the cache size. This is not used anymore by the current implementation. Let's remove this entirely
* Merge branch 'bschubert/stricter-thread-checks' into 'master'bst-marge-bot2020-08-233-6/+17
|\ | | | | | | | | _fixtures.py: Only get the normal number of threads at the start of session See merge request BuildStream/buildstream!2038
| * _fixtures.py: Only get the normal number of threads at the start of sessionBenjamin Schubert2020-08-233-6/+17
|/ | | | | This ensures that we also have our tests correctly shutting down background threads and not interferring with each other
* Merge branch 'bschubert/timed-suspendable' into 'master'bst-marge-bot2020-08-232-50/+39
|\ | | | | | | | | _messenger.py: Make `timed_suspendable` public and use it in job.py See merge request BuildStream/buildstream!2040
| * _messenger.py: Make `timed_suspendable` public and use it in job.pybschubert/timed-suspendableBenjamin Schubert2020-08-222-50/+39
|/ | | | This reduces the amount of code duplication
* Merge branch 'tristan/build-root-cache-key' into 'master'Tristan Van Berkom2020-08-2131-30/+42
|\ | | | | | | | | | | | | Consider %{build-root} in the cache key unconditionally Closes #1386 See merge request BuildStream/buildstream!2036
| * buildstream/element.py: Consider "build-root" variable in the cache key ↵tristan/build-root-cache-keyTristan van Berkom2020-08-2029-28/+40
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | unconditionally Not all elements use the "build-root" variable, but it is the standard variable to use for the build directory, and the build directory must be considered in the cache key. Handling this unilaterally in the core is safer than delegating this to element implementations, as we have less chance of plugin authors missing this detail and possibly introducing binary variance for artifacts where only the build directory differs (something which happens when the project name or element names change). This commit also updates the hard coded cache keys in the cache key test, so as to ensure every commit passes it's own tests. This fixes #1386.
| * tests/cachekey/cachekey.py: Use the same project name for cache key testsTristan van Berkom2020-08-201-1/+1
| | | | | | | | | | | | | | | | In the test_cache_key_fatal_warnings() test, use the same project name in both generated project directories in order to pass the tests in the case that elements are guaranteed to have differing cache keys for differeing element/project names (which is the case when we consider the "build-root" in the cache key unconditionally).
| * testing/_cachekeys.py: Fix formatting in cache key failure assertionTristan van Berkom2020-08-201-1/+1
|/
* Merge branch 'chandan/1.93.5' into 'master'1.93.5Chandan Singh2020-08-191-0/+8
|\ | | | | | | | | NEWS: Prepare for next development snapshot - 1.93.5 See merge request BuildStream/buildstream!2029
| * NEWS: Prepare for next development snapshot - 1.93.5chandan/1.93.5Chandan Singh2020-08-191-0/+8
|/ | | | | Among other things, this will allow BuildStream plugins that rely on our `testing` module to update to more recent pytest versions.
* Merge branch 'chandan/upgrade-ruamel' into 'master'bst-marge-bot2020-08-193-1/+37
|\ | | | | | | | | | | | | Bump minimum version requirement for ruamel.yaml Closes #1265 See merge request BuildStream/buildstream!2028
| * Bump minimum version requirement for ruamel.yamlchandan/upgrade-ruamelChandan Singh2020-08-183-1/+37
|/ | | | | | | | | | | | ruamel.yaml <= 0.16.6 suffers from a bug where ruamel's yaml dumper crashes when used on a sequence that has comments before it. In BuildStream, this manifests in form of issues like #1265. See upstream issue at https://sourceforge.net/p/ruamel-yaml/tickets/335. Also, add a regression test for it. Fixes #1265.
* Merge branch 'tristan/docs-fixes' into 'master'Tristan Van Berkom2020-08-172-5/+8
|\ | | | | | | | | Minor documentation fixes See merge request BuildStream/buildstream!2034
| * plugin.py: Fix documentation reference to MappingNode.validate_keys()tristan/docs-fixesTristan van Berkom2020-08-171-1/+1
| |
| * doc/source/format_declaring.rst: Documenting the ``all`` dependency typeTristan van Berkom2020-08-171-4/+7
|/ | | | | | The loader has supported dependencies explicitly being ``all`` for the dependency type (instead of only "build", "runtime" and unspecified) for a long time, but this was not properly documented.
* Merge branch 'chandan/cli-track-deps' into 'master'bst-marge-bot2020-08-1435-29/+96
|\ | | | | | | | | _frontend/cli: Add --deps `build` & `run` values for `source track --deps` See merge request BuildStream/buildstream!2027
| * NEWS: Add entry for `source track --deps build|run`chandan/cli-track-depsChandan Singh2020-08-141-0/+1
| |
| * Re-generate man pagesChandan Singh2020-08-1425-26/+26
| |
| * _frontend/cli: Add --deps `build` & `run` values for `source track --deps`Chandan Singh2020-08-149-3/+69
|/ | | | | | | This is part of #1349. This patch will conclude the first part of that issue, i.e. ensuring that the possible options for `--deps` are consistent across all commands (with the exception of `--deps plan` that we will handle separately).
* Merge branch 'dwinship/test_for_dot_bst_artifact_checkout_bug' into 'master'bst-marge-bot2020-08-141-0/+24
|\ | | | | | | | | regression-test for ".bst" artifact checkout bug See merge request BuildStream/buildstream!2014
| * regression-test for ".bst" artifact checkout bugdwinship/test_for_dot_bst_artifact_checkout_bugDouglas Winship2020-08-131-0/+24
|/
* Merge branch 'juerg/remote-asset' into 'master'bst-marge-bot2020-08-1334-2519/+3952
|\ | | | | | | | | | | | | Use Remote Asset API for artifact and source caches Closes #1273 and #1272 See merge request BuildStream/buildstream!1978
| * NEWS: Add note about the switch to the Remote Asset APIJürg Billeter2020-08-131-0/+8
| |
| * doc: Update for Remote Asset APIJürg Billeter2020-08-133-11/+10
| |
| * casremote.py: Migrate to GetInstanceNameForRemotes()Jürg Billeter2020-08-131-7/+17
| | | | | | | | GetInstanceNameForRemote() has been deprecated.
| * _protos: Update local_cas.proto from buildbox-commonJürg Billeter2020-08-133-8/+257
| |
| * _protos: Drop BuildStream Artifact and Source servicesJürg Billeter2020-08-139-809/+5
| | | | | | | | Replaced by Remote Asset API Fetch and Push services.
| * casserver.py: Drop BuildStream Artifact and Source servicesJürg Billeter2020-08-132-325/+0
| | | | | | | | Replaced by Remote Asset API Fetch and Push services.
| * _assetcache.py: Drop index_remote_class and storage_remote_classJürg Billeter2020-08-133-11/+6
| | | | | | | | They no longer differ among subclasses.
| * _sourcecache.py: Use AssetRemoteJürg Billeter2020-08-132-113/+48
| | | | | | | | | | This migrates the source cache from the BuildStream Source protocol to the Remote Asset API.
| * _artifactcache.py: Use AssetRemoteJürg Billeter2020-08-132-187/+117
| | | | | | | | | | | | | | This migrates the artifact cache from the BuildStream Artifact protocol to the Remote Asset API. Co-authored-by: Sander Striker <s.striker@striker.nl>
| * _assetcache.py: Add fetch_directory() and push_directory() methodsJürg Billeter2020-08-131-0/+78
| |
| * _assetcache.py: Add AssetRemote for Remote Asset APIJürg Billeter2020-08-131-3/+147
| | | | | | | | Co-authored-by: Sander Striker <s.striker@striker.nl>
View Lorry Controller Status