summaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
* artifactcache: implement new push methodsraoul/802-refactor-artifactcacheRaoul Hidalgo Charman2019-01-155-141/+166
| | | | | | | | Similar to the pull methods, this implements a yield_directory_digests methods that iterates over blobs in the local CAS, with the upload_blob sending blobs to a remote and batching them where appropriate. Part of #802
* artifactcache: Move pull logic into CASRemoteRaoul Hidalgo Charman2019-01-1510-250/+299
| | | | | | | | | | | | | Seperates the pull logic into a remote/local API, so that artifact cache iterates over blob digests checks whether it has them, and then requests them if not. The request command allows batching of blobs where appropriate. Tests have been updated to ensure the correct tmpdir is set up in process wrappers, else invalid cross link errors happen in the CI. Additional asserts have been added to check that the temporary directories are cleared by the end of a pull. Part of #802
* tmpdir: add tmpdir to context for CASRemoteRaoul Hidalgo Charman2019-01-155-11/+20
| | | | | | | | | | | As CASRemote is used in other places such as SandboxRemote it makes sense for there to be a tmpdir it uses. This dir is passed to CASRemote when initialized. This is currently in the artifactdir, but should be moved when artifactdir and builddir options are deprecated in favour of one top level directory containing the two. Part of #802
* cas: move remote only functions to CASRemoteRaoul Hidalgo Charman2019-01-154-144/+148
| | | | | | | | | | | | List of methods moved * Initialization check: made it a class method that is run in a subprocess, for when checking in the main buildstream process. * fetch_blobs * send_blobs * verify_digest_on_remote * push_method Part of #802
* casremote.py: Move remote CAS classes into its own fileRaoul Hidalgo Charman2019-01-154-257/+283
| | | | Part of #802
* _cas: Rename artifactcache folder and move that to a root moduleRaoul Hidalgo Charman2019-01-1513-21/+20
| | | | | | | | | Other components will start to reply on cas modules, and not the artifact cache modules so it should be organized to reflect this. All relevant imports have been changed. Part #802
* Merge branch 'tristan/element-processing-order' into 'master'Tristan Van Berkom2019-01-113-1/+128
|\ | | | | | | | | Make build plan element list stable See merge request BuildStream/buildstream!1058
| * tests/frontend/order.py: Adding tests ensuring stable element processing orderTristan Van Berkom2019-01-101-0/+109
| | | | | | | | | | | | | | | | | | | | | | | | | | For each sample project and expected result order, this test ensures that: * bst show --deps plan: Always shows the expected build order. * bst source fetch: Always executes in the expected build order. * bst build: Always builds in the expected order (disabled). The build part of the test is currently disabled as the scheduler seems to be messing with the order even in a `--builders 1` scenario.
| * testutils/runcli.py: Added Result.get_start_order()Tristan Van Berkom2019-01-101-0/+17
| | | | | | | | | | Added a function to report the list of elements which appeared in a given queue in the order of their first appearance.
| * _pipeline.py: Fix the planner to retain the original order when depth sortingTristan Van Berkom2019-01-101-1/+2
|/ | | | | | | | | | | The algorithm adds elements to a dictionary and then sorts the dictionary by the discovered depths while recursing - using an OrderedDict is enough to ensure a stable order. This fixes `bst show --deps plan ...` reporting different results on each invocation. This fixes an aspect of #824
* Merge branch 'aevri/contrib-typos' into 'master'Angelos Evripiotis2019-01-101-11/+11
|\ | | | | | | | | contributing: fix 'oprtation' and some other typos See merge request BuildStream/buildstream!1055
| * contributing: fix 'oprtation' and some other typosAngelos Evripiotis2019-01-101-11/+11
|/
* Merge branch 'juerg/arch' into 'master'Jürg Billeter2019-01-108-16/+146
|\ | | | | | | | | Accept architecture aliases See merge request BuildStream/buildstream!1034
| * tests/format/optionarch.py: Add tests for architecture aliasesJürg Billeter2019-01-105-0/+81
| |
| * element.py: Accept architecture aliases for sandbox configjuerg/archJürg Billeter2019-01-101-1/+7
| | | | | | | | | | Accept common architecture aliases for the sandbox config for consistency with arch options.
| * _options/optionarch.py: Accept architecture aliasesJürg Billeter2019-01-101-1/+30
| | | | | | | | | | | | | | | | Accept common architecture aliases for arch options instead of only accepting the canonicalized, OS-independent architecture name. This restores compatibility with existing projects and may simplify option handling for projects that only target a single OS (and thus, do not need OS-independent architecture names).
| * _platform/platform.py: Add canonicalize_arch() methodJürg Billeter2019-01-101-14/+28
|/
* Merge branch 'tristan/refine-coverage' into 'master'Tristan Van Berkom2019-01-091-1/+3
|\ | | | | | | | | .coveragerc: Omit some things which are irrelevant to cover. See merge request BuildStream/buildstream!1052
| * .coveragerc: Omit some things which are irrelevant to cover.Tristan Van Berkom2019-01-091-1/+3
|/ | | | | | | * Omit versioneer's _version.py * Omit our __main__.py which is used only internally for generating documentation
* Merge branch 'tristan/detox-tests' into 'master'Tristan Van Berkom2019-01-095-19/+76
|\ | | | | | | | | | | | | Allow using detox, and fix/refactor collection of coverage reports Closes #844 See merge request BuildStream/buildstream!1051
| * CONTRIBUTING.rst: Updated to mention new `coverage` tox environment.Tristan Van Berkom2019-01-091-0/+20
| | | | | | | | | | Also point out that it is possible to run test environments in parallel using the `detox` tool.
| * Support running test environments in parallel with `detox`Tristan Van Berkom2019-01-094-19/+56
|/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch namespaces the test temp directory and the output coverage report file with the name of the environment under test, such that separately run tests do not access the same files. When running tests without tox, directly through setup.py, then the tmp directory will still be `./tmp`. * .gitignore: Added new .coverage-reports/ directory * .gitlab-ci.yml: Rely on tox to combine and report coverage, only tell tox about the COVERAGE_PREFIX so that results can be namespaced by CI job name. This change also publishes the sources and final combined `.coverage` file in an output gitlab artifact for inspection, and lists some missing dependencies to the `coverage` job. * tox.ini: Add comments and refactor main [testenv] section so that other environments dont inherit too much unrelated cruft. Generate the coverate reports in the respective {envtmpdir} so that all per-process coverage files are prefixed with a full path, ensuring that concurrent runs don't mix reports and addressing concerns raised in #844. Also implemented new `tox -e coverage` environment to combine any found coverage and print a report. * .coveragerc: Omit .tox/ directory from coverage stats Fixes issue #844
* Merge branch 'jmac/rename_instance_name' into 'master'Jim MacArthur2019-01-091-2/+2
|\ | | | | | | | | Rename 'instance_name' option to 'instance-name' See merge request BuildStream/buildstream!1048
| * sandboxremote.py: Rename 'instance_name' option to 'instance-name'jmac/rename_instance_nameJim MacArthur2019-01-091-2/+2
|/ | | | It was 'instance-name' in the documentation.
* Merge branch 'valentindavid/script_virtual_directory' into 'master'Valentin David2019-01-091-0/+3
|\ | | | | | | | | | | | | buildstream/plugins/elements/script.py: Mark script as BST_VIRTUAL_DIRECTORY Closes #850 See merge request BuildStream/buildstream!1047
| * buildstream/plugins/elements/script.py: Mark script as BST_VIRTUAL_DIRECTORYvalentindavid/script_virtual_directoryValentin David2019-01-091-0/+3
|/ | | | | | | ScriptElement does not use Sandbox.get_directory. It works using it with remote execution. Fixes #850
* Merge branch 'valentindavid/remote_execution_configuration' into 'master'Valentin David2019-01-095-13/+115
|\ | | | | | | | | | | | | Remote execution configuration: HTTPS and user configuration Closes #780 and #631 See merge request BuildStream/buildstream!1030
| * Add support for https channel to remote execution and actions serversvalentindavid/remote_execution_configurationValentin David2019-01-092-11/+35
| | | | | | | | Fixes #780.
| * Add support for user remote execution configurationValentin David2019-01-095-4/+72
| | | | | | | | Fixes #631.
| * Use relative path to project directory for remote execution certificates/keysValentin David2019-01-091-0/+10
|/
* Merge branch 'tristan/sandbox-programming-error' into 'master'Tristan Van Berkom2019-01-081-4/+4
|\ | | | | | | | | sandbox/sandbox.py: Use assertions for programming errors instead of BstErrors. See merge request BuildStream/buildstream!1046
| * sandbox/sandbox.py: Use assertions for programming errors instead of BstErrors.tristan/sandbox-programming-errorTristan Van Berkom2019-01-081-4/+4
|/ | | | | When code is faulty, in a plugin or in the core, we should get a stack trace and a BUG message.
* Merge branch 'phil/remove-source-bundle-reference' into 'master'Jürg Billeter2019-01-081-1/+1
|\ | | | | | | | | element.py: remove documentation reference to source bundle command See merge request BuildStream/buildstream!1041
| * element.py: remove reference to source bundle commandphil/remove-source-bundle-referencePhil Dawson2019-01-081-1/+1
|/ | | | This command has been replacved by the bst source checkout command
* Merge branch 'tristan/fix-command-status-messages' into 'master'Tristan Van Berkom2019-01-081-1/+1
|\ | | | | | | | | sandbox/sandbox.py: Fix regression of command logging See merge request BuildStream/buildstream!1044
| * sandbox/sandbox.py: Fix regression of command loggingTristan Van Berkom2019-01-071-1/+1
|/ | | | | | | | | | Since we added batch commands, the batch commands print the text of the commands directly in the message text, but this is wrong. The detail string is the appropriate place for text of unknown lengths (the user can actually configure how many max lines of commands they want to see in their log), the message text itself should be controlled and brief enough to avoid text wrapping.
* Merge branch 'tristan/one-cache-size-job' into 'master'Tristan Van Berkom2019-01-0713-82/+133
|\ | | | | | | | | | | | | Only queue one cache size job Closes #753 See merge request BuildStream/buildstream!1040
| * _scheduler/jobs/job.py: Removed 'skipped' propertyTristan Van Berkom2019-01-072-20/+2
| | | | | | | | This is redundant now that we report it through the JobStatus.
| * Scheduler: Introduced JobStatus instead of simple success booleanTristan Van Berkom2019-01-0713-38/+68
| | | | | | | | | | | | | | | | | | | | This changes the deepest callback from when a Job completes to propagate a JobStatus value instead of a simple boolean, and updates all of the effected code paths which used to receive a boolean to now handle the JobStatus values. This further improves the situation for issue #753, as now we avoid queueing cache size jobs for pull jobs which are skipped.
| * _scheduler/scheduler.py: Only run one cache size job at a timeTristan Van Berkom2019-01-071-8/+45
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When queuing the special cache management related cleanup and cache size jobs, we now treat these jobs as special and do the following: * Avoid queueing a cleanup/cache_size job if one is already queued We just drop redundantly queued jobs here. * Ensure that jobs of this type only run one at a time This could have been done with the Resources mechanics, however as these special jobs have the same properties and are basically owned by the Scheduler, it seemed more straight forward to handle the behaviors of these special jobs together. This fixes issue #753
| * _scheduler/scheduler.py: Make _schedule_jobs() privateTristan Van Berkom2019-01-071-16/+18
|/ | | | | | | | This is not used anywhere outside of the Scheduler, currently only the Scheduler itself is allowed to queue a job at this level. If the highlevel business logic for automatic queueing of auxiliary jobs moves to another location, we can make this public again.
* Merge branch 'tristan/keyboard-interrupt-stack-trace' into 'master'Tristan Van Berkom2019-01-072-10/+6
|\ | | | | | | | | Fix stack traces discovered with ^C forceful termination. See merge request BuildStream/buildstream!1043
| * Fix stack traces discovered with ^C forceful termination.tristan/keyboard-interrupt-stack-traceTristan Van Berkom2019-01-072-10/+6
|/ | | | | | | | | | | | | | | | * utils.py:_kill_process_tree(): Ignore NoSuchProcess errors These are caused because we issue SIGTERM, and if the process has not exited after a timeout, we kill it. * _scheduler/jobs/job.py: Stop handling NoSuchProcess errors here redundantly, they are already ignored. It seems that we were ignoring it after sleeping when terminating tasks from the scheduler... but we were not ignoring it when performing the same pattern in the `Plugin.call()` -> `utils._call()` path, so we were still getting these exceptions at termination time from host tool processes launched by source plugins.
* Merge branch 'juerg/shell-buildtree' into 'master'Tristan Van Berkom2019-01-061-1/+1
|\ | | | | | | | | _frontend/app.py: Use buildtree for interactive shell on build failure See merge request BuildStream/buildstream!1039
| * _frontend/app.py: Use buildtree for interactive shell on build failureJürg Billeter2019-01-061-1/+1
|/ | | | Fixes: e29aea36 ("Basic options for shell --build to use buildtrees")
* Merge branch 'chandan/no-sdist' into 'master'Chandan Singh2019-01-041-44/+5
|\ | | | | | | | | .gitlab-ci.yml: Remove prepare stage See merge request BuildStream/buildstream!1037
| * .gitlab-ci.yml: Remove prepare stageChandan Singh2019-01-041-44/+5
|/ | | | | | | | | | As we now run tests using `tox`, we don't need to worry about manually packing and unpacking BuildStream. So, we can remove the preapre stage entirely. Update `coverage` and nightly jobs to appropriately cope with this change. Both these jobs now install all runtime dependencies from requirements files.
* Merge branch 'chandan/update-requirements-one-liner' into 'master'Tristan Van Berkom2019-01-0412-21/+60
|\ | | | | | | | | Add Makefile to update requirements files See merge request BuildStream/buildstream!1035
| * CONTRIBUTING.rst: Add instructions to update requirements fileschandan/update-requirements-one-linerChandan Singh2019-01-042-5/+23
| | | | | | | | | | | | | | | | Split the "The MANIFEST.in and setup.py" section in two: "Managing data files" and "Updating BuildStream's Python dependencies". Briefly explain the layout of `requirements` directory and add instructions to use the Makefile added in the last commit.
| * Add Makefile to update requirements filesChandan Singh2019-01-041-0/+21
| |
View Lorry Controller Status