summaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
* git.py: Update git plugin to fail temporarily on clone failuresQinusty/397Josh Smith2018-07-262-4/+8
| | | | | The git plugin will now make use of the fail_temporarily parameter to Plugin.call(), allowing failures to trigger a retry.
* plugin.py: Extend Plugin.call() APIJosh Smith2018-07-262-8/+13
| | | | | | | | Plugin.call() now takes fail_temporarily as an optional parameter, when supplied it will cause subsequent failures to trigger temporary errors as opposed to permanent errors. This also extends Plugin.check_output() which makes use of Plugin.call()
* cascache.py: Update cache push/pull errors to be temporaryJosh Smith2018-07-262-3/+3
| | | | Further work needs to be done for the current grpc exceptions which are reraised.
* tests/sources: deb, tar and zip all test for retry behaviourJosh Smith2018-07-263-3/+6
| | | | | | This is following 5d6418a06a3a0262df5f6b9e6e1a36578515fd4a where DownloadableFileSource was modified to raise 'temporary' errors when relating to file downloading.
* _downloadablefilesource.py: Update network errors to be temporaryJosh Smith2018-07-261-2/+2
| | | | This follows the change in 67ecd97a05279a3b7570ad59f05bf0a5973ef04c.
* _exceptions.py: Modify BstError API to allow optional retryJosh Smith2018-07-264-13/+40
| | | | | | | | | | | | job.py: Changes to the logic surrounding retry attempts and child process return codes element.py, source.py: ElementError and SourceError also implement this change. These exceptions now have an optional parameter of temporary which defaults to false. This will potentially break backwards compatibility where exceptions were previously raised and a retry was intended. To trigger a retry, one must now raise their SourceError or ElementError with temporary=True. This aims to fix #397.
* Merge branch 'edbaunton/remote-source' into 'master'466-optimize-bst-build-initialization-timeEd Baunton2018-07-2616-1/+262
|\ | | | | | | | | | | | | Add remote source plugin Closes #163 See merge request BuildStream/buildstream!541
| * Provide explicit reason for SourceError exception on pathEd Baunton2018-07-262-2/+3
| | | | | | | | When the user provides a path for the filename parameter, provide a reason
| * Add remote source pluginEd Baunton2018-07-2516-1/+261
|/ | | | | | | | Add a plugin that supports downloading files verbatim from a source with an optional overridable filename and destination directory. Bumps bst format version to 10. Fixes #163
* Merge branch 'bst_workspace_open_force_does_nothing' into 'master'Jonathan Maw2018-07-252-1/+57
|\ | | | | | | | | _stream.py: Added functionality for workspace open -f See merge request BuildStream/buildstream!549
| * _stream.py: Added functionality for workspace open -fPhillip Smyth2018-07-252-1/+57
|/ | | | tests/frontend/workspace.py: Added tests
* Merge branch 'phil/203-BuildStream-crashes-when-dependency-tree-too-deep' ↵Jim MacArthur2018-07-252-2/+62
|\ | | | | | | | | | | | | into 'master' Phil/203 BuildStream crashes when dependency tree too deep See merge request BuildStream/buildstream!512
| * app.py: Handle exception thrown when recursion limit is exceededphil/203-BuildStream-crashes-when-dependency-tree-too-deepPhil Dawson2018-07-251-0/+4
| | | | | | | | | | Because the RecursionError exception was introduced in Python 3.5, until we drop support for for Python 3.4, we must use RuntimeError.
| * tests/frontend/show.py: Add test case for maximum recursion depth being exceededPhil Dawson2018-07-251-2/+58
|/ | | | | | | Add test to ensure gracefull handling of exception thrown while loading a pipeline due the python's max recursion depth being exceeded. This is part of the work for issue #203
* Merge branch 'Qinusty/501' into 'master'Jonathan Maw2018-07-251-1/+0
|\ | | | | | | | | Remove misleading info message See merge request BuildStream/buildstream!567
| * cascache.py: Remove misleading info messageJosh Smith2018-07-251-1/+0
|/ | | | This addresses issue #501.
* Merge branch 'jjardon/ostree_repo' into 'master'501-misleading-info-message-for-cascache-pullJavier Jardón2018-07-241-22/+15
|\ | | | | | | | | | | | | doc/source/install_linux_distro.rst: buildstream doesn't hard depend on ostree or pygobject anymore Closes #492 See merge request BuildStream/buildstream!558
| * doc/source/install_linux_distro.rst: buildstream doesn't depend on ostree or ↵jjardon/ostree_repoJavier Jardón2018-07-231-22/+15
|/ | | | pygobject anymore
* Merge branch 'caching_build_trees_limited' into 'master'Tristan Van Berkom2018-07-2327-26/+33
|\ | | | | | | | | Adding caching build trees (limited) See merge request BuildStream/buildstream!560
| * Adding caching build treesPhillip Smyth2018-07-2327-26/+33
|/ | | | | buildstream/element.py: Adding build tree to cache buildstream/_versions.py: Bumping BST_CORE_ARTIFACT_VERSION
* Merge branch 'Qinusty/275' into 'master'Tristan Van Berkom2018-07-232-18/+25
|\ | | | | | | | | | | | | Indicate where artifacts are going to and coming from in the log Closes #275 See merge request BuildStream/buildstream!553
| * cascache.py: Add pull info detailing which remote is being pulled fromQinusty/275Josh Smith2018-07-231-0/+3
| | | | | | | | Work towards fixing issue #275
| * cascache.py: Add push info detailing which remote is being pushed toJosh Smith2018-07-231-0/+4
| | | | | | | | Work towards fixing issue #275
| * element.py: Make Element._get_brief_display_key() available to Internal APIJosh Smith2018-07-231-18/+18
|/ | | | | This change will allow for more detailed info messages throughout the CASCache.
* Merge branch 'tristan/git-stage-with-shared-clone' into 'master'Tristan Van Berkom2018-07-231-4/+4
|\ | | | | | | | | | | | | plugins/sources/git.py: Use --shared instead of --hardlinks Closes #488 See merge request BuildStream/buildstream!557
| * plugins/sources/git.py: Use --shared instead of --hardlinksTristan Van Berkom2018-07-231-4/+4
|/ | | | | | | | This improves staging performance by avoiding making any copies of the objects in the cloned repository in the staging area before removing the .git directory. Fixes #488
* HACKING.rst: Added convention for naming of branches in merge requests.Tristan Van Berkom2018-07-221-0/+3
|
* HACKING.rst: No longer requiring issue numbers in commit messages.Tristan Van Berkom2018-07-221-18/+15
| | | | | Instead, we are enabling the merge commit feature in gitlab and hopefully reducing some of the friction in review this way.
* git source plugin: Omit the .git directory completely.Tristan Van Berkom2018-07-201-0/+3
| | | | | | | | This will break builds which use `git describe` to determine their version number until a more elegant solution is implemented, and will unblock work on caching of build trees in the artifacts. Fixes issue #455
* Updating .bzr plugin to omit the .bzr dirPhillip Smyth2018-07-201-0/+2
| | | | Fixes issue #455
* Fix crash when --debug is passedSam Thursfield2018-07-201-2/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | I hit the following backtrace running `bst --debug push` and `bst --debug build`: pid:16736 id:000[--:--:--][][] START Push pid:16736 id:000[--:--:--][][] START Loading pipeline pid:16736 id:000[00:00:00][][] SUCCESS Loading pipeline pid:16736 id:000[--:--:--][][] START Resolving pipeline pid:16736 id:000[--:--:--][][] BUG 'MesonElement' object has no attribute '_Element__cache_key' Traceback (most recent call last): File "/home/sam/.local/bin/bst", line 8, in <module> sys.exit(cli()) File "/home/sam/.local/lib/python3.6/site-packages/click/core.py", line 722, in __call__ return self.main(*args, **kwargs) File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_frontend/cli.py", line 162, in override_main standalone_mode=standalone_mode, **extra) File "/home/sam/.local/lib/python3.6/site-packages/click/core.py", line 697, in main rv = self.invoke(ctx) File "/home/sam/.local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/sam/.local/lib/python3.6/site-packages/click/core.py", line 895, in invoke return ctx.invoke(self.callback, **ctx.params) File "/home/sam/.local/lib/python3.6/site-packages/click/core.py", line 535, in invoke return callback(*args, **kwargs) File "/home/sam/.local/lib/python3.6/site-packages/click/decorators.py", line 27, in new_func return f(get_current_context().obj, *args, **kwargs) File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_frontend/cli.py", line 471, in push app.stream.push(elements, selection=deps, remote=remote) File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_stream.py", line 341, in push fetch_subprojects=True) File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_stream.py", line 814, in _load fetch_subprojects=fetch_subprojects) File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_pipeline.py", line 119, in load for meta in meta_elements File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_pipeline.py", line 119, in <listcomp> for meta in meta_elements File "/home/sam/.local/lib/python3.6/site-packages/buildstream/element.py", line 894, in _new_from_meta element = project.create_element(artifacts, meta) File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_project.py", line 184, in create_element element = self._element_factory.create(self._context, self, artifacts, meta) File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_elementfactory.py", line 57, in create return element_type(context, project, artifacts, meta, default_config) File "/home/sam/.local/lib/python3.6/site-packages/buildstream/element.py", line 196, in __init__ super().__init__(meta.name, context, project, meta.provenance, "element") File "/home/sam/.local/lib/python3.6/site-packages/buildstream/plugin.py", line 171, in __init__ self.debug("Created: {}".format(self)) File "/home/sam/.local/lib/python3.6/site-packages/buildstream/plugin.py", line 407, in debug self.__message(MessageType.DEBUG, brief, detail=detail) File "/home/sam/.local/lib/python3.6/site-packages/buildstream/plugin.py", line 652, in __message self.__context.message(message) File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_context.py", line 338, in message self._message_handler(message, context=self) File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_frontend/app.py", line 665, in _message_handler text = self.logger.render(message) File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_frontend/widget.py", line 585, in render return self._render(message) File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_frontend/widget.py", line 620, in _render text += widget.render(message) File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_frontend/widget.py", line 228, in render _, key, missing = plugin._get_display_key() File "/home/sam/.local/lib/python3.6/site-packages/buildstream/element.py", line 1138, in _get_display_key cache_key = self._get_cache_key() File "/home/sam/.local/lib/python3.6/site-packages/buildstream/element.py", line 991, in _get_cache_key return self.__cache_key AttributeError: 'MesonElement' object has no attribute '_Element__cache_key' The issue here is that when `--debug` is enabled, Plugin.__init__() tries to log a message like this: pid:16929 id:001[--:--:--][????????][ main:tracker.bst ] DEBUG Created: meson element at tracker.bst [line 1 column 0] The log formatter is trying to get the cache key of the element in order to show it in the log, but the attribute hasn't yet been set because we've not got to the Element() constructor.
* setup.py: Specify minimum required version of pytest-cov pluginTristan Van Berkom2018-07-201-1/+1
| | | | | | This causes the new artifact tests to pass (unless you happened to already have a recent enough version of pytest-cov, in which case you didn't notice the breakage).
* Add support for creating a tarball on bst checkouttiagogomes/tarball_checkoutTiago Gomes2018-07-205-38/+227
| | | | | | | | One of the tests added is configured to be skipped for now, as dumping binary data is causing a bad descriptor exception when using the pytest capture module. Closes #263.
* Update checkout man page for new --deps optionTiago Gomes2018-07-201-1/+4
|
* install_linux_distro.rst: Document plugins deps for the rest of the systemsJavier Jardón2018-07-201-2/+16
|
* install_linux_distro.rst: document plugins installed by defaultJavier Jardón2018-07-201-0/+8
|
* Added dependency to the DocsWilliam Salmon2018-07-201-0/+5
| | | | | | | | | | | These dependencies are needed to use the plugins that are installed as part of this guild. As the dependencies added are not dependencies of the core package they have been added to a separate section. The lzip package is needed to build gnome. https://wiki.gnome.org/Newcomers/BuildSystemComponent
* Docs: Update the required build packages for the rest of the systemsJavier Jardón2018-07-201-9/+10
| | | | | | | Also remove the recomendation to install psutil as we need to build other python modules anyway (like ruamel) Completes 96d07153b7817cdaeda57dd163eed52b2b1b31e8
* Docs: Update the required build packages for fedora based systems.Jordan Petridis2018-07-201-1/+1
| | | | | | `ruamel.yaml` seems to require `Python.h` header file to build. `python3-devel` is what provides it for Fedora.
* source-bundle: Enable --except optionfranred/fix-except-argument-in-source-bundleFrancisco Redondo Marchena2018-07-192-2/+5
| | | | | | | Before this option was listed in help but was ignored when creating the source-bundle. Issue #468
* scheduler.py: Correct some anomalies within the docstringsQinusty/scheduler_docstring_fixJosh Smith2018-07-191-2/+4
|
* .gitlab-ci.yml: Use testsuite images for running testschandan/use-testsuite-fedoraChandan Singh2018-07-181-4/+4
| | | | | | | | | | | | | | | Instead of re-using the user-facing `buildstream/buildstream-fedora` image for tests (that comes with BuildStream pre-installed), use `buildstream/testsuite-fedora` image that is supposed to be used exclusively for these tests. Similarly, use the dedicated `buildstream/testsuite-debian` image instead of `buildstream/buildstream-debian`. Fixes #458. For related discussion, see https://gitlab.com/BuildStream/buildstream-docker-images/issues/8.
* Reorder app.py imports1.3.0Daniel Playle2018-07-181-3/+2
| | | | | | | | When testing locally on some systems, it appears that pylint fails on the order of imports for `buildstream/_frontend/app.py`. This commit changes the order of these imports. This commit addresses issue !453.
* _stream.py: Clarifying ambiguous FIXME comment about required artifactsbst-1-branchpointTristan Van Berkom2018-07-181-3/+6
|
* Refactor: Use new logging mechanism from Contexttristan/local-cache-expiryTristan Van Berkom2018-07-188-318/+55
| | | | | | | | | | | | | o element.py / plugin.py: Removed supporting logging code, and derive the log handle from Context. o _scheduler/scheduler.py, _scheduler/queues/queue.py: Adapt to new Job initializer API for the logfile o _scheduler/jobs/job.py: Run job activities within the new context manager which turns on logging for a job. Also removed a lot of custom logging abstract methods which are unneeded. o _scheduler/jobs: Job implementations need not implement so much custom logging.
* _context.py: Adding apis for message recording and task logging.Tristan Van Berkom2018-07-181-1/+138
| | | | | | | | | | | | Added the new Context.recorded_messages() context manager, this causes messages to be logged in a dedicated log file before being propagated to the log handler, and also provides an open log handle for the capturing and logging of stdout/stderr from child processes. This comes with the additional accessors: Context.get_log_handle() Context.get_log_filename()
* NEWS: Add news about cache expiryTristan Maat2018-07-181-0/+4
|
* tests/artifactcache/expiry.py: Add expiry testsTristan Maat2018-07-182-0/+278
|
* Clean the artifact cache when we hit the cache quotaTristan Maat2018-07-186-3/+201
| | | | | When the cache quota is hit, we will remove any elements not required for the current build, until our cache is only filled halfway.
* Compute the artifact cache size after each build/pullTristan Maat2018-07-187-2/+127
|