summaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
* .coveragerc: Omit .tox/ directory from coverage statschandan/toxify-1.2Chandan Singh2019-01-281-0/+2
| | | | | This is relevant bit of change from the MR !1051, to fix the coverage stats.
* .coveragerc: Omit some things which are irrelevant to cover.Tristan Van Berkom2019-01-281-1/+3
| | | | | | | | | * Omit versioneer's _version.py * Omit our __main__.py which is used only internally for generating documentation (cherry picked from commit 116c1070c893c4b5bd2505ea39157f1b37946092)
* bst2html.py: Name the second argument to avoid an errorJim MacArthur2019-01-281-1/+1
| | | | | | (cherry picked from commit 1fd38fb74c81d39ca7500d9b7f77c67430ed7852) Backport of !874.
* .gitlab-ci.yml: Run tests using toxChandan Singh2019-01-281-110/+35
| | | | | | | | | This commit a backport of a handful of merge requests to master. As these two branches have diverged a bit and the master branch has seen a fair bit of churn lately, it was not trivial to simply cherry-pick the changes. Some related merge reqtests are: !1027, !1037 and !1051.
* .gitignore: Ingnore tox, pytest, coverage filesChandan Singh2019-01-281-0/+3
|
* Add tox.ini from master branchChandan Singh2019-01-281-0/+70
| | | | | | | | | | | `tox.ini` has seen heavy churn lately, and backporting individual commits is turning out to be a more involved process as the two branches have diverged a bit. So, this is more or less copy-pasted from the master branch, without a separate lint job. Since switching to `pycodestyle` will require some code formatting changes, that does not seem reasonable to apply to this branch. So, let's stick with running the linters via `pytest`.
* Add requirements files for install, test and plugin dependenciesChandan Singh2019-01-199-12/+87
| | | | | | | | | | | | | | | | | | | | | Add `.in` and `.txt` requirements files for BuildStream's pure python dependencies. For each pair, the `.in` file is supposed to capture the loose version requirements, and the corresponding `.txt` file is supposed to have frozen requirements. We have 3 such sets: * `requirements`: BuildStream's runtime dependencies * `dev-requirements`: Dependencies for running tests * `plugin-requirements`: Dependencies for core plugins Note that the frozen requirements files will only be used for testing purposes, and `setup.py` will continue to read loose requirements. This commit comes with a Makefile to update the requirements files as well. --- !1027 is related MR for the master branch.
* conftest.py: Don't use deprecated get_marker() functionChandan Singh2019-01-191-1/+1
| | | | | | | | | | | | | | | | | Starting from `pytest` version 4.1.0, `Node.get_marker()` has been removed, and hence our tests break when running with newer versions of `pytest`. It was deprecated since a while back but it has recently been removed completely. Use `get_closest_marker()` as a replacement that is suggested in the changelog, and seems to work fine for our use case. See https://github.com/pytest-dev/pytest/pull/4564 for more context on the upstream issue. One way of verifying this change is that this should fix the recently added `tests-fedora-update-deps` job, that was failing before due to this issue. (cherry picked from commit c91784abcdfebef2e1e70612193b67efe7fa016a)
* Merge branch 'valentindavid/git_force_fetch_tags-1.2' into 'bst-1.2'Valentin David2018-12-113-2/+113
|\ | | | | | | | | [backport 1.2] Force updating tags when fetching git repository See merge request BuildStream/buildstream!1005
| * Force updating tags when fetching git repositoryValentin David2018-12-113-2/+113
|/ | | | | | | | | When using aliases there are multiple remotes used in the cache repository. When fetching, tags are not updated if the were previously fetched from a different remote. Commits that not in a branch and only tagged do not get fetched if the tag is not fetched. Fixes #812
* Merge branch 'tristan/avoid-unused-submodules-1.2' into 'bst-1.2'Tristan Van Berkom2018-12-101-7/+15
|\ | | | | | | | | [backport 1.2] Avoid downloading unused git submodules See merge request BuildStream/buildstream!1001
| * plugins/sources/git.py: Avoid downloading unused submodulesTristan Van Berkom2018-12-101-7/+15
|/ | | | | | | | | | | | | | Currently we have configuration in place to disable use of submodules, with the `checkout-submodules` git plugin option and the individual `checkout` options for each submodule, but these unused submodules are still downloaded at fetch time. This patch fixes the plugin to just completely ignore the submodules which are configured to be unused. This was previously fixed in the master branch by !996, and this patch backports the relevant fix to the 1.2 branch, fixing issue #804.
* Merge branch 'valentindavid/cache_server_fill_up-1.2' into 'bst-1.2'Valentin David2018-12-104-96/+255
|\ | | | | | | | | [backport 1.2] Fix cleanup of cache in server when disk is full See merge request BuildStream/buildstream!979
| * Lock cache cleanup in cas servervalentindavid/cache_server_fill_up-1.2Valentin David2018-11-291-46/+56
| | | | | | | | Cleaning up in parallel might slow down the cleaning process
| * Cleanup cache in cas server more agressivelyValentin David2018-11-293-67/+107
| | | | | | | | | | When there is less than 2GB left, it cleans up have 10GB available. These values are configurable.
| * Move cas server from ref-based to object-based garbage collection.Valentin David2018-11-292-3/+46
| |
| * Update mtimes of objects for requested references.Valentin David2018-11-293-11/+37
| | | | | | | | | | This also remove references when some objects are missing. This is in preparation for the move from reference to object garbage collection.
| * "Fallocate" object disk space to avoid getting NOSPC error when writingValentin David2018-11-291-6/+24
| | | | | | | | | | This locks the temporary object file so that cleanup does not need to be done for every write.
| * Avoid copying temporary file when adding object to CAS in server.Valentin David2018-11-192-18/+26
| | | | | | | | | | | | | | The file is already a temporary file and does not need copy. ENOSPC is thrown during that copy in issue #609. Fixes #678.
| * Fix type error in RPC messagesValentin David2018-11-191-2/+2
| |
| * Make cache clients not fail when a blob is not available.Valentin David2018-11-191-0/+14
| | | | | | | | | | | | | | We plan to make cache incomplete. That is some blobs are missing. For most of cases we will delete references when requested if they are incomplete. But there will be corner cases where objects are removed after the reference is requested.
| * Use f_bavail to query available space. Not f_bfree.Valentin David2018-11-191-1/+1
|/ | | | | | f_bfree space might not be usable. In practice we see failures in big disks because f_bfree is over 2GB and f_bavail is 0. We get ENOSPC if writing on disk then.
* Merge branch 'valentindavid/script-artifact-corruption-1.2' into 'bst-1.2'Valentin David2018-11-1410-8/+157
|\ | | | | | | | | [backport] Fix cache corruption by scripts when layout and integration commands are used See merge request BuildStream/buildstream!940
| * Fix cache corruption by scripts when layout and integration commands are usedvalentindavid/script-artifact-corruption-1.2Valentin David2018-11-0810-8/+157
|/ | | | | | | | Root directory was marked as a non-artifact mount, so not using SafeHardLink. However integration commands executed with write access to the root directory. Fixes #749
* Merge branch 'chandan/fix-checkout-none-1.2' into 'bst-1.2'Tristan Van Berkom2018-10-294-23/+34
|\ | | | | | | | | Ensure `--deps=none` option works for `bst checkout` See merge request BuildStream/buildstream!828
| * Ensure `--deps=none` option works for `bst checkout`Chandan Singh2018-10-294-23/+34
|/ | | | | | | | | | | This is a backport of !819. --- Currently, `bst checkout --deps none` command always produces empty output. Fix this issue and add regression test for the same. Fixes #670.
* Merge branch 'valentindavid/move_alpine_tarball-1.2' into 'bst-1.2'Javier Jardón2018-10-174-4/+4
|\ | | | | | | | | Change URL to the Alpine tarball See merge request BuildStream/buildstream!881
| * Change URL to the Alpine tarballValentin David2018-10-174-4/+4
|/
* NEWS: Updating for 1.2.3 release1.2.3Tristan Van Berkom2018-10-051-0/+20
|
* Merge branch 'valentindavid/rmtree_oserror-1.2' into 'bst-1.2'Valentin David2018-10-041-1/+1
|\ | | | | | | | | Catch correct exception from shutil.rmtree [bst-1.2] See merge request BuildStream/buildstream!861
| * Catch correct exception from shutil.rmtreevalentindavid/rmtree_oserror-1.2Valentin David2018-10-041-1/+1
|/ | | | | | | | | | | | | Python documentation is not clear on what shutil.rmtree is supposed to raise. However from the source code, it is obvious that OSError are raised, but never shutil.Error. It is not easy to test in normal condition because issues happen usually in combination with a FUSE filesystem, a probably a race condition where `fstatat` returns an error just before the filesystem being unmounted. Fixes #153.
* Merge branch 'tristan/fix-cache-size-race-1.2' into 'bst-1.2'Tristan Van Berkom2018-10-032-4/+8
|\ | | | | | | | | Tristan/fix cache size race 1.2 See merge request BuildStream/buildstream!855
| * utils.py: Document _get_dir_size() expectations.Tristan Van Berkom2018-10-031-0/+3
| | | | | | | | | | This function assumes that files do not disappear while walking the given directory.
| * _artifactcache/cascache.py: Don't create temporary files in the CAS storageTristan Van Berkom2018-10-031-1/+1
| | | | | | | | | | | | | | | | | | | | Use the designated tempdir when creating refs, we expect that temporary files are not created in the storage directory ever, they should be only ever created in the designated temporary directory. This fixes race conditions where utils._get_dir_size() throws an unhandled exception when attempting to stat the file which inadvertantly disappears.
| * utils.py: Give save_file_atomic() a tempdir argumentTristan Van Berkom2018-10-031-3/+4
|/ | | | Allow callers to decide where the temporary file will be created.
* Merge branch 'tristan/fix-double-terminate-prompt-1.2' into 'bst-1.2'Tristan Van Berkom2018-10-031-0/+9
|\ | | | | | | | | _scheduler/scheduler.py: Ignore interrupt events while terminating. See merge request BuildStream/buildstream!853
| * _scheduler/scheduler.py: Ignore interrupt events while terminating.Tristan Van Berkom2018-10-031-0/+9
|/ | | | | | | | | For some reason, we now receive a SIGINT from the main loop even when the SIGINT occurred with the handler disconnected in an interactive prompt. This patch simply ignores any received SIGINT events from the main loop in the case that we are already in the process of terminating. This fixes issue #693
* Merge branch 'tristan/fix-terminated-jobs-1.2' into 'bst-1.2'Tristan Van Berkom2018-10-032-9/+24
|\ | | | | | | | | _scheduler: Fix bookkeeping of terminated jobs See merge request BuildStream/buildstream!851
| * _scheduler: Fix bookkeeping of terminated jobsTristan Van Berkom2018-10-032-9/+24
|/ | | | | | | | | | | * Enhanced the base Job class to bookkeep which jobs have been terminated, and to consider them as `skipped` when asked via the `skipped` property. * Enhanced the base Queue class to bookkeep the job statuses more carefully. This fixes #479
* Merge branch 'juerg/cas-batch-1.2' into 'bst-1.2'Tristan Van Berkom2018-10-032-96/+224
|\ | | | | | | | | CAS: Implement BatchUpdateBlobs support See merge request BuildStream/buildstream!844
| * _artifactcache/cascache.py: Use BatchUpdateBlobsjuerg/cas-batch-1.2Jürg Billeter2018-10-031-4/+74
| | | | | | | | | | | | | | This uses BatchUpdateBlobs instead of individual blob upload to speed up artifact pushing, if the server supports it. Fixes #677.
| * _artifactcache/casserver.py: Implement BatchUpdateBlobsJürg Billeter2018-10-031-2/+43
| | | | | | | | Fixes #676.
| * cascache.py: Preparation for remote executionJim MacArthur2018-10-031-94/+111
|/ | | | | | | Refactor the push() and pull() implementations so that API additions needed for remote-execution is made easier. https://gitlab.com/BuildStream/buildstream/issues/454
* Merge branch 'tristan/fix-status-messages-1.2' into 'bst-1.2'Tristan Van Berkom2018-10-032-21/+49
|\ | | | | | | | | fix status messages (1.2) See merge request BuildStream/buildstream!846
| * source.py: Silence messages while consuming source fetcherstristan/fix-status-messages-1.2Tristan Van Berkom2018-10-021-17/+37
| | | | | | | | | | | | | | | | | | The source fetchers might be a list or a generator, when it is a generator (like the git source does), then we want to ensure that we silence the status messages which might occur as a result of consuming a source fetcher from the generator. This fixes the logs to be less verbose.
| * git.py: Added timed activity around fetching, displaying the URL being fetched.Tristan Van Berkom2018-10-021-4/+12
|/ | | | This will take care of silencing the status messages while checking submodules.
* Merge branch 'juerg/cas-batch-1.2' into 'bst-1.2'Jürg Billeter2018-10-015-39/+207
|\ | | | | | | | | _artifactcache/cascache.py: Use BatchReadBlobs See merge request BuildStream/buildstream!838
| * _artifactcache/cascache.py: Use BatchReadBlobsJürg Billeter2018-09-301-12/+137
| | | | | | | | | | | | | | This uses BatchReadBlobs instead of individual blob download to speed up artifact pulling, if the server supports it. Fixes #554.
| * _artifactcache/casserver.py: Harmonize payload size limitJürg Billeter2018-09-301-5/+6
| | | | | | | | | | Use 1 MiB as payload size limit on the server side for both individual downloads and batch uploads.
| * _artifactcache/cascache.py: Increase payload size limit for uploadsJürg Billeter2018-09-301-2/+7
| | | | | | | | | | gRPC can handle 1 MiB payloads. Increase size limit from 64 KiB to speed up uploads.`
View Lorry Controller Status