summaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
* setup.py: Add plugin requirements as extraschandan/extras-requireChandan Singh2019-11-151-0/+9
| | | | | | | | | | | | | | | Currently, there isn't an easy way to install BuildStream along with its plugin dependencies, using pip. Add plugin requirements as extras (https://setuptools.readthedocs.io/en/latest/setuptools.html#declaring-extras-optional-features-with-their-own-dependencies) so that they can be installed by specifying the relevant extras to `pip`. This repository has only one plugin - `deb` source - that has external dependencies, i.e. `arpy`. With this patch, one should be able to install BuildStream and dependencies for the `deb` plugin using something like `pip instal BuildStream[deb]`.
* Merge branch 'chandan/black' into 'master'Chandan Singh2019-11-15285-14002/+12216
|\ | | | | | | | | Use Black to format code See merge request BuildStream/buildstream!1703
| * Start linting doc/source/conf.pyChandan Singh2019-11-142-45/+40
| | | | | | | | | | | | | | | | | | | | | | Add `doc/source/conf.py` to the filelist for Black and Pylint. Previously this file was not covered by any of the linters, so this patch includes one-off sweeping changes for the formatting. To make pylint happy, we had to disable a warning about defining a variable called `copyright` since that's a built-in. It's unlikely that we will ever need the built-in `copyright()` in this module, so it seems safe to disable it.
| * doc/source/hacking: Remove pycodestyle, add BlackChandan Singh2019-11-142-16/+26
| | | | | | | | | | | | | | | | | | | | | | Now that code formatting is managed by Black, and we don't need to run `pycodestyle` separately, remove corresponding mentions from hacking documentation. Add documentation on how to run Black. Move out linting and formatting into a separate section for better readability.
| * Reformat code using BlackChandan Singh2019-11-14275-13933/+12097
| | | | | | | | | | | | | | As discussed over the mailing list, reformat code using Black. This is a one-off change to reformat all our codebase. Moving forward, we shouldn't expect such blanket reformats. Rather, we expect each change to already comply with the Black formatting style.
| * .gitlab-ci.yml: Check code formatting as part of CIChandan Singh2019-11-141-1/+4
| | | | | | | | As part of the `lint` CI job, also check code formatting.
| * Remove pycodestyleChandan Singh2019-11-144-8/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | All the formatting-related warnings from `pycodestyle` are covered by Black, and almost all of the remaining ones are covered by `pylint`. Based on that, remove `pycodestyle` so that we don't have to maintain another set of configuration. More details on warnings/errors covered by `pycodestyle` below. --- Here's the big list of pycodestyle error/warning codes: https://pycodestyle.readthedocs.io/en/latest/intro.html#error-codes. They are broken down into following categories: 1. Indentation 2. Whitespace 3. Blank line 4. Import 5. Line Length 6. Runtime 7. Deprecation warning 8. Statement Out of the above, 1-5 are purely formatting related so we don't needd to care about them. 6 is runtime issues like syntax error that are hard to miss (even mypy will complain about these). 7 is about deprecation warnings. Half of them don't apply to us since we only support Python 3.5+ and the other half is covered by `pylint`. That leaves us with the "E7*" codes, that pycodestyle groups under "Statement" heading. After verifying each of them, the only ones that is really not covered by either Black or Pylint is the following: * E731 do not assign a lambda expression, use a def This sole check does not seem worth keeping another tool.
| * .pylintrc: Disable formatting messagesChandan Singh2019-11-141-1/+13
| | | | | | | | | | | | | | Since we use Black to format our code, disable code-formatting messages from Pylint as they are already taken care of by Black. This ensures that we don't get double warnings for the same problem, and that Pylint and Black don't fight with each other over style.
| * Add configuration to run BlackChandan Singh2019-11-142-0/+38
|/ | | | | | | | | | Introduce two new tox environments - `format` and `format-check`. The `format` environment will reformat the code using `black` whereas the `format-check` envrionment will simply print the diff without modifying any files. Configure Black to use the same line lengths that we use currently, i.e. 119.
* Merge branch 'tpollard/profileerror' into 'master'bst-marge-bot2019-11-142-0/+30
|\ | | | | | | | | | | | | _profile.py: Raise exception if invalid BST_PROFILE topic Closes #1208 See merge request BuildStream/buildstream!1711
| * _profile.py: Raise exception if invalid BST_PROFILE topicsTom Pollard2019-11-142-0/+30
|/ | | | | This is implemented with a new ProfileError, ensuring a user doesn't wait for a profile to complete that was never valid.
* Merge branch 'bschubert/graceful-children-sigterm' into 'master'bst-marge-bot2019-11-132-28/+31
|\ | | | | | | | | | | | | Gracefully shutdown children on termination Closes #1185 See merge request BuildStream/buildstream!1692
| * scheduler.py: Extract timeout to kill as a global variableBenjamin Schubert2019-11-131-2/+5
| |
| * job.py: Gracefully handle killed subprocessesBenjamin Schubert2019-11-131-0/+8
| | | | | | | | | | This ensures that we don't show an unexpected error when we forcefully kill one of our workers
| * job.py: handle SIGTERM gracefullyBenjamin Schubert2019-11-131-0/+14
| | | | | | | | | | This allows showing a nice info to the user and stop showing that the return code was an unexpected 255.
| * job.py: Don't use 'terminate_wait', as it uses waitpid()Benjamin Schubert2019-11-132-28/+6
|/ | | | | | | | | | | Using `join()` on the subprocess calls `waitpid()` under the hood which breaks our child watcher. Instead, schedule a task for 20 seconds later that will effectively kill the tasks. Note that the task will only be called if we still have active jobs. Otherwise, it will just be skipped and we won't wait as long.
* Merge branch 'chandan/pylint' into 'master'bst-marge-bot2019-11-138-16/+12
|\ | | | | | | | | | | | | Fix pylint configuration Closes #1206 See merge request BuildStream/buildstream!1708
| * tox.ini: Fix pylint configurationChandan Singh2019-11-131-1/+1
| | | | | | | | | | | | | | Running it like `src/buildstream` apparently doesn't report the warnings correctly, as pylint does not like file paths. Fixes #1206.
| * src/buildstream: Fix pylint warning 'arguments-differ'Chandan Singh2019-11-136-10/+10
| | | | | | | | | | Pylint failed to catch these earlier due to misconfiguration, so fix them now.
| * src/buildstream: Fix/silence attribute-defined-outside-initChandan Singh2019-11-131-1/+1
| | | | | | | | | | Pylint failed to catch these earlier due to misconfiguration, so fix them now.
| * src/buildstream: Remove unused importsChandan Singh2019-11-132-4/+0
|/ | | | | Pylint failed to catch these earlier due to misconfiguration, so fix them now.
* Merge branch 'traveltissues/1204' into 'master'bst-marge-bot2019-11-131-5/+3
|\ | | | | | | | | | | | | app: Pass color setting to Status Closes #1204 See merge request BuildStream/buildstream!1707
| * Pass color setting to StatusDarius Makovsky2019-11-131-5/+3
|/
* Merge branch 'bschubert/fix-capa-check' into 'master'bst-marge-bot2019-11-126-71/+225
|\ | | | | | | | | | | | | Fix capabilities check of remotes Closes #1178 See merge request BuildStream/buildstream!1705
| * _sourcecache.py: Fix reporting of capabilities for the source cacheBenjamin Schubert2019-11-122-12/+63
| | | | | | | | This also adds a test to ensure we are correctly setting the cache up
| * _remote: Ensure checks done in the subclasses are propagatedBenjamin Schubert2019-11-124-59/+162
|/ | | | | | | | | | | Currently, the `BaseRemote` would call `_check()` on the children, which states that errors should be sent back as a string. However, `BaseRemote` doesn't check the return of `_check()`. This changes the contract so that subclasses throw `RemoteError` themselves. This also fixes the `ArtifactShare` and add a test.
* Merge branch 'chandan/fix-no-color' into 'master'bst-marge-bot2019-11-123-13/+12
|\ | | | | | | | | | | | | _frontend: Simplify color handling Closes #1200 See merge request BuildStream/buildstream!1704
| * _frontend: Simplify color handlingChandan Singh2019-11-123-13/+12
|/ | | | | | | | | | | | Currently we store color configuration in our App object, and have to remember to pass it around to `click.echo()` when printing things on the screen. This is error-prone as we can forget to do so. This leads to bugs like #1200, where `bst init` was not respecting `--no-colors` flag. Instead of doing that, this patch controls colors in output by configuring the `click.Context` object. Fixes #1200.
* Merge branch 'chandan/fix-implicit-string-concat' into 'master'bst-marge-bot2019-11-121-2/+2
|\ | | | | | | | | tests/frontend/completions: Fix implicit string concatenation See merge request BuildStream/buildstream!1702
| * tests/frontend/completions: Fix implicit string concatenationChandan Singh2019-11-121-2/+2
|/ | | | | | | | | | We expect `INVALID_ELEMENTS` to be a list of element names. However we are missing a comma in between the names, so they end up being concatenated into a single strings. The test passes either way (which is not ideal) because it just checks that the suggested completion is _different_ than the expected in case of errors.
* Merge branch 'juerg/sandbox-reapi' into 'master'bst-marge-bot2019-11-112-178/+226
|\ | | | | | | | | Add SandboxREAPI class See merge request BuildStream/buildstream!1699
| * sandbox: Move SandboxRemoteBatch to SandboxREAPIJürg Billeter2019-11-112-69/+68
| |
| * sandbox: Move _run() from SandboxRemote to SandboxREAPIJürg Billeter2019-11-112-102/+111
| |
| * sandbox: Add abstract SandboxREAPI classJürg Billeter2019-11-112-6/+32
| | | | | | | | | | This provides a skeleton for sandbox implementations based on the Remote Execution API.
| * _sandboxremote.py: Fetch outputs in _execute_action()Jürg Billeter2019-11-111-8/+17
| | | | | | | | This makes process_job_output() reusable for local execution.
| * _sandboxremote.py: Extract _execute_action() method from _run()Jürg Billeter2019-11-111-22/+32
| | | | | | | | This will allow making _run() reusable for local execution.
| * _sandboxremote.py: Simplify exit code returnJürg Billeter2019-11-111-6/+3
| |
| * _sandboxremote.py: Send command and action together with the other blobsJürg Billeter2019-11-111-14/+7
| | | | | | | | This reduces the number of round trips.
| * _sandboxremote.py: Ensure working directory existsJürg Billeter2019-11-111-0/+5
|/
* Merge branch 'bschubert/fix-children-termination' into 'master'bst-marge-bot2019-11-113-16/+85
|\ | | | | | | | | scheduler.py: Prevent the asyncio loop from leaking into subprocesses See merge request BuildStream/buildstream!1691
| * scheduler.py: Remove FIXME message and add explanation insteadBenjamin Schubert2019-11-111-5/+3
| |
| * scheduler.py: Prevent the asyncio loop from leaking into subprocessesBenjamin Schubert2019-11-112-11/+82
|/ | | | | | | | | | | Having a running asyncio loop while forking a program is not supported in python and doesn't work as expected. This leads to file descriptors leaking and the subprocesses sharing the same loop as the parents. This also leads to the parent receiving all signals the children receive. This ensures we don't leek our asyncio loop in the workers we fork.
* Merge branch 'chandan/no-command-general-elements' into 'master'bst-marge-bot2019-11-112-0/+6
|\ | | | | | | | | Indicate that compose & stack plugins do not run commands See merge request BuildStream/buildstream!1697
| * Indicate that compose & stack plugins do not run commandsChandan Singh2019-11-112-0/+6
|/ | | | | | Both compose and stack elements do not run commands in the build sandbox. Indicate so by setting the `BST_RUN_COMMANDS` attribute accordingly.
* Merge branch 'coldtom/allow-dev-shm' into 'master'bst-marge-bot2019-11-114-2/+61
|\ | | | | | | | | | | | | _sandboxbwrap.py: Create /dev/shm in the sandbox Closes #1197 See merge request BuildStream/buildstream!1694
| * _sandboxbwrap.py: Create /dev/shm in the sandboxThomas Coldrick2019-11-114-2/+61
|/ | | | | | | | | | | | Creates /dev/shm as a tmpfs in the sandbox. Before now access to /dev/shm was only available by a plugin using `Sandbox.mark_directory()` or adding to `Sandbox.DEVICES`, either of which would _mount_ /dev/shm into the sandbox, allowing pollution from the host. This adds it as a tmpfs by default, which seems sensible as it is required for POSIX support. Also adds a test which makes sure that we can open a shared memory object inside the build sandbox with some (probably poor) C code.
* Merge branch 'traveltissues/1186-3' into 'master'bst-marge-bot2019-11-113-16/+30
|\ | | | | | | | | | | | | skip tracking elements without trackable sources Closes #1186 See merge request BuildStream/buildstream!1689
| * _stream: Remove exception case for removed optionDarius Makovsky2019-11-111-10/+0
| |
| * Add _is_trackable() method to Source()Darius Makovsky2019-11-113-6/+30
|/ | | | | | | | | | | | | | | | This method reports whether the source can be tracked. This would be false for sources advertising BST_KEY_REQUIRES_STAGE. Element tracking can be skipped if none of the held sources can be tracked. This is determined by the value of the `Element.__tracking_scheduled` attribute which is set in `Element._schedule_tracking()`. This is set to `True` if at least one source can be tracked. Also remove some of the tracking handling from `_stream._load` to `_stream.track` where it is more relevant. closes #1186
* Merge branch 'bschubert/expand-path-configs' into 'master'bst-marge-bot2019-11-112-2/+82
|\ | | | | | | | | | | | | _remote.py: Expand user in certificates paths Closes #741 See merge request BuildStream/buildstream!1696
View Lorry Controller Status