summaryrefslogtreecommitdiff
path: root/src/buildstream/sandbox
Commit message (Collapse)AuthorAgeFilesLines
* sandbox/_sandboxbuildboxrun.py: Use Messenger conveinence functionstristan/messenger-convenience-functionsTristan van Berkom2020-12-221-5/+2
|
* sandbox/_sandboxremote.py: Use Messenger convenience functionsTristan van Berkom2020-12-221-5/+2
|
* sandbox/sandbox.py: Use Messenger convenience functionsTristan van Berkom2020-12-221-7/+2
|
* Use CASCache.open()juerg/casJürg Billeter2020-12-091-2/+2
|
* cascache.py: Generalize remote_missing_blobs() into missing_blobs()Jürg Billeter2020-12-091-3/+3
| | | | | The implementation can be reused to replace `local_missing_blobs()` and simplify `contains_files()`.
* sandbox/_config.py, element.py: Refactor SandboxConfigTristan van Berkom2020-12-071-30/+101
| | | | | | | | | | | | | | | | | | | | | This commit changes SandboxConfig such that it now has a simple constructor and a new SandboxConfig.new_from_node() classmethod to load it from a YAML configuration node. The new version of SandboxConfig now uses type annotations. SandboxConfig also now sports a to_dict() method to help in serialization in artifacts, this replaces SandboxConfig.get_unique_key() since it does exactly the same thing, but uses the same names as expected in the YAML configuration to achieve it. The element.py code has been updated to use the classmethod, and to use the to_dict() method when constructing cache keys. This refactor is meant to allow instantiating a SandboxConfig without any MappingNode, such that we can later load a SandboxConfig from an Artifact instead of from an parsed Element. This commit also updates the cache keys in the cache key test, as the cache key format is slightly changed by the to_dict() method.
* scheduler.py: Use threads instead of processes for jobsBenjamin Schubert2020-12-042-6/+12
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This changes how the scheduler works and adapts all the code that needs adapting in order to be able to run in threads instead of in subprocesses, which helps with Windows support, and will allow some simplifications in the main pipeline. This addresses the following issues: * Fix #810: All CAS calls are now made in the master process, and thus share the same connection to the cas server * Fix #93: We don't start as many child processes anymore, so the risk of starving the machine are way less * Fix #911: We now use `forkserver` for starting processes. We also don't use subprocesses for jobs so we should be starting less subprocesses And the following highlevel changes where made: * cascache.py: Run the CasCacheUsageMonitor in a thread instead of a subprocess. * casdprocessmanager.py: Ensure start and stop of the process are thread safe. * job.py: Run the child in a thread instead of a process, adapt how we stop a thread, since we ca't use signals anymore. * _multiprocessing.py: Not needed anymore, we are not using `fork()`. * scheduler.py: Run the scheduler with a threadpool, to run the child jobs in. Also adapt how our signal handling is done, since we are not receiving signals from our children anymore, and can't kill them the same way. * sandbox: Stop using blocking signals to wait on the process, and use timeouts all the time. * messenger.py: Use a thread-local context for the handler, to allow for multiple parameters in the same process. * _remote.py: Ensure the start of the connection is thread safe * _signal.py: Allow blocking entering in the signal's context managers by setting an event. This is to ensure no thread runs long-running code while we asked the scheduler to pause. This also ensures all the signal handlers is thread safe. * source.py: Change check around saving the source's ref. We are now running in the same process, and thus the ref will already have been changed.
* Sandbox: Remove Sandbox.set_output_directory()Tristan van Berkom2020-09-211-10/+0
| | | | | This was a dead codepath, not used by any sandbox implementation for any reason, and used to be called inconsistently by some elements.
* Sandbox: Assert absolute paths where they are setTristan van Berkom2020-09-211-0/+4
|
* _sandboxremote.py: Include details in UNAVAILABLE error loggingJosh Smith2020-09-071-3/+5
|
* sandbox.py: Remove unused importBenjamin Schubert2020-07-061-1/+1
|
* Completely abolish job pickling.tristan/nuke-pickle-jobberTristan van Berkom2020-06-151-4/+0
|
* sandbox/_sandboxremote.py: Improved log messagesscottclarke/remote-build-messageScott Clarke2020-06-101-27/+30
| | | | | | | | | | | | | | The log message which shows that the remote build is ongoing has been extended to include the name of the element to which the remote build corresponds. Likewise the message indicating that an action result has been found in the action cache will also no include the element name. A new timed_activity() call has been added to show when missing blobs are being determined and uploaded. Fixed #1336
* sandbox: Remove unused root and scratch directoriesJürg Billeter2020-06-031-22/+0
|
* sandbox: Drop unused _mount.pyJürg Billeter2020-06-031-148/+0
|
* sandbox: Drop _use_cas_based_directory()Jürg Billeter2020-06-032-26/+2
| | | | It is always true.
* Drop SandboxBwrapJürg Billeter2020-06-031-511/+0
| | | | Replaced by buildbox-run.
* Move sandbox YAML processing from Element to SandboxConfigJürg Billeter2020-05-281-5/+22
|
* _sandboxbuildboxrun.py: Pass --no-logs-capture to buildbox-runjuerg/no-logs-captureJürg Billeter2020-05-121-0/+4
| | | | | | buildbox-common master no longer forwards stdout/stderr of the action command by default. The buildbox-run capabilities mechanism is used to support old and new versions.
* sandbox: Remove Sandbox.get_directory()Jürg Billeter2020-04-282-36/+4
| | | | Require element plugins to use `Sandbox.get_virtual_directory()`.
* Remove bst shell --sysrootJürg Billeter2020-04-251-12/+4
| | | | This is no longer needed now that we support caching buildtrees in CAS.
* src: Removing all pre 2.x "Since" documentation annotations.Tristan Van Berkom2020-04-211-3/+1
| | | | | | | | | | | | | | | This does not make sense to keep in the public API surface documentation. As we are heading towards a release of 2.0, this represents a "reset" in public API, and older annotations only serve to clutter the documentation with information that is not relevant to the reader. Everything which is public at the time of the 2.0 release can be considered available "Since: 2.0" implicitly (as this is going to be the starting point of this new stable API). It will make sense to start adding these annotations again for any added API in 2.2 and forward.
* cascache.py: Add allow_partial parameter to fetch_blobs()Jürg Billeter2020-04-201-6/+1
| | | | This fixes handling of missing blobs in `ArtifactCache.pull()`.
* _sandboxbuildboxrun.py: Detect platforms supported by buildbox-runjuerg/platformJürg Billeter2020-04-141-5/+19
| | | | | | This allows builds where the host OS or architecture doesn't match the build OS or architecture, if the buildbox-run implementation supports it. E.g., this allows x86-32 builds on x86-64 systems.
* sandbox: Reinitialize class when pickle is usedJürg Billeter2020-04-142-0/+8
| | | | | | Class data is not pickled and thus, class variables set by `check_available()` are not set when `check_sandbox_config()` is called by a job subprocess.
* Drop concept of partially supported sandbox configurationJürg Billeter2020-04-142-4/+0
| | | | | | | | | | | | Allowing builds without affecting the cache key but disallowing push, when the sandbox configuration is not fully compatible, results in an inconsistent user experience and may lead to unexpected build issues. Especially as push is allowed or disallowed based on the sandbox available at the time of push, not at the time of build. The previous commit making build-uid and build-gid configuration optional allows each project to decide whether the sandbox is required to support UID/GID configuration or not.
* sandbox: Make build-uid and build-gid configuration optionalJürg Billeter2020-04-144-20/+18
| | | | | This allows use of sandbox implementations that don't support configuring sandbox UID/GID such as buildbox-run-userchroot.
* _sandboxbuildboxrun.py: Reduce SIGTERM timeout to 15sJürg Billeter2020-04-091-1/+1
| | | | | | | The scheduler timeout for `SIGTERM` of job processes is 20s. The timeout in the sandbox backend should be shorter to allow graceful termination of the job process even when `buildbox-run` fails to terminate gracefully.
* _sandboxbuildboxrun.py: Check for buildbox-run initialization errorsjuerg/buildbox-run-errorJürg Billeter2020-03-311-2/+6
| | | | | | If `buildbox-run --capabilities` reports an error, we should consider the sandbox to be unavailable, forwarding the error message from buildbox-run.
* _protos: Update remote_execution.protojuerg/reapiJürg Billeter2020-03-161-2/+1
| | | | Move output_node_properties from Action to Command.
* storage: Add public Directory.exists() methodJürg Billeter2020-03-102-3/+3
|
* _sandboxreapi.py: Sort platform properties by nameJürg Billeter2020-03-092-12/+20
| | | | This is required by the REAPI specification.
* _sandboxbuildboxrun.py: Fix signal handlingjuerg/buildbox-signalsJürg Billeter2020-03-021-2/+17
| | | | | This fixes resume in non-interactive mode and job control in interactive shells, matching the behavior of `SandboxBwrap`.
* _sandboxreapi.py: Support read-only rootjuerg/reapi-readonlyJürg Billeter2020-02-271-3/+10
|
* _sandboxreapi.py: Ensure mount points existJürg Billeter2020-02-271-4/+18
|
* _sandboxreapi.py: Support optional network accessJürg Billeter2020-02-272-1/+4
| | | | This is used by `bst shell`.
* _sandboxreapi.py: Set unixUID and unixGID platform propertiesJürg Billeter2020-02-272-6/+20
|
* _sandboxreapi.py: Set OSFamily and ISA platform propertiesJürg Billeter2020-02-271-1/+6
|
* _sandboxremote.py: Support stdout and stderr digestsjuerg/reapi-logJürg Billeter2020-02-201-2/+12
| | | | Fetch blobs from remote CAS and then forward them to the sandbox output.
* sandbox: Move log forwarding from SandboxREAPI to SandboxRemoteJürg Billeter2020-02-202-9/+10
| | | | | SandboxBuildBoxRun doesn't require log forwarding as we use stdout and stderr of the buildbox-run process.
* sandbox: Add _create_empty_file() methodJürg Billeter2020-02-112-1/+48
|
* _sandboxreapi.py: Support setting output_node_propertiesJürg Billeter2020-02-111-0/+7
|
* sandbox: Remove unused _SandboxBatchCallJürg Billeter2020-02-112-20/+0
|
* _sandboxreapi.py: Optimize output_directoriesjuerg/sandbox-reapi-captureJürg Billeter2020-02-111-32/+33
| | | | | | Do not capture read-only directories. Fixes #1223.
* sandbox: Remove unused _mounter.pyJürg Billeter2020-02-101-148/+0
|
* _sandboxbuildboxrun.py: Bind mount /dev in interactive modeJürg Billeter2020-02-031-0/+5
|
* _sandboxbuildboxrun.py: Support host-filesJürg Billeter2020-02-031-1/+20
|
* _sandboxbuildboxrun.py: Check buildbox-run capabilitiesJürg Billeter2020-02-031-1/+9
|
* _sandboxreapi.py: Do not create directories for host-filesJürg Billeter2020-02-031-0/+3
| | | | This breaks if host-files is used to bind mount regular files.
* _sandboxreapi.py: Improve batch command loggingJürg Billeter2020-01-131-2/+9
| | | | | | | | This adds a timed activity message to bring batch command logging for REAPI-based sandboxes in line with the default sandbox as far as possible. Thix fixes 3 xfails with buildbox-run.
View Lorry Controller Status