| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
| |
This also adjusts the very strange tests in tests/internals/cascache.py
which use unittest's MagicMock interface to inspect what happened on
specific python methods instead of doing proper end-to-end testing.
|
| | |
|
| | |
|
| |
|
|
|
| |
The implementation can be reused to replace `local_missing_blobs()` and
simplify `contains_files()`.
|
| | |
|
| |
|
|
|
| |
This allows adding multiple objects in a single batch, avoiding extra
gRPC round trips to buildbox-casd.
|
| | |
|
| |
|
|
|
| |
It's only used by `_fetch_tree()` and can be replaced by a single
additional line.
|
| |
|
|
|
| |
This eliminates code duplication in `ArtifactCache`, `SourceCache` and
`ElementSourcesCache`.
|
| |
|
|
| |
This simplifies the code, delegating the logic to buildbox-casd.
|
| |
|
|
| |
It's not used outside testutils.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This changes how the scheduler works and adapts all the code that needs
adapting in order to be able to run in threads instead of in
subprocesses, which helps with Windows support, and will allow some
simplifications in the main pipeline.
This addresses the following issues:
* Fix #810: All CAS calls are now made in the master process, and thus
share the same connection to the cas server
* Fix #93: We don't start as many child processes anymore, so the risk
of starving the machine are way less
* Fix #911: We now use `forkserver` for starting processes. We also
don't use subprocesses for jobs so we should be starting less
subprocesses
And the following highlevel changes where made:
* cascache.py: Run the CasCacheUsageMonitor in a thread instead of a
subprocess.
* casdprocessmanager.py: Ensure start and stop of the process are thread
safe.
* job.py: Run the child in a thread instead of a process, adapt how we
stop a thread, since we ca't use signals anymore.
* _multiprocessing.py: Not needed anymore, we are not using `fork()`.
* scheduler.py: Run the scheduler with a threadpool, to run the child
jobs in. Also adapt how our signal handling is done, since we are not
receiving signals from our children anymore, and can't kill them the
same way.
* sandbox: Stop using blocking signals to wait on the process, and use
timeouts all the time.
* messenger.py: Use a thread-local context for the handler, to allow for
multiple parameters in the same process.
* _remote.py: Ensure the start of the connection is thread safe
* _signal.py: Allow blocking entering in the signal's context managers
by setting an event. This is to ensure no thread runs long-running
code while we asked the scheduler to pause. This also ensures all the
signal handlers is thread safe.
* source.py: Change check around saving the source's ref. We are now
running in the same process, and thus the ref will already have been
changed.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* This modifies the signal terminator so that it can be called from any
thread.
This checks that either:
- The signal handler is already in place
- Or the caller is in the main thread, allowing to set the signal
handler.
This also removes the exact callback that was added instead of removing
the last one, and fixes the `suspend_handler` to do the same.
This is required, as we don't know which interleaving of calls will be
done, and we can't guarantee that the last one is the right one to
remove
|
| |
|
|
| |
GetInstanceNameForRemote() has been deprecated.
|
| |
|
|
| |
Replaced by Remote Asset API Fetch and Push services.
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
| |
Do not copy file mode from casd object file. Do not change
non-executable mode bits if file is executable.
|
| |
|
|
|
|
| |
an indentation error meant it was only passed when quota was defined
Fixes #1268
|
| |
|
|
|
|
| |
buildbox-casd `UploadMissingBlobs` will call `FindMissingBlobs` before
the actual upload. Having BuildStream call `FindMissingBlobs` as well
adds unnecessary overhead.
|
| | |
|
| | |
|
| |
|
|
| |
Also fix linting errors coming with new version of pylint
|
| |
|
|
| |
for connection
|
| | |
|
| | |
|
| |
|
|
| |
Like `contains_file()` but checks multiple files in a single request.
|
| |
|
|
| |
This fixes handling of missing blobs in `ArtifactCache.pull()`.
|
| |
|
|
| |
this lets buildbox-casd know it is needed and not delete it when pruning
|
| |
|
|
|
|
| |
buildbox-casd should handle the creation of its own directories. This is
especially needed when buildbox-casd and buildstream don't run under the
same user.
|
| |
|
|
|
|
| |
This moves the cas logs outside of the cas directory. Since BuildStream
is responsible for creating and handling the logs, it should not be in
the cas directory.
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
| |
If checking out files from a CasBasedDirectory which holds
node_properties in the index files are explicitly copied instead of
being hardlinked and the mtime is updated to the stored value.
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
| |
This is necessary to allow using buildbox-run with userchroot in the
near future, since currently only the owner of the BuildStream process
can access the CASD socket, but the buildbox-casd binary will need to
be setuid' to another user.
This gets around this limitation by allowing the group to access a
symlink, which in turn should point to a directory owned by the CASD
user.
|
| |
|
|
|
|
| |
This also involves a number of changes to tests and other parts of the
codebase since they were hacking about wit API that shouldn't have
existed.
|
| | |
|
| | |
|
| |
|
|
|
| |
It doesn't seem like we need to call GetCapabilities here, as we're not
querying anything. Remove the call to save some time.
|
| | |
|
| |
|
|
|
|
| |
Change _CASCacheUsageMonitor to hold a CASDChannel instead of
it's parent CASCache. This means that when in spawn mode, we don't need
to pickle the CASCache, only the CASDChannel.
|
