summaryrefslogtreecommitdiff
path: root/tests
Commit message (Collapse)AuthorAgeFilesLines
* _stream.py: Pull missing artifacts in push()juerg/pushJürg Billeter2020-09-292-12/+2
| | | | | | | | | | | As per #819, BuildStream should pull missing artifacts by default. The previous behavior was to only pull missing buildtrees. A top-level `--no-pull` option can easily be supported in the future. This change makes it possible to use a single scheduler session (with concurrent pull and push jobs). This commit also simplifies the code as it removes the `sched_error_action` emulation, using the regular frontend code path instead.
* tests/format/dependencies.py: Test bad filename configurationTristan van Berkom2020-09-282-0/+24
| | | | Test a dictionary instead of a string when given to the filename list.
* tests/format/dependencies.py: Testing shorthand dependency configurationsTristan van Berkom2020-09-289-0/+51
|
* tests/format/dependencies3/elements/supported2.bst: Adding commentTristan van Berkom2020-09-281-0/+6
| | | | Adding some clarifications about an existing test
* tests/integration/filter.py: Add test_filter_pass_integration_uncachedJürg Billeter2020-09-241-0/+32
| | | | | This tests that built filter artifacts don't depend on build dependencies for integration.
* tests/frontend/track.py: Test behavior of SKIP messagesTristan van Berkom2020-09-241-0/+79
| | | | | | | | This test checks that: * We get SKIP messages for tracking local sources * We get SKIP messages for tracking workspaced elements * We go no messages at all for elemenents which have no sources
* tests/integration/manual.py: Test staging files in custom directoriesTristan van Berkom2020-09-183-0/+37
|
* tests/cachekey: Added new cachekey test for new BuildElement configurationTristan van Berkom2020-09-184-1/+16
|
* buildelement.py: Remove legacy command stepsTristan van Berkom2020-09-183-3/+3
| | | | | | | | | | | This was actually deadcode, since node.validate_keys() was called on the configure dictionary without the legacy command steps. If any element was using the legacy commands, they would have been met with a load time error anyway. This commit also updates the cache key test, since removing these legacy commands affects BuildElement internally in such a way as to affect the cache keys.
* ScriptElement: Porting to new APITristan van Berkom2020-09-188-44/+20
| | | | | | | | | | | | | | | | | | | | | | | | Instead of relying on Element.search(), use Element.configure_dependencies() to configure the layout. Summary of changes: * scriptelement.py: Change ScriptElement.layout_add() API to take an Element instead of an element name, this is now not optional (one cannot specify a `None` element). This is an API breaking change * plugins/elements/script.py: Implement Element.configure_dependencies() in order to call ScriptElement.layout_add(). This is a breaking YAML format change. * tests/integration: Script integration tests updated to use the new YAML format * tests/cachekey: Updated for `script` element changes
* tests/frontend/overlaps.py: Test multiple calls to ↵Tristan van Berkom2020-09-188-2/+192
| | | | | | | | | | | | | | | | | | | | Element.stage_dependency_artifacts() This patch adds a new test plugin which implements Element.configure_dependencies() in order to offer better flexibility for testing overlaps. Newly added tests: * Test overlap warnings and errors when staging elsewhere than in the sandbox root. * Test unstaged files failure modes when staging elsewhere than in the sandbox root. * Test various overlap behaviors of OverlapAction, when different calls to Element.stage_dependency_artifacts() cause overlaps to occur after staging files into separate directories.
* tests/format/dependencies.py: Adding tests for dependency `config` attributesTristan van Berkom2020-09-189-0/+119
|
* tests/format/dependencies.py: Test merging of multiple dependenciesTristan van Berkom2020-09-183-0/+34
| | | | | | Test that when the same dependency is added as a build and runtime dependency separately, they end up being the same dependency which is both a build & runtime dependency in the loaded build graph.
* tests/format/dependencies.py: Remove tests for ↵Tristan van Berkom2020-09-189-97/+0
| | | | LoadErrorReason.DUPLICATE_DEPENDENCY
* tests/frontend/overlaps.py: Test CoreWarnings.UNSTAGED_FILESTristan van Berkom2020-09-186-2/+38
|
* tests/frontend/overlaps.py: Minor cleanup and refactoringTristan van Berkom2020-09-181-37/+32
| | | | | | | | | | | | | | | | | This commit: * Removes testing of the deprecated `fail-on-overlap` project configuration option, this is going away soon and unneeded. * Tests that warnings are issued whenever they should be (some tests were happy to see a successful run but failed to check for an expected warning). * Test error/warning more evenly across tests, some were missing the warning mode. * Use `bst show` instead of `bst build` for the undefined_variable test, it should fail without needing a build.
* tests/elements/filter/basic/element_plugins/dynamic.py: Stage in Element.stage()Tristan van Berkom2020-09-181-5/+2
| | | | | This test element was also staging artifacts in Element.assemble(), which is now illegal.
* tests/frontend/rebuild.py: Add test_modify_and_revertJürg Billeter2020-09-161-0/+41
| | | | | | This tests that, in non-strict mode, a cached artifact matching the strict cache key is preferred to a more recent artifact matching only the weak cache key.
* tests/frontend/rebuild.py: Add assertion for built elementsJürg Billeter2020-09-161-0/+8
|
* tests/frontend/push.py: Test artifact push with globJürg Billeter2020-09-101-0/+30
|
* tests/frontend/buildcheckout.py: Add non-strict cache key testjuerg/cache-keyJürg Billeter2020-09-081-0/+34
| | | | | This is a regression test for the previously broken dependency cache key check in non-strict mode.
* element.py: Hide dependencies which are irrelevant to the ElementTristan van Berkom2020-09-042-4/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | This is a large breaking change, a summary of the changes are that: * The Scope type is now private, since Element plugins do not have the choice to view any other scopes. * Element.dependencies() API change Now it accepts a "selection" (sequence) of dependency elements, so that Element.dependencies() can iterate over a collection of dependencies, ensuring that we iterate over every element only once even when we need to iterate over multiple element's dependencies. The old API is moved to Element._dependencies() and still used internally. * Element.stage_dependency_artifacts() API change This gets the same treatment as Element.dependencies(), and the old API is also preserved as Element._stage_dependency_artifacts(), so that the CLI can stage things for `bst artifact checkout` and such. * Element.search() API change The Scope argument is removed, and the old API is preserved as Element._search() temporarily, until we can remove this completely.
* Element.dependencies() now yields ElementProxy wrappers by default.Tristan van Berkom2020-09-041-1/+1
| | | | | | | | | | | | | | | This prepares the ground for policing the dependencies which are visible to an Element plugin, such that plugins are only allowed to see the elements in their Scope.BUILD scope, even if they call Element.dependencies() on a dependency. This commit does the following: * Element.dependencies() is now a user facing frontend which yields ElementProxy elements instead of Elements. * Various core codepaths have been updated to call the internal Element._dependencies() codepath which still returns Elements.
* Move handling of the source `directory` configuration to ElementSourcesjuerg/element-source-cacheJürg Billeter2020-09-0328-28/+28
| | | | | | | | | | | | | | | | | The `directory` value determines where a source is staged within the build root of an element, however, it does not directly affect individual sources. With this change the sources will individually be cached in CAS independent of the value of `directory`. `ElementSources` will use the value of `directory` when staging all element sources into the build root. This results in a cache key change as the `directory` value is moved from the unique key of individual sources to the unique key of `ElementSources`. This is in preparation for #1274.
* Add ElementSourcesCacheJürg Billeter2020-09-034-20/+44
| | | | | | | | | | | | | | | | | | Sources have been cached in CAS individually, except for sources that transform other sources, which have been cached combined with all previous sources of the element. This caching structure may be confusing as sources are specified in the element as a list and this is not a good fit for #1274 where we want to support caching individual sources in a Remote Asset server with a BuildStream-independent URI (especially the `directory` configuration would be problematic). This replaces the combined caching of 'previous' sources with an element-level source cache, which caches all sources of an element staged together. Sources that don't depend on previous sources are still cached individually. This also makes it possible to add a list of all element sources to the source proto used by the element-level source cache.
* tests/sourcecache: Do not inspect uninitialized elementsJürg Billeter2020-09-033-0/+10
|
* tests/sourcecache/push.py: Add test_push_missing_source_after_build()Jürg Billeter2020-09-011-0/+30
| | | | | Test that source push succeeds if the source needs to be fetched even if the artifact of the corresponding element is already cached.
* runcli.py: Configurable index/storage artifact typesJosh Smith2020-08-261-0/+8
| | | | | This enables for seperate index/storage artifact servers to be configured by environment variables passed through tox.
* _elementsources.py: Fix source names in unique keyjuerg/source-cache-keyJürg Billeter2020-08-2528-28/+28
| | | | | | | | | | | | | | The goal was to include the source plugin kind in the element cache key as the unique key of a source may not be unique across different source plugins. This is the source equivalent of the `element-plugin-name` value in the element cache key. However, `Source._get_source_name()` was the wrong method for this as that also includes the key itself, which may not even be set yet. This results in a cache key change. Fixes: 3953bcc6 ("element.py: clobber sources with workspace")
* _fixtures.py: Only get the normal number of threads at the start of sessionBenjamin Schubert2020-08-231-1/+5
| | | | | This ensures that we also have our tests correctly shutting down background threads and not interferring with each other
* buildstream/element.py: Consider "build-root" variable in the cache key ↵tristan/build-root-cache-keyTristan van Berkom2020-08-2028-28/+28
| | | | | | | | | | | | | | | | | | | unconditionally Not all elements use the "build-root" variable, but it is the standard variable to use for the build directory, and the build directory must be considered in the cache key. Handling this unilaterally in the core is safer than delegating this to element implementations, as we have less chance of plugin authors missing this detail and possibly introducing binary variance for artifacts where only the build directory differs (something which happens when the project name or element names change). This commit also updates the hard coded cache keys in the cache key test, so as to ensure every commit passes it's own tests. This fixes #1386.
* tests/cachekey/cachekey.py: Use the same project name for cache key testsTristan van Berkom2020-08-201-1/+1
| | | | | | | | In the test_cache_key_fatal_warnings() test, use the same project name in both generated project directories in order to pass the tests in the case that elements are guaranteed to have differing cache keys for differeing element/project names (which is the case when we consider the "build-root" in the cache key unconditionally).
* Bump minimum version requirement for ruamel.yamlchandan/upgrade-ruamelChandan Singh2020-08-182-0/+36
| | | | | | | | | | | | ruamel.yaml <= 0.16.6 suffers from a bug where ruamel's yaml dumper crashes when used on a sequence that has comments before it. In BuildStream, this manifests in form of issues like #1265. See upstream issue at https://sourceforge.net/p/ruamel-yaml/tickets/335. Also, add a regression test for it. Fixes #1265.
* _frontend/cli: Add --deps `build` & `run` values for `source track --deps`Chandan Singh2020-08-148-2/+65
| | | | | | | This is part of #1349. This patch will conclude the first part of that issue, i.e. ensuring that the possible options for `--deps` are consistent across all commands (with the exception of `--deps plan` that we will handle separately).
* regression-test for ".bst" artifact checkout bugdwinship/test_for_dot_bst_artifact_checkout_bugDouglas Winship2020-08-131-0/+24
|
* casserver.py: Drop BuildStream Artifact and Source servicesJürg Billeter2020-08-131-107/+0
| | | | Replaced by Remote Asset API Fetch and Push services.
* _sourcecache.py: Use AssetRemoteJürg Billeter2020-08-131-11/+23
| | | | | This migrates the source cache from the BuildStream Source protocol to the Remote Asset API.
* _artifactcache.py: Use AssetRemoteJürg Billeter2020-08-131-11/+38
| | | | | | | This migrates the artifact cache from the BuildStream Artifact protocol to the Remote Asset API. Co-authored-by: Sander Striker <s.striker@striker.nl>
* tests/frontend/push.py: Pull complete artifactJürg Billeter2020-08-131-7/+8
| | | | | | | | Use a separate Cli instance with a separate local cache for the second pull in `test_recently_pulled_artifact_does_not_expire()` to ensure the complete artifact is pulled. If only a part of the artifact is pulled, there is no guarantee that the other blobs of that artifact won't expire.
* Completely remove MetaElementTristan van Berkom2020-08-131-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This dramatically affects the load process and removes one hoop we had to jump through, which is the creation of the extra intermediate MetaElement objects. This allows us to more easily carry state discovered by the Loader over to the Element constructor, as we need not add additional state to the intermediate MetaElement for this. Instead we have the Element initializer understand the LoadElement directly. Summary of changes: * _loader/metaelement.py: Removed * _loader/loadelement.py: Added some attributes previously required on MetaElement * _loader/loader.py: Removed _collect_element() and collect_element_no_deps(), removing the process of Loader.load() which translates LoadElements into MetaElements completely. * _loader/init.py: Export LoadElement, Dependency and Symbol types, stop exporting MetaElement * _loader/metasource.py: Now take the 'first_pass' parameter as an argument * _artifactelement.py: Use a virtual LoadElement instead of a virtual MetaElement to instantiate the ArtifactElement objects. * _pluginfactory/elementfactory.py: Adjust to now take a LoadElement * _project.py: Adjust Project.create_element() to now take a LoadElement, and call the new Element._new_from_load_element() instead of the old Element._new_from_meta() function * element.py: - Now export Element._new_from_load_element() instead of Element._new_from_meta() - Adjust the constructor to do the LoadElement toplevel node parsing instead of expecting members on the MetaElement object - Added __load_sources() which parses out and creates MetaSource objects for the sake of instantiating the element's Source objects. Consequently this simplifies the scenario where workspaces are involved. * source.py: Adjusted to use the new `first_pass` parameter to MetaSource when creating a duplicate clone.
* element.py: Early return in _initialize_state()Tristan van Berkom2020-08-131-3/+1
| | | | | | | | | | | | | | | | | | | | Instead of having an assertion here, lets just have an early return and make the __resolved_initial_state variable internal private (with two leading underscores). We also stop checking for it in _pipeline.py before resolving state. Some background: * We only defer _initialize_state() to a later stage because it can be a resource intensive task which interrogates the disk or the local CAS, thus we have the Pipeline iterate over the instantiated elements and resolve them separately for better user feedback. * Some "first pass" elements must have their state initialized earlier, i.e. the "link" and "junction" elements need to be usable during the load sequence.
* Add tests for checking the working of a remote cacheWilliam Salmon2020-08-1111-0/+194
| | | | | These basic tests are ment to be run with a remote cache and can be used to check bst conpatiblity with a remote cache server.
* _frontend/cli.py: Fix `artifact checkout` with no argumentsChandan Singh2020-08-111-0/+12
| | | | | | | | | | | | | | | When no arguments are passed to `artifact checkout` command, it currently crashes. We actually have a check of this, but we crash before that becasue this method tries to read the value of `target` before we've had time to check for it. That can only be done correctly after the app has been initialized. So, refactor that bit of the method to run after we've checked that we are working with a non-empty target. Also, add a regression test for it. Fixes #1367.
* _yaml.pyx: Make shortname a mandatory argument to _yaml.load()Tristan van Berkom2020-08-107-46/+46
| | | | | | | | | | | | | This ensures that important calls to this function do give some thought to providing a reasonable shortname, which will be used as a display name in errors. This continues to support `None` as a shortname, which is used in various tests which don't need to provide a reasonable user facing error. The buildstream.testing module now exports a `load_yaml` function which only takes a filename and no shortname.
* Extract ElementSources classjuerg/element-sourcesJürg Billeter2020-08-061-1/+1
| | | | An ElementSources object represents the combined sources of an element.
* element.py: Drop cache checks from `_tracking_done()`Jürg Billeter2020-08-061-3/+2
| | | | | | This was a workaround for inconsistent error handling in the frontend. As the error handling is now fixed, drop these cache checks after tracking.
* source.py: Validate cache when it's used, not in `_is_cached()`Jürg Billeter2020-08-061-31/+17
| | | | | | | `_is_cached()` is indirectly called by the frontend, which is not optimal for handling per-plugin errors. Instead, call `validate_cache()` right before the cache is used: in fetch jobs and when opening a workspace.
* tests/format/variables.py: Increase timeoutTristan van Berkom2020-08-031-1/+1
| | | | | | | With sometimes very slow runners, this test has been found to timeout more often after the recent refactoring. Double the timeout to avoid erronously failing CI.
* tests/integration/interactive_build.py: make a new test for retrying failed ↵abderrahim/cached-failureAbderrahim Kitouni2020-07-291-1/+16
| | | | | | builds The old one tested that retrying the failed build doesn't actually retry
* tests/format/include.py: Test preservation of conditionals at include timetristan/options-orderTristan van Berkom2020-07-2926-0/+179
| | | | | | Added tests to ensure that conditional statements don't get overwritten when performing composition of one dictionary on top of another due to include processing.
View Lorry Controller Status