============ (unreleased) ============ CLI --- o Add `bst source push` subcommand. This command pushes element sources to a remote source cache. Plugins ------- o tar: filter out 'dev' nodes from the tar archives when extracting them ================== buildstream 1.93.4 ================== Core ---- o The bwrap sandboxing backend and the SafeHardlinks FUSE filesystem have been dropped. buildbox-run is now the only sandboxing backend. CLI --- o `bst shell --build` will now automatically fetch missing sources. o `bst build --deps` now also accepts "build" as an input. o `bst source fetch --deps` now also accepts "build" and "run" as inputs. o `bst artifact pull --deps` now also accepts "build" and "run" as inputs. o `bst artifact push --deps` now also accepts "build" and "run" as inputs. o Full paths can now be used to address elements across multiple junction boundaries Format ------ o BREAKING CHANGE: Now deprecation warnings are suppressed using the `allow-deprecated` configuration with the plugin origins in project.conf, instead of on the source/element overrides section (See issue #1291) o BREAKING CHANGE: The `target` option of junctions has been completely removed, Use `link` elements instead. o Variables from an element can now be used in source configurations o Full paths can now be used to address elements across multiple junction boundaries o A new "junction" plugin origin allows loading of plugins across junction boundaries o Support buildstream2.conf as a user configuration, allowing parallel installation of bst 1 & bst 2 Plugins ------- o Cache keys will change for all elements that have defined the `command-subdir` variable. This is the result of fixing a bug where this variable was not included in the cache key correctly. o The `pip` element has been removed. Please use the one from bst-plugins-experimental o Introduced new `link` element which can be used as a symbolic link to other elements or junctions, in the local project or in subprojects. o The DownloadableFileSource is now public API API --- o `Element.node_subst_vars` and `Element.node_subst_sequence_vars` are now deprecated and will get removed in the next version. All config entries are now resolved so there is no need to use them anymore. ================== buildstream 1.93.3 ================== o BREAKING CHANGE: Removed BST_REQUIRED_VERSION_[MAJOR/MINOR] from plugin base class, this is now replaced with BST_MIN_VERSION which is now a hard requirement. o BREAKING CHANGE: Removed BST_FORMAT_VERSION from plugin base class, this is no longer supported. o buildbox-run is now the default sandbox backend on all platforms ================== buildstream 1.93.2 ================== Core ---- o BuildStream now requires Python >= 3.6. o BREAKING CHANGE: Conditional directives `(?)` from files included through junctions are evaluated with the options defined in the sub project the file comes from. o BREAKING CHANGE: "format-version" is removed and replaced with "min-version", which is now required to be specified in project.conf. o Incremental workspace builds are supported again. CLI --- o BREAKING CHANGE: `bst shell --sysroot` has been removed. This is no longer needed now that we support caching buildtrees in CAS. API --- o The `Directory` API has been extended to cover the use cases of more element plugins. ================== buildstream 1.93.1 ================== API --- o `ErrorDomain` and `LoadErrorReason` are now available in the public `buildstream.exceptions` module for use in test suites of external plugins. Plugins ------- o The 'git' source will now only fetch a single revision when its ref is set to an exact tag in git-describe format (...-0-g...). o When a 'git' source has a ref in git-describe format, only the commit ID is used when calculating the element's cache key. This means you can change between ref formats without having to rebuild the element. ================== buildstream 1.93.0 ================== CLI --- o BREAKING CHANGE: `bst show` will now output `junction` instead of `buildable` for the state of junction elements, as they can't be built. API --- o BREAKING CHANGE: The entry point groups used for element and source plugins are now separate. Elements must be defined in `buildstream.plugins.elements` group, and sources in `buildstream.plugins.sources`. o External plugins can now register a `buildstream.tests.source_plugins` entrypoint. The entry point can have an arbitrary name, but its value should point to a module containing a `register_sources()` method. This method should call `register_repo_kind` for all sources you want to have tested in BuildStream. Plugins authors that do this and believe BuildStream should be testing that part of their plugins should open an issue on BuildStream. o BREAKING CHANGE: `Consistency` has been removed and `Source.get_consistency()` has been replaced by `Source.is_resolved()` and `Source.is_cached()`. `Source.is_resolved()` has a default of `self.get_ref() is not None`, so if the only thing your plugin requires for being resolved is to have a valid ref, you don't need to do anything there. `Source.is_cached()` is there to replace a `Consistency.CACHED` implementation and will need to be implemented by plugin authors. Core ---- o BREAKING CHANGE: Once a source has been put in the internal BuildStream cache, its content will not get checked for validity unless a workspace is opened on it. If you see a warning that was not fatal as you fetch your source and want to re-test it to make sure it's gone without changing its ref (most plugins would handle that correctly), you will need to delete the internal source cache first. Plugins ------- o BREAKING CHANGE: deb plugin have been moved to the bst-plugins-experimental repository. Miscellaneous ------------- o By default the update rate of builstream console output (e.g. messages & status updates) when executing a scheduler driven task is restricted to once per second, known as the tick, with messages being batched in the intervening periods if necessary. This restriction can be lifted with `throttle-ui-updates: False` in user configuration. Logging behaviour remains unaffected by this configuration. ================== buildstream 1.91.3 ================== CLI --- o BREAKING CHANGE: Commands no longer accept any options related to tracking. Please use `bst source track` separately prior to running commands, if you need similar functionality. The full list of removed options is as follows: * `bst build --track` * `bst build --track-all` * `bst build --track-except` * `bst build --track-cross-junctions` / `bst build -J` * `bst build --track-save` * `bst source fetch --track` * `bst source fetch --track-cross-junctions` / `bst source fetch -J` * `bst workspace open --track` * `bst workspace reset --track` Plugins ------- o The 'git' source will now fetch submodules recursively when its 'checkout-submodules' option is enabled. API --- o Sources may force early staging to cache by advertising `BST_KEY_REQUIRES_STAGE`. Sources setting this are staged to the cache to generate unique keys. `WorkspaceSource` and `LocalSource` set this. Core ---- o BuildStream now supports Python 3.8. Note that due to a bug in the 3.8.0 series, BuildStream requires Python >= 3.8.1 when running on Python 3.8. o BuildStream now also supports the following machine architectures: * powerpc * powerpc64 * powerpc64le * sun4v ================== buildstream 1.91.2 ================== o Various bug fixes. ================== buildstream 1.91.1 ================== API --- o BREAKING CHANGE: 'Element.node_subst_member' has been removed. Please use 'Element.node_subst_vars' instead. o BREAKING CHANGE: 'Element.node_subst_list' has been removed. Please use 'Element.node_subst_sequence_vars' instead. o BREAKING CHANGE: Workspace incremental builds have been temporarily disabled o A new 'Node.strip_node_info()' is available and allows getting the underlying data structure for the given node. o BuildStream now requires Buildbox-casd 0.0.3 at minimum ================== buildstream 1.91.0 ================== This release is a development snapshot, without any compatibility guarantees. (The list of changes is non-exhaustive) CLI --- o BREAKING CHANGE: `bst track` and `bst fetch` commands are now obsolete. Their functionality is provided by `bst source track` and `bst source fetch` respectively. o BREAKING CHANGE: The top level commands `checkout`, `push` and `pull` have been moved to the `bst artifact` subcommand group and are now obsolete. For example, you must now use `bst artifact pull hello.bst`. The behaviour of `checkout` has changed. The previously mandatory LOCATION argument should now be specified with the `--directory` option. In addition to this, `--tar` is no longer a flag, it is a mutually incompatible option to `--directory`. For example, `bst artifact checkout foo.bst --tar foo.tar.gz`. o BREAKING CHANGE: `bst workspace open` now supports the creation of multiple elements and allows the user to set a default location for their creation. This has meant that the new CLI is no longer backwards compatible with BuildStream 1.2. o BREAKING CHANGE: The `bst build` command no longer accepts the `--all` option. It now accepts `--deps` with a choice of 'plan' (default) and 'all', for equivalent behaviour. o BREAKING CHANGE: `bst init` no longer uses the `--directory` or `-C` option. Instead, it (optionally) takes a directory as an argument. o BREAKING CHANGE: The `bst source-bundle` command has been removed. The functionality it provided has been replaced by the `--include-build-scripts` option of the `bst source checkout` command. To produce a tarball containing an element's sources and generated build scripts you can do the command `bst source checkout --include-build-scripts --tar foo.tar foo.bst`. A `--compression` option is also supported when using `--tar` which supports xz, gz and bz2 compression. o BREAKING CHANGE: The 'auto-init' functionality has been removed. This would offer to create a project in the event that bst was run against a directory without a project, to be friendly to new users. It has been replaced with an error message and a hint instead, to avoid bothering folks that just made a mistake. o BREAKING CHANGE: The unconditional 'Are you sure?' prompts have been removed. These would always ask you if you were sure when running 'bst workspace close --remove-dir' or 'bst workspace reset'. They got in the way too often. o Added `bst artifact show` subcommand which shows the cached status of an artifact. If project/user remotes are available, they are checked for the target elements (and their deps, if specified). Artifacts available in remotes are displayed as "available". o Added `bst artifact list-contents` subcommand which can display the names of files in artifacts in your artifact cache, either by element name or by direct artifact reference. --long option can be used to display more information; file type and size. o Added `bst artifact delete` subcommand. This command removes artifacts from the local cache. Multiple element names and artifact refs can be specified as arguments. o Added `bst artifact log` subcommand for viewing build logs. o Added new `bst source checkout` command to checkout sources of an element. o Running commands without elements specified will now attempt to use the default targets defined in the project configuration. If no default target is defined, all elements in the project will be used. o bst 'build' now has '--remote, -r' option, inline with bst 'push' & 'pull'. Providing a remote will limit build's pull/push remote actions to the given remote specifically, ignoring those defined via user or project configuration. o `bst shell --sysroot` now takes any directory that contains a sysroot, instead of just a specially-formatted build-root with a `root` and `scratch` subdirectory. o Creating a build shell through the interactive mode or `bst shell --build` will now use the cached build tree if available locally. It is now easier to debug local build failures. Plugins ------- o BREAKING CHANGE: cmake, make, makemaker, meson, modulebuild and qmake plugins have been moved to the bst-plugins-experimental repository. o BREAKING CHANGE: The 'manual' element lost its default 'MAKEFLAGS' and 'V' environment variables. There is already a 'make' element with the same variables. Note that this is a breaking change, it will require users to make changes to their .bst files if they are expecting these environment variables to be set. o BREAKING CHANGE: The `git` plugin does not create a local `.git` repository by default. If `git describe` is required to work, the plugin has now a tag tracking feature instead. This can be enabled by setting 'track-tags'. o Added new `pip` source plugin for downloading python packages using pip, based on requirements files from previous sources. API --- o BREAKING CHANGE: The yaml API has been rewritten entirely. When accessing configuration from YAML, please use the new `Node` classes exposed in the `buildstream` package. See the documentation for how to use it. This change has also removed the need of a YAML cache as it proved to no longer be of benefit. o There is a new sandbox API for command batching. It is used for build, script, and compose elements. o Source plugins may now request access access to previous during track and fetch by setting `BST_REQUIRES_PREVIOUS_SOURCES_TRACK` and/or `BST_REQUIRES_PREVIOUS_SOURCES_FETCH` attributes. Configuration Format -------------------- o Elements may now specify 'build-depends' and 'runtime-depends' fields to avoid having to specify the dependency type for every entry in 'depends'. o Junction elements may now specify another junction as their target, using the `target` configuration option. o Elements may now specify cross-junction dependencies as simple strings using the format '{junction-name}:{element-name}'. Core ---- o BREAKING CHANGE: Reverted the default behaviour of junctions. Subproject elements will no longer interact with the parent project's remote (by default). To enable this behaviour, a new "cache-junction-elements" boolean can be optionally declared as part of your junction element's configuration. Additionally, a new "ignore-junction-remotes" option has also been introduced. This allows you to completely ignore subproject remotes when using the parent project. o BREAKING CHANGE: Symlinks are no longer resolved during staging and absolute symlinks are now preserved instead of being converted to relative symlinks. o BREAKING CHANGE: Overlap whitelists now require absolute paths. This allows use of variables such as %{prefix} and matches the documentation. o BREAKING CHANGE: Default strip-commands have been removed as they are too specific. Recommendation if you are building in Linux is to use the ones being used in freedesktop-sdk project, for example o All elements should now be suffixed with `.bst`. Attempting to use an element that does not have the `.bst` extension, will result in a warning. o Failed builds are included in the cache as well. `bst artifact checkout` will provide anything in `%{install-root}`. A build including cached fails will cause any dependant elements to not be scheduled and fail during artifact assembly, and display the retry prompt during an interactive session. o Added Basic support for the BuildBox sandbox. The sand box will only be used if the environment variable BST_FORCE_SANDBOX is set to `buildbox`. This is the first step in transitioning to only using BuildBox for local sandboxing. Artifacts --------- o BREAKING CHANGE: Artifact as a Proto. The caching of BuildStream artifacts has changed from a reference based impelementation. Existing artifacts and artifact servers are not compatible, as such remote artifact servers need to be updated to the latest version which will then allow them to be repopulated with new artifacts. o BREAKING CHANGE: The project name of dependencies have been included when calculating the cache key. This was required to keep inline with the artifact proto. Additionally, for strict mode, the name of dependencies have also been included in the calculation of the cache key. This is again to keep inline with the proto, but also to maintain symmetry in the behaviour between the strict and non-strict modes. o Due to the element `build tree` being cached in the respective artifact their size in some cases has significantly increased. In *most* cases the build trees are not utilised when building targets, as such by default bst 'pull' & 'build' will not fetch build trees from remotes. This behaviour can be overridden with the cli main option '--pull-buildtrees', or the user configuration cache group option 'pull-buildtrees = True'. The override will also add the build tree to already cached artifacts. When attempting to populate an artifactcache server with cached artifacts, only 'complete' elements can be pushed. If the element is expected to have a populated build tree then it must be cached before pushing. o Artifacts can now be cached explicitly with an empty `build tree` when built. Element types without a build-root were already cached with an empty build tree directory, this can now be extended to all or successful artifacts to save on cache overheads. The cli main option '--cache-buildtrees' or the user configuration cache group option 'cache-buildtrees' can be set as 'always', 'auto' or 'never', with the default being 'auto'. Note, as the cache-key for the artifact is independent of the cached build tree input it will remain unaltered, however the availbility of the build tree content may differ. Workspaces ---------- o Due to enabling the use of relative workspaces, "Legacy" workspaces may need to be closed and remade before the changes will affect them. Downgrading after using this feature may result in workspaces not functioning correctly o Opening a workspace now creates a .bstproject.yaml file that allows BuildStream commands to be run from a workspace that is not inside a project. o Specifying an element is now optional for most commands when BuildStream is run from inside a workspace. See CLI reference for more details. o Added Documentation on how to create out of source builds. This includes the new the `conf-root` variable to make the process easier. And there has been a bug fix to workspaces so they can be build in workspaces too. Miscellaneous ------------- o Generate Docker images from built artifacts using `contrib/bst-docker-import` script. ================= buildstream 1.1.5 ================= o Added a `--tar` option to `bst checkout` which allows a tarball to be created from the artifact contents. o Fetching and tracking will consult mirrors defined in project config, and the preferred mirror to fetch from can be defined in the command line or user config. o Added new `remote` source plugin for downloading file blobs o Added support for the new include '(@)' directive in project.conf and .bst files ================= buildstream 1.1.4 ================= o `bst workspace` commands and `bst track` will substitute their source elements when performing those operations, e.g. performing `bst track` on a filter element will track the sources on the element that it depends on (if it has sources). o Added new simple `make` element o Switch to Remote Execution CAS-based artifact cache on all platforms. Artifact servers need to be migrated. o BuildStream now requires python version >= 3.5 o BuildStream will now automatically clean up old artifacts when it runs out of space. The exact behavior is configurable in the user's buildstream.conf. ================= buildstream 1.1.3 ================= o Added new `bst init` command to initialize a new project. o Cross junction tracking is now disabled by default for projects which can support this by using project.refs ref-storage New options have been added to explicitly enable cross-junction tracking. o Failed jobs are now summarised at the end of a build. Use `--verbose` and `--no-verbose` to adjust the amount of detail given. o BuildElements' `configure-commands` are only run once for workspaces now, which allows for incremental builds. Appropriate API for plugins is also exposed through `Element.prepare`. o The `cmake` plugin now supports building with ninja with the newly added `generator` configuration option. o `bst workspace close` and `bst workspace reset` now support multiple elements. All elements can be specified using `--all`. o The elements whose cache keys had to be determined during the build are summarised at the end of the build. o Fixed versioning introspection to be dynamic, many users use a developer install mode so they can update with git, now the version information is always up to date in logs. This causes a minor API break: The --version output now only outputs the version. ================= buildstream 1.1.2 ================= o New ref-storage option allows one to store source refs, such as git shas, in one central project.refs file instead of inline with the source declarations. o Deprecated `--track-save` optionality in `bst build`, this does not make sense to support now that we have project.refs. o Added the `sandbox` configuration option which can be used in `project.conf` and elements, to control the user ID and group ID used in build sandboxes. o Added new `deb` source implementation, for staging of downloaded deb package files. ================= buildstream 1.1.1 ================= o New project configuration controlling how the sandbox behaves when `bst shell` is used; allowing projects to provide a more functional shell environment. o The `bst shell` command now has a `--mount` option allowing users to mount files and directories into the sandbox for testing purposes. o Log lines are now configurable with the new "message-format" user configuration, allowing one to express optional fields such as microsecond precision and wallclock time. o Newly added filter element o Git source plugin now allows disabling of submodule checkouts o In the same way we allow overriding element configurations by their 'kind' in project.conf, we now support the same for source plugin configurations. o Tar and zip sources now automatically recall an `etag` from the http headers, optimizing tracking of tarballs significantly (issue #62) ================= buildstream 1.1.0 ================= o Multiple artifact caches are now supported in project and user configuration with a priority order (issue #85) o Add junction support for subprojects o Changes towards incremental builds in workspaces o `bst shell --build` now creates true build sandbox o Many bug fixes ================= buildstream 1.0.0 ================= First stable release of BuildStream BuildStream 1.0.0 is all about API stability - for the past months we have been reviewing our various API surfaces, implementing strategies for revisioning of our interfaces and cleaning up. Long term stability is very important for build reproducibility over time, and this release is the first promise we are making on any API surfaces. Stable API surfaces include: o The command line interface o The YAML user configuration file format o The YAML project `.bst` file format o The core Python module imported by external plugins