path: root/NEWS

============
(unreleased)
============

Format
------
  o The `script` element no longer has a `layout` configuration directly, and now
    exposes a new `location` dependency configuration instead.

  o The BuildElement now has a new `location` dependency configuration, allowing
    BuildElement plugins to also stage dependencies into custom locations in
    the sandbox.

  o BREAKING CHANGE: Stack element dependencies are now hard required to be
    both build and runtime dependencies.

Core
----

  o BuildStream now also supports Python 3.9.

  o BREAKING CHANGE: Changed API signatures to remove Scope type from the API
    - Element.dependencies()
    - Element.stage_dependency_artifacts()
    - Element.search()
    Elements now can only ever see dependencies in their build scope.

  * BREAKING CHANGE: Changed ScriptElement.layout_add() API to take Element instances
                     in place of Element names

CLI
---

  o BREAKING CHANGE: `bst shell --use-buildtree` is now a boolean flag.
    `--use-buildtree=ask` and `--use-buildtree=try` are no longer supported.


==================
buildstream 1.93.5
==================

Format
------

  o BREAKING CHANGE: Some project.conf keys are no longer allowed to be specified outside
    of the project.conf file (they cannot be specified in an include file), these include:
    - name
    - element-path
    - min-version
    - plugins

CLI
---

  o Add `bst source push` subcommand. This command pushes element sources to a
    remote source cache.
  o `bst source track` now also accepts "build" and "run" as inputs.

Artifacts
---------

  o BREAKING CHANGE: Use Remote Asset API for remote artifact and source caches.
    Existing artifact servers are not compatible and need to be updated to the
    latest version which will then allow them to be repopulated with new
    artifacts.

Plugins
-------

  o tar: filter out 'dev' nodes from the tar archives when extracting them

  o `buildstream.testing` module now requires pytest >= 6.0.1.


==================
buildstream 1.93.4
==================

Core
----

  o The bwrap sandboxing backend and the SafeHardlinks FUSE filesystem
    have been dropped. buildbox-run is now the only sandboxing backend.


CLI
---

  o `bst shell --build` will now automatically fetch missing sources.
  o `bst build --deps` now also accepts "build" as an input.
  o `bst source fetch --deps` now also accepts "build" and "run" as inputs.
  o `bst artifact pull --deps` now also accepts "build" and "run" as inputs.
  o `bst artifact push --deps` now also accepts "build" and "run" as inputs.
  o Full paths can now be used to address elements across multiple junction boundaries


Format
------

  o BREAKING CHANGE: Now deprecation warnings are suppressed using the `allow-deprecated`
    configuration with the plugin origins in project.conf, instead of on the
    source/element overrides section (See issue #1291)
  o BREAKING CHANGE: The `target` option of junctions has been completely removed,
    Use `link` elements instead.
  o Variables from an element can now be used in source configurations
  o Full paths can now be used to address elements across multiple junction boundaries
  o A new "junction" plugin origin allows loading of plugins across junction boundaries
  o Support buildstream2.conf as a user configuration, allowing parallel installation
    of bst 1 & bst 2


Plugins
-------

  o Cache keys will change for all elements that have defined the
    `command-subdir` variable. This is the result of fixing a bug where this
    variable was not included in the cache key correctly.
  o The `pip` element has been removed. Please use the one from bst-plugins-experimental
  o Introduced new `link` element which can be used as a symbolic link to other
    elements or junctions, in the local project or in subprojects.
  o The DownloadableFileSource is now public API

API
---

  o `Element.node_subst_vars` and `Element.node_subst_sequence_vars` are now deprecated
     and will get removed in the next version. All config entries are now resolved so there
     is no need to use them anymore.


==================
buildstream 1.93.3
==================

  o BREAKING CHANGE: Removed BST_REQUIRED_VERSION_[MAJOR/MINOR] from plugin base class,
    this is now replaced with BST_MIN_VERSION which is now a hard requirement.

  o BREAKING CHANGE: Removed BST_FORMAT_VERSION from plugin base class, this is no
    longer supported.

  o buildbox-run is now the default sandbox backend on all platforms


==================
buildstream 1.93.2
==================

Core
----

  o BuildStream now requires Python >= 3.6.

  o BREAKING CHANGE: Conditional directives `(?)` from files included
    through junctions are evaluated with the options defined in the
    sub project the file comes from.

  o BREAKING CHANGE: "format-version" is removed and replaced with "min-version",
    which is now required to be specified in project.conf.

  o Incremental workspace builds are supported again.

CLI
---

  o BREAKING CHANGE: `bst shell --sysroot` has been removed. This is no longer
    needed now that we support caching buildtrees in CAS.

API
---

  o The `Directory` API has been extended to cover the use cases of more
    element plugins.


==================
buildstream 1.93.1
==================

API
---

  o `ErrorDomain` and `LoadErrorReason` are now available in the public
    `buildstream.exceptions` module for use in test suites of external plugins.

Plugins
-------

  o The 'git' source will now only fetch a single revision when its ref is set
    to an exact tag in git-describe format (...-0-g...).

  o When a 'git' source has a ref in git-describe format, only the commit
    ID is used when calculating the element's cache key. This means you can
    change between ref formats without having to rebuild the element.


==================
buildstream 1.93.0
==================

CLI
---

  o BREAKING CHANGE: `bst show` will now output `junction` instead
    of `buildable` for the state of junction elements, as they can't
    be built.

API
---

  o BREAKING CHANGE: The entry point groups used for element and source plugins
    are now separate. Elements must be defined in `buildstream.plugins.elements`
    group, and sources in `buildstream.plugins.sources`.

  o External plugins can now register a `buildstream.tests.source_plugins` entrypoint.
    The entry point can have an arbitrary name, but its value should point to a module
    containing a `register_sources()` method.
    This method should call `register_repo_kind` for all sources you want to have
    tested in BuildStream.
    Plugins authors that do this and believe BuildStream should be testing that
    part of their plugins should open an issue on BuildStream.

  o BREAKING CHANGE: `Consistency` has been removed and
    `Source.get_consistency()` has been replaced by `Source.is_resolved()`
    and `Source.is_cached()`.

    `Source.is_resolved()` has a default of `self.get_ref() is not None`,
    so if the only thing your plugin requires for being resolved is to
    have a valid ref, you don't need to do anything there.

    `Source.is_cached()` is there to replace a `Consistency.CACHED`
    implementation and will need to be implemented by plugin authors.

Core
----

  o BREAKING CHANGE: Once a source has been put in the internal BuildStream
    cache, its content will not get checked for validity unless a workspace
    is opened on it. If you see a warning that was not fatal as you fetch
    your source and want to re-test it to make sure it's gone without changing
    its ref (most plugins would handle that correctly), you will need to delete
    the internal source cache first.

Plugins
-------

  o BREAKING CHANGE: deb plugin have been moved to the bst-plugins-experimental
    repository.

Miscellaneous
-------------

  o By default the update rate of builstream console output
    (e.g. messages & status updates) when executing a scheduler driven task is
    restricted to once per second, known as the tick, with messages being batched
    in the intervening periods if necessary. This restriction can be lifted with
    `throttle-ui-updates: False` in user configuration. Logging behaviour remains
    unaffected by this configuration.


==================
buildstream 1.91.3
==================

CLI
---

  o BREAKING CHANGE: Commands no longer accept any options related to
    tracking. Please use `bst source track` separately prior to
    running commands, if you need similar functionality. The full list
    of removed options is as follows:

      * `bst build --track`
      * `bst build --track-all`
      * `bst build --track-except`
      * `bst build --track-cross-junctions` / `bst build -J`
      * `bst build --track-save`
      * `bst source fetch --track`
      * `bst source fetch --track-cross-junctions` / `bst source fetch -J`
      * `bst workspace open --track`
      * `bst workspace reset --track`

Plugins
-------

  o The 'git' source will now fetch submodules recursively when
    its 'checkout-submodules' option is enabled.

API
---

  o Sources may force early staging to cache by advertising
    `BST_KEY_REQUIRES_STAGE`. Sources setting this are staged to the cache to
    generate unique keys. `WorkspaceSource` and `LocalSource` set this.

Core
----

  o BuildStream now supports Python 3.8. Note that due to a bug in the 3.8.0
    series, BuildStream requires Python >= 3.8.1 when running on Python 3.8.

  o BuildStream now also supports the following machine architectures:

    * powerpc
    * powerpc64
    * powerpc64le
    * sun4v


==================
buildstream 1.91.2
==================

  o Various bug fixes.


==================
buildstream 1.91.1
==================

API
---

  o BREAKING CHANGE: 'Element.node_subst_member' has been removed. Please use
    'Element.node_subst_vars' instead.

  o BREAKING CHANGE: 'Element.node_subst_list' has been removed. Please use
    'Element.node_subst_sequence_vars' instead.

  o BREAKING CHANGE: Workspace incremental builds have been temporarily disabled

  o A new 'Node.strip_node_info()' is available and allows getting the
    underlying data structure for the given node.

  o BuildStream now requires Buildbox-casd 0.0.3 at minimum


==================
buildstream 1.91.0
==================
This release is a development snapshot, without any compatibility guarantees.
(The list of changes is non-exhaustive)

CLI
---

  o BREAKING CHANGE: `bst track` and `bst fetch` commands are now obsolete.
    Their functionality is provided by `bst source track` and
    `bst source fetch` respectively.

  o BREAKING CHANGE: The top level commands `checkout`, `push` and `pull` have
    been moved to the `bst artifact` subcommand group and are now obsolete.
    For example, you must now use `bst artifact pull hello.bst`.

    The behaviour of `checkout` has changed. The previously mandatory LOCATION
    argument should now be specified with the `--directory` option. In addition
    to this, `--tar` is no longer a flag, it is a mutually incompatible option
    to `--directory`. For example, `bst artifact checkout foo.bst --tar foo.tar.gz`.

  o BREAKING CHANGE: `bst workspace open` now supports the creation of multiple
    elements and allows the user to set a default location for their creation.
    This has meant that the new CLI is no longer backwards compatible with
    BuildStream 1.2.

  o BREAKING CHANGE: The `bst build` command no longer accepts the `--all`
    option. It now accepts `--deps` with a choice of 'plan' (default) and
    'all', for equivalent behaviour.

  o BREAKING CHANGE: `bst init` no longer uses the `--directory` or `-C`
    option. Instead, it (optionally) takes a directory as an argument.

  o BREAKING CHANGE: The `bst source-bundle` command has been removed. The
    functionality it provided has been replaced by the `--include-build-scripts`
    option of the `bst source checkout` command. To produce a tarball containing
    an element's sources and generated build scripts you can do the command
    `bst source checkout --include-build-scripts --tar foo.tar foo.bst`.

    A `--compression` option is also supported when using `--tar` which supports
    xz, gz and bz2 compression.

  o BREAKING CHANGE: The 'auto-init' functionality has been removed. This would
    offer to create a project in the event that bst was run against a directory
    without a project, to be friendly to new users. It has been replaced with
    an error message and a hint instead, to avoid bothering folks that just
    made a mistake.

  o BREAKING CHANGE: The unconditional 'Are you sure?' prompts have been
    removed. These would always ask you if you were sure when running
    'bst workspace close --remove-dir' or 'bst workspace reset'. They got in
    the way too often.

  o Added `bst artifact show` subcommand which shows the cached status
    of an artifact. If project/user remotes are available, they are checked
    for the target elements (and their deps, if specified). Artifacts available
    in remotes are displayed as "available".

  o Added `bst artifact list-contents` subcommand which can display the names
    of files in artifacts in your artifact cache, either by element name
    or by direct artifact reference. --long option can be used to display more
    information; file type and size.

  o Added `bst artifact delete` subcommand. This command removes artifacts from
    the local cache. Multiple element names and artifact refs can be specified
    as arguments.

  o Added `bst artifact log` subcommand for viewing build logs.

  o Added new `bst source checkout` command to checkout sources of an element.

  o Running commands without elements specified will now attempt to use
    the default targets defined in the project configuration.
    If no default target is defined, all elements in the project will be used.

  o bst 'build' now has '--remote, -r' option, inline with bst 'push' & 'pull'.
    Providing a remote will limit build's pull/push remote actions to the given
    remote specifically, ignoring those defined via user or project configuration.

  o `bst shell --sysroot` now takes any directory that contains a sysroot,
    instead of just a specially-formatted build-root with a `root` and `scratch`
    subdirectory.

  o Creating a build shell through the interactive mode or `bst shell --build`
    will now use the cached build tree if available locally. It is now easier to
    debug local build failures.


Plugins
-------

  o BREAKING CHANGE: cmake, make, makemaker, meson, modulebuild and qmake
    plugins have been moved to the bst-plugins-experimental repository.

  o BREAKING CHANGE: The 'manual' element lost its default 'MAKEFLAGS' and 'V'
    environment variables. There is already a 'make' element with the same
    variables. Note that this is a breaking change, it will require users to
    make changes to their .bst files if they are expecting these environment
    variables to be set.

  o BREAKING CHANGE: The `git` plugin does not create a local `.git`
    repository by default.  If `git describe` is required to work, the
    plugin has now a tag tracking feature instead. This can be enabled
    by setting 'track-tags'.

  o Added new `pip` source plugin for downloading python packages using pip,
    based on requirements files from previous sources.


API
---

  o BREAKING CHANGE: The yaml API has been rewritten entirely. When accessing
    configuration from YAML, please use the new `Node` classes exposed in the
    `buildstream` package. See the documentation for how to use it.

    This change has also removed the need of a YAML cache as it proved to no
    longer be of benefit.

  o There is a new sandbox API for command batching. It is used for build,
    script, and compose elements.

  o Source plugins may now request access access to previous during track and
    fetch by setting `BST_REQUIRES_PREVIOUS_SOURCES_TRACK` and/or
    `BST_REQUIRES_PREVIOUS_SOURCES_FETCH` attributes.


Configuration Format
--------------------

  o Elements may now specify 'build-depends' and 'runtime-depends' fields
    to avoid having to specify the dependency type for every entry in
    'depends'.

  o Junction elements may now specify another junction as their target, using
    the `target` configuration option.

  o Elements may now specify cross-junction dependencies as simple strings
    using the format '{junction-name}:{element-name}'.


Core
----

  o BREAKING CHANGE: Reverted the default behaviour of junctions. Subproject
    elements will no longer interact with the parent project's remote (by
    default). To enable this behaviour, a new "cache-junction-elements" boolean
    can be optionally declared as part of your junction element's configuration.
    Additionally, a new "ignore-junction-remotes" option has also been
    introduced. This allows you to completely ignore subproject remotes when
    using the parent project.

  o BREAKING CHANGE: Symlinks are no longer resolved during staging and absolute
    symlinks are now preserved instead of being converted to relative symlinks.

  o BREAKING CHANGE: Overlap whitelists now require absolute paths. This allows
    use of variables such as %{prefix} and matches the documentation.

  o BREAKING CHANGE: Default strip-commands have been removed as they are too
    specific. Recommendation if you are building in Linux is to use the
    ones being used in freedesktop-sdk project, for example

  o All elements should now be suffixed with `.bst`.
    Attempting to use an element that does not have the `.bst` extension,
    will result in a warning.

  o Failed builds are included in the cache as well.
    `bst artifact checkout` will provide anything in `%{install-root}`.
    A build including cached fails will cause any dependant elements
    to not be scheduled and fail during artifact assembly,
    and display the retry prompt during an interactive session.

  o Added Basic support for the BuildBox sandbox. The sand box will only be
    used if the environment variable BST_FORCE_SANDBOX is set to `buildbox`.
    This is the first step in transitioning to only using BuildBox for local
    sandboxing.


Artifacts
---------

  o BREAKING CHANGE: Artifact as a Proto. The caching of BuildStream artifacts
    has changed from a reference based impelementation. Existing artifacts and
    artifact servers are not compatible, as such remote artifact servers need to
    be updated to the latest version which will then allow them to be repopulated
    with new artifacts.

  o BREAKING CHANGE: The project name of dependencies have been included when
    calculating the cache key. This was required to keep inline with the
    artifact proto. Additionally, for strict mode, the name of dependencies have
    also been included in the calculation of the cache key. This is again to
    keep inline with the proto, but also to maintain symmetry in the behaviour
    between the strict and non-strict modes.

  o Due to the element `build tree` being cached in the respective artifact their
    size in some cases has significantly increased. In *most* cases the build trees
    are not utilised when building targets, as such by default bst 'pull' & 'build'
    will not fetch build trees from remotes. This behaviour can be overridden with
    the cli main option '--pull-buildtrees', or the user configuration cache group
    option 'pull-buildtrees = True'. The override will also add the build tree to
    already cached artifacts. When attempting to populate an artifactcache server
    with cached artifacts, only 'complete' elements can be pushed. If the element
    is expected to have a populated build tree then it must be cached before pushing.

  o Artifacts can now be cached explicitly with an empty `build tree` when built.
    Element types without a build-root were already cached with an empty build tree
    directory, this can now be extended to all or successful artifacts to save on cache
    overheads. The cli main option '--cache-buildtrees' or the user configuration cache
    group option 'cache-buildtrees' can be set as 'always', 'auto' or 'never', with
    the default being 'auto'. Note, as the cache-key for the artifact is independent of
    the cached build tree input it will remain unaltered, however the availbility of the
    build tree content may differ.


Workspaces
----------

  o Due to enabling the use of relative workspaces, "Legacy" workspaces
    may need to be closed and remade before the changes will affect them.
    Downgrading after using this feature may result in workspaces
    not functioning correctly

  o Opening a workspace now creates a .bstproject.yaml file that allows BuildStream
    commands to be run from a workspace that is not inside a project.

  o Specifying an element is now optional for most commands when BuildStream is run
    from inside a workspace. See CLI reference for more details.

  o Added Documentation on how to create out of source builds. This includes the
    new the `conf-root` variable to make the process easier. And there has been
    a bug fix to workspaces so they can be build in workspaces too.


Miscellaneous
-------------

  o Generate Docker images from built artifacts using
    `contrib/bst-docker-import` script.


=================
buildstream 1.1.5
=================

  o Added a `--tar` option to `bst checkout` which allows a tarball to be
    created from the artifact contents.

  o Fetching and tracking will consult mirrors defined in project config,
    and the preferred mirror to fetch from can be defined in the command
    line or user config.

  o Added new `remote` source plugin for downloading file blobs

  o Added support for the new include '(@)' directive in project.conf and .bst files


=================
buildstream 1.1.4
=================

  o `bst workspace` commands and `bst track` will substitute their
    source elements when performing those operations, e.g. performing
    `bst track` on a filter element will track the sources on the
    element that it depends on (if it has sources).

  o Added new simple `make` element

  o Switch to Remote Execution CAS-based artifact cache on all platforms.

    Artifact servers need to be migrated.

  o BuildStream now requires python version >= 3.5

  o BuildStream will now automatically clean up old artifacts when it
    runs out of space. The exact behavior is configurable in the user's
    buildstream.conf.


=================
buildstream 1.1.3
=================

  o Added new `bst init` command to initialize a new project.

  o Cross junction tracking is now disabled by default for projects
    which can support this by using project.refs ref-storage

    New options have been added to explicitly enable cross-junction
    tracking.

  o Failed jobs are now summarised at the end of a build.

    Use `--verbose` and `--no-verbose` to adjust the amount of detail given.

  o BuildElements' `configure-commands` are only run once for
    workspaces now, which allows for incremental builds.

    Appropriate API for plugins is also exposed through
    `Element.prepare`.

  o The `cmake` plugin now supports building with ninja with the
    newly added `generator` configuration option.

  o `bst workspace close` and `bst workspace reset` now support multiple
    elements. All elements can be specified using `--all`.

  o The elements whose cache keys had to be determined during the build
    are summarised at the end of the build.

  o Fixed versioning introspection to be dynamic, many users use
    a developer install mode so they can update with git, now the
    version information is always up to date in logs.

    This causes a minor API break: The --version output now only
    outputs the version.


=================
buildstream 1.1.2
=================

  o New ref-storage option allows one to store source refs, such
    as git shas, in one central project.refs file instead of inline
    with the source declarations.

  o Deprecated `--track-save` optionality in `bst build`, this
    does not make sense to support now that we have project.refs.

  o Added the `sandbox` configuration option which can be used in
    `project.conf` and elements, to control the user ID and group ID
    used in build sandboxes.

  o Added new `deb` source implementation, for staging of downloaded
    deb package files.


=================
buildstream 1.1.1
=================

  o New project configuration controlling how the sandbox behaves
    when `bst shell` is used; allowing projects to provide a more
    functional shell environment.

  o The `bst shell` command now has a `--mount` option allowing
    users to mount files and directories into the sandbox for
    testing purposes.

  o Log lines are now configurable with the new "message-format"
    user configuration, allowing one to express optional fields
    such as microsecond precision and wallclock time.

  o Newly added filter element

  o Git source plugin now allows disabling of submodule checkouts

  o In the same way we allow overriding element configurations
    by their 'kind' in project.conf, we now support the same
    for source plugin configurations.

  o Tar and zip sources now automatically recall an `etag`
    from the http headers, optimizing tracking of tarballs
    significantly (issue #62)


=================
buildstream 1.1.0
=================

  o Multiple artifact caches are now supported in project and
    user configuration with a priority order (issue #85)
  o Add junction support for subprojects
  o Changes towards incremental builds in workspaces
  o `bst shell --build` now creates true build sandbox
  o Many bug fixes


=================
buildstream 1.0.0
=================
First stable release of BuildStream

BuildStream 1.0.0 is all about API stability - for the past months we
have been reviewing our various API surfaces, implementing strategies
for revisioning of our interfaces and cleaning up. Long term stability
is very important for build reproducibility over time, and this release
is the first promise we are making on any API surfaces.

Stable API surfaces include:
  o The command line interface
  o The YAML user configuration file format
  o The YAML project `.bst` file format
  o The core Python module imported by external plugins