1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
|
About
-----
.. image:: https://buildstream.gitlab.io/buildstream/_static/release.svg
:target: https://gitlab.com/BuildStream/buildstream/commits/bst-1.2
.. image:: https://buildstream.gitlab.io/buildstream/_static/snapshot.svg
:target: https://gitlab.com/BuildStream/buildstream/commits/master
.. image:: https://gitlab.com/BuildStream/buildstream/badges/master/pipeline.svg
:target: https://gitlab.com/BuildStream/buildstream/commits/master
.. image:: https://gitlab.com/BuildStream/buildstream/badges/master/coverage.svg?job=coverage
:target: https://gitlab.com/BuildStream/buildstream/commits/master
.. image:: https://img.shields.io/pypi/v/BuildStream.svg
:target: https://pypi.org/project/BuildStream
What is BuildStream?
====================
`BuildStream <https://buildstream.build>`_ is a Free Software tool for
building/integrating software stacks.
It takes inspiration, lessons and use-cases from various projects including
OBS, Reproducible Builds, Yocto, Baserock, Buildroot, Aboriginal, GNOME Continuous,
JHBuild, Flatpak Builder and Android repo.
BuildStream supports multiple build-systems (e.g. autotools, cmake, cpan, distutils,
make, meson, qmake), and can create outputs in a range of formats (e.g. debian packages,
flatpak runtimes, sysroots, system images) for multiple platforms and chipsets.
Why should I use BuildStream?
=============================
BuildStream offers the following advantages:
* **Declarative build instructions/definitions**
BuildStream provides a flexible and extensible framework for the modelling
of software build pipelines in a declarative YAML format, which allows you to
manipulate filesystem data in a controlled, reproducible sandboxed environment.
* **Support for developer and integrator workflows**
BuildStream provides traceability and reproducibility for integrators handling
stacks of hundreds/thousands of components, as well as workspace features and
shortcuts to minimise cycle-time for developers.
* **Fast and predictable**
BuildStream can cache previous builds and track changes to source file content
and build/config commands. BuildStream only rebuilds the things that have changed.
* **Extensible**
You can extend BuildStream to support your favourite build-system.
* **Bootstrap toolchains and bootable systems**
BuildStream can create full systems and complete toolchains from scratch, for
a range of ISAs including x86_32, x86_64, ARMv7, ARMv8, MIPS.
Mission statement
=================
* **Deterministic build environment**
Ideally, the build result (or *artifact*) of any given build will always be
bit-for-bit identical if it is given exactly the same inputs, regardless of
the host environment, so long as BuildStream supports the given host.
To this end, we go to some lengths to ensure a clean execution environment
for building, and we bump the core cache key version if ever we change or
improve sanitizing of the execution environment, so that everything needs
to be rebuilt.
While BuildStream cannot itself guarantee a bit-for-bit identical result
for every identical input, we can help in the majority of the work needed
to ensure your builds are determinstic and reproducible.
* **Reusable build instructions**
Our declarative format is designed with the intention that the same BuildStream
project can be used to accomplish various things for the set of software it
defines the integration for.
For instance, using project options; it should be possible to reuse the same
project to deploy the same software stack, or bundle, in various ways and
on various platforms. This should be possible with BuildStream using only some
conditional statements, with minimal redundance and maximum reuse.
* **Backwards compatibility**
BuildStream provides various backwards compatible stable API surfaces, in this
way we ensure that nobody's project can ever break as a result of upgrading to
a new version of BuildStream.
These stable API surfaces include:
* The command line interface. BuildStream is intended to be scriptable and integratable
into third party tooling. For this reason the command line interface may be extended
from version to version but existing interfaces cannot be modified or removed.
* The Python plugin facing API surface. In order to avoid breaking anyone's project
who uses a custom or third party plugin, the plugin interfaces may be extended but
can never be modified or removed.
* The YAML format. As the main API surface for project authors, interfaces can be
extended in the YAML format but never modified or removed.
Beyond stability of the API surfaces, there is also the stability of the cache
keys. Currently BuildStream guarantees that artifact cache keys will never change
in a given stable release of BuildStream (e.g., all versions of 1.2.x will produce
the same cache key for the same project).
It is a long term goal to also make artifact cache keys stable, that any later version
of BuildStream produces the same cache key for an artifact which was built by any other
version of BuildStream
* **Build avoidance**
It is a general goal to reduce builds as much as possible. Whenever we can
guarantee that we already have a cached artifact which has identical inputs,
we should always prefer the existing artifact.
* **Convenient developer experience**
As an integration tool, we place focus on determinism first, but recognize that
developers need to have the same guarantees of determinism as integrators do, but
normally lack the tooling perform edit, compile and test cycles inside a well
defined deterministic build environment.
BuildStream aims to bring the deterministic and predictable target system
environment to the developer's fingertips, while also bringing convinent
developer tools to the integrator.
* **Decoupling of tooling and payload**
BuildStream is a generic build and integration tool which does not make any
assumptions about which software platform or machine architecture is going to
be used.
Some plugin elements invoking platform specific tooling such as autotools or
cmake, these provide configurable defaults for and are designed with maximum
configurability in mind, while the core application should not become biased
towards specific platforms.
* **Project Modularity**
From various experiences with tooling used to produce customized Linux based
appliances, we have recognized a trend that is to combine all build metadata
for the whole stack (from kernel to the user facing applications) in a single
repository, we see this as problematic as it does not allow inter-organizational
knowledge sharing easily, or separation of teams which produce and maintain separate
parts of the operating system stack.
BuildStream aims to make it easier to produce and maintain systems in a modular
fashion, where organizations or teams can maintain and share parts of the stack.
* **Easy to use**
Part of the mission is to be well documented, and as simple and straightforward
to use as possible.
As a part of this, we place great emphasis on error reporting, and try to fail
as early as possible when we know that we can fail; and provide as much useful
context to the user as possible to allow them to easily figure out what went
wrong.
* **Core simplicity, maximum flexibility**
BuildStream aims to be a generic core which simply processes an abstract pipeline
of elements which perform filesystem permutations inside a sandboxed environment.
We address the problem of scope creep in the following ways:
- **Many simple tools exposed in the CLI**
The command line interface is composed mostly of simple commands which
are API stable.
In this way we allow more complex and user specific constructs to be implemented
as shell scripts which invoke BuildStream one or more times, instead of growing
user specific features directly in the BuildStream CLI.
- **Stable Plugin API**
By providing a stable plugin API with strong guarantees that BuildStream will
not break external plugins, we hope to encourage and develop a healthy ecosystem
of useful plugins which users can reliably use in their project.
How do I use BuildStream?
=========================
Please refer to the `documentation <https://buildstream.gitlab.io/buildstream/>`_
for information about installing BuildStream, and about the BuildStream YAML format
and plugin options.
How does BuildStream work?
==========================
BuildStream operates on a set of YAML files (.bst files), as follows:
* Loads the YAML files which describe the target(s) and all dependencies.
* Evaluates the version information and build instructions to calculate a build
graph for the target(s) and all dependencies and unique cache-keys for each
element.
* Retrieves previously built elements (artifacts) from a local/remote cache, or
builds the elements in a sandboxed environment using the instructions declared
in the .bst files.
* Transforms/configures and/or deploys the resulting target(s) based on the
instructions declared in the .bst files.
How can I get started?
======================
To get started, first `install BuildStream by following the installation guide
<https://buildstream.gitlab.io/buildstream/main_install.html>`_
and then follow our tutorial in the
`user guide <https://buildstream.gitlab.io/buildstream/main_using.html>`_.
We also recommend exploring some existing BuildStream projects:
* https://gitlab.gnome.org/GNOME/gnome-build-meta/
* https://gitlab.com/freedesktop-sdk/freedesktop-sdk
* https://gitlab.com/baserock/definitions
If you have any questions please ask on our `#buildstream <irc://irc.gnome.org/buildstream>`_ channel in `irc.gnome.org <irc://irc.gnome.org>`_
|
