1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
|
# Gitlab Continuous Integration (CI) for at-spi2-core
Summary: make the robots set up an environment for running the test
suite, run it, and report back to us.
If you have questions about the CI, mail federico@gnome.org, or [file
an issue](https://gitlab.gnome.org/GNOME/at-spi2-core/-/issues) and
mention `@federico` in it.
Table of contents:
[[_TOC_]]
# Quick overview
By having a [`.gitlab-ci.yml`](../.gitlab-ci.yml) file in the toplevel
directory of a project, Gitlab knows that it must run a continuous
integration pipeline when certain events occur, for example, when
someone creates a merge request, or pushes to a branch.
What's a pipeline? It is an automated version of the following.
Running the test suite for at-spi2-core involves some repetitive
steps:
* Create a pristine and minimal environment for testing, without all the random
gunk from one's development system. Gitlab CI uses Linux containers,
with pre-built operating system images in the [Open Container
Initiative][OCI] format — this is what Docker, Podman, etc. all use.
* Install the build-time dependencies (gcc, meson, libfoo-devel,
etc.), the test-time dependencies (dbus-daemon, etc.) in that
pristine environment, as well as special tools (lcov, libasan,
clang-tools).
* Run the build and install it, and run the test suite.
* Run variations of the build and test suite with other tools — for
example, using static analysis during compilation, or with address
sanitizer (asan), or with a code coverage tool. Gitlab can collect
the analysis results of each of these tools and present them as part
of the merge request that is being evaluated. It also lets
developers obtain those useful results without dealing with a lot of
fiddly tools on their own computers.
Additionally, on each pipeline run we'd like to do extra repetitive
work like building the reference documentation, and publishing it on a
web page.
The `.gitlab-ci.yml` file defines the CI pipeline, the jobs it will
run (build/test, coverage, asan, static-scan, etc.), and the locations
where each job's artifacts will be stored.
What's an artifact or a job? Read on!
# A little glossary
**Pipeline** - A collection of **jobs**, which can be run in parallel
or sequentially. For example, a pair of "build" and "test" jobs would
need to run sequentially, but maybe a "render documentation" job can
run in parallel with them. Similarly, "build" jobs for various
distributions or configurations could be run in parallel.
**Job** - Think of it as running a shell script within a container.
It can have input from other previous jobs: if you have separate
"build" and "test" jobs, then the "build" job will want to keep around
its compiled artifacts so that the "test" job can use them. It can
provide output artifacts that can be stored for human perusal, or for
use by other jobs.
**Artifact** - Something produced from a job. If your job compiles
binaries, those binaries could be artifacts if you decide to keep them
around for use later. A documentation job can produce HTML artifacts
from the rendered documentation. A code coverage job will produce a
coverage report artifact.
**Runner** - An operating system setup for running jobs.
Gitlab.gnome.org provides runners for Linux, BSD, Windows, and MacOS.
For example. the Linux runners let you use any OCI image, so you can
test on openSUSE, Fedora, a custom distro, etc. You don't normally
need to be concerned with runners; Gitlab assigns the shared runners
automatically to your pipeline.
**Container** - You can think of it as a chroot with extra isolation,
or a lightweight virtual machine. Basically, the Linux kernel can
isolate groups of processes in control groups (cgroups). Each cgroup
can have a different view of the file system, as if you had a
different chroot for each cgroup. Cgroups can be isolated to be in
their own PID namespace, so running "ps" in the container will not
show all the processes in the system, but only those inside the
container's cgroup. File system overlays allow you to have read-only
images for the operating system (the OCI images we talked about above)
plus a read-write overlay that is kept around only during the lifetime
of a container, or persistently if one wants. For Gitlab CI one does
not need to deal with containers directly, but keep in mind that your
jobs will run inside a container, which is more limited than e.g. a
shell session on a graphical, development machine.
# The CI pipeline for at-spi2-core
The `.gitlab-ci.yml` file is a more-or-less declarative description
the CI pipeline, with some `script` sections which are imperative
commands to actually *do stuff*.
Jobs are run in `stages`, and the names of the stages are declared
near the beginning of the YAML file. The stage names are arbitrary;
the ones here follow some informal GNOME conventions.
Jobs are declared at the toplevel of the YAML file, and they are
distinguished from other declarations by having a container `image`
declared for them, as well as a `script` to execute.
Many jobs need exactly the same kind of setup (same container images,
mostly same package dependencies), so they use `extends:` to use a
declared template with all that stuff instead of redeclaring it each
time.
The `build` stage has these jobs:
* `opensuse-x86_64` - Extends the `.only-default` rule,
builds/installs the code, and runs the tests. Generally this is the
job that one cares about during regular development.
* `fedora-x86_64` - Same as the previous job, but runs on Fedora
instead of openSUSE. The intention is to run the build
configuration for `dbus-broker` here instead of `dbus-daemon`.
The `analysis` stage has these jobs:
* `static-scan` - Runs static analysis during compilation, which
performs interprocedural analysis to detect things like double
`free()` or uninitialized struct fields across functions.
* `asan-build` - Builds and runs with Address Sanitizer (libasan).
* `coverage` - Instruments the build to get code coverage information,
and runs the test suite to see how much of the code it manages to
exercise. This is to see which code paths may be untested
automatically, and to decide which ones would require manual
testing, or refactoring to allow automated testing.
As of 2022/Apr/19 there are some commented-out jobs to generate
documentation and publish it; this needs to be made to work.
# CI templates
The task of setting up CI for a particular distro or build
configuration is rather repetitive. One has to start with a "bare"
distro image, then install the build-time dependencies that your
project requires, then that is slow, then you want to build a
container image instead of installing packages every time, then you
want to test another distro, then you want to make those container
images easily available to your project's forks, and then you start
pulling your hair.
[Fredesktop CI Templates][ci-templates]
([documentation][ci-templates-docs]) are a solution to this. They can
automatically build container images for various distros, make them
available to forks of your project, and have some nice amenities to
reduce the maintenance burden.
At-spi2-core uses CI templates to test its various build
configurations, since it actually works differently depending on a
distro's choice of `dbus-daemon` versus `dbus-broker`.
The prebuilt container images are stored here:
https://gitlab.gnome.org/GNOME/at-spi2-core/container_registry
They get updated automatically thanks to the CI Templates machinery.
# General advice and future work
A failed run of a CI pipeline should trouble you; it either means that
some test broke, or that something is not completely deterministic.
Fix it at once.
Try not to accept merge requests that fail the CI, as this will make
`git bisect` hard in the future. There are tools like Marge-bot to
enforce this; ask federico@gnome.org about it. Read ["The Not Rocket
Science Rule Of Software
Engineering"](https://graydon.livejournal.com/186550.html), which can
be summarized as "automatically maintain a repository of code that
always passes all the tests" for inspiration. Marge-bot is an
implementation of that, and can be used with Gitlab.
If your software can be configured to build with substantial changes,
the CI pipeline should have jobs that test each of those
configurations. For example, at-spi-bus-launcher operates differently
depending on whether `dbus-daemon` or `dbus-broker` are being used. As of
2022/Apr/19 the CI only tests `dbus-daemon`; there should be a test for
dbus-broker, too, in the `fedora-x86_64` job since that is one of the
distros that uses `dbus-broker`.
Although the YAML syntax for `.gitlab-ci.yml` is a bit magic, the
scripts and configuration are quite amenable to refactoring. Do it
often!
Minimizing the amount of time that CI takes to run is a good goal. It
reduces energy consumption in the build farm, and allows you to have a
faster feedback loop. Instead of installing package dependencies on
each job, we can move to prebuilt container images.
# References
Full documentation for Gitlab CI: https://docs.gitlab.com/ee/ci/
Introduction to Gitlab CI: https://docs.gitlab.com/ee/ci/quick_start/index.html
Freedesktop CI templates: https://gitlab.freedesktop.org/freedesktop/ci-templates/
[OCI]: https://opencontainers.org/
[ci-templates]: https://gitlab.freedesktop.org/freedesktop/ci-templates/
[ci-templates-docs]: https://freedesktop.pages.freedesktop.org/ci-templates/
|
