1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
|
Gitlab Continuous Integration (CI) for at-spi2-core
===================================================
Summary: make the robots set up an environment for running the test
suite, run it, and report back to us.
If you have questions about the CI, mail federico@gnome.org, or `file an
issue <https://gitlab.gnome.org/GNOME/at-spi2-core/-/issues>`__ and
mention ``@federico`` in it.
Table of contents:
[[*TOC*]]
Quick overview
==============
By having a ```.gitlab-ci.yml`` <../.gitlab-ci.yml>`__ file in the
toplevel directory of a project, Gitlab knows that it must run a
continuous integration pipeline when certain events occur, for example,
when someone creates a merge request, or pushes to a branch.
What’s a pipeline? It is an automated version of the following. Running
the test suite for at-spi2-core involves some repetitive steps:
- Create a pristine and minimal environment for testing, without all
the random gunk from one’s development system. Gitlab CI uses Linux
containers, with pre-built operating system images in the `Open
Container Initiative <https://opencontainers.org/>`__ format — this
is what Docker, Podman, etc. all use.
- Install the build-time dependencies (gcc, meson, libfoo-devel, etc.),
the test-time dependencies (dbus-daemon, etc.) in that pristine
environment, as well as special tools (lcov, libasan, clang-tools).
- Run the build and install it, and run the test suite.
- Run variations of the build and test suite with other tools — for
example, using static analysis during compilation, or with address
sanitizer (asan), or with a code coverage tool. Gitlab can collect
the analysis results of each of these tools and present them as part
of the merge request that is being evaluated. It also lets developers
obtain those useful results without dealing with a lot of fiddly
tools on their own computers.
Additionally, on each pipeline run we’d like to do extra repetitive work
like building the reference documentation, and publishing it on a web
page.
The ``.gitlab-ci.yml`` file defines the CI pipeline, the jobs it will
run (build/test, coverage, asan, static-scan, etc.), and the locations
where each job’s artifacts will be stored.
What’s an artifact or a job? Read on!
A little glossary
=================
**Pipeline** - A collection of **jobs**, which can be run in parallel or
sequentially. For example, a pair of “build” and “test” jobs would need
to run sequentially, but maybe a “render documentation” job can run in
parallel with them. Similarly, “build” jobs for various distributions or
configurations could be run in parallel.
**Job** - Think of it as running a shell script within a container. It
can have input from other previous jobs: if you have separate “build”
and “test” jobs, then the “build” job will want to keep around its
compiled artifacts so that the “test” job can use them. It can provide
output artifacts that can be stored for human perusal, or for use by
other jobs.
**Artifact** - Something produced from a job. If your job compiles
binaries, those binaries could be artifacts if you decide to keep them
around for use later. A documentation job can produce HTML artifacts
from the rendered documentation. A code coverage job will produce a
coverage report artifact.
**Runner** - An operating system setup for running jobs.
Gitlab.gnome.org provides runners for Linux, BSD, Windows, and MacOS.
For example. the Linux runners let you use any OCI image, so you can
test on openSUSE, Fedora, a custom distro, etc. You don’t normally need
to be concerned with runners; Gitlab assigns the shared runners
automatically to your pipeline.
**Container** - You can think of it as a chroot with extra isolation, or
a lightweight virtual machine. Basically, the Linux kernel can isolate
groups of processes in control groups (cgroups). Each cgroup can have a
different view of the file system, as if you had a different chroot for
each cgroup. Cgroups can be isolated to be in their own PID namespace,
so running “ps” in the container will not show all the processes in the
system, but only those inside the container’s cgroup. File system
overlays allow you to have read-only images for the operating system
(the OCI images we talked about above) plus a read-write overlay that is
kept around only during the lifetime of a container, or persistently if
one wants. For Gitlab CI one does not need to deal with containers
directly, but keep in mind that your jobs will run inside a container,
which is more limited than e.g. a shell session on a graphical,
development machine.
The CI pipeline for at-spi2-core
================================
The ``.gitlab-ci.yml`` file is a more-or-less declarative description
the CI pipeline, with some ``script`` sections which are imperative
commands to actually *do stuff*.
Jobs are run in ``stages``, and the names of the stages are declared
near the beginning of the YAML file. The stage names are arbitrary; the
ones here follow some informal GNOME conventions.
Jobs are declared at the toplevel of the YAML file, and they are
distinguished from other declarations by having a container ``image``
declared for them, as well as a ``script`` to execute.
Many jobs need exactly the same kind of setup (same container images,
mostly same package dependencies), so they use ``extends:`` to use a
declared template with all that stuff instead of redeclaring it each
time.
The ``build`` stage has these jobs:
- ``opensuse-x86_64`` - Extends the ``.only-default`` rule,
builds/installs the code, and runs the tests. Generally this is the
job that one cares about during regular development.
- ``fedora-x86_64`` - Same as the previous job, but runs on Fedora
instead of openSUSE. The intention is to run the build configuration
for ``dbus-broker`` here instead of ``dbus-daemon``.
The ``analysis`` stage has these jobs:
- ``static-scan`` - Runs static analysis during compilation, which
performs interprocedural analysis to detect things like double
``free()`` or uninitialized struct fields across functions.
- ``asan-build`` - Builds and runs with Address Sanitizer (libasan).
- ``coverage`` - Instruments the build to get code coverage
information, and runs the test suite to see how much of the code it
manages to exercise. This is to see which code paths may be untested
automatically, and to decide which ones would require manual testing,
or refactoring to allow automated testing.
As of 2022/Apr/19 there are some commented-out jobs to generate
documentation and publish it; this needs to be made to work.
CI templates
============
The task of setting up CI for a particular distro or build configuration
is rather repetitive. One has to start with a “bare” distro image, then
install the build-time dependencies that your project requires, then
that is slow, then you want to build a container image instead of
installing packages every time, then you want to test another distro,
then you want to make those container images easily available to your
project’s forks, and then you start pulling your hair.
`Fredesktop CI
Templates <https://gitlab.freedesktop.org/freedesktop/ci-templates/>`__
(`documentation <https://freedesktop.pages.freedesktop.org/ci-templates/>`__)
are a solution to this. They can automatically build container images
for various distros, make them available to forks of your project, and
have some nice amenities to reduce the maintenance burden.
At-spi2-core uses CI templates to test its various build configurations,
since it actually works differently depending on a distro’s choice of
``dbus-daemon`` versus ``dbus-broker``.
The prebuilt container images are stored here:
https://gitlab.gnome.org/GNOME/at-spi2-core/container_registry
They get updated automatically thanks to the CI Templates machinery.
General advice and future work
==============================
A failed run of a CI pipeline should trouble you; it either means that
some test broke, or that something is not completely deterministic. Fix
it at once.
Try not to accept merge requests that fail the CI, as this will make
``git bisect`` hard in the future. There are tools like Marge-bot to
enforce this; ask federico@gnome.org about it. Read `“The Not Rocket
Science Rule Of Software
Engineering” <https://graydon.livejournal.com/186550.html>`__, which can
be summarized as “automatically maintain a repository of code that
always passes all the tests” for inspiration. Marge-bot is an
implementation of that, and can be used with Gitlab.
If your software can be configured to build with substantial changes,
the CI pipeline should have jobs that test each of those configurations.
For example, at-spi-bus-launcher operates differently depending on
whether ``dbus-daemon`` or ``dbus-broker`` are being used. As of
2022/Apr/19 the CI only tests ``dbus-daemon``; there should be a test
for dbus-broker, too, in the ``fedora-x86_64`` job since that is one of
the distros that uses ``dbus-broker``.
Although the YAML syntax for ``.gitlab-ci.yml`` is a bit magic, the
scripts and configuration are quite amenable to refactoring. Do it
often!
Minimizing the amount of time that CI takes to run is a good goal. It
reduces energy consumption in the build farm, and allows you to have a
faster feedback loop. Instead of installing package dependencies on each
job, we can move to prebuilt container images.
References
==========
Full documentation for Gitlab CI: https://docs.gitlab.com/ee/ci/
Introduction to Gitlab CI:
https://docs.gitlab.com/ee/ci/quick_start/index.html
Freedesktop CI templates:
https://gitlab.freedesktop.org/freedesktop/ci-templates/
|
