1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
|
# Building with a dedicated Jenkins CI instance
## History
Around April 2019, once Travis CI performance became so abysmal that the
development team couldn't tolerate it any longer. We decided to move off
of it entirely and onto our own dedicated Jenkins instance. Performance
was bad because:
* The Travis CI VMs are severely underpowered, leading to many
unexpected test failures
* The ASF wait queue for Travis was exceedingly long, with jobs
sometimes taking 30-45 minutes just to _launch_
* Our test suites (all 3 of Erlang, JavaScript and Elixir) had
sporadic failures, requiring many re-launches of builds before
we could move forward
We also had other reasons to want to move to a dedicated server:
* Credentials on the main Jenkins instance could be reused by *anyone*
in the ASF
* Credentials on the Travis CI setup should not contain credentials
for things such as building binaries, to ensure the best possible
security
Other options we explored:
* ASF BuildBot. This required a lot more work and couldn't guarantee
availability of sufficient workers.
* Circle CI, GitHub CI, etc.: Lack of availability of alternate binary
platforms we've been asked to support (arm64v8, ppc64le) or alternate
OSes (OSX, FreeBSD, Windows), though there is some support for either
or both.
## Cloudbees Core
End of November 2019, ASF and CloudBees reached an agreement to allow
the Foundation to use CloudBees Core to have a farm of managed Jenkins
masters. This allows the ASF to give larger projects their own dedicated
Jenkins master, which can be custom configured for the project. They can
readily manage this farm of Jenkins masters centrally, including push
updates to all masters and their plugins. Naturally, this also reduces
contention, as well as providing increased security for project-level
credentials. CouchDB is the first project to use this setup, via
https://ci-couchdb.apache.org/ (aka https://jenkins-cm1.apache.org/)
Only members of the ASF LDAP group `couchdb-pmc` have write access to
the Jenkins CI server, and all jobs are set to not trust changes to the
Jenkinsfile from forked repos (see below).
Further, IBM is sponsoring CouchDB with cloud-based worker nodes, for
the project's exclusive use. Combined with the FreeBSD and OSX nodes the
project already had, this will provide the necessary compute resources
to meet the needs of the project for some time to come.
# Jenkins configuration
All jobs on the new Jenkins master are contained in a CouchDB folder.
Credentials for the project are placed within this folder; this is a
unique capability of CloudBees Core at this time.
## Pull Request job configuration
Blue Ocean link: https://ci-couchdb.apache.org/blue/organizations/jenkins/jenkins-cm1%2FPullRequests/activity/
To implement build-a-PR functionality in the same way that Travis
performs builds, Jenkins offers the Multibranch Pipeline job. Here's how
it's configured for CouchDB through the GUI:
* Job name: PullRequests (jobs shouldn't have spaces in them, because
the job name is used for workspace path naming.)
* Display Name: "Pull Requests"
* Description: "This job builds all GitHub pull requests against
apache/couchdb."
* Branch sources: Github
* Credentials: a GitHub API key from wohali. These credentials are
stored in the top-level Jenkins CouchDB folder on the server.
The API token credentials are `user:email` and `repo:status`.
* URL https://github.com/apache/couchdb
* Behaviors
* Discover branches: Exclude branches that are also filed as PRs
* Discover PRs from origin: Merging the PR with the current target
branch revision
* Discover PRs from works: Merging the PR with the current target
branch revision, trust Nobody [2]
* Advanced clone behaviours:
* Fetch tags
* Clear "Shallow clone" [1]
* Clean before checkout [1]
* Prune stale remote-tracking branches
* Property strategy: All branches get the same properties
* Build Configuration
* Mode: by Jenkinsfile
* Script path: `build-aux/Jenkinsfile.pr`
* Scan Repository Triggers
* Periodically if not other wise run
* Interval: 1 day (do not set this too high, GitHub has an API token
throttle that can cause issues!)
* Orphaned Item Strategy
* Discard old items
* Days to keep old items: <blank>
* Max # of old items to keep: 10
* Everything else set as defaults.
[1]: https://issues.jenkins-ci.org/browse/JENKINS-44598 explains why we have the build set to Clean Before Checkout every time, and why clones are not shallow.
[2]: https://issues.apache.org/jira/browse/INFRA-17449 explains why "Discover pull requests from forks/Trust" has been set to "Nobody."
## `master` and release branch configuration
Our old Jenkins job (formerly `/Jenkinsfile`) is now
`build-aux/Jenkinsfile.full`. This builds CouchDB on `master`, all of
our release branches (`2.*`, `3.*`, etc.) as well as `jenkins-*` for
testing.
Settings are as follows:
* Job name: FullPlatformMatrix (jobs shouldn't have spaces in them, because
the job name is used for workspace path naming.)
* Display Name: "Full Platform Builds"
* Description: "This job builds on our master and release branches,
and builds packages on all."
* Branch sources: Github
* Credentials: a GitHub API key from wohali. These credentials are
stored in the top-level Jenkins CouchDB folder on the server.
The API token credentials are `user:email` and `repo:status`.
* URL https://github.com/apache/couchdb
* Behaviors
* Discover branches: All branches
* Filter by name (with wildcards): Include: master 2.*.x 3.*.x 4.*.x jenkins-*
* Advanced clone behaviours:
* Fetch tags
* Clear "Shallow clone" [1]
* Clean before checkout [1]
* Prune stale remote-tracking branches
* Property strategy: All branches get the same properties
* Build Configuration
* Mode: by Jenkinsfile
* Script path: `build-aux/Jenkinsfile.pull`
* Scan Repository Triggers
* none
* Orphaned Item Strategy
* Discard old items
* Days to keep old items: <blank>
* Max # of old items to keep: 10
* Everything else set as defaults.
|
