1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
|
<!-- $Id$ -->
<body>
<html>
<head>
<title>The realm of the build czar</title>
</head>
<h2>Build Czar Duties</h2>
<p>
The main duties of the Build Czar are summarized as follows
<li>
Continuously monitor the builds using the <a href="http://www.dre.vanderbilt.edu/scoreboard">
Scoreboard </a>
as one of the primary source of information.
<li>
Notify developers who broke compilation to fix the errors as soon as possible,
ideally by the next day. A red color in the "Compile" column is not at all
acceptable - the Build Czar needs to ensure that these problems are identified
and fixed in a timely manner. If possible, the Build Czar should let developers
know what the source of problems might be. It is quite possible that developers
who checked in the code or users who provided the patch may not have resources
to investigate the issues, so the Builds Czar's help is essential to keep
things moving ahead.
<li>
Keep an eye on the tests that are run in every build. Anything abnormal needs
to be notified to the right developer. The Build Czar should try helping the
developer by providing stack traces (in case of crashes) or other details like
printouts with debugging level turned on.
<li>
Some tests fail in the daily builds for many reasons like known bugs, transient
timeouts etc. Make sure that no new test failures show up. This <a href="mailto:jwillemsen@remedy.nl">
guy</a>
knows most of the information. Ask him to help you out with known problems.
<li>
Keep an eye on the <a href="http://www.dre.vanderbilt.edu/Stats">footprint and
performance</a> stats. Any abnormal changes should be brought to the
attention of the developer resposible for it or to the <a href="mailto:devo-group@list.isis.vanderbilt.edu">
devo group</a>.
<li>
Keep the builds ticking. Any red on the "Last Finished" column in the
Scoreboard should be fixed. The link to the "Build Name" indicates the machine
where the build is being run.
<li>
The builds don't cover all the possible configurations. If you get a bug report
about a compile error in a particular configuration, try setting up a build to
make sure that it doesn't show up again if it has been fixed.
<li>
Keep an eye on the <a href="http://bugzilla.dre.vanderbilt.edu/">bugzilla
</a>
entries that are registered by users and developers. Decide on the bugs that
need to be fixed for the beta and pain developers for an ETA.
</p>
<P>
The document <a href="./privileges.html">here </a>talks about the powers of a
build Czar.
</P>
<P>
The Build Czar is empowered to set up more builds on his own for his
convenience. This <a href="https://svn.dre.vanderbilt.edu/viewvc/ACE_autobuild/trunk/README?revision=HEAD">
page </a>has a step by step instructions on how to do that.
</P>
<P>
The build czar can get the build configuration by looking at the config portion
of the scoreboard.
</P>
<p>Pro-active involvement by the build czar is necessary. Being a pro-active
build czar requires monitoring the subversion archive carefully and responding
quickly to suspected changes to keep the repo stays stable.</p>
<hr>
<h2>Recipe for Cutting a Beta/Minor Kit</h2>
<P>
The build czar is also in charge for the release of the beta. Cutting a beta is
as simple as cutting butter if things go well. Here is the procedure followed
while cutting a beta:
<ol>
<li>
The whole process takes somewhere between 8-9 hours, it now takes much longer
than the original 2 hours for making the release itself where as generating the
doxygen documentation now completes in around this 2-3 hours mark.</li>
<li>
I suggest you take advantage of GNU Screen so that even if your SSH session is
interrupted, the cutting process can continue. This command is available on
both of the two machines we use to cut the release.
<ul>
<li>
type <code>screen</code> to start screen.</li>
<li>
execute commands as normal. Note that Ctrl-A is special in screen, so you need
to type Ctrl-A-A to send a Ctrl-A to the shell</li>
<li>
should your session be interrupted, reconnect and type <code>screen -x</code></li>
<li>
when finished, just type exit twice</li>
</ul>
<li>
Prior to starting this, gather aggregate release notes from all developers.
This is usually in the form of an email plea asking for a writeup of
significant changes since the last beta. Add these notes to the NEWS files
before cutting the release so that all notes are part of the release.</li>
<li>
Checkout a new workspace on <tt>anduril.dre.vanderbilt.edu</tt></li>
<ul>
<li>
The best place to create the workspace is under /export/anduriltmp/bczar. Don't
use the home directory itself, it is an NFS share and not really fast.
</li>
<li>
Checkout like this:
<ul>
<li>
svn co --username <your user id>
https://svn.dre.vanderbilt.edu/DOC/Middleware/trunk DOC_ROOT</li>
<li>
svn co --username <your user id>
https://svn.dre.vanderbilt.edu/DOC/MPC/trunk DOC_ROOT/ACE/MPC</li>
</ul>
</ul>
<li>
Set $DOC_ROOT to point to the new workspace you checked out.</li>
<li>
Set an environment variable SIGNATURE indicating your full name. This is used
to fill the ChangeLog entry.</li>
<ul>
<li>
For example,<tt>export SIGNATURE="Chris Cleeland"</tt></li></ul>
<li>
Set an environment variable MAILID indicating your mail id. This is used to
fill the mail id portion of the ChangeLog entry.</li>
<ul>
<li>
For example,<tt>export MAILID="cleeland@ociweb.com"</tt></li></ul>
<li>
Change directories to <tt>$DOC_ROOT</tt>
</li>
<li>
Tag the release by executing<br>
<code>ACE/bin/make_release.py --beta --update --tag</code><br>
This will only take a couple minutes to complete and once done successfully,
you can carry on with BOTH creating the kits and generating the doxygen
documentation in parallel. NOTE that <code>--beta</code> should be replaced
with <code>--minor</code> or <code>--major</code> as appropriate.</li>
<br>
<br>
In the <em>EXTREMELY</em> unlikely event that something goes wrong during the <em>
tagging</em> of the repo, the following files must be returned to the state
they were in before the release process started and then checked back into SVN:<br>
<ul>
<code>
<li>
ACE/ChangeLog</li>
<li>
ACE/PROBLEM-REPORT-FORM</li>
<li>
ACE/VERSION</li>
<li>
ACE/TAO/ChangeLog</li>
<li>
ACE/TAO/PROBLEM-REPORT-FORM</li>
<li>
TAO/VERSION</li>
<li>
CIAO/ChangeLog</li>
<li>
CIAO/PROBLEM-REPORT-FORM</li>
<li>
CIAO/VERSION</li>
<li>
CIAO/ciao/Version.h</li>
<li>
TAO/tao/Version.h</li>
ace/Version.h</li> </code>
</ul>
In most cases, a<br>
<code>svn revert -R *</code><br>
from DOC_ROOT will suffice.<br />
The tag will also need to be removed (both in Middleware and MPC):
ACE+TAO+CIAO-X_Y_Z (where X is the ACE Major version number, and Y & Z are the
Minor and Beta release numbers of the release that is to be restarted).<p>
E.g.:<br>
<code>svn rm
https://svn.dre.vanderbilt.edu/DOC/Middleware/tags/ACE+TAO+CIAO-X_Y_Z<br />
svn rm https://svn.dre.vanderbilt.edu/DOC/MPC/tags/ACE+TAO+CIAO-X_Y_Z<br />
</code>
<br>
Note that this <em>only</em> needs to be done if the <em>tagging</em> fails. If
kit creation fails, simply restart that process.
<li>
Create the kits by executing<br>
<code>ACE/bin/make_release.py --kit</code><br>
This will take somewhere arround 4-8 hours to complete.
<ul>
<li>
These commands only tags and creates the kits for the software itself, not
documentation, this can be started in parrellel with this activity.
</li>
<li>
The kits end up in <tt>/export/anduriltmp/bczar/packages</tt></li>
</ul>
<p>
To summarize, the following is a transcript of the steps up to this point
executing successfully:
<p><code>$ ssh bczar@anduril.dre.vanderbilt.edu<br>
No default printer<br>
-bash-3.00$ screen<br>
-bash-3.00$ cd /export/anduriltmp/bczar<br>
-bash-3.00$ rm -rf DOC_ROOT<br>
-bash-3.00$ mkdir DOC_ROOT<br>
-bash-3.00$ export DOC_ROOT=$PWD/DOC_ROOT<br>
-bash-3.00$ export SIGNATURE="Johnny Willemsen"<br>
-bash-3.00$ export MAILID=jwillemsen@remedy.nl<br>
-bash-3.00$ svn co --username johnnyw https://svn.dre.vanderbilt.edu/DOC/Middleware/trunk DOC_ROOT<br>
-bash-3.00$ svn co --username johnnyw https://svn.dre.vanderbilt.edu/DOC/MPC/trunk
DOC_ROOT/ACE/MPC<br>
-bash-3.00$ cd DOC_ROOT/<br>
-bash-3.00$ ACE/bin/make_release.py --beta --update --tag<br>
-bash-3.00$ ACE/bin/make_release.py --kit<br>
</code>
<p>
Feel free to cut and paste with suitable edits.
<li>
The packages end up by default under $DOC_ROOT/package-<PID>, you can
copy them to the webserver using the following commands. (Note that <PID>
needs to be the numerical pid of the process that created the kit, use<br>
<code>ls -ald</code><br>
to determine the correct filename.) At the moment you execute these commands
all users can download these packages.</li><br>
<code>cp $DOC_ROOT/package-<PID>/ACE*
/export/www/download.dre/ACE+TAO-distribution<br>
</code>
<li>
After the repository is tagged you can also start generating the doxygen
documentation in parrellel with the kit generation above.<br>
<ul>
<li>
Login to naboo.dre.vanderbilt.edu as bczar and start a screen session:<br>
<code>ssh bczar@naboo.dre.vanderbilt.edu</code><br>
<code>screen</code></li>
<li>
After login check that you can,
<br>
<code>ssh bczar@download.dre.vanderbilt.edu</code><br>
to ensure that this succeeds. If not fix the problem, if ok exit again back to
naboo. The make script tries to copy the tar.gz files to the website using ssh.</li>
<li>
<code>cd /web/users/isisbuilds/tmp/ACE_wrappers</code><br>
and remove the contents of this directory with</li><br>
<code>rm -rf *</code>
<li>
Update the workspace with the right version tag (replace the X_Y_Z with the ACE
version number being released e.g. 5_6_7)
<br>
<code>svn co
svn://svn.dre.vanderbilt.edu/DOC/Middleware/tags/ACE+TAO+CIAO-X_Y_Z/ACE
ACE_wrappers<br>
svn co svn://svn.dre.vanderbilt.edu/DOC/Middleware/tags/ACE+TAO+CIAO-X_Y_Z/TAO
ACE_wrappers/TAO<br>
svn co svn://svn.dre.vanderbilt.edu/DOC/Middleware/tags/ACE+TAO+CIAO-X_Y_Z/CIAO
ACE_wrappers/TAO/CIAO </code>
</li>
<li>
Set the needed environment variables using<br>
<code>export ACE_ROOT=/web/users/isisbuilds/tmp/ACE_wrappers/ACE_wrappers</code></li>
<li>
Check the doxygen version at the shell by executing the command:<br>
<code>doxygen --version</code>
</li>
<li>
Open up <code>$ACE_ROOT/bin/generate_rel_manpages</code> in your favorite
editor.<br>
Search for the string '<code><b>doxy_version</b></code>'.<br>
Check that the version specified in the file is the same as the one you got
using the shell command. If it is different change the value of the
doxy_version to the one installed on naboo.dre.vanderbilt.edu.</li> Then exit the editor.
Check the file, generate_rel_manpages back into the repository if you have made some
changes to it.
<br>
Now you are ready to create documentation
</li>
<li>
<code>cd $ACE_ROOT</code><br>
<code>make -f Release manpages</code>
</li></ul><br>
<li>
While doxygen churns, format a release announcement, including the release
notes gathered from developers.
<ul>
<li>Get from bugzilla the bugs fixed. Use the following
<a href="http://bugzilla.dre.vanderbilt.edu/buglist.cgi?query_format=advanced&short_desc_type=allwordssubstr&short_desc=&product=ACE&product=CIAO&product=CoSMIC&product=MPC&product=OpenDDS&product=TAO&long_desc_type=allwordssubstr&long_desc=&bug_file_loc_type=allwordssubstr&bug_file_loc=&status_whiteboard_type=allwordssubstr&status_whiteboard=&bug_status=RESOLVED&bug_status=VERIFIED&bug_status=CLOSED&emailassigned_to1=1&emailtype1=substring&email1=&emailreporter2=1&emailtype2=substring&email2=&bugidtype=include&bug_id=&votes=&chfieldfrom=2008-11-28&chfieldto=Now&chfield=resolution&chfieldvalue=FIXED&cmdtype=doit&order=Reuse+same+sort+as+last+time&field0-0-0=noop&type0-0-0=noop&value0-0-0=">query</a> as start query and update the start date.</li>
<li>
Let <a href="mailto:schmidt@cs.wustl.edu">Doug Schmidt</a> review these before
you do anything with them.</li></ul>
<li>
Make sure the new version is available in Bugzilla.</li>
<ul>
<li>
We now have a bczar Bugzilla user (bczar@dre.vanderbilt.edu) with full
privileges which points to the bczar user at ISIS. To gain access to this
account as a new build czar, you could update the ~/.forward file on one of the
ISIS hosts (for example bczar@naboo.dre.vanderbilt.edu) with your own email
address (but be aware that if you leave this ~/.forward file in effect, you
will get innundated with cron mail messages from all of the ISIS lab build
machines, so it is probably best to remove it after obtaining the Bugzilla
password). From the "Bugzilla - Main Page"
(http://bugzilla.dre.vanderbilt.edu/index.cgi) click on one of the various
"LogIn" links, and from this login page you should be able to "Submit Request"
to change a forgotten password. If you enter bczar@dre.vanderbilt.edu and click
"Submit Request" this will email you the password change link to
bczar@dre.vanderbilt.edu, which will then in turn be forwarded to your own
email account. Simply copy this link into your browser to set your own bczar
password for the next steps.</li>
<li>
here is the description of how to add a new version through Bugzilla.</li>
<li>
go to any Bugzilla "Query" page, you should see a grey box at the bottom. click
"Log in" link to log in as bczar@dre.vanderbilt.edu.</li>
<li>
look at the grey box at bottom again. You will see several links following
"Edit". Click on the "Product" link.</li>
<li>
you are then at "Select product" page. You should see a list components, i.e.,
ACE CIAO TAO. Click on the product you want to edit.</li>
<li>
you are then at "Edit product" page. Scroll down a bit, you should see a list
of all versions coming with this product. At the very beginning of the list,
you should see "Edit versions" link. Click this link.</li>
<li>
you should see a "Add a new version" text box and a "Add" link just above the
grey box at the bottom. Click on this link</li>
<li>
you are then at "Add version of [Name of the product]" page.</li>
<li>
you are able to add the new version now.</li>
</ul><br>
<li>
Back on <b>naboo.dre.vanderbilt.edu</b> once the doxygen generation has finished, update the documentation at www.dre.vanderbilt.edu/Doxygen.</li>
<ul>
<li>
<code>cd /web/www/Doxygen</code></li>
<li>
Create a temporary directory. eg. tmp and change Directory to this- <br><code>mkdir tmp</code><br><code>cd tmp</code></li>
<li>
scp file from the download server -<br><code>scp
bczar@download.dre.vanderbilt.edu:/export/www/download.dre/ACE+TAO-distribution/ACE-html.tar.bz2 .</code>
</li>
<li>
Unzip and untar the files - <br><code>bunzip2 ACE-html.tar.bz2;tar -xvf ACE-html.tar</code></li>
<li>back out of the temporary directory<br>
<code>cd ..</code></li>
<li>
Create directory 'Current Version No' for example 5.6.7 and change directory into this new one<br><code>mkdir 5.6.7<br>cd 5.6.7</code></li>
<li>
Move contents of the temporary directory's html to this directory -<br><code>mv ../tmp/html .</code></li>
<li>
Now back our of this directory and remove the already existing softlink to the "Micro" directory -<br><code>cd ..<br>rm Micro</code></li>
<li>
Create softlink "Micro" linking it to new Documentation using -<br><code>ln -s 5.6.7/html
Micro</code></li>
<li>
Remove the directory tmp -<br><code>rm -rf tmp</code></lib>
</ul><br>
<li>
Back on <b>anduril.dre.vanderbilt.edu</b> where the kit was being generated and once <b>BOTH</b> the kit
and doxygen generation have finished their work, you should also move the packages to the
previous versions directory with the appropriate decorators.
<ul>
<li>
<code>cd /export/www/download.dre/ACE+TAO-distribution</code></li>
<li>
Check that the packages in this directory are all from today</li>
<li>
Modify <b><code>/export/anduriltmp/bczar/copy_script.sh</code></b>
to use the correct ACE version X.Y.Z and run it.
<li>
Update the copy_script.sh file for the new micro release</li>
</ul><br>
<li>
Validate the packages on the webserver whether they are really containing the new release. Make at least
one build where you run the TAO Hello world test and check if the libraries are having the
correct version number.</li>
<li>
Update in the autobuild archive the file configs/scoreboard/releases.xml with
the made release (version number and release date). This is used by the integrated scoreboard on http://scoreboard.theaceorb.nl Remember to do a changelog entry.</li>
<li>
Update the ACE_wrappers repo (remember to create a changelog entry, and possiably archive the old changelog to the changelog directory if this has become too long):<ul>
<li>docs/Download.html to show the new release. Make sure you refer to the
previous_versions directory, that way we can exactly track how many people
download a specific version.</li>
<li>
etc/index.hml to show the new doxygen package you installed</li>
<li>bin/diff-builds-and-group-fixed-tests-only.sh to give the correct default old_date for this release.</li></ul>
<li>
Update the NEWS, TAO/NEWS, and TAO/CIAO/NEWS files to have a new section for the next release.</li>
<li>
Mail the approved release announcement out to, at minimum the following: <tt>ciao-users@list.isis.vanderbilt.edu</tt>,
<tt>tao-users@list.isis.vanderbilt.edu</tt>, <tt>tao-announce@list.isis.vanderbilt.edu</tt>,
<tt>ace-users@list.isis.vanderbilt.edu</tt>, <tt>ace-announce@list.isis.vanderbilt.edu</tt>.
Do this as yourself (not as bugzilla). <b>N.B.</b>
You will not be able to post to the users' lists unless you are subscribed to
them. Odds are you will not be able to post to the announce lists at all. Ask
someone else (like Doug or Johnny) to do this step.<br>
</ol>
</p>
<hr>
<h2>
Tips to being a Build Czar</h2>
<p>
1. Trust no one.<br>
2. Be careful with <a href="http://www.cs.wustl.edu/~schmidt">this guy</a>, he
is notorious in breaking builds (and fixing them as well...Rumour has it that
it's actually a super-scalar, super-pipelined processor capable of out-of-order
execution, in human incarnation).<br>
3. Don't forgive people who break ACE :-)<br>
4. If a build hasn't run in a long time (symptoms are a "red" in the Last Run
column of the build scoreboard), delete the .disable file in
/path/to/build/directory/BUILD_NAME/ by hand.<br>
5. Think of the group who wrote the scoreboard update script, every time you
catch an otherwise not so obvious error with the help of the scoreboard. Tell <a href="mailto:devo-group@list.isis.vanderbilt.edu">
DEVO group</a> about it.<br>
6. Send a note to <a href="mailto:sysadmin@isis.vanderbilt.edu">sysadmin@isis.vanderbilt.edu</a> asking for the repo to be frozen. Provide them a list of names, including yourself and bczar to be granted write permission.
<br>
7. Compile once on Win32, Linux and Solaris before cutting a beta.<br>
8. Trust the release script when making a release. Don't make tar balls by
hand. Make sure that the public ftp directories
(/project/beguine/ftp/pub/ACE+TAO-distribution and
/project/beguine/ftp/pub/ACE+TAO-distribution/diffs) have the right
permissions, so that the release script can copy the tar balls.<br>
9. When all hell breaks loose, don't wait for the nightly builds to monitor
improvement. Instead manually start the builds.<br>
10. Maintain private up-to-date workspaces for problem platforms (read as
Solaris).<br>
11. Don't hesitate to ask for help.<br>
12. When you get an account to access the svn repo, make sure you are added to
the correct groups, for example,
gid=100(users),5000(doc),5002(acetaodev),5003(cvs). Otherwise you will have
problem to checkout various modules.<br>
13. Install your public key to the different machines you have frequent access
to avoid typing password.<br>
14. Update this page if you have any more tips for future build czars :-). This
page is in svn under <code>ACE_wrappres/docs/bczar/bczar.html</code><br>
</p>
<hr>
<Center>
<h1>The Realm of the Build Czar</h1>
</Center>
<hr>
<h2>Build Czar Arthur</h2>
<p>Many years have passes since the days of the legendary Build Czar Arthur. His
duties were given to him by the mystical Lady of the Lake, who outlined the
first responsibilities of the Build Czar.</p>
<tt>
<br>
Then bespake the Lady of the Lake,<br>
And these were the words said shee:<br>
"I knoweth of thy deeds, thou noble Arthur,<br>
but thy task hath not finished for thee"<br>
<br>
"Thou shalt feitch thy trusty steed,<br>
And cleanse thy builds againe;<br>
Then shallt thy ryde hath finnished,<br>
When new kits released thee cann."<br>
<br>
Then bespake him noble Arthur<br>
And these were the words said he:<br>
"With what weapons shallt I use,<br>
To asure these from the devil free?"<br>
<br>
Then appeered before noble Arthur,<br>
Uppon the ground a sacred scroll<br>
Conjurred by the Lady of the Lake<br>
Borne of the earth in a roll.<br>
<br>
She saies, "Clasp this to thine selfe<br>
For thee shallt find need for it.<br>
It shall keep others in the cold,<br>
Only to be ressurected when thee sees fit."<br>
<br>
"Others shall join thy person,<br>
To ryde with thee in thy quest;<br>
Thee shallt be thankful of theire help,<br>
And to alsoe hold them steadfast."<br>
<br>
"But if theire talke too lodly rise,<br>
And causeth much damage to thine cuntry,<br>
He must come forth, and make proclamation,<br>
For the next one he shall be."<br>
<br>
So hath Arthur to the Lady spoke:<br>
"For I sweare, and save my othe,<br>
While enimes and evils I seeke,<br>
I shall fight against them bothe.<br>
<br>
</tt>
<hr>
</html>
</body>
|
