1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
|
<HTML>
<!-- this file was generated by troffcvt and tc2html -->
<HEAD>
<TITLE>
include_server manual page
</TITLE>
</HEAD>
<BODY>
<H1>
include_server manual page<BR>
</H1>
<P>
<H2>
Table of Contents
</H2>
<!-- INSERT TOC HERE, MAYBE -->
<!-- TOC BEGIN -->
<UL>
<LI>
<A HREF=#TOC_1> NAME</A>
<LI>
<A HREF=#TOC_2> SYNOPSIS</A>
<LI>
<A HREF=#TOC_3> DESCRIPTION</A>
<LI>
<A HREF=#TOC_4> OPTION SUMMARY</A>
<LI>
<A HREF=#TOC_5> INCLUDE SERVER SYMPTOMS AND ISSUES</A>
<LI>
<A HREF=#TOC_6> DISTCC DISCREPANCY SYMPTOMS</A>
<LI>
<A HREF=#TOC_7> EXIT CODES</A>
<LI>
<A HREF=#TOC_8> ENVIRONMENT VARIABLES</A>
<LI>
<A HREF=#TOC_9> BUGS</A>
<LI>
<A HREF=#TOC_10> AUTHOR</A>
<LI>
<A HREF=#TOC_11> LICENCE</A>
<LI>
<A HREF=#TOC_12> SEE ALSO</A>
</UL>
<!-- TOC END -->
<BR>
<H2>
<A NAME=TOC_1>
NAME</A>
</H2>
include_server.py - conservative approximation of include dependencies
for C/C++<BR>
<H2>
<A NAME=TOC_2>
SYNOPSIS</A>
</H2>
<B>include_server</B> --port <I> INCLUDE_SERVER_PORT [OPTIONS]</I><BR>
<H2>
<A NAME=TOC_3>
DESCRIPTION</A>
</H2>
include_server.py starts an include server process. This process
answers queries from <B>distcc</B>(1) clients about what files
to include in C/C++ compilations. The include_server.py command
itself terminates as soon as the include server has been spawned.<BR>
<P>
The INCLUDE_SERVER_PORT argument is the name of a socket used
for all communication between distcc clients and the include server.
The <B>pump</B>(1) command is responsible for creating the socket
location, for passing it to this script, and for passing it to
all distcc clients via the environment variable named INCLUDE_SERVER_PORT.<BR>
<P>
The protocol used by the include server uses distcc's RPC implementation.
Each distcc request consists of (1) the current directory and
(2) the list of arguments of the compilation command.<BR>
<P>
If the include server is able to process the request, then it
answers the distcc client by sending a list of filepaths. The
filepaths are those of the compressed source and header files
found to be necessary for compilation through include analysis.
The list also comprises symbolic links and even dummy files needed
for the compilation server to construct an accurate replica of
the parts of the filesystem needed for compilation. In this way,
a needed header file like /path/foo.h is compressed, renamed,
and stored in a temporary location, such as /dev/shm/tmpiAvfGv.include_server-9368-1/path/foo.h.lzo.
The distcc client will pass these files on to a compilation server,
where they will be uncompressed and mounted temporarily.<BR>
<P>
If the include server is not able to process the request, then
it returns the empty list to the distcc client. There are two
kinds of failures that relate to the include server. The include
server may fail to compute the includes or fail in other ways,
see section <B>INCLUDE SERVER SYMPTOMS</B>. Also, the compilation
on the remove server may fail due to inadequacy of the calculated
include closure, but then succeed when locally retried, see section
<B>DISTCC DISCREPANCY SYMPTOMS</B>.<BR>
<P>
<H2>
<A NAME=TOC_4>
OPTION SUMMARY</A>
</H2>
The following options are understood by include_server.py.<BR>
<DL>
<DT>
<B>-dPAT, --debug_pattern=PAT</B>
</DT>
<DD>
Bit vector for turning on warnings and debugging<BR>
1 = warnings<BR>
2 = trace some functions<BR>
other powers of two: see include_server/basics.py.<BR>
</DD>
<DT>
<B>-e, --email</B>
</DT>
<DD>
Send email to 'distcc-pump-errors' or if defined, the value of
enviroment variable DISTCC_EMAILLOG_WHOM_TO_BLAME, when include
server gets in trouble. The default is to not send email.<BR>
</DD>
<DT>
<B>--email_bound NUMBER</B>
</DT>
<DD>
Maximal number of emails to send (in addition to a final email).
Default: 3.<BR>
</DD>
<DT>
<B>--no-email</B>
</DT>
<DD>
Do not send email. This is the default.<BR>
</DD>
<DT>
<B>--path_observation_re=RE</B>
</DT>
<DD>
Issue warning message whenever a filename is resolved to a realpath
that is matched by RE, which is a regular expression in Python
syntax. This is useful for finding out where files included actually
come from. Use RE="" to find them all. Note: warnings
must be enabled with at least -d1.<BR>
</DD>
<DT>
<B>--pid_file FILEPATH</B>
</DT>
<DD>
The pid of the include server is written to file FILEPATH. This
allows a script such a <B>pump</B> to tear down the include server.<BR>
</DD>
<DT>
<B>-s, --statistics</B>
</DT>
<DD>
Print information to stdout about include analysis.<BR>
</DD>
<DT>
<B>--stat_reset_triggers=LIST</B>
</DT>
<DD>
Flush stat caches when the timestamp of any filepath in LIST changes
or the filepath comes in or out of existence. LIST is a colon
separated string of filepaths, possibly containing simple globs
(as allowed by Python's glob module). Print a warning whenever
such a change happens (if warnings are enabled). This option allows
limited exceptions to distcc_pump's normal assumption that source
files are not modified during the build.<BR>
</DD>
<DT>
<B>-t, --time</B>
</DT>
<DD>
Print elapsed, user, and system time to stderr.<BR>
</DD>
<DT>
<B>--unsafe_absolute_includes</B>
</DT>
<DD>
Do preprocessing on the compilation server even if includes of
absolute filepaths are encountered. Normally the include-server
will fall back on local preprocessing if it detects any absolute
includes. Thus, this flag is useful for preventing such fallbacks
when the absolute includes are a false alarm, either because the
absolute include is discarded during preprocessing or because
the absolutely included file exists on the compilation servers.<BR>
<P>
More precisely, with --unsafe_absolute_includes absolute includes
are ignored for the purposes of gathering the include closure.
Using this option may lead to incorrect results because (1) the
header may actually be included on the compilation server and
it may not be the same as on the client, (2) the include directives
of the header are not further analyzed.<BR>
<P>
The option is useful for compiling code that has such hardcoded
absolute locations of header files inside conditional directives
(e.g. "#ifdef") that render the includes irrelevant.
More precisely, these includes must be eliminated during preprocessing
for the actual configuration. Then the question of existence of
the header file is moot and the remote compilation is sound. This
is often the case if such includes are meant for unusual configurations
different from the actual configuration.<BR>
</DD>
<DT>
<B>-v, --verify</B>
</DT>
<DD>
Verify that files in CPP closure are contained in closure calculated
by include processor.<BR>
</DD>
<DT>
<B>-w, --write_include_closure</B>
</DT>
<DD>
Write a .d_approx file which lists all the included files calculated
by the include server; with -x, additionally write the included
files as calculated by CPP to a .d_exact file.<BR>
</DD>
<DT>
<B>-x, --exact_analysis</B>
</DT>
<DD>
Use CPP instead, do not omit system headers files.<BR>
<P>
</DD>
</DL>
<H2>
<A NAME=TOC_5>
INCLUDE SERVER SYMPTOMS AND ISSUES</A>
</H2>
<BR>
<P>
<P>
The most likely messages and warnings to come from the include
processor are listed below.<BR>
<P>
<DL>
<DT>
<B>Preprocessing locally. Include server not covering: Couldn't
determine default system include directories</B>
</DT>
<DD>
To determine the default system header directories, the include
server runs the compiler once for each language needed during
its session. This message indicates that the compiler specified
to distcc is not present on the client.<BR>
<P>
</DD>
<DT>
<B>Preprocessing locally. Include server not covering: Bailing
out because include server spent more than ...s user time handling
request</B>
</DT>
<DD>
In uncommon situations, the include server fails to analyze very
complicated macro expressions. The distcc client will use plain
distcc mode.<BR>
<P>
</DD>
<DT>
<B>Warning: Filepath must be relative but isn't</B>
</DT>
<DD>
The include server does not accept absolute filepaths, such as
/usr/include/stdio.h, in include directives, because there is
no guarantee that this header on the compilation server machine
will be the same as that on the client. The include server gives
up analyzing the include closure. The distcc client cannot use
pump-mode.<BR>
<P>
To overcome this problem in a not always reliable way, set the
environment variable INCLUDE_SERVER_ARGS='--unsafe_absolute_includes'
when invoking the pump script to pass the --unsafe_absolute_includes
option to the include server.<BR>
</DD>
<DT>
<B>Warning: Absolute filepath ... was IGNORED</B>
</DT>
<DD>
The --unsafe_absolute_includes is in use. This situation happens
under the same circumstances as when "Filepath must be relative
but isn't" is issued, but in this case the include will provide
an answer to the distcc client.<BR>
</DD>
<DT>
<B>Warning: Path '/PATH/FILE' changed/came into existence/no longer
exists</B>
</DT>
<DD>
These warnings are issued when using stat reset triggers. Because
/PATH/FILE changed, the include server clears its caches; the
new version of the file (or the lack of it) renders the include
analysis invalid. This message can usually be ignored; it does
signify a somewhat precarious use of files by the build system.
It is recommended to fix the build system so that files are not
rewritten.<BR>
<P>
</DD>
<DT>
<B>Warning: For translation unit ..., lookup of file ... resolved
to ... whose realpath is ...</B>
</DT>
<DD>
This warning occurs with --path_observation_re when a new realpath
matching a source or header file is observed.<BR>
<P>
<P>
</DD>
</DL>
<H2>
<A NAME=TOC_6>
DISTCC DISCREPANCY SYMPTOMS</A>
</H2>
<BR>
<P>
<P>
The interactions between the build system, distcc, and the include
server is somewhat complex. When a distcc commands receives a
failing compilation from the remote server it retries the compilation
locally. This section discusses the causes of discrepancies between
remote and local compilation. These are flagged by the demotion
message:<BR>
<P>
<B>__________Warning: ... pump-mode compilation(s) failed on server,</B>
<B>but succeeded locally.</B><BR>
<B>__________Distcc-pump was demoted to plain mode.</B> <B> See
the Distcc Discrepancy Symptoms section in the include_server(1)
man</B> <B> page.</B><BR>
<P>
The pump script issues this message at the end of the build. This
means that for at least one distcc invocation a local compilation
succeeded after the remote compilation failed. Each distcc invocation
for which such a discrepancy occurred in turn also issues a message
such as:<BR>
<P>
<B>Warning: remote compilation of '...' failed,</B> <B> retried
locally and got a different result.</B><BR>
<P>
The demotion makes subsequent distcc invocations use plain distcc
mode. Thus preprocessing will take place on the local machine
for the remainder of the build. This technique prevents very
slow builds where all compilations end up on the local machine
after failing remotely.<BR>
<P>
Of course, if the local compilations fails after the remote failures,
then the distcc invocation exits with the non-zero status of the
local compilation. The error messages printed are also those of
the local compilation.<BR>
<P>
The fallback behavior for distcc-pump mode to local compilation
can be disabled by setting the environment variable DISTCC_FALLBACK
to 1, which makes the distcc command fail as soon as the remote
compilation has failed. This setting is very useful for debugging
why the remote compilation went wrong, because now the output
from the server will be printed.<BR>
<P>
Next we discuss the possible causes of discrepancies.<BR>
<P>
<DL>
<DT>
<B>The user changed a source or header file during the build.</B>
</DT>
<DD>
This yields inconsistent results of course.<BR>
<P>
</DD>
<DT>
<B>A source or header file changed during the build.</B>
</DT>
<DD>
The build system rewrites a file. For Linux kernel 2.6, this
happens for 'include/linux/compile.h' and 'include/asm/asm-offsets.h'.
This condition is fixed by letting the include server know that
it must reset its caches when a stat of any of the files changes.
Practically, this is done by gathering the files in a colon-separated
list and then setting the INCLUDE_SERVER_ARGS command as in:<BR>
<P>
include_server_args="--stat_reset_triggers=include/linux/compile.h:include/asm/asm-offsets.h"<BR>
<P>
</DD>
<DT>
<B>A header file is potentially included, but does not exist,
and is then later included.</B>
</DT>
<DD>
<BR>
<P>
This occurs when some header foo.h includes another header file
trick.h, but the trick.h file has not yet been generated and the
inclusion is actually ignored because of preprocessing directives.
The include server will probe for the existence of trick.h, because
it overapproximates all possible ways directives actually evaluate.
The file trick.h is determined not to exist. If it is later generated,
and then really included, then the include server will falsely
believe that the file still does not exist. The solution to this
problem is to make the build system generate trick.h before the
first time any header file is included that makes a syntactic
reference to trick.h<BR>
<P>
</DD>
<DT>
<B>The include server was started with --unsafe_absolute_includes</B>.<B>
</DT>
<DD>
This is a problem if there are header files locally that do not
exist remotely and that are actually used. Such includes are
often protected by conditional directives that evaluate so that
are actually used on only specific and often uncommon platforms.
If you are not compiling for such a platform, then it may be correct
to use --unsafe_absolute_include</B>.<BR>
<P>
</DD>
<DT>
<B>The include server has calculated the wrong includes.</B>
</DT>
<DD>
We do not know of such a situation.<BR>
<P>
<P>
</DD>
</DL>
<H2>
<A NAME=TOC_7>
EXIT CODES</A>
</H2>
The exit code of include_server.py is usually 0. That the include
server has been started properly is communicated through the existence
of the pid_file.<BR>
<P>
<H2>
<A NAME=TOC_8>
ENVIRONMENT VARIABLES</A>
</H2>
<BR>
<P>
<B>DISTCC_EMAILLOG_WHOM_TO_BLAME</B> The email address to use
for include server automated emails. The default is 'distcc-pump-errors'
(which is an email address that probably will not exist in your
domain).<BR>
<P>
Additionally, the invocation of the compiler may use additional
environment variables.<BR>
<P>
<P>
<H2>
<A NAME=TOC_9>
BUGS</A>
</H2>
If you think you have found a distcc bug, please see the file
<I>reporting-bugs.txt</I> in the documentation directory for information
on how to report it.<BR>
<P>
<P>
In distcc-pump mode, the include server is unable to handle certain
very complicated computed includes as found in parts of the Boost
library. The include server will time out and distcc will revert
to plain mode.<BR>
<P>
Other known bugs may be documented on <I> http://code.google.com/p/distcc/</I><BR>
<P>
<H2>
<A NAME=TOC_10>
AUTHOR</A>
</H2>
The include server was written by Nils Klarlund, with assistance
from Fergus Henderson, Manos Renieris, and Craig Silverstein.
Please report bugs to <distcc@lists.samba.org>.<BR>
<P>
<H2>
<A NAME=TOC_11>
LICENCE</A>
</H2>
You are free to use distcc. distcc (including this manual) may
be copied, modified or distributed only under the terms of the
GNU General Public Licence version 2 or later. distcc comes with
absolutely no warrany. A copy of the GPL is included in the file
COPYING.<BR>
<P>
<H2>
<A NAME=TOC_12>
SEE ALSO</A>
</H2>
<B>distcc</B>(1), <B>distccd</B>(1), <B>include_server</B>(1),
and <B>gcc</B>(1). http://code.google.com/p/distcc/ http://ccache.samba.org/<BR>
</BODY>
</HTML>
|
