diff options
| -rw-r--r-- | Makefile.in | 9 | ||||
| -rw-r--r-- | doc/web/man/distcc_1.html | 40 | ||||
| -rw-r--r-- | doc/web/man/include_server_1.html | 484 | ||||
| -rw-r--r-- | packaging/RedHat/rpm.spec | 1 |
4 files changed, 526 insertions, 8 deletions
diff --git a/Makefile.in b/Makefile.in index f18153b..9194411 100644 --- a/Makefile.in +++ b/Makefile.in @@ -362,10 +362,11 @@ conf_files = $(conf_dir)/hosts \ default_dir = packaging/RedHat/default default_files = $(default_dir)/distcc -man1_MEN = man/distcc.1 man/distccd.1 man/distccmon-text.1 man/pump.1 +man1_MEN = man/distcc.1 man/distccd.1 man/distccmon-text.1 \ + man/pump.1 man/include_server.1 man_HTML = man/distcc_1.html man/distccd_1.html man/distccmon_text_1.html \ - man/pump_1.html -MEN = $(man1_MEN) + man/pump_1.html man/include_server_1.html +MEN = $(man1_MEN) gnome_data = gnome/distccmon-gnome-icon.png \ gnome/distccmon-gnome.desktop @@ -650,6 +651,8 @@ man/distccmon_text_1.html: man/distccmon-text.1 man/pump_1.html: man/pump.1 troff2html -man "$(srcdir)"/man/pump.1 > $@ +man/include_server_1.html: man/include_server.1 + troff2html -man "$(srcdir)"/man/include_server.1 > $@ ###################################################################### ## CHECK targets for code that has been build. diff --git a/doc/web/man/distcc_1.html b/doc/web/man/distcc_1.html index bac83ad..74eccf6 100644 --- a/doc/web/man/distcc_1.html +++ b/doc/web/man/distcc_1.html @@ -243,12 +243,12 @@ are being invoked, and whether the job can be distributed.<BR> <A NAME=TOC_7> HOW DISTCC-PUMP MODE WORKS</A> </H2> -In pump mode, distcc runs the prepreprocessor remotely too. To -do so, the preprocessor must have access to all the files that -it would have accessed if had been running locally. In pump mode, +In pump mode, distcc runs the preprocessor remotely too. To do +so, the preprocessor must have access to all the files that it +would have accessed if had been running locally. In pump mode, therefore, distcc gathers all of the recursively included headers, -except the ones that are part of the compiler installation, and -sends them along with the source file to the compilation server.<BR> +except the ones that are default system headers, and sends them +along with the source file to the compilation server.<BR> <P> In distcc-pump mode, the server unpacks the set of all source files in a temporary directory, which contains a directory tree @@ -293,6 +293,29 @@ such as that for Linux kernel 2.6, do not quite satisfy this requirement. To overcome such issues, and other corner cases such as absolute filepaths in includes, see the include_server(1) man page.<BR> <P> +Another important assumption is that the include configuration +of all machines must be identical. Thus the headers under the +default system path must be the same on all servers and all clients. +If a standard GNU compiler installation is used, then this requirement +applies to all libraries whose header files are installed under +/usr/include or /usr/local/include/. Note that installing software +packages often lead to additional headers files being placed in +subdirectories of either.<BR> +<P> +If this assumption does not hold, then it is possible to break +builds with distcc-pump mode, or worse, to get wrong results without +warning. Presently this condition is not verified, and it is +on our TODO list to address this issue. Also, currently, explicit +include specifications that resolve to something under the default +system search path such as -I/usr/include/foolib do not work except +if this directory is specified with -isystem.<BR> +<P> +An easy way to guarantee that the include configurations are identical +is to use a crosscompiler that defines a default system search +path restricted to directories of the compiler installation.<BR> +<P> +See the <B>include_server</B>(1) manual for more information on +symptoms and causes of violations of distcc-pump mode assumptions.<BR> <P> <H2> <A NAME=TOC_8> @@ -991,6 +1014,13 @@ very complicated computed includes as found in parts of the boost library. The include server will time out and distcc will revert to plain mode.<BR> <P> +In distcc-pump mode, includes of the form -I/usr/include/foolib +do not usually work, see discussion in section HOW DISTCC-PUMP +MODE WORKS above.<BR> +<P> +In distcc-pump mode, certain assumptions are made that source +and header files do not change during the build. See discussion +in section DISTCC DISCREPANCY SYMPTOMS of <B>include_server</B>(1().<BR> <P> Other known bugs may be documented on <I> http://code.google.com/p/distcc/</I><BR> <H2> diff --git a/doc/web/man/include_server_1.html b/doc/web/man/include_server_1.html new file mode 100644 index 0000000..39f5696 --- /dev/null +++ b/doc/web/man/include_server_1.html @@ -0,0 +1,484 @@ +<HTML> +<!-- this file was generated by troffcvt and tc2html --> +<HEAD> +<TITLE> +include_server manual page +</TITLE> +</HEAD> +<BODY> +<H1> +include_server manual page<BR> +</H1> +<P> +<H2> +Table of Contents +</H2> +<!-- INSERT TOC HERE, MAYBE --> +<!-- TOC BEGIN --> +<UL> +<LI> +<A HREF=#TOC_1> NAME</A> +<LI> +<A HREF=#TOC_2> SYNOPSIS</A> +<LI> +<A HREF=#TOC_3> DESCRIPTION</A> +<LI> +<A HREF=#TOC_4> OPTION SUMMARY</A> +<LI> +<A HREF=#TOC_5> INCLUDE SERVER SYMPTOMS AND ISSUES</A> +<LI> +<A HREF=#TOC_6> DISTCC DISCREPANCY SYMPTOMS</A> +<LI> +<A HREF=#TOC_7> EXIT CODES</A> +<LI> +<A HREF=#TOC_8> ENVIRONMENT VARIABLES</A> +<LI> +<A HREF=#TOC_9> BUGS</A> +<LI> +<A HREF=#TOC_10> AUTHOR</A> +<LI> +<A HREF=#TOC_11> LICENCE</A> +<LI> +<A HREF=#TOC_12> SEE ALSO</A> +</UL> +<!-- TOC END --> +<BR> +<H2> +<A NAME=TOC_1> +NAME</A> +</H2> +include_server.py - conservative approximation of include dependencies +for C/C++<BR> +<H2> +<A NAME=TOC_2> +SYNOPSIS</A> +</H2> +<B>include_server</B> --port <I> INCLUDE_SERVER_PORT [OPTIONS]</I><BR> +<H2> +<A NAME=TOC_3> +DESCRIPTION</A> +</H2> +include_server.py starts an include server process. This process +answers queries from <B>distcc</B>(1) clients about what files +to include in C/C++ compilations. The include_server.py command +itself terminates as soon as the include server has been spawned.<BR> +<P> +The INCLUDE_SERVER_PORT argument is the name of a socket used +for all communication between distcc clients and the include server. +The <B>pump</B>(1) command is responsible for creating the socket +location, for passing it to this script, and for passing it to +all distcc clients via the environment variable named INCLUDE_SERVER_PORT.<BR> +<P> +The protocol used by the include server uses distcc's RPC implementation. +Each distcc request consists of (1) the current directory and +(2) the list of arguments of the compilation command.<BR> +<P> +If the include server is able to process the request, then it +answers the distcc client by sending a list of filepaths. The +filepaths are those of the compressed source and header files +found to be necessary for compilation through include analysis. +The list also comprises symbolic links and even dummy files needed +for the compilation server to construct an accurate replica of +the parts of the filesystem needed for compilation. In this way, +a needed header file like /path/foo.h is compressed, renamed, +and stored in a temporary location, such as /dev/shm/tmpiAvfGv.include_server-9368-1/path/foo.h.lzo. +The distcc client will pass these files on to a compilation server, +where they will be uncompressed and mounted temporarily.<BR> +<P> +If the include server is not able to process the request, then +it returns the empty list to the distcc client. There are two +kinds of failures that relate to the include server. The include +server may fail to compute the includes or fail in other ways, +see section <B>INCLUDE SERVER SYMPTOMS</B>. Also, the compilation +on the remove server may fail due to inadequacy of the calculated +include closure, but then succeed when locally retried, see section +<B>DISTCC DISCREPANCY SYMPTOMS</B>.<BR> +<P> +<H2> +<A NAME=TOC_4> +OPTION SUMMARY</A> +</H2> +The following options are understood by include_server.py.<BR> +<DL> +<DT> +<B>-dPAT, --debug_pattern=PAT</B> +</DT> +<DD> +Bit vector for turning on warnings and debugging<BR> +1 = warnings<BR> +2 = trace some functions<BR> +other powers of two: see include_server/basics.py.<BR> +</DD> +<DT> +<B>-e, --email</B> +</DT> +<DD> +Send email to 'distcc-pump-errors' or if defined, the value of +enviroment variable DISTCC_EMAILLOG_WHOM_TO_BLAME, when include +server gets in trouble. The default is to not send email.<BR> +</DD> +<DT> +<B>--email_bound NUMBER</B> +</DT> +<DD> +Maximal number of emails to send (in addition to a final email). +Default: 3.<BR> +</DD> +<DT> +<B>--no-email</B> +</DT> +<DD> +Do not send email. This is the default.<BR> +</DD> +<DT> +<B>--path_observation_re=RE</B> +</DT> +<DD> +Issue warning message whenever a filename is resolved to a realpath +that is matched by RE, which is a regular expression in Python +syntax. This is useful for finding out where files included actually +come from. Use RE="" to find them all. Note: warnings +must be enabled with at least -d1.<BR> +</DD> +<DT> +<B>--pid_file FILEPATH</B> +</DT> +<DD> +The pid of the include server is written to file FILEPATH. This +allows a script such a <B>pump</B> to tear down the include server.<BR> +</DD> +<DT> +<B>-s, --statistics</B> +</DT> +<DD> +Print information to stdout about include analysis.<BR> +</DD> +<DT> +<B>--stat_reset_triggers=LIST</B> +</DT> +<DD> +Flush stat caches when the timestamp of any filepath in LIST changes +or the filepath comes in or out of existence. LIST is a colon +separated string of filepaths, possibly containing simple globs +(as allowed by Python's glob module). Print a warning whenever +such a change happens (if warnings are enabled). This option allows +limited exceptions to distcc_pump's normal assumption that source +files are not modified during the build.<BR> +</DD> +<DT> +<B>-t, --time</B> +</DT> +<DD> +Print elapsed, user, and system time to stderr.<BR> +</DD> +<DT> +<B>--unsafe_absolute_includes</B> +</DT> +<DD> +Do preprocessing on the compilation server even if includes of +absolute filepaths are encountered. Normally the include-server +will fall back on local preprocessing if it detects any absolute +includes. Thus, this flag is useful for preventing such fallbacks +when the absolute includes are a false alarm, either because the +absolute include is discarded during preprocessing or because +the absolutely included file exists on the compilation servers.<BR> +<P> +More precisely, with --unsafe_absolute_includes absolute includes +are ignored for the purposes of gathering the include closure. +Using this option may lead to incorrect results because (1) the +header may actually be included on the compilation server and +it may not be the same as on the client, (2) the include directives +of the header are not further analyzed.<BR> +<P> +The option is useful for compiling code that has such hardcoded +absolute locations of header files inside conditional directives +(e.g. "#ifdef") that render the includes irrelevant. +More precisely, these includes must be eliminated during preprocessing +for the actual configuration. Then the question of existence of +the header file is moot and the remote compilation is sound. This +is often the case if such includes are meant for unusual configurations +different from the actual configuration.<BR> +</DD> +<DT> +<B>-v, --verify</B> +</DT> +<DD> +Verify that files in CPP closure are contained in closure calculated +by include processor.<BR> +</DD> +<DT> +<B>-w, --write_include_closure</B> +</DT> +<DD> +Write a .d_approx file which lists all the included files calculated +by the include server; with -x, additionally write the included +files as calculated by CPP to a .d_exact file.<BR> +</DD> +<DT> +<B>-x, --exact_analysis</B> +</DT> +<DD> +Use CPP instead, do not omit system headers files.<BR> +<P> +</DD> +</DL> +<H2> +<A NAME=TOC_5> +INCLUDE SERVER SYMPTOMS AND ISSUES</A> +</H2> +<BR> +<P> +<P> +The most likely messages and warnings to come from the include +processor are listed below.<BR> +<P> +<DL> +<DT> +<B>Preprocessing locally. Include server not covering: Couldn't +determine default system include directories</B> +</DT> +<DD> +To determine the default system header directories, the include +server runs the compiler once for each language needed during +its session. This message indicates that the compiler specified +to distcc is not present on the client.<BR> +<P> +</DD> +<DT> +<B>Preprocessing locally. Include server not covering: Bailing +out because include server spent more than ...s user time handling +request</B> +</DT> +<DD> +In uncommon situations, the include server fails to analyze very +complicated macro expressions. The distcc client will use plain +distcc mode.<BR> +<P> +</DD> +<DT> +<B>Warning: Filepath must be relative but isn't</B> +</DT> +<DD> +The include server does not accept absolute filepaths, such as +/usr/include/stdio.h, in include directives, because there is +no guarantee that this header on the compilation server machine +will be the same as that on the client. The include server gives +up analyzing the include closure. The distcc client cannot use +pump-mode.<BR> +<P> +To overcome this problem in a not always reliable way, set the +environment variable INCLUDE_SERVER_ARGS='--unsafe_absolute_includes' +when invoking the pump script to pass the --unsafe_absolute_includes +option to the include server.<BR> +</DD> +<DT> +<B>Warning: Absolute filepath ... was IGNORED</B> +</DT> +<DD> +The --unsafe_absolute_includes is in use. This situation happens +under the same circumstances as when "Filepath must be relative +but isn't" is issued, but in this case the include will provide +an answer to the distcc client.<BR> +</DD> +<DT> +<B>Warning: Path '/PATH/FILE' changed/came into existence/no longer +exists</B> +</DT> +<DD> +These warnings are issued when using stat reset triggers. Because +/PATH/FILE changed, the include server clears its caches; the +new version of the file (or the lack of it) renders the include +analysis invalid. This message can usually be ignored; it does +signify a somewhat precarious use of files by the build system. +It is recommended to fix the build system so that files are not +rewritten.<BR> +<P> +</DD> +<DT> +<B>Warning: For translation unit ..., lookup of file ... resolved +to ... whose realpath is ...</B> +</DT> +<DD> +This warning occurs with --path_observation_re when a new realpath +matching a source or header file is observed.<BR> +<P> +<P> +</DD> +</DL> +<H2> +<A NAME=TOC_6> +DISTCC DISCREPANCY SYMPTOMS</A> +</H2> +<BR> +<P> +<P> +The interactions between the build system, distcc, and the include +server is somewhat complex. When a distcc commands receives a +failing compilation from the remote server it retries the compilation +locally. This section discusses the causes of discrepancies between +remote and local compilation. These are flagged by the demotion +message:<BR> +<P> +<B>__________Warning: ... pump-mode compilation(s) failed on server,</B> +<B>but succeeded locally.</B><BR> +<B>__________Distcc-pump was demoted to plain mode.</B> <B> See +the Distcc Discrepancy Symptoms section in the include_server(1) +man</B> <B> page.</B><BR> +<P> +The pump script issues this message at the end of the build. This +means that for at least one distcc invocation a local compilation +succeeded after the remote compilation failed. Each distcc invocation +for which such a discrepancy occurred in turn also issues a message +such as:<BR> +<P> +<B>Warning: remote compilation of '...' failed,</B> <B> retried +locally and got a different result.</B><BR> +<P> +The demotion makes subsequent distcc invocations use plain distcc +mode. Thus preprocessing will take place on the local machine +for the remainder of the build. This technique prevents very +slow builds where all compilations end up on the local machine +after failing remotely.<BR> +<P> +Of course, if the local compilations fails after the remote failures, +then the distcc invocation exits with the non-zero status of the +local compilation. The error messages printed are also those of +the local compilation.<BR> +<P> +The fallback behavior for distcc-pump mode to local compilation +can be disabled by setting the environment variable DISTCC_FALLBACK +to 1, which makes the distcc command fail as soon as the remote +compilation has failed. This setting is very useful for debugging +why the remote compilation went wrong, because now the output +from the server will be printed.<BR> +<P> +Next we discuss the possible causes of discrepancies.<BR> +<P> +<DL> +<DT> +<B>The user changed a source or header file during the build.</B> +</DT> +<DD> +This yields inconsistent results of course.<BR> +<P> +</DD> +<DT> +<B>A source or header file changed during the build.</B> +</DT> +<DD> +The build system rewrites a file. For Linux kernel 2.6, this +happens for 'include/linux/compile.h' and 'include/asm/asm-offsets.h'. +This condition is fixed by letting the include server know that +it must reset its caches when a stat of any of the files changes. +Practically, this is done by gathering the files in a colon-separated +list and then setting the INCLUDE_SERVER_ARGS command as in:<BR> +<P> +include_server_args="--stat_reset_triggers=include/linux/compile.h:include/asm/asm-offsets.h"<BR> +<P> +</DD> +<DT> +<B>A header file is potentially included, but does not exist, +and is then later included.</B> +</DT> +<DD> +<BR> +<P> +This occurs when some header foo.h includes another header file +trick.h, but the trick.h file has not yet been generated and the +inclusion is actually ignored because of preprocessing directives. +The include server will probe for the existence of trick.h, because +it overapproximates all possible ways directives actually evaluate. +The file trick.h is determined not to exist. If it is later generated, +and then really included, then the include server will falsely +believe that the file still does not exist. The solution to this +problem is to make the build system generate trick.h before the +first time any header file is included that makes a syntactic +reference to trick.h<BR> +<P> +</DD> +<DT> +<B>The include server was started with --unsafe_absolute_includes</B>.<B> +</DT> +<DD> +This is a problem if there are header files locally that do not +exist remotely and that are actually used. Such includes are +often protected by conditional directives that evaluate so that +are actually used on only specific and often uncommon platforms. +If you are not compiling for such a platform, then it may be correct +to use --unsafe_absolute_include</B>.<BR> +<P> +</DD> +<DT> +<B>The include server has calculated the wrong includes.</B> +</DT> +<DD> +We do not know of such a situation.<BR> +<P> +<P> +</DD> +</DL> +<H2> +<A NAME=TOC_7> +EXIT CODES</A> +</H2> +The exit code of include_server.py is usually 0. That the include +server has been started properly is communicated through the existence +of the pid_file.<BR> +<P> +<H2> +<A NAME=TOC_8> +ENVIRONMENT VARIABLES</A> +</H2> +<BR> +<P> +<B>DISTCC_EMAILLOG_WHOM_TO_BLAME</B> The email address to use +for include server automated emails. The default is 'distcc-pump-errors' +(which is an email address that probably will not exist in your +domain).<BR> +<P> +Additionally, the invocation of the compiler may use additional +environment variables.<BR> +<P> +<P> +<H2> +<A NAME=TOC_9> +BUGS</A> +</H2> +If you think you have found a distcc bug, please see the file +<I>reporting-bugs.txt</I> in the documentation directory for information +on how to report it.<BR> +<P> +<P> +In distcc-pump mode, the include server is unable to handle certain +very complicated computed includes as found in parts of the Boost +library. The include server will time out and distcc will revert +to plain mode.<BR> +<P> +Other known bugs may be documented on <I> http://code.google.com/p/distcc/</I><BR> +<P> +<H2> +<A NAME=TOC_10> +AUTHOR</A> +</H2> +The include server was written by Nils Klarlund, with assistance +from Fergus Henderson, Manos Renieris, and Craig Silverstein. +Please report bugs to <distcc@lists.samba.org>.<BR> +<P> +<H2> +<A NAME=TOC_11> +LICENCE</A> +</H2> +You are free to use distcc. distcc (including this manual) may +be copied, modified or distributed only under the terms of the +GNU General Public Licence version 2 or later. distcc comes with +absolutely no warrany. A copy of the GPL is included in the file +COPYING.<BR> +<P> +<H2> +<A NAME=TOC_12> +SEE ALSO</A> +</H2> +<B>distcc</B>(1), <B>distccd</B>(1), <B>include_server</B>(1), +and <B>gcc</B>(1). http://code.google.com/p/distcc/ http://ccache.samba.org/<BR> +</BODY> +</HTML> diff --git a/packaging/RedHat/rpm.spec b/packaging/RedHat/rpm.spec index 457a232..e39e4db 100644 --- a/packaging/RedHat/rpm.spec +++ b/packaging/RedHat/rpm.spec @@ -82,6 +82,7 @@ rm -rf $RPM_BUILD_ROOT %doc %{_mandir}/man1/distcc.1.gz %doc %{_mandir}/man1/distccmon-text.1.gz %doc %{_mandir}/man1/pump.1.gz +%doc %{_mandir}/man1/include_server.1.gz %doc %{_docdir} |