diff options
Diffstat (limited to 'doc/python10/design.sgml')
| -rw-r--r-- | doc/python10/design.sgml | 898 |
1 files changed, 898 insertions, 0 deletions
diff --git a/doc/python10/design.sgml b/doc/python10/design.sgml new file mode 100644 index 00000000..cb58af97 --- /dev/null +++ b/doc/python10/design.sgml @@ -0,0 +1,898 @@ +<para> + + The &SCons; architecture consists of three layers: + +</para> + +<mediaobject> + <imageobject> + <imagedata fileref="arch" format="eps" align="center"> + </imageobject> + <imageobject> + <imagedata fileref="arch.jpg" format="jpg" align="center"> + </imageobject> + <!-- PDF files? + <imageobject> + <imagedata fileref="arch.pdf" align="center"> + </imageobject> + --> +</mediaobject> + +<itemizedlist> + + <listitem> + <para> + + The &SCons; <emphasis>Build Engine</emphasis>, a package of Python + modules that handle dependency management and updating out-of-date + objects. + + </para> + </listitem> + + <listitem> + <para> + + The &SCons; <emphasis>API</emphasis> (applications programming + interface) between the Build Engine + and the user interface. + + </para> + </listitem> + + <listitem> + <para> + + The &scons; <emphasis>script</emphasis> itself (note lower case + <emphasis>sc</emphasis>), which is the pre-provided interface to + the Build Engine. + + </para> + </listitem> + +</itemizedlist> + +<para> + + Notice that this architecture separates the internal workings of + &SCons; (the Build Engine) from the + external user interface. The benefit is that the &SCons; Build Engine + can be imported into any other software package written in Python + to support a variety of user interfaces—or, to look at it + in reverse, other software interfaces can use the &SCons; Build + Engine to manage dependencies between their objects. + +</para> + +<para> + + Because the + &SCons; package itself is modular, only those parts of the package + relevant to the embedding interface need be imported; for example, + a utility that wants to use only file timestamps for checking + whether a file is up-to-date + need not import the MD5 signature module. + +</para> + +<section> + <title>The &SCons; Build Engine</title> + + <para> + + The Build Engine is a package of Python modules that + form the heart of &SCons;. + + The Build Engine can be broadly divided into five + architectural subsystems, each responsible + for a crucial part of &SCons; functionality: + + </para> + + <itemizedlist> + + <listitem> + <para> + + A <emphasis>node</emphasis> subsystem, responsible for managing + the files (or other objects) to be built, and the dependency + relationships between them. + + </para> + </listitem> + + <listitem> + <para> + + A <emphasis>scanner</emphasis> subsystem, responsible for + scanning various file types for implicit dependencies. + + </para> + </listitem> + + <listitem> + <para> + + A <emphasis>signature</emphasis> subsystem, responsible for + deciding whether a given file (or other object) requires + rebuilding. + + </para> + </listitem> + + <listitem> + <para> + + A <emphasis>builder</emphasis> subsystem, responsible for + actually executing the necessary command or function to + build a file (or other object). + + </para> + </listitem> + + <listitem> + <para> + + A <emphasis>job/task</emphasis> subsystem, responsible for + handling parallelization of builds. + + </para> + </listitem> + + </itemizedlist> + + <para> + + The rest of this section will provide a high-level overview of the + class structure of each of these Build Engine subsystems. + + </para> + + <section> + <title>Node Subsystem</title> + + <para> + + The node subsystem of the Build Engine is + responsible for managing the knowledge in &SCons; of + the relationships among the external objects + (files) it is responsible for updating. + The most important of these relationships is + the dependency relationship between various &Node; objects, + which &SCons; uses to determine the order + in which builds should be performed. + + </para> + + <mediaobject> + <imageobject> + <imagedata fileref="node" format="eps" align="center"> + </imageobject> + <imageobject> + <imagedata fileref="node.jpg" format="jpg" align="center"> + </imageobject> + <!-- PDF files? + <imageobject> + <imagedata fileref="node.pdf" align="center"> + </imageobject> + --> + </mediaobject> + + <para> + + The &scons; script (or other + user interface) + tells the Build Engine + about dependencies + through its &consenv; API. + The Build Engine also discovers + dependencies automatically through the use of &Scanner; objects. + + </para> + + <para> + + Subclasses of the &Node; class maintain additional + relationships that reflect the real-world + existence of these objects. + For example, the &Node_FS; subclass + is responsible for managing a + representation of the directory hierarchy + of a file system. + + </para> + + <para> + + A &Walker; class is used by other subsystems + to walk the dependency tree maintained by the &Node; class. + The &Walker; class maintains a stack of &Node; objects + visited during its depth-first traversal of the + dependency tree, + and uses an intermediate node &Wrapper; class + to maintain state information about a + &Node; object's dependencies. + + </para> + + </section> + + <section> + <title>Scanner Subsystem</title> + + <para> + + The scanner subsystem is responsible for maintaining + objects that can scan the contents of a &Node;'s + for implicit dependencies. + + </para> + + <mediaobject> + <imageobject> + <imagedata fileref="scanner" format="eps" align="center"> + </imageobject> + <imageobject> + <imagedata fileref="scanner.jpg" format="jpg" align="center"> + </imageobject> + <!-- PDF files? + <imageobject> + <imagedata fileref="scanner.pdf" align="center"> + </imageobject> + --> + </mediaobject> + + <para> + + In practice, a given &Scanner; subclass object + functions as a prototype, + returning clones of itself + depending on the &consenv; + values governing how the &Node; + should be scanned. + + </para> + + </section> + + <section> + <title>Signature Subsystem</title> + + <para> + + The signature subsystem is responsible for computing + signature information for &Node; objects. + The signature subsystem in &SCons; + supports multiple ways to + determine whether a &Node is up-to-date + by using an abstract &Sig; class + as a strategy wrapper: + + </para> + + <mediaobject> + <imageobject> + <imagedata fileref="sig" format="eps" align="center"> + </imageobject> + <imageobject> + <imagedata fileref="sig.jpg" format="jpg" align="center"> + </imageobject> + <!-- PDF files? + <imageobject> + <imagedata fileref="sig.pdf" align="center"> + </imageobject> + --> + </mediaobject> + + <para> + + By default, &SCons; tracks dependencies by computing and + maintaining MD5 signatures for the contents of each source file + (or other object). The signature of a <emphasis>derived</emphasis> + file consists of the aggregate of the signatures of all the source + files <emphasis>plus</emphasis> the command-line string used to + build the file. These signatures are stored in a &sconsign; file + in each directory. + + </para> + + <para> + + If the contents of any of the source files changes, the change to its + MD5 signature is propogated to the signature of the derived file(s). The + simple fact that the new signature does not match the stored signature + indicates that the derived file is not up to date and must be rebuilt. + + </para> + + <para> + + A separate &TimeStamp; subclass of the &Sig; class supports + the use of traditional file timestamps for + deciding whether files are up-to-date. + + </para> + + </section> + + <section> + <title>Builder Subsystem</title> + + <para> + + The &SCons; Build Engine records how out-of-date files + (or other objects) should be rebuilt in &Builder; objects, + maintained by the builder subsystem: + + </para> + + <mediaobject> + <imageobject> + <imagedata fileref="builder" format="eps" align="center"> + </imageobject> + <imageobject> + <imagedata fileref="builder.jpg" format="jpg" align="center"> + </imageobject> + <!-- PDF files? + <imageobject> + <imagedata fileref="builder.pdf" align="center"> + </imageobject> + --> + </mediaobject> + + <para> + + The actual underlying class name is &BuilderBase;, + and there are subclasses that can encapsulate + multiple &Builder; objects for special purposes. + One subclass + (&CompositeBuilder;) + selects an appropriate encapsulated &Builder; + based on the file suffix of the target object. + The other + (&MultiStepBuilder;). + can chain together multiple + &Builder; objects, + for example, + to build an executable program from a source file + through an implicit intermediate object file. + + </para> + + <para> + + A &BuilderBase; object has an associated + &ActionBase; object + responsible for actually executing + the appropriate steps + to update the target file. + There are three subclasses, + one for externally executable commands + (&CommandAction;), + one for Python functions + (&FunctionAction;), + and one for lists of + multiple &Action; objects + (&ListAction;). + + </para> + + </section> + + <section> + <title>Job/Task Subsystem</title> + + <para> + + &SCons; supports parallel builds with a thread-based tasking + model, managed by the job/task subsystem. + + </para> + + <mediaobject> + <imageobject> + <imagedata fileref="job-task" format="eps" align="center"> + </imageobject> + <imageobject> + <imagedata fileref="job-task.jpg" format="jpg" align="center"> + </imageobject> + <!-- PDF files? + <imageobject> + <imagedata fileref="job-task.pdf" align="center"> + </imageobject> + --> + </mediaobject> + + <para> + + Instead of performing an outer-loop recursive descent + of the dependency tree and then forking a task when it finds a + file that needs updating, &SCons; starts as many threads as are + requested, each thread managed by the &Jobs; class. + As a performance optimization, + the &Jobs; class maintains an internal + distinction between + &Serial; and &Parallel; + build jobs, + so that serial builds + don't pay any performance penalty + by using a multi-threaded implementation + written for &Parallel; builds. + + </para> + + <para> + + Each &Jobs; object, running in its own thread, + then requests a &Task; from a central &Taskmaster;, + which is responsible + for handing out available &Task; objects for (re-)building + out-of-date nodes. A condition variable + makes sure that the &Jobs; objects + query the &Taskmaster; one at a time. + + </para> + + <para> + + The &Taskmaster uses the node subsystem's + &Walker; class to walk the dependency tree, + and the &Sig; class to use the + appropriate method + of deciding if a &Node; is up-to-date. + + </para> + + <para> + + This scheme has many advantages over the standard &Make; + implementation of <option>-j</option>. + Effective use of <option>-j</option> is difficult + with the usual recursive use of Make, + because the number of jobs started by <option>-j</option> multiply + at each level of the source tree. + This makes the actual number of jobs + executed at any moment very dependent on the size and layout of + the tree. &SCons;, in contrast, starts only as many jobs as are + requested, and keeps them constantly busy (excepting jobs that + block waiting for their dependency files to finish building). + + </para> + + </section> + +</section> + +<section> + <title>The &SCons; API</title> + + <para> + + This section provides an overview of the &SCons; interface. The + complete interface specification is both more detailed and flexible + than this overview. + + </para> + + <section> + <title>&ConsVars;</title> + + <para> + + In &SCons;, a &consenv; is an object through which an external + interface (such as the &scons; script) communicates dependency + information to the &SCons; Build Engine. + + </para> + + <para> + + A construction environment is implemented as a dictionary + containing: + + </para> + + <itemizedlist> + + <listitem> + <para> + + construction variables, string values that are substituted + into command lines or used by builder functions; + + </para> + </listitem> + + <listitem> + <para> + + one or more &Builder; objects that can be invoked to update a + file or other object; + + </para> + </listitem> + + <listitem> + <para> + + one or more &Scanner; objects that can be used to + scan a file automatically for dependencies (such as + files specified on <literal>#include</literal> lines). + + </para> + </listitem> + + </itemizedlist> + + <para> + + &Consenvs; are instantiated as follows: + + </para> + + <programlisting> + env = Environment() + env_debug = Environment(CCFLAGS = '-g') + </programlisting> + + </section> + + <section> + <title>&Builder; Objects</title> + + <para> + + An &SCons; &Builder; object encapsulates information about how to + build a specific type of file: an executable program, an object + file, a library, etc. A &Builder; object is associated with a + file through an associated &consenv; method and later invoked to + actually build the file. The &Builder; object will typically use + construction variables (such as &CCFLAGS;, &LIBPATH;) to influence + the specific build execution. + + </para> + + <para> + + &Builder; objects are instantiated as follows: + + </para> + + <programlisting> + bld = Builder(name = 'Program', action = "$CC -o $TARGET $SOURCES") + </programlisting> + + <para> + + In the above example, the <literal>action</literal> is a + command-line string in which the Build Engine will + interpolate the values of construction + variables before execution. The actual + <literal>action</literal> specified, though, + may be a function: + + </para> + + <programlisting> + def update(dest): + # [code to update the object] + return 0 + + bld = Builder(name = 'Program', function = update) + </programlisting> + + <para> + + Or a callable Python object (or class): + + </para> + + <programlisting> + class class_a: + def __call__(self, kw): + # build the desired object + return 0 + + builder = SCons.Builder.Builder(action = class_a()) + </programlisting> + + <para> + + A &Builder; object may have the <literal>prefix</literal> and + <literal>suffix</literal> of its target file type specified + as keyword arguments at instantiation. Additionally, the + suffix of the <emphasis>source files</emphasis> used by this + &Builder; to build its target files may be specified using the + <literal>src_suffix</literal> keyword argument: + + </para> + + <programlisting> + bld_lib = Builder(name = 'Library', action = "$AR r $TARGET $SOURCES", + prefix = 'lib', suffix = '.a', src_suffix = '.o') + </programlisting> + + <para> + + The specified <literal>prefix</literal> and + <literal>suffix</literal> will be appended to the name of any + target file built by this &Builder; object, if they are not + already part of the file name. The <literal>src_suffix</literal> + is used by the &SCons; Build Engine to chain together + multiple &Builder; objects to create, + for example, a library from the original source + files without having to specify the + intermediate <literal>.o</literal> files. + + </para> + + <para> + + &Builder; objects are associated with a &consenv; through a + &consvar; named &BUILDERS;, a list of the &Builder objects that + will be available for execution through the &consenv: + + </para> + + <programlisting> + env = Environment(BUILDERS = [ Object, Library, WebPage, Program ]) + </programlisting> + + </section> + + <section> + <title>&Scanner; Objects</title> + + <para> + + &Scanner; objects perform automatic checking for dependencies + by scanning the contents of files. The canonical + example is scanning a C source file or header file for + files specified on <literal>#include</literal> lines. + + </para> + + <para> + + A &Scanner; object is instantiated as follows: + + </para> + + <programlisting> + def c_scan(contents): + # scan contents of file + return # list of files found + + c_scanner = Scanner(name = 'CScan', function = c_scan, + argument = None, + skeys = ['.c', '.C', '.h', '.H') + </programlisting> + + <para> + + The <literal>skeys</literal> argument specifies a list of file + suffixes for file types that this &Scanner; knows how to scan. + + </para> + + <para> + + &Scanner; objects are associated with a &consenv; through a + &consvar; named &SCANNERS;, a list of the &Scanner; objects that + will be available through the &consenv: + + </para> + + <programlisting> + env = Environment(SCANNERS = [ CScan, M4Scan ]) + </programlisting> + + <para> + + For utilities that will build files with a variety of file + suffixes, or which require unusual scanning rules, a &Scanner; + object may be associated explicitly with a &Builder; object as + follows: + + </para> + + <programlisting> + def tool_scan(contents): + # scan contents of file + return # list of files found + + tool_scanner = Scanner(name = 'TScan', function = tool_scan) + + bld = Builder(name = 'Tool', scanner = tool_scanner) + </programlisting> + + </section> + + <section> + <title>&BuildDir;</title> + + <para> + + &SCons; supports a flexible mechanism for building target + files in a separate build directory from the source files. + The &BuildDir; syntax is straightforward: + + </para> + + <programlisting> + BuildDir(source = 'src', build = 'bld') + </programlisting> + + <para> + + By + default, source files are linked or copied into the build + directory, because exactly replicating the source directory + is sometimes necessary for certain combinations of use of + <literal>#include "..."</literal> and <option>-I</option> search + paths. + + An option exists to specify that only output files should be placed in + the build directory: + + </para> + + <programlisting> + BuildDir(source = 'src', build = 'bld', no_sources = 1) + </programlisting> + + </section> + + <section> + <title>&Repository;</title> + + <para> + + &SCons; supports the ability to search a list of code repositories + for source files and derived files. This works much like + &Make;'s <varname>VPATH</varname> feature, as implemented in + recent versions of GNU &Make;. + (The POSIX standard for &Make; specifies slightly + different behavior for <varname>VPATH</varname>.) + The syntax is: + + </para> + + <programlisting> + Repository('/home/source/1.1', '/home/source/1.0') + </programlisting> + + <para> + + A command-line <option>-Y</option> option exists to allow + repositories to be specified on the command line, or in the + &SCONSFLAGS; environment variable (not construction variable!). + This avoids a chicken-and-egg situation and allows the top-level + &SConstruct; file to be found in a repository as well. + + </para> + + </section> + + <section> + <title>&Cache;</title> + + <para> + + &SCons; supports a way for developers to share derived files. Again, the + syntax is straightforward: + + </para> + + <programlisting> + Cache('/var/build.cache/i386') + </programlisting> + + <para> + + Copies of any derived files built will be placed in the specified + directory with their MD5 signature. If another build results in an + out-of-date derived file with the same signature, the derived file + will be copied from the cache instead of being rebuilt. + + </para> + + </section> + +</section> + +<section> + <title>The &scons; Script</title> + + <para> + + The &scons; script provides an interface + that looks roughly equivalent to the + classic &Make; utility—that is, execution from the command + line, and dependency information read from configuration files. + + </para> + + <para> + + The most noticeable difference between &scons; and &Make;, or most + other build tools, is that the configuration files are actually + Python scripts, generically called "SConscripts" (although the + top-level "Makefile" is named &SConstruct). Users do not have to + learn a new language syntax, but instead configure dependency + information by making direct calls to the Python API of the + &SCons; Build Engine. Here is an example &SConstruct file which + builds a program in side-by-side normal and debug versions: + + </para> + + <programlisting> + env = Environment() + debug = env.Copy(CCFLAGS = '-g') + + source_files = ['f1.c', 'f2.c', 'f3.c'] + + env.Program(target = 'foo', sources = source_files) + debug.Program(target = 'foo-debug', sources = source_files) + </programlisting> + + <para> + + Notice the fact that this file is a Python script, which allows us + to define and re-use an array that lists the source files. + + </para> + + <para> + + Because quoting individul strings in long + lists of files can get tedious and error-prone, the &SCons; + methods support a short-cut of listing multiple files in a single + string, separated by white space. + This would change + the assignment in the above example to a more easily-readable: + + </para> + + <programlisting> + source_files = 'f1.c f2.c f3.c' + </programlisting> + + <para> + + The mechanism to establish hierarchical builds is to "include" any + subsidiary configuration files in the build by listing them explicitly + in a call to the &SConscript; function: + + </para> + + <programlisting> + SConscript('src/SConscript', 'lib/SConscript') + </programlisting> + + <para> + + By convention, configuration files in subdirectories are named + &SConscript;. + + </para> + + <para> + + The &scons; script has intentionally been made to look, from + the outside, as much like &Make; as is practical. To this + end, the &scons; script supports all of the same command-line + options supported by GNU &Make;: <option>-f</option> FILE, + <option>-j</option>, <option>-k</option>, <option>-s</option>, + etc. For compatibility, &scons; ignores those GNU &Make; options + that don't make sense for the &SCons; architecture, such as + <option>-b</option>, <option>-m</option>, <option>-S</option>, + and <option>-t</option>. The + intention is that, given an equivalent &SConstruct; file for a + &Makefile;, a user could use &SCons; as a drop-in replacement for + &Make;. Additional command-line options are, where possible, taken + from the Perl &Cons; utility on which the &SCons; design is based. + + </para> + +</section> |