Build Engine API

General Principles

Keyword arguments All methods and functions in this API will support the use of keyword arguments in calls, for the sake of explicitness and readability. For brevity in the hands of experts, most methods and functions will also support positional arguments for their most-commonly-used arguments. As an explicit example, the following two lines will each arrange for an executable program named foo (or foo.exe on a Win32 system) to be compiled from the foo.c source file: env.Program(target = 'foo', source = 'foo.c') env.Program('foo', 'foo.c')

Internal object representation All methods and functions use internal (Python) objects that represent the external objects (files, for example) for which they perform dependency analysis. All methods and functions in this API that accept an external object as an argument will accept either a string description or an object reference. For example, the two following two-line examples are equivalent: env.Object(target = 'foo.o', source = 'foo.c') env.Program(target = 'foo', 'foo.o') # builds foo from foo.o foo_obj = env.Object(target = 'foo.o', source = 'foo.c') env.Program(target = 'foo', foo_obj) # builds foo from foo.o

&ConsEnvs; A &consenv; is the basic means by which a software system interacts with the &SCons; Python API to control a build process. A &consenv; is an object with associated methods for generating target files of various types (&Builder; objects), other associated object methods for automatically determining dependencies from the contents of various types of source files (&Scanner; objects), and a dictionary of values used by these methods. Passing no arguments to the &Environment; instantiation creates a &consenv; with default values for the current platform: env = Environment()

&Consvars; A &consenv; has an associated dictionary of &consvars; that control how the build is performed. By default, the &Environment; method creates a &consenv; with values that make most software build "out of the box" on the host system. These default values will be generated at the time &SCons; is installed using functionality similar to that provided by GNU &Autoconf;. It would be nice if we could avoid re-inventing the wheel here by using some other Python-based tool &Autoconf; replacement--like what was supposed to come out of the Software Carpentry configuration tool contest. It will probably be most efficient to roll our own logic initially and convert if something better does come along. At a minimum, there will be pre-configured sets of default values that will provide reasonable defaults for UNIX and Windows NT. The default &consenv; values may be overridden when a new &consenv; is created by specifying keyword arguments: env = Environment(CC = 'gcc', CCFLAGS = '-g', CPPPATH = ['.', 'src', '/usr/include'], LIBPATH = ['/usr/lib', '.'])

Fetching &consvars; A copy of the dictionary of &consvars; can be returned using the &Dictionary; method: env = Environment() dict = env.Dictionary() If any arguments are supplied, then just the corresponding value(s) are returned: ccflags = env.Dictionary('CCFLAGS') cc, ld = env.Dictionary('CC', 'LD')

Copying a &consenv; A method exists to return a copy of an existing environment, with any overridden values specified as keyword arguments to the method: env = Environment() debug = env.Copy(CCFLAGS = '-g')

Multiple &consenvs; Different external objects often require different build characteristics. Multiple &consenvs; may be defined, each with different values: env = Environment(CCFLAGS = '') debug = Environment(CCFLAGS = '-g') env.Make(target = 'hello', source = 'hello.c') debug.Make(target = 'hello-debug', source = 'hello.c') Dictionaries of values from multiple &consenvs; may be passed to the &Environment; instantiation or the &Copy; method, in which case the last-specified dictionary value wins: env1 = Environment(CCFLAGS = '-O', LDFLAGS = '-d') env2 = Environment(CCFLAGS = '-g') new = Environment(env1.Dictionary(), env2.Dictionary()) The new environment in the above example retains LDFLAGS = '-d' from the env1 environment, and CCFLAGS = '-g' from the env2 environment.

Variable substitution Within a construction command, any variable from the &consenv; may be interpolated by prefixing the name of the construction with $: MyBuilder = Builder(command = "$XX $XXFLAGS -c $_INPUTS -o $target") env.Command(targets = 'bar.out', sources = 'bar.in', command = "sed '1d' < $source > $target") Variable substitution is recursive: the command line is expanded until no more substitutions can be made. Variable names following the $ may be enclosed in braces. This can be used to concatenate an interpolated value with an alphanumeric character: VerboseBuilder = Builder(command = "$XX -${XXFLAGS}v > $target") The variable within braces may contain a pair of parentheses after a Python function name to be evaluated (for example, ${map()}). &SCons; will interpolate the return value from the function (presumably a string): env = Environment(FUNC = myfunc) env.Command(target = 'foo.out', source = 'foo.in', command = "${FUNC($<)}") If a referenced variable is not defined in the &consenv;, the null string is interpolated. The following special variables can also be used: $targets All target file names. If multiple targets are specified in an array, $targets expands to the entire list of targets, separated by a single space. Individual targets from a list may be extracted by enclosing the targets keyword in braces and using the appropriate Python array index or slice: ${targets[0]} # expands to the first target ${targets[1:]} # expands to all but the first target ${targets[1:-1]} # expands to all but the first and last targets $target A synonym for ${targets[0]}, the first target specified. $sources All input file names. Any input file names that are used anywhere else on the current command line (via ${sources[0]}, ${sources{[1]}, etc.) are removed from the expanded list. Any of the above special variables may be enclosed in braces and followed immediately by one of the following attributes to select just a portion of the expanded path name: .base Basename: the directory plus the file name, minus any file suffix. .dir The directory in which the file lives. This is a relative path, where appropriate. .file The file name, minus any directory portion. .suffix The file name suffix (that is, the right-most dot in the file name, and all characters to the right of that). .filebase The file name (no directory portion), minus any file suffix. .abspath The absolute path to the file.

&Builder; Objects By default, &SCons; supplies (and uses) a number of pre-defined &Builder; objects: &Object; compile or assemble an object file &Library; archive files into a library &SharedLibrary; archive files into a shared library &Program; link objects and/or libraries into an executable &MakeBuilder; build according to file suffixes; see below A &consenv; can be explicitly initialized with associated &Builder; objects that will be bound to the &consenv; object: env = Environment(BUILDERS = ['Object', 'Program']) &Builder; objects bound to a &consenv; can be called directly as methods. When invoked, a &Builder; object returns a (list of) objects that it will build: obj = env.Object(target ='hello.o', source = 'hello.c') lib = env.Library(target ='libfoo.a', source = ['aaa.c', 'bbb.c']) slib = env.SharedLibrary(target ='libbar.so', source = ['xxx.c', 'yyy.c']) prog = env.Program(target ='hello', source = ['hello.o', 'libfoo.a', 'libbar.so'])

Specifying multiple inputs Multiple input files that go into creating a target file may be passed in as a single string, with the individual file names separated by white space: env.Library(target = 'foo.a', source = 'aaa.c bbb.c ccc.c') env.Object(target = 'yyy.o', source = 'yyy.c') env.Program(target = 'bar', source = 'xxx.c yyy.o foo.a') Alternatively, multiple input files that go into creating a target file may be passed in as an array. This allows input files to be specified using their object representation: env.Library(target = 'foo.a', source = ['aaa.c', 'bbb.c', 'ccc.c']) yyy_obj = env.Object(target = 'yyy.o', source = 'yyy.c') env.Program(target = 'bar', source = ['xxx.c', yyy_obj, 'foo.a']) Individual string elements within an array of input files are not further split into white-space separated file names. This allows file names that contain white space to be specified by putting the value into an array: env.Program(target = 'foo', source = ['an input file.c'])

Specifying multiple targets Conversely, the generated target may be a string listing multiple files separated by white space: env.Object(target = 'grammar.o y.tab.h', source = 'grammar.y') An array of multiple target files can be used to mix string and object representations, or to accomodate file names that contain white space: env.Program(target = ['my program'], source = 'input.c')

File prefixes and suffixes For portability, if the target file name does not already have an appropriate file prefix or suffix, the &Builder; objects will append one appropriate for the file type on the current system: # builds 'hello.o' on UNIX, 'hello.obj' on Windows NT: obj = env.Object(target ='hello', source = 'hello.c') # builds 'libfoo.a' on UNIX, 'foo.lib' on Windows NT: lib = env.Library(target ='foo', source = ['aaa.c', 'bbb.c']) # builds 'libbar.so' on UNIX, 'bar.dll' on Windows NT: slib = env.SharedLibrary(target ='bar', source = ['xxx.c', 'yyy.c']) # builds 'hello' on UNIX, 'hello.exe' on Windows NT: prog = env.Program(target ='hello', source = ['hello.o', 'libfoo.a', 'libbar.so'])

&Builder; object exceptions &Builder; objects raise the following exceptions on error:

User-defined &Builder; objects Users can define additional &Builder; objects for specific external object types unknown to &SCons;. A &Builder; object may build its target by executing an external command: WebPage = Builder(command = 'htmlgen $HTMLGENFLAGS $sources > $target', suffix = '.html', src_suffix = '.in') Alternatively, a &Builder; object may also build its target by executing a Python function: def update(dest): # [code to update the object] return 1 OtherBuilder1 = Builder(function = update, src_suffix = ['.in', '.input']) An optional argument to pass to the function may be specified: def update_arg(dest, arg): # [code to update the object] return 1 OtherBuilder2 = Builder(function = update_arg, function_arg = 'xyzzy', src_suffix = ['.in', '.input']) Both an external command and an internal function may be specified, in which case the function will be called to build the object first, followed by the command line. User-defined &Builder; objects can be used like the default &Builder; objects to initialize &consenvs;. WebPage = Builder(command = 'htmlgen $HTMLGENFLAGS $sources > $target', suffix = '.html', src_suffix = '.in') env = Environment(BUILDERS = ['WebPage']) env.WebPage(target = 'foo.html', source = 'foo.in') # Builds 'bar.html' on UNIX, 'bar.htm' on Windows NT: env.WebPage(target = 'bar', source = 'bar.in') The command-line specification can interpolate variables from the &consenv;; see "Variable substitution," above. A &Builder; object may optionally be initialized with a list of: the prefix of the target file (e.g., 'lib' for libraries) the suffix of the target file (e.g., '.a' for libraries) the expected suffixes of the input files (e.g., '.o' for object files) These arguments are used in automatic dependency analysis and to generate output file names that don't have suffixes supplied explicitly.

Copying &Builder; Objects A &Copy; method exists to return a copy of an existing &Builder; object, with any overridden values specified as keyword arguments to the method: build = Builder(function = my_build) build_out = build.Copy(suffix = '.out') Typically, &Builder; objects will be supplied by a tool-master or administrator through a shared &consenv;.

Special-purpose build rules A pre-defined &Command; builder exists to associate a target file with a specific command or list of commands for building the file: env.Command(target = 'foo.out', source = command = 'foo.in', "foo.process $sources > $target") commands = [ "bar.process -o .tmpfile $sources", "mv .tmpfile $target" ] env.Command(target = 'bar.out', source = 'bar.in', command = commands) This is useful when it's too cumbersome to create a &Builder; object just to build a single file in a special way.

The &MakeBuilder; &Builder; A pre-defined &Builder; object named &MakeBuilder; exists to make simple builds as easy as possible for users, at the expense of sacrificing some build portability. The following minimal example builds the 'hello' program from the 'hello.c' source file: Environment().Make('hello', 'hello.c') Users of the &MakeBuilder; &Builder; object are not required to understand intermediate steps involved in generating a file--for example, the distinction between compiling source code into an object file, and then linking object files into an executable. The details of intermediate steps are handled by the invoked method. Users that need to, however, can specify intermediate steps explicitly: env = Environment() env.Make(target = 'hello.o', source = 'hello.c') env.Make(target = 'hello', source = 'hello.o') The &MakeBuilder; method understands the file suffixes specified and "does the right thing" to generate the target object and program files, respectively. It does this by examining the specified output suffixes for the &Builder; objects bound to the environment. Because file name suffixes in the target and source file names must be specified, the &MakeBuilder; method can't be used portably across operating systems. In other words, for the example above, the &MakeBuilder; builder will not generate hello.exe on Windows NT.

&Builder; maps The env.Make method "does the right thing" to build different file types because it uses a dictionary from the &consenv; that maps file suffixes to the appropriate &Builder; object. This &BUILDERMAP; can be initialized at instantiation: env = Environment(BUILDERMAP = { '.o' : Object, '.a' : Library, '.html' : WebPage, '' : Program, }) With the &BUILDERMAP; properly initialized, the env.Make method can be used to build additional file types: env.Make(target = 'index.html', source = 'index.input') &Builder; objects referenced in the &BUILDERMAP; do not need to be listed separately in the BUILDERS variable. The &consenv; will bind the union of the &Builder; objects listed in both variables.

Dependencies

Automatic dependencies By default, &SCons; assumes that a target file has automatic dependencies on the:

tool used to build the target file contents of the input files command line used to build the target file

If any of these changes, the target file will be rebuilt.

Implicit dependencies Additionally, &SCons; can scan the contents of files for implicit dependencies on other files. For example, &SCons; will scan the contents of a .c file and determine that any object created from it is dependent on any .h files specified via #include. &SCons;, therefore, "does the right thing" without needing to have these dependencies listed explicitly: % cat Construct env = Environment() env.Program('hello', 'hello.c') % cat hello.c #include "hello_string.h" main() { printf("%s\n", STRING); } % cat > hello_string.h #define STRING "Hello, world!\n" % scons . gcc -c hello.c -o hello.o gcc -o hello hello.c % ./hello Hello, world! % cat > hello_string.h #define STRING "Hello, world, hello!\n" % scons . gcc -c hello.c -o hello.o gcc -o hello hello.c % ./hello Hello, world, hello! %

Ignoring dependencies Undesirable automatic dependencies or implicit dependencies may be ignored: env.Program(target = 'bar', source = 'bar.c') env.Ignore('bar', '/usr/bin/gcc', 'version.h') In the above example, the bar program will not be rebuilt if the /usr/bin/gcc compiler or the version.h file change.

Explicit dependencies Dependencies that are unknown to &SCons; may be specified explicitly in an &SCons; configuration file: env.Dependency(target = 'output1', dependency = 'input_1 input_2') env.Dependency(target = 'output2', dependency = ['input_1', 'input_2']) env.Dependency(target = 'output3', dependency = ['white space input']) env.Dependency(target = 'output_a output_b', dependency = 'input_3') env.Dependency(target = ['output_c', 'output_d'], dependency = 'input_4') env.Dependency(target = ['white space output'], dependency = 'input_5') Just like the target keyword argument, the dependency keyword argument may be specified as a string of white-space separated file names, or as an array. A dependency on an &SCons; configuration file itself may be specified explicitly to force a rebuild whenever the configuration file changes: env.Dependency(target = 'archive.tar.gz', dependency = 'SConstruct')

&Scanner; Objects Analagous to the previously-described &Builder; objects, &SCons; supplies (and uses) &Scanner; objects to search the contents of a file for implicit dependency files: CScan scan .{c,C,cc,cxx,cpp} files for #include dependencies A &consenv; can be explicitly initialized with associated &Scanner; objects: env = Environment(SCANNERS = ['CScan', 'M4Scan']) &Scanner; objects bound to a &consenv; can be associated directly with specified files: env.CScan('foo.c', 'bar.c') env.M4Scan('input.m4')

User-defined &Scanner; objects A user may define a &Scanner; object to scan a type of file for implicit dependencies: def scanner1(file_contents): # search for dependencies return dependency_list FirstScan = Scanner(function = scanner1) The scanner function must return a list of dependencies that its finds based on analyzing the file contents it is passed as an argument. The scanner function, when invoked, will be passed the calling environment. The scanner function can use &consenvs; from the passed environment to affect how it performs its dependency scan--the canonical example being to use some sort of search-path construction variable to look for dependency files in other directories: def scanner2(file_contents, env): path = env.{'SCANNERPATH'} # XXX # search for dependencies using 'path' return dependency_list SecondScan = Scanner(function = scanner2) The user may specify an additional argument when the &Scanner; object is created. When the scanner is invoked, the additional argument will be passed to the scanner funciton, which can be used in any way the scanner function sees fit: def scanner3(file_contents, env, arg): # skip 'arg' lines, then search for dependencies return dependency_list Skip_3_Lines_Scan = Scanner(function = scanner2, argument = 3) Skip_6_Lines_Scan = Scanner(function = scanner2, argument = 6)

Copying &Scanner; Objects A method exists to return a copy of an existing &Scanner; object, with any overridden values specified as keyword arguments to the method: scan = Scanner(function = my_scan) scan_path = scan.Copy(path = '%SCANNERPATH') Typically, &Scanner; objects will be supplied by a tool-master or administrator through a shared &consenv;.

&Scanner; maps Each &consenv; has a &SCANNERMAP;, a dictionary that associates different file suffixes with a scanner object that can be used to generate a list of dependencies from the contents of that file. This &SCANNERMAP; can be initialized at instantiation: env = Environment(SCANNERMAP = { '.c' : CScan, '.cc' : CScan, '.m4' : M4Scan, }) &Scanner; objects referenced in the &SCANNERMAP; do not need to be listed separately in the SCANNERS variable. The &consenv; will bind the union of the &Scanner; objects listed in both variables.

Targets The methods in the build engine API described so far merely establish associations that describe file dependencies, how a file should be scanned, etc. Since the real point is to actually build files, &SCons; also has methods that actually direct the build engine to build, or otherwise manipulate, target files.

Building targets One or more targets may be built as follows: env.Build(target = ['foo', 'bar']) Note that specifying a directory (or other collective object) will cause all subsidiary/dependent objects to be built as well: env.Build(target = '.') env.Build(target = 'builddir') By default, &SCons; explicitly removes a target file before invoking the underlying function or command(s) to build it.

Removing targets A "cleanup" operation of removing generated (target) files is performed as follows: env.Clean(target = ['foo', 'bar']) Like the &Build; method, the &Clean; method may be passed a directory or other collective object, in which case the subsidiary target objects under the directory will be removed: env.Clean(target = '.') env.Clean(target = 'builddir') (The directories themselves are not removed.)

Suppressing cleanup removal of build-targets By default, &SCons; explicitly removes all build-targets when invoked to perform "cleanup". Files that should not be removed during "cleanup" can be specified via the &NoClean; method: env.Library(target = 'libfoo.a', source = ['aaa.c', 'bbb.c', 'ccc.c']) env.NoClean('libfoo.a') The NoClean operation has precedence over the Clean operation. A target that is specified as both Clean and NoClean, will not be removed during a clean. In the following example, target 'foo' will not be removed during "cleanup": env.Clean(target = 'foo') env.NoClean('foo')

Suppressing build-target removal As mentioned, by default, &SCons; explicitly removes a target file before invoking the underlying function or command(s) to build it. Files that should not be removed before rebuilding can be specified via the &Precious; method: env.Library(target = 'libfoo.a', source = ['aaa.c', 'bbb.c', 'ccc.c']) env.Precious('libfoo.a')

Default targets The user may specify default targets that will be built if there are no targets supplied on the command line: env.Default('install', 'src') Multiple calls to the &Default; method (typically one per &SConscript; file) append their arguments to the list of default targets.

File installation Files may be installed in a destination directory: env.Install('/usr/bin', 'program1', 'program2') Files may be renamed on installation: env.InstallAs('/usr/bin/xyzzy', 'xyzzy.in') Multiple files may be renamed on installation by specifying equal-length lists of target and source files: env.InstallAs(['/usr/bin/foo', '/usr/bin/bar'], ['foo.in', 'bar.in'])

Target aliases In order to provide convenient "shortcut" target names that expand to a specified list of targets, aliases may be established: env.Alias(alias = 'install', targets = ['/sbin', '/usr/lib', '/usr/share/man']) In this example, specifying a target of install will cause all the files in the associated directories to be built (that is, installed). An &Alias; may include one or more other &Aliases; in its list: env.Alias(alias = 'libraries', targets = ['lib']) env.Alias(alias = 'programs', targets = ['libraries', 'src'])

Customizing output The &SCons; API supports the ability to customize, redirect, or suppress its printed output through user-defined functions. &SCons; has several pre-defined points in its build process at which it calls a function to (potentially) print output. User-defined functions can be specified for these call-back points when &Build; or &Clean;is invoked: env.Build(target = '.', on_analysis = dump_dependency, pre_update = my_print_command, post_update = my_error_handler) on_error = my_error_handler) The specific call-back points are: on_analysis Called for every object, immediately after the object has been analyzed to see if it's out-of-date. Typically used to print a trace of considered objects for debugging of unexpected dependencies. pre_update Called for every object that has been determined to be out-of-date before its update function or command is executed. Typically used to print the command being called to update a target. post_update Called for every object after its update function or command has been executed. Typically used to report that a top-level specified target is up-to-date or was not remade. on_error Called for every error returned by an update function or command. Typically used to report errors with some string that will be identifiable to build-analysis tools. Functions for each of these call-back points all take the same arguments: my_dump_dependency(target, level, status, update, dependencies) where the arguments are: target The target object being considered. level Specifies how many levels the dependency analysis has recursed in order to consider the target. A value of 0 specifies a top-level target (that is, one passed to the &Build; or &Clean; method). Objects which a top-level target is directly dependent upon have a level of <1>, their direct dependencies have a level of <2>, etc. Typically used to indent output to reflect the recursive levels. status A string specifying the current status of the target ("unknown", "built", "error", "analyzed", etc.). A complete list will be enumerated and described during implementation. update The command line or function name that will be (or has been) executed to update the target. dependencies A list of direct dependencies of the target.

Separate source and build trees &SCons; allows target files to be built completely separately from the source files by "linking" a build directory to an underlying source directory: env.Link('build', 'src') SConscript('build/SConscript') &SCons; will copy (or hard link) necessary files (including the &SConscript; file) into the build directory hierarchy. This allows the source directory to remain uncluttered by derived files.

Variant builds The &Link; method may be used in conjunction with multiple &consenvs; to support variant builds. The following &SConstruct; and &SConscript; files would build separate debug and production versions of the same program side-by-side: % cat SConstruct env = Environment() env.Link('build/debug', 'src') env.Link('build/production', 'src') flags = '-g' SConscript('build/debug/SConscript', Export(env)) flags = '-O' SConscript('build/production/SConscript', Export(env)) % cat src/SConscript env = Environment(CCFLAGS = flags) env.Program('hello', 'hello.c') The following example would build the appropriate program for the current compilation platform, without having to clean any directories of object or executable files for other architectures: % cat SConstruct build_platform = os.path.join('build', sys.platform) Link(build_platform, 'src') SConscript(os.path.join(build_platform, 'SConscript')) % cat src/SConscript env = Environment env.Program('hello', 'hello.c')

Code repositories &SCons; may use files from one or more shared code repositories in order to build local copies of changed target files. A repository would typically be a central directory tree, maintained by an integrator, with known good libraries and executables. Repository('/home/source/1.1', '/home/source/1.0') Specified repositories will be searched in-order for any file (configuration file, input file, target file) that does not exist in the local directory tree. When building a local target file, &SCons; will rewrite path names in the build command to use the necessary repository files. This includes modifying lists of or flags to specify an appropriate set of include paths for dependency analysis. &SCons; will modify the Python sys.path variable to reflect the addition of repositories to the search path, so that any imported modules or packages necessary for the build can be found in a repository, as well. If an up-to-date target file is found in a code repository, the file will not be rebuilt or copied locally. Files that must exist locally (for example, to run tests) may be specified: Local('program', 'libfoo.a') in which case &SCons; will copy or link an up-to-date copy of the file from the appropriate repository.

Derived-file caching &SCons; can maintain a cache directory of target files which may be shared among multiple builds. This reduces build times by allowing developers working on a project together to share common target files: Cache('/var/tmp/build.cache/i386') When a target file is generated, a copy is added to the cache. When generating a target file, if &SCons; determines that a file that has been built with the exact same dependencies already exists in the specified cache, &SCons; will copy the cached file rather than re-building the target. Command-line options exist to modify the &SCons; caching behavior for a specific build, including disabling caching, building dependencies in random order, and displaying commands as if cached files were built.

Job management A simple API exists to inform the Build Engine how many jobs may be run simultaneously: Jobs(limit = 4)