Add a timings/Java configuration to time simple Java compilation.

Update timings/README.txt with doc of how to add timing configurations.

1 files changed, 111 insertions, 0 deletions

diff --git a/timings/README.txt b/timings/README.txt
index e612aa1f..443d235b 100644
--- a/timings/README.txt
+++ b/timings/README.txt
@@ -32,3 +32,114 @@ in-line data.  This is typically done when it's necessary to create
 hundreds of identical input files or directories before running the
 timing test, to avoid cluttering our SCM system with hundreds of otherwise
 meaningless files.
+
+
+STRUCTURE OF A TIMING CONFIGURATION
+===================================
+
+A timing configuration should be in a subdirectory and should contain
+at least the following three files:
+
+    TimeSCons-run.py
+
+        The script that controls the timing run.  It looks a lot
+        like the test scripts (since it uses the same infrastructure)
+        except that you instantiate TestSCons.TimeSCons object, not a
+        TestSCons.TestSCons object.
+
+        Typically you want to initialize the object with a "variables"
+        dict containing one or more parameters whose values control the
+        scale of the configuration.  This would typically be the number
+        of source files, directories to scan, etc.  The TimeSCons-run.py
+        script can then use the value of those variables to generate that
+        many copies of input source files, or directories, or what have,
+        from in-line data, instead of having to check in a couple hundred
+        files for a large configuration.
+
+        These variables get passed to the timed SCons invocation as
+        ARGUMENT= arguments on the command line, so the SConstruct
+        file can use it to loop through the right number of files /
+        directories / what have you.
+
+    SConstruct
+
+        This is the actual SCons configuration that gets tested.  It has
+        access to the variable(s) that control the configuration as
+        ARGUMENTS from the command line.
+
+        It's possible for the SConstruct file to do additional set up of
+        input files and the like, but in general that should be kept to
+        a minimum.  We want what the SConscript file does to be dominated
+        by the actual execution we're timing, not initialization stuff,
+        so most initialization is better left in TimeSCons-run.py.
+
+    config.js
+
+        This gives our buildbot information about the timing configuration
+        (specifically, the title) for display.
+
+Note that it's perfectly acceptable to check in additional files that
+may be necessary for your configuration.  They'll get copied to the
+temporary directory used to run the timing.
+
+
+CALIBRATING YOUR TIMING CONFIGURATION
+=====================================
+
+One goal we have for timing configurations is that they should take
+about 10 seconds to run on our buildbot timing system, which is an older,
+slower system than most.
+
+Per above, you presumably defined one or more variables that control the
+"size" of your configuration:  the number of input files, directories,
+etc.  The timing infrastructure actually reports the value of these
+variables in a way that lets us automate the process of adjusting the
+variable values to run within a specified amount of time.
+
+The bin/calibrate.py will run your configuration repeatedly, adjusting
+the value(s) of the variable(s) that control your configuration until
+it gets three successive runs that take between 9.5 and 10.0 seconds
+(by default, options let you adjust the range):
+
+    $ python bin/calibrate.py timings/MyNewTimingConfiguration/TimeSCons-run.py
+    run   1:   3.124:  TARGET_COUNT=50
+    run   2:  11.936:  TARGET_COUNT=160
+    run   3:   9.175:  TARGET_COUNT=134
+    run   4:  10.489:  TARGET_COUNT=146
+    run   5:   9.798:  TARGET_COUNT=139
+    run   6:   9.695:  TARGET_COUNT=139
+    run   7:   9.670:  TARGET_COUNT=139
+    $
+
+If you have multiple variables, it will adjust *all* of the variables
+on each run.  In other words, the proportion between your variables will
+remain (relatively) constant.
+
+Of course, this needs to be run on a quiet system for the numbers
+to converge.  And what you really need to do before committing a
+configuration is run bin/calibrate.py on the actual system that runs
+our Buildbot timings.  For that, see Bill Deegan or Steven Knight.
+
+Once you have "good" values for your variables, put them in your
+TimeSCons-run.py and you should be good to go.  Note that we've started a
+convention of also pasting the run output from calibrate.py into comments
+in the TimeSCons-run.py, just to preserve some of the historical context
+that led to certain values being chosen.
+
+
+ADDING A NEW TIMING CONFIGURATION
+=================================
+
+In addition to creating a subdirectory with at least the pieces listed
+above in the "STRUCTURE" section and "CALIBRATING" your variable(s),
+you need to update the following file in this directory:
+
+    index.html
+
+        Add an entry to the test_map dictionary for the subdirectory
+        you just created.
+
+That should be it before checkin.  After checkin, one of the Buildbot
+administrators (currently Bill Deegan or Steven Knight) needs to update
+and restart the Buildbot master so that it will start executing the
+build step to run the new timing configuration.