perf/README: Add notes on using cairo-perf-diff

1 files changed, 93 insertions, 12 deletions

diff --git a/perf/README b/perf/README
index 6965c3f94..b713dd08c 100644
--- a/perf/README
+++ b/perf/README
@@ -1,24 +1,105 @@
 This is cairo's performance test suite.
 
-To run the performance tests simply type:
+One of the simplest ways to run the performance suite is:
 
-	make perf
+    make perf
 
-and the tests results will be printed on stdout
+while will give a report of the speed of each indivudual test. See
+more details on other options for running the suite below.
 
-TODO: We'll want to add support for charting the results. Various
-charts would be interesting:
+Running the cairo performance suite
+-----------------------------------
+The lowest-level means of running the test suite is with the
+cairo-perf program, (which is what "make perf" executes). Some
+examples of running it:
 
-  * For a given test, how does its performance scale as a function of
-    image size
+    # Report on all tests with default number of iterations:
+    ./cairo-perf
 
-  * For a given test, how do the various backends perform
+    # Report on 100 iterations of all gradient tests:
+    ./cairo-perf -i 100 gradient
 
-  * For a given test, how has the performance changed throughout the
-    history of cairo development.
+    # Generate raw results for 10 iterations into cairo.perf
+    ./cairo-perf -r -i 10 > cairo.perf
+    # Append 10 more iterations of the paint test
+    ./cairo-perf -r -i 10 paint >> cairo.perf
 
-TODO: We'll also want to make it easy to run individual tests for
-profiling, etc.
+Raw results aren't useful for reading directly, but are quite useful
+when using cairo-perf-diff to compare spearate runs (see more
+below). The advantage of using the raw mode is that test runs can be
+generated incrementally and appended to existing reports.
+
+Generating comparisons of separate runs
+---------------------------------------
+It's often useful to generate a chart showing the comparison of two
+separate runs of the cairo performance suite, (for example, after
+applying a patch intended to improve cairo's performance). The
+cairo-perf-diff script can be used to compare two report files
+generated by cairo-perf.
+
+Again, by way of example:
+
+    # Show performance changes from cairo-orig.perf to cairo-patched.perf
+    ./cairo-perf-diff cairo-orig.perf cairo-patched.perf
+
+This will work whether the data files were generate in raw mode (with
+cairo-perf -r) or cooked, (cairo-perf without -r).
+
+Finally, in its most powerful mode, cairo-perf-diff accepts two git
+revisions and will do all the work of checking each revision out,
+building it, running cairo-perf for each revision, and finally
+generating the report. Obviously, this mode only works if you are
+using cairo within a git repository, (and not from a tar file). Using
+this mode is as simple as passing the git revisions to be compared to
+cairo-perf-diff:
+
+    # Compare cairo 1.2.6 to cairo 1.4.0
+    ./cairo-perf-diff 1.2.6 1.4.0
+
+    # Measure the impact of the latest commit
+    ./cairo-perf-diff HEAD~1 HEAD
+
+As a convenience, this common desire to measure a single commit is
+supported by passing a single revision to cairo-perf-diff, in which
+case it will compare it to the immediately preceeding commit. So for
+example:
+
+    # Measure the impact of the latest commit
+    ./cairo-perf-diff HEAD
+
+    # Measure the impact of an arbitrary commit by SHA-1
+    ./cairo-perf-diff aa883123d2af90
+
+Also, when passing git revisions to cairo-perf-diff like this, it will
+automatically cache results and re-use them rather than re-rerunning
+cairo-perf over and over on the same versions. This means that if you
+ask for a report that you've generated in the past, cairo-perf-diff
+should return it immediately.
+
+Now, sometimes it is desirable to generate more iterations rather than
+re-using cached results. In this case, the -f flag can be used to
+force cairo-perf-diff to generate additional results in addition to
+what has been cached:
+
+    # Measure the impact of latest commit (force more measurement)
+    ./cairo-perf-diff -f
+
+And finally, the -f mode is most useful in conjunction with the --
+option to cairo-perf-diff which allows you to pass options to the
+underlying cairo-perf runs. This allows you to restrict the additonal
+test runs to a limited subset of the tests.
+
+For example, a frequently used trick is to first generate a chart with
+a very small number of iterations for all tests:
+
+    ./cairo-perf-diff HEAD
+
+Then, if any of the results look suspicious, (say there's a slowdown
+reported in the text tests, but you think the text test shouldn't be
+affected), then you can force more iterations to be tested for only
+those tests:
+
+    ./cairo-perf-diff -f HEAD -- text
 
 Creating a new performance test
 -------------------------------