More testing notes

1 files changed, 50 insertions, 16 deletions

diff --git a/TESTING b/TESTING
index 4a195d7..75ad654 100644
--- a/TESTING
+++ b/TESTING
@@ -2,9 +2,8 @@ How the test suites work in Gitano
 ==================================
 
 Gitano's test suite is a functional black-box suite built in the `yarn` testing
-language.  Yarn is part of the cmdtest suite by Lars Wirzenius and can be found
-in the `cmdtest` Debian package.  The testing stuff all lives in the testing
-directory at the top level of the Gitano git repository.
+language.  The testing stuff all lives in the testing directory at the top
+level of the Gitano git repository.
 
 The test suite can run in both an SSH mode and an HTTP mode.  In the former the
 accesses to the Gitano instances created during the tests are made by simulated
@@ -32,22 +31,57 @@ to HTTP mode if you are changing the CGI code by running:
 The protocol being used during a test run is exposed in the GTT_PROTO
 environment variable passed to implementations.
 
-HTTP mode test dependencies
----------------------------
+Yarn implementations
+--------------------
+
+The original (and primary) Yarn is part of the `cmdtest` suite by Lars
+Wirzenius, is written in Python 2, and can be found in the `cmdtest` Debian
+package or at <http://git.liw.fi/cgi-bin/cgit/cgit.cgi/cmdtest/> at the time of
+writing this document.
+
+The Gitano test suite is also compatible with `rsyarn` by Daniel Silverstone,
+is written in Rust (needs 1.13 or better), and can be found at
+<https://git.gitano.org.uk/rsyarn.git> at the time of writing this document.
+
+Other Yarn implementations may be compatible, but the Makefile may need
+tweaking if they do not expose the same cmdline arguments.  Patches are welcome
+to support other Yarn implementations' cmdline syntax.
 
-HTTP tests work via password access rather than key access,
-so the `htpasswd` binary must be in your `PATH`.
-In debian this can be accomplised by installing `apache2-utils`.
+You may select a different Yarn tool by setting the 'YARN' make variable on
+the command line, and you may pass additional arguments as may be necessary by
+setting the 'YARN_ARGS' variable on the command line, for example if you wish
+to use `rsyarn` and have it run five tests in parallel, you might run:
 
-HTTP tests require starting an HTTP server on an unused port.
-The test suite implements this using `lighttpd`,
-so that binary must be in your `PATH`.
-In debian this can be accomplished by installing `lighttpd`.
-This starts it by default, so you probably want to run
-`systemctl disable --now lighttpd.service` to stop it.
+    make test YARN=rsyarn YARN_ARGS=-j5
+
+This can be a significant speed boost when your computer is capable of running
+multiple of the scenarios in parallel, for example on the author's slow old
+laptop, the full yarn suite takes over two minutes to run using a single
+threaded approach, or only 40 seconds if five scenarios are run in parallel.
+
+Note: The test suite was originally written with single-threaded execution in
+mind, and though the Yarns allows for arbitrary numbers of scenarios to run in
+parallel, you should be careful not to overload your system when running the
+test suite.
+
+HTTP mode test dependencies
+---------------------------
 
-This defaults to starting at 8080 and walking up to find a free one.
-If there is a better port to use set `HTTP_FIRST_TEST_PORT`.
+HTTP tests work via password access rather than key access, so the `htpasswd`
+binary must be in your `PATH`.  In Debian (and derived distributions) this can
+be accomplised by installing the `apache2-utils` package.
+
+HTTP tests require starting an HTTP server on an unused port.  The test suite
+implements this using `lighttpd`, so that binary must be in your `PATH`.  In
+Debian (and derived distributions) this can be accomplished by installing the
+`lighttpd` package.  This package may start the server by default, so you are
+likely to want to run `systemctl disable --now lighttpd.service` to stop it.
+
+The tests default to running servers on ports starting at 8080 and walking up
+to find a free one (up to a maximum of ten times).  If there is a better port
+to use set `HTTP_FIRST_TEST_PORT`.  This limitation means that if your `yarn`
+implementation parallelises scenarios, you must limit the parallelism to ten or
+else the HTTP tests will not run successfully.
 
 The fundamental parts of the `testing/` directory are
 =====================================================