Science fiction.

1 files changed, 40 insertions, 0 deletions

diff --git a/doc/MANUAL.txt b/doc/MANUAL.txt
index a0f0bcb..98150dd 100644
--- a/doc/MANUAL.txt
+++ b/doc/MANUAL.txt
@@ -185,6 +185,46 @@ testr can ignore such tests if they are tagged, using the filter_tags
 configuration option. Tests tagged with any tag in that (space separated) list
 will only be included in counts and reports if the test failed (or errored).
 
+Automated test isolation bisection
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+As mentioned above, its possible to manually analyze test isolation issues by
+interrogating the repository for which tests ran on which worker, and then 
+creating a list file with those tests, re-running only half of them, checking
+the error still happens, rinse and repeat.
+
+However that is tedious. testr can perform this analysis for you::
+
+  $ testr run --analyze-isolation 
+
+will perform that analysis for you. (This requires that your test runner is
+(mostly) deterministic on test ordering). The process is::
+
+1. The last run in the repository is used as a basis for analysing against -
+   tests are only cross checked against tests run in the same worker in that
+   run.
+
+1. Each test that is currently listed as a failure is run in a test process
+   given just that id to run.
+
+1. Tests that fail are excluded from analysis - they are broken on their own.
+
+1. The remaining failures are then individually analysed one by one.
+
+1. For each failing, it gets run in one work along with the first 1/2 of the
+   tests that were previously run prior to it.
+
+1. If the test now passes, that set of prior tests are discarded, and the
+   other half of the tests is promoted to be the full list. If the test fails
+   then other other half of the tests are discarded and the current set
+   promoted.
+
+1. Go back to running the failing test along with 1/2 of the current list of
+   priors unless the list only has 1 test in it. If the failing test still
+   failed with that test, we have found the isolation issue. If it did not
+   then either the isolation issue is racy, or it is a 3-or-more test
+   isolation issue. Neither of those cases are automated today.
+
 Repositories
 ~~~~~~~~~~~~