Add more parallelism to t/harness

This commit causes test files in cpan/, etc to run in parallel provided
there is no file in the test directory whose name begins with zero.
Experience shows that things like 01test, 02test, ... are supposed to be
executed serially.  Sometimes there is a single file beginning with 0,
and this is supposed to be executed first as setup for the remaining
tests (which probably could be executed in parallel, but that
enhancement is much harder to do, so isn't tried here)

This is a heuristic, and this commit adds the ability to explicitly
specify directories to the serial list that don't correspond to the
above rule.  One directory is currently on that list: ext/Pod-Html/t.
It follows a different paradigm to order things.  We could rename the
tests to follow the normal paradigm since this is in ext/, but I thought
it best to keep it as an example of what to do should the need for other
exceptions arise.

This can save a bunch of wallclock time on slower systems with many
cores, such as dromedary.  On that system, the test suite finishes 17%
faster with this commit on the first run.  (This is a savings of over
100 wallclock seconds; a very noticeable improvement.)  On subsequent
runs when it knows approximately how long each test takes and sorts the
order accordingly, the suite runs 8% faster.

On machines with few cores, there isn't much of a difference.

1 files changed, 44 insertions, 2 deletions

diff --git a/t/harness b/t/harness
index 5ae2702ec1..7733d2bb78 100644
--- a/t/harness
+++ b/t/harness
@@ -160,18 +160,60 @@ if (@ARGV) {
 
 	my %dir;
 	my %total_time;
+        my %serials;
+        my %all_dirs;
 
+        # Preprocess the list of tests
 	for (@last) {
 	    if ($^O eq 'MSWin32') {
 		s,\\,/,g; # canonicalize path
 	    };
-	    # Treat every file matching lib/*.t as a "directory"
-	    m! \A ( \.\. / (?: lib | ext/XS-APItest/t )
+
+            # Keep a list of the distinct directory names, and another list of
+            # those which contain a file whose name begins with a 0
+            if ( m! \A \.\. /
+                                ( .*? )         # $1 is the directory path name
+                            /
+                                ( [^/]* \.t )   # $2 is the .t name
+                    \z !x)
+            {
+                my $path = $1;
+
+                $all_dirs{$path} = 1;
+                $serials{$path} = 1 if $2 =~ / \A 0 /x;
+            }
+        }
+
+        # We assume that the reason a test file's name begins with a 0 is to
+        # order its execution among the tests in its directory.  Hence, a
+        # directory containing such files should be tested in serial order.
+        #
+        # Add exceptions to the above rule
+        for (qw(ext/Pod-Html/t)) {
+            $serials{$_} = 1;
+        }
+
+        # Remove the serial testing directories from the list of all
+        # directories.  The remaining ones are testable in parallel.  Make the
+        # parallel list a scalar with names separated by '|' so that below
+        # they will be added to a regular expression.
+        my $non_serials = join "|", grep { not exists $serials{$_} } keys %all_dirs;
+        undef %all_dirs;
+        undef %serials;
+
+	for (@last) {
+            # Treat every file in each non-serial directory as its own
+            # "directory", so that it can be executed in parallel
+            m! \A ( \.\. / (?: $non_serials )
                          / [^/]+ \.t \z | .* [/] ) !x
                 or die "'$_'";
 	    push @{$dir{$1}}, $_;
+
+            # This file contributes time to the total needed for the directory
+            # as a whole
 	    $total_time{$1} += $times{$_} || 0;
 	}
+        #print STDERR __LINE__, join "\n", sort { $total_time{$b} <=> $total_time{$a} } keys %dir, "  ";
 
 	push @tests, @last;