tests: add basic support for running tests in parallel

In the beginning it didn't matter that we were running the regression
tests serially, but as we are now running +16k tests the run time is
getting rather long.

As there is no good reason why we can't run these tests in parallel,
let's add some basic support to do so.  This patch adds support for
running multiple tests jobs at once using the '-j <NUM>' flag,
similar to the "make" command.  If the number of jobs specified is
invalid/zero then it is set to the number of CPUs present on the
system.  If the '-j <NUM>' flag is not specified then the tests are
executed in serial fashion as they are done now.

If the '-l <LOG>' option is specified the test run reverts to serial
execution regardless of the command line in order to preserve the log
output.  While the normal console output is preserved regardless of
the number of jobs, the logfile output is handled differently and
this restriction was the easiest solution.  We should consider
removing the '-l <LOG>' option at some point since output capture and
redirection is likely best handled by the shell anyway.

As far as the performance improvements are concerned, the results
speak for themselves.  On my eight core laptop the runtime drops from
~14 minutes to ~4 minutes (!).

* Existing code, single threaded (14 minutes, 14 seconds)

% time -- ./regression -m c -m python
=============== Tue Jan  5 06:11:52 PM EST 2021 ===============
Regression Test Report ("regression -m c -m python")
 batch name: 01-sim-allow
 test mode:  c
 test type:  bpf-sim
Test 01-sim-allow%%001-00001 result:   SUCCESS
...
Test 58-live-tsync_notify%%001-00001 result:   SKIPPED (must specify live tests)
Regression Test Summary
 tests run: 16412
 tests skipped: 130
 tests passed: 16412
 tests failed: 0
 tests errored: 0
============================================================
real 854.37
user 693.87
sys 269.25

* Patched code, 8 jobs (4 minutes, 7 seconds)

% time -- ./regression -j 8 -m c -m python
=============== Tue Jan  5 06:27:56 PM EST 2021 ===============
Regression Test Report ("regression -j 8 -m c -m python")
 batch name: 01-sim-allow
 test mode:  c
 test type:  bpf-sim
Test 01-sim-allow%%001-00001 result:   SUCCESS
...
Test 58-live-tsync_notify%%001-00001 result:   SKIPPED (must specify live tests)
Regression Test Summary
 tests run: 16412
 tests skipped: 130
 tests passed: 16412
 tests failed: 0
 tests errored: 0
============================================================
real 246.96
user 966.08
sys 251.27

Reviewed-by: Tom Hromatka <tom.hromatka@oracle.com>
Signed-off-by: Paul Moore <paul@paul-moore.com>

0 files changed, 0 insertions, 0 deletions