mapreduce| Fixes -jN for map/reduce Checkers (e.g. SimilarChecker)

This integrate the map/reduce functionality into lint.check_process().

We previously had `map` being invoked, here we add `reduce` support.

We do this by collecting the map-data by worker and then passing it to a
reducer function on the Checker object, if available - determined by
whether they confirm to the `mapreduce_checker.MapReduceMixin` mixin
interface or nor.

This allows Checker objects to function across file-streams when using
multiprocessing/-j2+. For example SimilarChecker needs to be able to
compare data across all files.

The tests, that we also add here, check that a Checker instance returns
and reports expected data and errors, such as error-messages and stats -
at least in a exit-ok (0) situation.

On a personal note, as we are copying more data across process
boundaries, I suspect that the memory implications of this might cause
issues for large projects already running with -jN and duplicate code
detection on. That said, given that it takes a long time to perform
lints of large code bases that is an issue for the [near?] future and
likely to be part of the performance work. Either way but let's get it
working first and deal with memory and perforamnce considerations later
- I say this as there are many quick wins we can make here, e.g.
file-batching, hashing lines, data compression and so on.

1 files changed, 9 insertions, 3 deletions

diff --git a/tests/test_self.py b/tests/test_self.py
index cd138e789..e8f9f848f 100644
--- a/tests/test_self.py
+++ b/tests/test_self.py
@@ -46,7 +46,7 @@ from unittest import mock
 
 import pytest
 
-from pylint.constants import MAIN_CHECKER_NAME
+from pylint.constants import MAIN_CHECKER_NAME, MSG_TYPES_STATUS
 from pylint.lint import Run
 from pylint.reporters import JSONReporter
 from pylint.reporters.text import BaseReporter, ColorizedTextReporter, TextReporter
@@ -243,13 +243,19 @@ class TestRunTC:
         )
 
     def test_parallel_execution(self):
+        out = StringIO()
         self._runtest(
             [
                 "-j 2",
                 join(HERE, "functional", "a", "arguments.py"),
-                join(HERE, "functional", "a", "arguments.py"),
             ],
-            code=2,
+            out=out,
+            # We expect similarities to fail and an error
+            code=MSG_TYPES_STATUS["E"],
+        )
+        assert (
+            "Unexpected keyword argument 'fourth' in function call"
+            in out.getvalue().strip()
         )
 
     def test_parallel_execution_missing_arguments(self):