Refactor file checking for a simpler parallel API (#3016)

This change prepares the code for enabling Prospector to take advantage
of running PyLint parallel.

Iterating files is moved into generator (_iterate_file_descrs) so that
parallel checking can use the same implementation (_check_file) just
by providing different kind of generator that reads the files from parent
process.

The refactoring removes code duplication that existed in PyLinter._do_check
method; checking module content from stdin had identical implementation to
checking content from a source file.

Made PyLinter.expand_files a private method.

The previous implementation of parallel linting created new PyLinter
objects in the worker (child) process causing failure when running under
Prospector because Prospector uses a custom PyLinter class
(a class inherited from PyLinter)
and PyLint naturally just creates PyLinter object. This caused linting to
fail because there is options for Prospector's IndentChecker which was not
created in the worker process.

The new implementation passes the original PyLinter object into workers
when the workers are created. See https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods

Note that as Windows uses spawn method by default, PyLinter object (and
its) members need to be pickleable from now on with the exception being
PyLinter.reporter which is not passed to child processes.

The performance has remained about the same based on quick tests done with
Django project containing about 30 000 lines of code; with the old
implementation linting took 26-28 seconds with 8 jobs on quad core i7 and
24-27 seconds with the new implementation.

1 files changed, 2 insertions, 2 deletions

diff --git a/pylint/message/message_definition.py b/pylint/message/message_definition.py
index e54c15aae..43c219631 100644
--- a/pylint/message/message_definition.py
+++ b/pylint/message/message_definition.py
@@ -23,7 +23,7 @@ class MessageDefinition:
         maxversion=None,
         old_names=None,
     ):
-        self.checker = checker
+        self.checker_name = checker.name
         self.check_msgid(msgid)
         self.msgid = msgid
         self.symbol = symbol
@@ -63,7 +63,7 @@ class MessageDefinition:
         """return the help string for the given message id"""
         desc = self.description
         if checkerref:
-            desc += " This message belongs to the %s checker." % self.checker.name
+            desc += " This message belongs to the %s checker." % self.checker_name
         title = self.msg
         if self.minversion or self.maxversion:
             restr = []