Added a new option for controlling the peephole optimizer in astroid.

The option ``--optimize-ast`` will control the peephole optimizer,
which is used to optimize a couple of AST subtrees. The current problem
solved by the peephole optimizer is when multiple joined strings,
with the addition operator, are encountered. If the numbers of such
strings is high enough, Pylint will then fail with a maximum recursion
depth exceeded error, due to its visitor architecture. The peephole
just transforms such calls, if it can, into the final resulting string
and this exhibit a problem, because the visit_binop method stops being
called (in the optimized AST it will be a Const node).

3 files changed, 41 insertions, 2 deletions

diff --git a/ChangeLog b/ChangeLog
index 6b68696..261054e 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -7,6 +7,18 @@ ChangeLog for Pylint
       abstract-class-not-used. These warnings don't add any real value
       and they don't imply errors or problems in the code.
 
+    * Added a new option for controlling the peephole optimizer in astroid.
+      The option ``--optimize-ast`` will control the peephole optimizer,
+      which is used to optimize a couple of AST subtrees. The current problem
+      solved by the peephole optimizer is when multiple joined strings,
+      with the addition operator, are encountered. If the numbers of such
+      strings is high enough, Pylint will then fail with a maximum recursion
+      depth exceeded error, due to its visitor architecture. The peephole
+      just transforms such calls, if it can, into the final resulting string
+      and this exhibit a problem, because the visit_binop method stops being
+      called (in the optimized AST it will be a Const node).           
+
+
 2015-03-11 -- 1.4.2
 
     * Don't require a docstring for empty modules. Closes issue #261.
diff --git a/doc/faq.rst b/doc/faq.rst
index a42080e..1ce9a3e 100644
--- a/doc/faq.rst
+++ b/doc/faq.rst
@@ -261,12 +261,25 @@ mangled sys.path.  Pylint doesn't import any of the candidate modules and
 thus doesn't include any of import's side effects (good and bad).  It
 traverses an AST representation of the code.
 
-6.3 I think I found a bug in Pylint. What should I do?
+6.3 Pylint keeps crashing with `Maximum recursion depth exceeded`
+-----------------------------------------------------------------
+
+Pylint can crash with this error if you have a string in your analyzed
+program, created by joining a lot of strings with the addition operator.
+Due to how Pylint works, visiting nodes on a AST tree and due to how
+the BinOp node is represented (the node which represents the string '1+1'
+for instance), the same visit method will be called over and over again, leading
+to a maximum recursion error. You can alleviate this problem by passing
+the flag `--optimize-ast=y` to Pylint. This will activate an optimization
+which will transform such AST subtrees into the final resulting string.
+This flag is off by default. If this is not the case, please report a bug!
+
+6.4 I think I found a bug in Pylint. What should I do?
 -------------------------------------------------------
 
 Read http://docs.pylint.org/contribute#bug-reports-feedback
 
-6.4 I have a question about Pylint that isn't answered here.
+6.5 I have a question about Pylint that isn't answered here.
 ------------------------------------------------------------
 
 Read http://docs.pylint.org/contribute#mailing-lists
diff --git a/pylint/lint.py b/pylint/lint.py
index eb8f706..b96a4ca 100644
--- a/pylint/lint.py
+++ b/pylint/lint.py
@@ -406,6 +406,19 @@ class PyLinter(configuration.OptionsManagerMixIn,
                             ' loading into the active Python interpreter and may run'
                             ' arbitrary code')}
                   ),
+
+                ('optimize-ast',
+                  {'type': 'yn', 'metavar': '<yn>', 'default': False,
+                   'help': ('Allow optimization of some AST trees. This will '
+                            'activate a peephole AST optimizer, which will '
+                            'apply various small optimizations. For instance, '
+                            'it can be used to obtain the result of joining '
+                            'multiple strings with the addition operator. '
+                            'Joining a lot of strings can lead to a maximum '
+                            'recursion error in Pylint and this flag can prevent '
+                            'that. It has one side effect, the resulting AST '
+                            'will be different than the one from reality.')}
+                ),
                )
 
     option_groups = (
@@ -940,6 +953,7 @@ class PyLinter(configuration.OptionsManagerMixIn,
         self.stats = {'by_module' : {},
                       'by_msg' : {},
                      }
+        MANAGER.optimize_ast = self.config.optimize_ast
         MANAGER.always_load_extensions = self.config.unsafe_load_any_extension
         MANAGER.extension_package_whitelist.update(
             self.config.extension_pkg_whitelist)