Reorganize the documentation table of content (#6589)

* [doc] Reorganize the doc table of content with four levels

User, developer, contributor, maintainer.

Co-authored-by: Jacob Walls <jacobtylerwalls@gmail.com>

16 files changed, 677 insertions, 12 deletions

diff --git a/doc/development_guide/contribute.rst b/doc/development_guide/contributor_guide/contribute.rst
index b09414a12..b09414a12 100644
--- a/doc/development_guide/contribute.rst
+++ b/doc/development_guide/contributor_guide/contribute.rst
diff --git a/doc/development_guide/contributor_guide/index.rst b/doc/development_guide/contributor_guide/index.rst
new file mode 100644
index 000000000..8becbe3a6
--- /dev/null
+++ b/doc/development_guide/contributor_guide/index.rst
@@ -0,0 +1,13 @@
+Contributing to pylint
+======================
+
+The contributor guide will help you if you want to contribute to pylint itself.
+
+.. toctree::
+   :maxdepth: 2
+   :titlesonly:
+
+   contribute
+   tests/index
+   profiling
+   release
diff --git a/doc/development_guide/profiling.rst b/doc/development_guide/contributor_guide/profiling.rst
index bb49038ff..bb49038ff 100644
--- a/doc/development_guide/profiling.rst
+++ b/doc/development_guide/contributor_guide/profiling.rst
diff --git a/doc/development_guide/contributor_guide/release.md b/doc/development_guide/contributor_guide/release.md
new file mode 100644
index 000000000..6953e7e3d
--- /dev/null
+++ b/doc/development_guide/contributor_guide/release.md
@@ -0,0 +1,89 @@
+# Releasing a pylint version
+
+So, you want to release the `X.Y.Z` version of pylint ?
+
+## Releasing a major or minor version
+
+**Before releasing a major or minor version check if there are any unreleased commits on
+the maintenance branch. If so, release a last patch release first. See
+`Releasing a patch version`.**
+
+- Write the `Summary -- Release highlights` in `doc/whatsnew` and upgrade the release
+  date.
+- Remove the empty changelog for the last unreleased patch version `X.Y-1.Z'`. (For
+  example: `v2.3.5`)
+- Check the result of `git diff vX.Y-1.Z' ChangeLog`. (For example:
+  `git diff v2.3.4 ChangeLog`)
+- Install the release dependencies: `pip3 install -r requirements_test.txt`
+- Bump the version and release by using `tbump X.Y.0 --no-push --no-tag`. (For example:
+  `tbump 2.4.0 --no-push --no-tag`)
+- Check the commit created with `git show` amend the commit if required.
+- Create a new `What's new in Pylint X.Y+1` document. Add it to `doc/index.rst`. Take a
+  look at the examples from `doc/whatsnew`. Commit that with `git commit -am "wip"`.
+- Move the `main` branch up to a dev version with `tbump`:
+
+```bash
+tbump X.Y+1.0-dev0 --no-tag --no-push  # You can interrupt after the first step
+git commit -am "Upgrade the version to x.y+1.0-dev0 following x.y.0 release"
+```
+
+For example:
+
+```bash
+tbump 2.5.0-dev0 --no-tag --no-push
+git commit -am "Upgrade the version to 2.5.0-dev0 following 2.4.0 release"
+```
+
+Check the commit, fixup the 'wip' commit with the what's new then push to a release
+branch
+
+- Open a merge request with the two commits (no one can push directly on `main`)
+- After the merge, recover the merged commits on `main` and tag the first one (the
+  version should be `X.Y.Z`) as `vX.Y.Z` (For example: `v2.4.0`)
+- Push the tag.
+- Release the version on GitHub with the same name as the tag and copy and paste the
+  appropriate changelog in the description. This triggers the PyPI release.
+- Delete the `maintenance/X.Y-1.x` branch. (For example: `maintenance/2.3.x`)
+- Create a `maintenance/X.Y.x` (For example: `maintenance/2.4.x` from the `v2.4.0` tag.)
+- Close the current milestone and create the new ones (For example: close `2.4.0`,
+  create `2.4.1` and `2.6.0`)
+
+## Backporting a fix from `main` to the maintenance branch
+
+Whenever a commit on `main` should be released in a patch release on the current
+maintenance branch we cherry-pick the commit from `main`.
+
+- During the merge request on `main`, make sure that the changelog is for the patch
+  version `X.Y-1.Z'`. (For example: `v2.3.5`)
+- After the PR is merged on `main` cherry-pick the commits on the `maintenance/X.Y.x`
+  branch (For example: from `maintenance/2.4.x` cherry-pick a commit from `main`)
+- Remove the "need backport" label from cherry-picked issues
+- Release a patch version
+
+## Releasing a patch version
+
+We release patch versions when a crash or a bug is fixed on the main branch and has been
+cherry-picked on the maintenance branch.
+
+- Check the result of `git diff vX.Y-1.Z-1 ChangeLog`. (For example:
+  `git diff v2.3.4 ChangeLog`)
+- Install the release dependencies: `pip3 install -r requirements_test.txt`
+- Bump the version and release by using `tbump X.Y-1.Z --no-push`. (For example:
+  `tbump 2.3.5 --no-push`)
+- Check the result visually with `git show`.
+- Open a merge request to run the CI tests for this branch
+- Create and push the tag.
+- Release the version on GitHub with the same name as the tag and copy and paste the
+  appropriate changelog in the description. This triggers the PyPI release.
+- Merge the `maintenance/X.Y.x` branch on the main branch. The main branch should have
+  the changelog for `X.Y-1.Z+1` (For example `v2.3.6`). This merge is required so
+  `pre-commit autoupdate` works for pylint.
+- Fix version conflicts properly, or bump the version to `X.Y.0-devZ` (For example:
+  `2.4.0-dev6`) before pushing on the main branch
+- Close the current milestone and create the new one (For example: close `2.3.5`, create
+  `2.3.6`)
+
+## Milestone handling
+
+We move issues that were not done to the next milestone and block releases only if there
+are any open issues labelled as `blocker`.
diff --git a/doc/development_guide/tests/index.rst b/doc/development_guide/contributor_guide/tests/index.rst
index b612ccea2..2b069857b 100644
--- a/doc/development_guide/tests/index.rst
+++ b/doc/development_guide/contributor_guide/tests/index.rst
@@ -15,4 +15,4 @@ unless they include tests.
 
    install
    launching_test
-   writting_test
+   writing_test
diff --git a/doc/development_guide/tests/install.rst b/doc/development_guide/contributor_guide/tests/install.rst
index c52301ce3..c52301ce3 100644
--- a/doc/development_guide/tests/install.rst
+++ b/doc/development_guide/contributor_guide/tests/install.rst
diff --git a/doc/development_guide/tests/launching_test.rst b/doc/development_guide/contributor_guide/tests/launching_test.rst
index c4b014d98..c4b014d98 100644
--- a/doc/development_guide/tests/launching_test.rst
+++ b/doc/development_guide/contributor_guide/tests/launching_test.rst
diff --git a/doc/development_guide/tests/writting_test.rst b/doc/development_guide/contributor_guide/tests/writing_test.rst
index 2d9844b16..2d9844b16 100644
--- a/doc/development_guide/tests/writting_test.rst
+++ b/doc/development_guide/contributor_guide/tests/writing_test.rst
diff --git a/doc/development_guide/how_tos/custom_checkers.rst b/doc/development_guide/how_tos/custom_checkers.rst
new file mode 100644
index 000000000..eff8cf543
--- /dev/null
+++ b/doc/development_guide/how_tos/custom_checkers.rst
@@ -0,0 +1,312 @@
+.. _write_a_checker:
+
+How to Write a Checker
+======================
+You can find some simple examples in the distribution
+(`custom.py <https://github.com/PyCQA/pylint/blob/main/examples/custom.py>`_
+,
+`custom_raw.py <https://github.com/PyCQA/pylint/blob/main/examples/custom_raw.py>`_
+and
+`deprecation_checker.py <https://github.com/PyCQA/pylint/blob/main/examples/deprecation_checker.py>`_).
+
+.. TODO Create custom_token.py
+
+There are three kinds of checkers:
+
+* Raw checkers, which analyse each module as a raw file stream.
+* Token checkers, which analyse a file using the list of tokens that
+  represent the source code in the file.
+* AST checkers, which work on an AST representation of the module.
+
+The AST representation is provided by the ``astroid`` library.
+``astroid`` adds additional information and methods
+over ``ast`` in the standard library,
+to make tree navigation and code introspection easier.
+
+.. TODO Writing a Raw Checker
+
+.. TODO Writing a Token Checker
+
+Writing an AST Checker
+----------------------
+Let's implement a checker to make sure that all ``return`` nodes in a function
+return a unique constant.
+Firstly we will need to fill in some required boilerplate:
+
+.. code-block:: python
+
+  import astroid
+  from astroid import nodes
+  from typing import TYPE_CHECKING, Optional
+
+  from pylint.checkers import BaseChecker
+
+  if TYPE_CHECKING:
+      from pylint.lint import PyLinter
+
+
+  class UniqueReturnChecker(BaseChecker):
+
+      name = "unique-returns"
+      msgs = {
+          "W0001": (
+              "Returns a non-unique constant.",
+              "non-unique-returns",
+              "All constants returned in a function should be unique.",
+          ),
+      }
+      options = (
+          (
+              "ignore-ints",
+              {
+                  "default": False,
+                  "type": "yn",
+                  "metavar": "<y or n>",
+                  "help": "Allow returning non-unique integers",
+              },
+          ),
+      )
+
+
+So far we have defined the following required components of our checker:
+
+* A name. The name is used to generate a special configuration
+   section for the checker, when options have been provided.
+
+* A message dictionary. Each checker is being used for finding problems
+   in your code, the problems being displayed to the user through **messages**.
+   The message dictionary should specify what messages the checker is
+   going to emit. It has the following format::
+
+       msgs = {
+           "message-id": (
+               "displayed-message", "message-symbol", "message-help"
+           )
+       }
+
+
+   * The ``message-id`` should be a 4-digit number,
+     prefixed with a **message category**.
+     There are multiple message categories,
+     these being ``C``, ``W``, ``E``, ``F``, ``R``,
+     standing for ``Convention``, ``Warning``, ``Error``, ``Fatal`` and ``Refactoring``.
+     The 4 digits should not conflict with existing checkers
+     and the first 2 digits should consistent across the checker.
+
+   * The ``displayed-message`` is used for displaying the message to the user,
+     once it is emitted.
+
+   * The ``message-symbol`` is an alias of the message id
+     and it can be used wherever the message id can be used.
+
+   * The ``message-help`` is used when calling ``pylint --help-msg``.
+
+We have also defined an optional component of the checker.
+The options list defines any user configurable options.
+It has the following format::
+
+    options = (
+        ("option-symbol", {"argparse-like-kwarg": "value"}),
+    )
+
+
+* The ``option-symbol`` is a unique name for the option.
+  This is used on the command line and in config files.
+  The hyphen is replaced by an underscore when used in the checker,
+  similarly to how you would use  ``argparse.Namespace``.
+
+Next we'll track when we enter and leave a function.
+
+.. code-block:: python
+
+  def __init__(self, linter: Optional["PyLinter"] = None) -> None:
+      super().__init__(linter)
+      self._function_stack = []
+
+  def visit_functiondef(self, node: nodes.FunctionDef) -> None:
+      self._function_stack.append([])
+
+  def leave_functiondef(self, node: nodes.FunctionDef) -> None:
+      self._function_stack.pop()
+
+In the constructor we initialise a stack to keep a list of return nodes
+for each function.
+An AST checker is a visitor, and should implement
+``visit_<lowered class name>`` or ``leave_<lowered class name>``
+methods for the nodes it's interested in.
+In this case we have implemented ``visit_functiondef`` and ``leave_functiondef``
+to add a new list of return nodes for this function,
+and to remove the list of return nodes when we leave the function.
+
+Finally we'll implement the check.
+We will define a ``visit_return`` function,
+which is called with an ``.astroid.nodes.Return`` node.
+
+.. _astroid_extract_node:
+.. TODO We can shorten/remove this bit once astroid has API docs.
+
+We'll need to be able to figure out what attributes an
+``.astroid.nodes.Return` node has available.
+We can use ``astroid.extract_node`` for this::
+
+  >>> node = astroid.extract_node("return 5")
+  >>> node
+  <Return l.1 at 0x7efe62196390>
+  >>> help(node)
+  >>> node.value
+  <Const.int l.1 at 0x7efe62196ef0>
+
+We could also construct a more complete example::
+
+  >>> node_a, node_b = astroid.extract_node("""
+  ... def test():
+  ...     if True:
+  ...         return 5 #@
+  ...     return 5 #@
+  ... """)
+  >>> node_a.value
+  <Const.int l.4 at 0x7efe621a74e0>
+  >>> node_a.value.value
+  5
+  >>> node_a.value.value == node_b.value.value
+  True
+
+For ``astroid.extract_node``, you can use ``#@`` at the end of a line to choose which statements will be extracted into nodes.
+
+For more information on ``astroid.extract_node``,
+see the `astroid documentation <https://pylint.pycqa.org/projects/astroid/en/latest/>`_.
+
+Now we know how to use the astroid node, we can implement our check.
+
+.. code-block:: python
+
+  def visit_return(self, node: nodes.Return) -> None:
+      if not isinstance(node.value, nodes.Const):
+          return
+      for other_return in self._function_stack[-1]:
+          if node.value.value == other_return.value.value and not (
+              self.config.ignore_ints and node.value.pytype() == int
+          ):
+              self.add_message("non-unique-returns", node=node)
+
+      self._function_stack[-1].append(node)
+
+Once we have established that the source code has failed our check,
+we use ``~.BaseChecker.add_message`` to emit our failure message.
+
+Finally, we need to register the checker with pylint.
+Add the ``register`` function to the top level of the file.
+
+.. code-block:: python
+
+  def register(linter: "PyLinter") -> None:
+      """This required method auto registers the checker during initialization.
+      :param linter: The linter to register the checker to.
+      """
+      linter.register_checker(UniqueReturnChecker(linter))
+
+We are now ready to debug and test our checker!
+
+Debugging a Checker
+-------------------
+It is very simple to get to a point where we can use ``pdb``.
+We'll need a small test case.
+Put the following into a Python file:
+
+.. code-block:: python
+
+  def test():
+      if True:
+          return 5
+      return 5
+
+  def test2():
+      if True:
+          return 1
+      return 5
+
+After inserting pdb into our checker and installing it,
+we can run pylint with only our checker::
+
+  $ pylint --load-plugins=my_plugin --disable=all --enable=non-unique-returns test.py
+  (Pdb)
+
+Now we can debug our checker!
+
+.. Note::
+
+    ``my_plugin`` refers to a module called ``my_plugin.py``.
+    This module can be made available to pylint by putting this
+    module's parent directory in your ``PYTHONPATH``
+    environment variable or by adding the ``my_plugin.py``
+    file to the ``pylint/checkers`` directory if running from source.
+
+Parallelize a Checker
+---------------------
+
+``BaseChecker`` has two methods ``get_map_data`` and ``reduce_map_data`` that
+permit to parallelize the checks when used with the ``-j`` option. If a checker
+actually needs to reduce data it should define ``get_map_data`` as returning
+something different than ``None`` and let its ``reduce_map_data`` handle a list
+of the types returned by ``get_map_data``.
+
+An example can be seen by looking at ``pylint/checkers/similar.py``.
+
+Testing a Checker
+-----------------
+Pylint is very well suited to test driven development.
+You can implement the template of the checker,
+produce all of your test cases and check that they fail,
+implement the checker,
+then check that all of your test cases work.
+
+Pylint provides a ``pylint.testutils.CheckerTestCase``
+to make test cases very simple.
+We can use the example code that we used for debugging as our test cases.
+
+.. code-block:: python
+
+  import my_plugin
+  import pylint.testutils
+
+
+  class TestUniqueReturnChecker(pylint.testutils.CheckerTestCase):
+      CHECKER_CLASS = my_plugin.UniqueReturnChecker
+
+      def test_finds_non_unique_ints(self):
+          func_node, return_node_a, return_node_b = astroid.extract_node("""
+          def test(): #@
+              if True:
+                  return 5 #@
+              return 5 #@
+          """)
+
+          self.checker.visit_functiondef(func_node)
+          self.checker.visit_return(return_node_a)
+          with self.assertAddsMessages(
+              pylint.testutils.MessageTest(
+                  msg_id="non-unique-returns",
+                  node=return_node_b,
+              ),
+          ):
+              self.checker.visit_return(return_node_b)
+
+      def test_ignores_unique_ints(self):
+          func_node, return_node_a, return_node_b = astroid.extract_node("""
+          def test(): #@
+              if True:
+                  return 1 #@
+              return 5 #@
+          """)
+
+          with self.assertNoMessages():
+              self.checker.visit_functiondef(func_node)
+              self.checker.visit_return(return_node_a)
+              self.checker.visit_return(return_node_b)
+
+
+Once again we are using ``astroid.extract_node`` to
+construct our test cases.
+``pylint.testutils.CheckerTestCase`` has created the linter and checker for us,
+we simply simulate a traversal of the AST tree
+using the nodes that we are interested in.
diff --git a/doc/development_guide/how_tos/index.rst b/doc/development_guide/how_tos/index.rst
new file mode 100644
index 000000000..7ec3a2f36
--- /dev/null
+++ b/doc/development_guide/how_tos/index.rst
@@ -0,0 +1,10 @@
+How To Guides
+=============
+
+.. toctree::
+   :maxdepth: 2
+   :titlesonly:
+
+   custom_checkers
+   plugins
+   transform_plugins
diff --git a/doc/development_guide/how_tos/plugins.rst b/doc/development_guide/how_tos/plugins.rst
new file mode 100644
index 000000000..bc2c0f14c
--- /dev/null
+++ b/doc/development_guide/how_tos/plugins.rst
@@ -0,0 +1,89 @@
+.. -*- coding: utf-8 -*-
+
+How To Write a Pylint Plugin
+============================
+
+Pylint provides support for writing two types of extensions.
+First, there is the concept of **checkers**,
+which can be used for finding problems in your code.
+Secondly, there is also the concept of **transform plugin**,
+which represents a way through which the inference and
+the capabilities of Pylint can be enhanced
+and tailored to a particular module, library of framework.
+
+In general, a plugin is a module which should have a function ``register``,
+which takes an instance of ``pylint.lint.PyLinter`` as input.
+
+A plugin can optionally define a function, ``load_configuration``,
+which takes an instance of ``pylint.lint.PyLinter`` as input. This
+function is called after Pylint loads configuration from configuration
+file and command line interface. This function should load additional
+plugin specific configuration to Pylint.
+
+So a basic hello-world plugin can be implemented as:
+
+.. sourcecode:: python
+
+  # Inside hello_plugin.py
+  from typing import TYPE_CHECKING
+
+  import astroid
+
+  if TYPE_CHECKING:
+      from pylint.lint import PyLinter
+
+
+  def register(linter: "PyLinter") -> None:
+    """This required method auto registers the checker during initialization.
+
+    :param linter: The linter to register the checker to.
+    """
+    print('Hello world')
+
+
+We can run this plugin by placing this module in the PYTHONPATH and invoking
+**pylint** as:
+
+.. sourcecode:: bash
+
+  $ pylint -E --load-plugins hello_plugin foo.py
+  Hello world
+
+We can extend hello-world plugin to ignore some specific names using
+``load_configuration`` function:
+
+.. sourcecode:: python
+
+  # Inside hello_plugin.py
+  from typing import TYPE_CHECKING
+
+  import astroid
+
+  if TYPE_CHECKING:
+      from pylint.lint import PyLinter
+
+
+  def register(linter: "PyLinter") -> None:
+    """This required method auto registers the checker during initialization.
+
+    :param linter: The linter to register the checker to.
+    """
+    print('Hello world')
+
+  def load_configuration(linter):
+
+    name_checker = get_checker(linter, NameChecker)
+    # We consider as good names of variables Hello and World
+    name_checker.config.good_names += ('Hello', 'World')
+
+    # We ignore bin directory
+    linter.config.black_list += ('bin',)
+
+Depending if we need a **transform plugin** or a **checker**, this might not
+be enough. For the former, this is enough to declare the module as a plugin,
+but in the case of the latter, we need to register our checker with the linter
+object, by calling the following inside the ``register`` function::
+
+    linter.register_checker(OurChecker(linter))
+
+For more information on writing a checker see :ref:`write_a_checker`.
diff --git a/doc/development_guide/how_tos/transform_plugins.rst b/doc/development_guide/how_tos/transform_plugins.rst
new file mode 100644
index 000000000..031faa0f1
--- /dev/null
+++ b/doc/development_guide/how_tos/transform_plugins.rst
@@ -0,0 +1,117 @@
+
+Transform plugins
+^^^^^^^^^^^^^^^^^
+
+Why write a plugin?
+-------------------
+
+Pylint is a static analysis tool and Python is a dynamically typed language.
+So there will be cases where Pylint cannot analyze files properly (this problem
+can happen in statically typed languages also if reflection or dynamic
+evaluation is used).
+
+The plugins are a way to tell Pylint how to handle such cases,
+since only the user would know what needs to be done. They are usually operating
+on the AST level, by modifying or changing it in a way which can ease its
+understanding by Pylint.
+
+Example
+-------
+
+Let us run Pylint on a module from the Python source: `warnings.py`_ and see what happens:
+
+.. sourcecode:: shell
+
+  amitdev$ pylint -E Lib/warnings.py
+  E:297,36: Instance of 'WarningMessage' has no 'message' member (no-member)
+  E:298,36: Instance of 'WarningMessage' has no 'filename' member (no-member)
+  E:298,51: Instance of 'WarningMessage' has no 'lineno' member (no-member)
+  E:298,64: Instance of 'WarningMessage' has no 'line' member (no-member)
+
+
+Did we catch a genuine error? Let's open the code and look at ``WarningMessage`` class:
+
+.. sourcecode:: python
+
+  class WarningMessage(object):
+
+    """Holds the result of a single showwarning() call."""
+
+    _WARNING_DETAILS = ("message", "category", "filename", "lineno", "file",
+                        "line")
+
+    def __init__(self, message, category, filename, lineno, file=None,
+                    line=None):
+      local_values = locals()
+      for attr in self._WARNING_DETAILS:
+        setattr(self, attr, local_values[attr])
+      self._category_name = category.__name__ if category else None
+
+    def __str__(self):
+      ...
+
+Ah, the fields (``message``, ``category`` etc) are not defined statically on the class.
+Instead they are added using ``setattr``. Pylint would have a tough time figuring
+this out.
+
+Enter Plugin
+------------
+
+We can write a transform plugin to tell Pylint how to analyze this properly.
+
+One way to fix our example with a plugin would be to transform the ``WarningMessage`` class,
+by setting the attributes so that Pylint can see them. This can be done by
+registering a transform function. We can transform any node in the parsed AST like
+Module, Class, Function etc. In our case we need to transform a class. It can be done so:
+
+.. sourcecode:: python
+
+  from typing import TYPE_CHECKING
+
+  import astroid
+
+  if TYPE_CHECKING:
+      from pylint.lint import PyLinter
+
+
+  def register(linter: "PyLinter") -> None:
+    """This required method auto registers the checker during initialization.
+
+    :param linter: The linter to register the checker to.
+    """
+    pass
+
+  def transform(cls):
+    if cls.name == 'WarningMessage':
+      import warnings
+      for f in warnings.WarningMessage._WARNING_DETAILS:
+        cls.locals[f] = [astroid.ClassDef(f, None)]
+
+  astroid.MANAGER.register_transform(astroid.ClassDef, transform)
+
+Let's go through the plugin. First, we need to register a class transform, which
+is done via the ``register_transform`` function in ``MANAGER``. It takes the node
+type and function as parameters. We need to change a class, so we use ``astroid.ClassDef``.
+We also pass a ``transform`` function which does the actual transformation.
+
+``transform`` function is simple as well. If the class is ``WarningMessage`` then we
+add the attributes to its locals (we are not bothered about type of attributes, so setting
+them as class will do. But we could set them to any type we want). That's it.
+
+Note: We don't need to do anything in the ``register`` function of the plugin since we
+are not modifying anything in the linter itself.
+
+Lets run Pylint with this plugin and see:
+
+.. sourcecode:: bash
+
+  amitdev$ pylint -E --load-plugins warning_plugin Lib/warnings.py
+  amitdev$
+
+All the false positives associated with ``WarningMessage`` are now gone. This is just
+an example, any code transformation can be done by plugins.
+
+See `astroid/brain`_ for real life examples of transform plugins.
+
+.. _`warnings.py`: https://hg.python.org/cpython/file/2.7/Lib/warnings.py
+.. _`astroid/brain`: https://github.com/PyCQA/astroid/tree/main/astroid/brain
diff --git a/doc/development_guide/index.rst b/doc/development_guide/index.rst
deleted file mode 100644
index 9eefc992c..000000000
--- a/doc/development_guide/index.rst
+++ /dev/null
@@ -1,11 +0,0 @@
-Development
-===========
-
-.. toctree::
-   :maxdepth: 2
-   :titlesonly:
-
-   contribute
-   tests/index
-   api/index
-   profiling
diff --git a/doc/development_guide/technical_reference/checkers.rst b/doc/development_guide/technical_reference/checkers.rst
new file mode 100644
index 000000000..de703ac05
--- /dev/null
+++ b/doc/development_guide/technical_reference/checkers.rst
@@ -0,0 +1,7 @@
+Checkers
+--------
+All of the default pylint checkers exist in ``pylint.checkers``.
+This is where most of pylint's brains exist.
+Most checkers are AST based and so use ``astroid``.
+``pylint.checkers.utils`` provides a large number of utility methods for
+dealing with ``astroid``.
diff --git a/doc/development_guide/technical_reference/index.rst b/doc/development_guide/technical_reference/index.rst
new file mode 100644
index 000000000..9ec307902
--- /dev/null
+++ b/doc/development_guide/technical_reference/index.rst
@@ -0,0 +1,16 @@
+.. _technical-reference:
+
+Technical Reference
+===================
+
+.. TODO Configuration
+.. TODO Messages
+.. TODO Reports
+.. extensions.rst and features.rst are generated.
+
+.. toctree::
+  :maxdepth: 2
+  :titlesonly:
+
+  startup
+  checkers
diff --git a/doc/development_guide/technical_reference/startup.rst b/doc/development_guide/technical_reference/startup.rst
new file mode 100644
index 000000000..22a395ad3
--- /dev/null
+++ b/doc/development_guide/technical_reference/startup.rst
@@ -0,0 +1,23 @@
+Startup and the Linter Class
+----------------------------
+
+The two main classes in ``pylint.lint`` are
+``.pylint.lint.Run`` and ``.pylint.lint.PyLinter``.
+
+The ``.pylint.lint.Run`` object is responsible for starting up pylint.
+It does some basic checking of the given command line options to
+find the initial hook to run,
+find the config file to use,
+and find which plugins have been specified.
+It can then create the main ``.pylint.lint.PyLinter`` instance
+and initialise it with the config file and plugins that were discovered
+when preprocessing the command line options.
+Finally the ``.pylint.lint.Run`` object launches any child linters
+for parallel jobs, and starts the linting process.
+
+The ``.pylint.lint.PyLinter`` is responsible for coordinating the
+linting process.
+It parses the configuration and provides it for the checkers and other plugins,
+it handles the messages emitted by the checkers,
+it handles the output reporting,
+and it launches the checkers.