Merge pull request #5678 from ArtFlag/docs/extension-tutorials

Docs: Extension tutorial portal + helloworld example

2 files changed, 110 insertions, 400 deletions

diff --git a/doc/extdev/index.rst b/doc/extdev/index.rst
index b5336373d..da0d0e2d0 100644
--- a/doc/extdev/index.rst
+++ b/doc/extdev/index.rst
@@ -52,6 +52,115 @@ Note that it is still necessary to register the builder using
 
 .. _entry points: https://setuptools.readthedocs.io/en/latest/setuptools.html#dynamic-discovery-of-services-and-plugins
 
+.. _important-objects:
+
+Important objects
+-----------------
+
+There are several key objects whose API you will use while writing an
+extension. These are:
+
+**Application**
+   The application object (usually called ``app``) is an instance of
+   :class:`.Sphinx`.  It controls most high-level functionality, such as the
+   setup of extensions, event dispatching and producing output (logging).
+
+   If you have the environment object, the application is available as
+   ``env.app``.
+
+**Environment**
+   The build environment object (usually called ``env``) is an instance of
+   :class:`.BuildEnvironment`.  It is responsible for parsing the source
+   documents, stores all metadata about the document collection and is
+   serialized to disk after each build.
+
+   Its API provides methods to do with access to metadata, resolving references,
+   etc.  It can also be used by extensions to cache information that should
+   persist for incremental rebuilds.
+
+   If you have the application or builder object, the environment is available
+   as ``app.env`` or ``builder.env``.
+
+**Builder**
+   The builder object (usually called ``builder``) is an instance of a specific
+   subclass of :class:`.Builder`.  Each builder class knows how to convert the
+   parsed documents into an output format, or otherwise process them (e.g. check
+   external links).
+
+   If you have the application object, the builder is available as
+   ``app.builder``.
+
+**Config**
+   The config object (usually called ``config``) provides the values of
+   configuration values set in :file:`conf.py` as attributes.  It is an instance
+   of :class:`.Config`.
+
+   The config is available as ``app.config`` or ``env.config``.
+
+To see an example of use of these objects, refer to :doc:`../development/tutorials/index`.
+
+.. _build-phases:
+
+Build Phases
+------------
+
+One thing that is vital in order to understand extension mechanisms is the way
+in which a Sphinx project is built: this works in several phases.
+
+**Phase 0: Initialization**
+
+   In this phase, almost nothing of interest to us happens.  The source
+   directory is searched for source files, and extensions are initialized.
+   Should a stored build environment exist, it is loaded, otherwise a new one is
+   created.
+
+**Phase 1: Reading**
+
+   In Phase 1, all source files (and on subsequent builds, those that are new or
+   changed) are read and parsed.  This is the phase where directives and roles
+   are encountered by docutils, and the corresponding code is executed.  The
+   output of this phase is a *doctree* for each source file; that is a tree of
+   docutils nodes.  For document elements that aren't fully known until all
+   existing files are read, temporary nodes are created.
+
+   There are nodes provided by docutils, which are documented `in the docutils
+   documentation <http://docutils.sourceforge.net/docs/ref/doctree.html>`__.
+   Additional nodes are provided by Sphinx and :ref:`documented here <nodes>`.
+
+   During reading, the build environment is updated with all meta- and cross
+   reference data of the read documents, such as labels, the names of headings,
+   described Python objects and index entries.  This will later be used to
+   replace the temporary nodes.
+
+   The parsed doctrees are stored on the disk, because it is not possible to
+   hold all of them in memory.
+
+**Phase 2: Consistency checks**
+
+   Some checking is done to ensure no surprises in the built documents.
+
+**Phase 3: Resolving**
+
+   Now that the metadata and cross-reference data of all existing documents is
+   known, all temporary nodes are replaced by nodes that can be converted into
+   output using components called tranform.  For example, links are created for
+   object references that exist, and simple literal nodes are created for those
+   that don't.
+
+**Phase 4: Writing**
+
+   This phase converts the resolved doctrees to the desired output format, such
+   as HTML or LaTeX.  This happens via a so-called docutils writer that visits
+   the individual nodes of each doctree and produces some output in the process.
+
+.. note::
+
+   Some builders deviate from this general build plan, for example, the builder
+   that checks external links does not need anything more than the parsed
+   doctrees and therefore does not have phases 2--4.
+
+To see an example of application, refer to :doc:`../development/tutorials/todo`.
+
 .. _ext-metadata:
 
 Extension metadata
@@ -82,8 +191,8 @@ APIs used for writing extensions
 --------------------------------
 
 .. toctree::
+   :maxdepth: 2
 
-   tutorial
    appapi
    projectapi
    envapi
diff --git a/doc/extdev/tutorial.rst b/doc/extdev/tutorial.rst
deleted file mode 100644
index 33b45035e..000000000
--- a/doc/extdev/tutorial.rst
+++ /dev/null
@@ -1,399 +0,0 @@
-.. _exttut:
-
-Tutorial: Writing a simple extension
-====================================
-
-This section is intended as a walkthrough for the creation of custom extensions.
-It covers the basics of writing and activating an extension, as well as
-commonly used features of extensions.
-
-As an example, we will cover a "todo" extension that adds capabilities to
-include todo entries in the documentation, and to collect these in a central
-place.  (A similar "todo" extension is distributed with Sphinx.)
-
-
-Important objects
------------------
-
-There are several key objects whose API you will use while writing an
-extension.  These are:
-
-**Application**
-   The application object (usually called ``app``) is an instance of
-   :class:`.Sphinx`.  It controls most high-level functionality, such as the
-   setup of extensions, event dispatching and producing output (logging).
-
-   If you have the environment object, the application is available as
-   ``env.app``.
-
-**Environment**
-   The build environment object (usually called ``env``) is an instance of
-   :class:`.BuildEnvironment`.  It is responsible for parsing the source
-   documents, stores all metadata about the document collection and is
-   serialized to disk after each build.
-
-   Its API provides methods to do with access to metadata, resolving references,
-   etc.  It can also be used by extensions to cache information that should
-   persist for incremental rebuilds.
-
-   If you have the application or builder object, the environment is available
-   as ``app.env`` or ``builder.env``.
-
-**Builder**
-   The builder object (usually called ``builder``) is an instance of a specific
-   subclass of :class:`.Builder`.  Each builder class knows how to convert the
-   parsed documents into an output format, or otherwise process them (e.g. check
-   external links).
-
-   If you have the application object, the builder is available as
-   ``app.builder``.
-
-**Config**
-   The config object (usually called ``config``) provides the values of
-   configuration values set in :file:`conf.py` as attributes.  It is an instance
-   of :class:`.Config`.
-
-   The config is available as ``app.config`` or ``env.config``.
-
-
-Build Phases
-------------
-
-One thing that is vital in order to understand extension mechanisms is the way
-in which a Sphinx project is built: this works in several phases.
-
-**Phase 0: Initialization**
-
-   In this phase, almost nothing of interest to us happens.  The source
-   directory is searched for source files, and extensions are initialized.
-   Should a stored build environment exist, it is loaded, otherwise a new one is
-   created.
-
-**Phase 1: Reading**
-
-   In Phase 1, all source files (and on subsequent builds, those that are new or
-   changed) are read and parsed.  This is the phase where directives and roles
-   are encountered by docutils, and the corresponding code is executed.  The
-   output of this phase is a *doctree* for each source file; that is a tree of
-   docutils nodes.  For document elements that aren't fully known until all
-   existing files are read, temporary nodes are created.
-
-   There are nodes provided by docutils, which are documented `in the docutils
-   documentation <http://docutils.sourceforge.net/docs/ref/doctree.html>`__.
-   Additional nodes are provided by Sphinx and :ref:`documented here <nodes>`.
-
-   During reading, the build environment is updated with all meta- and cross
-   reference data of the read documents, such as labels, the names of headings,
-   described Python objects and index entries.  This will later be used to
-   replace the temporary nodes.
-
-   The parsed doctrees are stored on the disk, because it is not possible to
-   hold all of them in memory.
-
-**Phase 2: Consistency checks**
-
-   Some checking is done to ensure no surprises in the built documents.
-
-**Phase 3: Resolving**
-
-   Now that the metadata and cross-reference data of all existing documents is
-   known, all temporary nodes are replaced by nodes that can be converted into
-   output using components called tranform.  For example, links are created for
-   object references that exist, and simple literal nodes are created for those
-   that don't.
-
-**Phase 4: Writing**
-
-   This phase converts the resolved doctrees to the desired output format, such
-   as HTML or LaTeX.  This happens via a so-called docutils writer that visits
-   the individual nodes of each doctree and produces some output in the process.
-
-.. note::
-
-   Some builders deviate from this general build plan, for example, the builder
-   that checks external links does not need anything more than the parsed
-   doctrees and therefore does not have phases 2--4.
-
-
-Extension Design
-----------------
-
-We want the extension to add the following to Sphinx:
-
-* A "todo" directive, containing some content that is marked with "TODO", and
-  only shown in the output if a new config value is set.  (Todo entries should
-  not be in the output by default.)
-
-* A "todolist" directive that creates a list of all todo entries throughout the
-  documentation.
-
-For that, we will need to add the following elements to Sphinx:
-
-* New directives, called ``todo`` and ``todolist``.
-* New document tree nodes to represent these directives, conventionally also
-  called ``todo`` and ``todolist``.  We wouldn't need new nodes if the new
-  directives only produced some content representable by existing nodes.
-* A new config value ``todo_include_todos`` (config value names should start
-  with the extension name, in order to stay unique) that controls whether todo
-  entries make it into the output.
-* New event handlers: one for the :event:`doctree-resolved` event, to replace
-  the todo and todolist nodes, and one for :event:`env-purge-doc` (the reason
-  for that will be covered later).
-
-
-The Setup Function
-------------------
-
-.. currentmodule:: sphinx.application
-
-The new elements are added in the extension's setup function.  Let us create a
-new Python module called :file:`todo.py` and add the setup function::
-
-   def setup(app):
-       app.add_config_value('todo_include_todos', False, 'html')
-
-       app.add_node(todolist)
-       app.add_node(todo,
-                    html=(visit_todo_node, depart_todo_node),
-                    latex=(visit_todo_node, depart_todo_node),
-                    text=(visit_todo_node, depart_todo_node))
-
-       app.add_directive('todo', TodoDirective)
-       app.add_directive('todolist', TodolistDirective)
-       app.connect('doctree-resolved', process_todo_nodes)
-       app.connect('env-purge-doc', purge_todos)
-
-       return {'version': '0.1'}   # identifies the version of our extension
-
-The calls in this function refer to classes and functions not yet written.  What
-the individual calls do is the following:
-
-* :meth:`~Sphinx.add_config_value` lets Sphinx know that it should recognize the
-  new *config value* ``todo_include_todos``, whose default value should be
-  ``False`` (this also tells Sphinx that it is a boolean value).
-
-  If the third argument was ``'html'``, HTML documents would be full rebuild if the
-  config value changed its value.  This is needed for config values that
-  influence reading (build phase 1).
-
-* :meth:`~Sphinx.add_node` adds a new *node class* to the build system.  It also
-  can specify visitor functions for each supported output format.  These visitor
-  functions are needed when the new nodes stay until phase 4 -- since the
-  ``todolist`` node is always replaced in phase 3, it doesn't need any.
-
-  We need to create the two node classes ``todo`` and ``todolist`` later.
-
-* :meth:`~Sphinx.add_directive` adds a new *directive*, given by name and class.
-
-  The handler functions are created later.
-
-* Finally, :meth:`~Sphinx.connect` adds an *event handler* to the event whose
-  name is given by the first argument.  The event handler function is called
-  with several arguments which are documented with the event.
-
-
-The Node Classes
-----------------
-
-Let's start with the node classes::
-
-   from docutils import nodes
-
-   class todo(nodes.Admonition, nodes.Element):
-       pass
-
-   class todolist(nodes.General, nodes.Element):
-       pass
-
-   def visit_todo_node(self, node):
-       self.visit_admonition(node)
-
-   def depart_todo_node(self, node):
-       self.depart_admonition(node)
-
-Node classes usually don't have to do anything except inherit from the standard
-docutils classes defined in :mod:`docutils.nodes`.  ``todo`` inherits from
-``Admonition`` because it should be handled like a note or warning, ``todolist``
-is just a "general" node.
-
-.. note::
-
-   Many extensions will not have to create their own node classes and work fine
-   with the nodes already provided by `docutils
-   <http://docutils.sourceforge.net/docs/ref/doctree.html>`__ and :ref:`Sphinx
-   <nodes>`.
-
-
-The Directive Classes
----------------------
-
-A directive class is a class deriving usually from
-:class:`docutils.parsers.rst.Directive`.  The directive interface is also
-covered in detail in the `docutils documentation`_; the important thing is that
-the class should have attributes that configure the allowed markup,
-and a ``run`` method that returns a list of nodes.
-
-The ``todolist`` directive is quite simple::
-
-   from docutils.parsers.rst import Directive
-
-   class TodolistDirective(Directive):
-
-       def run(self):
-           return [todolist('')]
-
-An instance of our ``todolist`` node class is created and returned.  The
-todolist directive has neither content nor arguments that need to be handled.
-
-The ``todo`` directive function looks like this::
-
-   from sphinx.locale import _
-
-   class TodoDirective(Directive):
-
-       # this enables content in the directive
-       has_content = True
-
-       def run(self):
-           env = self.state.document.settings.env
-
-           targetid = "todo-%d" % env.new_serialno('todo')
-           targetnode = nodes.target('', '', ids=[targetid])
-
-           todo_node = todo('\n'.join(self.content))
-           todo_node += nodes.title(_('Todo'), _('Todo'))
-           self.state.nested_parse(self.content, self.content_offset, todo_node)
-
-           if not hasattr(env, 'todo_all_todos'):
-               env.todo_all_todos = []
-           env.todo_all_todos.append({
-               'docname': env.docname,
-               'lineno': self.lineno,
-               'todo': todo_node.deepcopy(),
-               'target': targetnode,
-           })
-
-           return [targetnode, todo_node]
-
-Several important things are covered here. First, as you can see, you can refer
-to the build environment instance using ``self.state.document.settings.env``.
-
-Then, to act as a link target (from the todolist), the todo directive needs to
-return a target node in addition to the todo node.  The target ID (in HTML, this
-will be the anchor name) is generated by using ``env.new_serialno`` which
-returns a new unique integer on each call and therefore leads to unique target
-names.  The target node is instantiated without any text (the first two
-arguments).
-
-On creating admonition node, the content body of the directive are parsed using
-``self.state.nested_parse``.  The first argument gives the content body, and
-the second one gives content offset.  The third argument gives the parent node
-of parsed result, in our case the ``todo`` node.
-
-Then, the todo node is added to the environment.  This is needed to be able to
-create a list of all todo entries throughout the documentation, in the place
-where the author puts a ``todolist`` directive.  For this case, the environment
-attribute ``todo_all_todos`` is used (again, the name should be unique, so it is
-prefixed by the extension name).  It does not exist when a new environment is
-created, so the directive must check and create it if necessary.  Various
-information about the todo entry's location are stored along with a copy of the
-node.
-
-In the last line, the nodes that should be put into the doctree are returned:
-the target node and the admonition node.
-
-The node structure that the directive returns looks like this::
-
-   +--------------------+
-   | target node        |
-   +--------------------+
-   +--------------------+
-   | todo node          |
-   +--------------------+
-     \__+--------------------+
-        | admonition title   |
-        +--------------------+
-        | paragraph          |
-        +--------------------+
-        | ...                |
-        +--------------------+
-
-
-The Event Handlers
-------------------
-
-Finally, let's look at the event handlers.  First, the one for the
-:event:`env-purge-doc` event::
-
-   def purge_todos(app, env, docname):
-       if not hasattr(env, 'todo_all_todos'):
-           return
-       env.todo_all_todos = [todo for todo in env.todo_all_todos
-                             if todo['docname'] != docname]
-
-Since we store information from source files in the environment, which is
-persistent, it may become out of date when the source file changes.  Therefore,
-before each source file is read, the environment's records of it are cleared,
-and the :event:`env-purge-doc` event gives extensions a chance to do the same.
-Here we clear out all todos whose docname matches the given one from the
-``todo_all_todos`` list.  If there are todos left in the document, they will be
-added again during parsing.
-
-The other handler belongs to the :event:`doctree-resolved` event.  This event is
-emitted at the end of phase 3 and allows custom resolving to be done::
-
-   def process_todo_nodes(app, doctree, fromdocname):
-       if not app.config.todo_include_todos:
-           for node in doctree.traverse(todo):
-               node.parent.remove(node)
-
-       # Replace all todolist nodes with a list of the collected todos.
-       # Augment each todo with a backlink to the original location.
-       env = app.builder.env
-
-       for node in doctree.traverse(todolist):
-           if not app.config.todo_include_todos:
-               node.replace_self([])
-               continue
-
-           content = []
-
-           for todo_info in env.todo_all_todos:
-               para = nodes.paragraph()
-               filename = env.doc2path(todo_info['docname'], base=None)
-               description = (
-                   _('(The original entry is located in %s, line %d and can be found ') %
-                   (filename, todo_info['lineno']))
-               para += nodes.Text(description, description)
-
-               # Create a reference
-               newnode = nodes.reference('', '')
-               innernode = nodes.emphasis(_('here'), _('here'))
-               newnode['refdocname'] = todo_info['docname']
-               newnode['refuri'] = app.builder.get_relative_uri(
-                   fromdocname, todo_info['docname'])
-               newnode['refuri'] += '#' + todo_info['target']['refid']
-               newnode.append(innernode)
-               para += newnode
-               para += nodes.Text('.)', '.)')
-
-               # Insert into the todolist
-               content.append(todo_info['todo'])
-               content.append(para)
-
-           node.replace_self(content)
-
-It is a bit more involved.  If our new "todo_include_todos" config value is
-false, all todo and todolist nodes are removed from the documents.
-
-If not, todo nodes just stay where and how they are.  Todolist nodes are
-replaced by a list of todo entries, complete with backlinks to the location
-where they come from.  The list items are composed of the nodes from the todo
-entry and docutils nodes created on the fly: a paragraph for each entry,
-containing text that gives the location, and a link (reference node containing
-an italic node) with the backreference.  The reference URI is built by
-``app.builder.get_relative_uri`` which creates a suitable URI depending on the
-used builder, and appending the todo node's (the target's) ID as the anchor
-name.
-
-.. _docutils documentation: http://docutils.sourceforge.net/docs/ref/rst/directives.html