sandbox/code-block-directive: update docs and example:

the "latex2e" bug is fixed!

git-svn-id: http://svn.code.sf.net/p/docutils/code/trunk@5870 929543f6-e4f2-0310-98a6-ba3bd3dd1d04

3 files changed, 179 insertions, 166 deletions

diff --git a/sandbox/code-block-directive/README.txt b/sandbox/code-block-directive/README.txt
index b13237e85..6ffe7e4dd 100644
--- a/sandbox/code-block-directive/README.txt
+++ b/sandbox/code-block-directive/README.txt
@@ -4,34 +4,34 @@
 Proposal for a code-block directive in docutils
 ===============================================
 
-:Author: Guenter Milde <milde@users.berlios.de>
+:Author: Günter Milde
+:Contact: milde@users.berlios.de
 :Date: $Date$
 
-
 This sandbox project contains experimental code and documentation related to
 the proposal for syntax highlight of source code in docutils using a 
 "code-block" directive.
 
-See `syntax-highlight.html`_ for the full picture.
-
-`pygments_code_block_directive.py`_ 
-   Working example: defines and registers a
-   code-block directive using the Pygments_  syntax highlighter. 
-   
-   colourful literal code (maybe outdated):
-   `pygments_code_block_directive-bunt.py.htm`_
+See `<docs/syntax-highlight.html>`_ for a full description.
 
-`rst2html-highlight`_ 
+`<rst2html-highlight>`_ 
    front end for reStructuredText -> HTML conversion supporting the
    "code-block" directive.
 
-data_
+`<data>`_
    Style sheets
 
-docs_
+`<docs>`_
    Documentation, concepts, discussion, examples...
+
+   `<docs/pygments_code_block_directive.py>`_ 
+     Working example: defines and registers a
+     code-block directive using the Pygments_  syntax highlighter. 
+     
+   `<docs/pygments_code_block_directive-bunt.py.htm>`_
+     Colourful literal code (maybe outdated).
    
-tools_
+`<tools>`_
    Alternative (legacy) front ends, 
    script for interactive testing.
 
@@ -40,15 +40,6 @@ tools_
 
 .. _pygments: http://pygments.org/
 
-.. _pygments_code_block_directive.py: pygments_code_block_directive.py
-.. _pygments_code_block_directive-bunt.py.htm:
-     docs/pygments_code_block_directive-bunt.py.htm
-.. _rst2html-highlight: rst2html-highlight
-.. _data: data
-.. _docs: docs
-.. _tools: tools
-.. _syntax-highlight.html: docs/syntax-highlight.html
-
 
 
 
diff --git a/sandbox/code-block-directive/docs/myfunction.py.tex b/sandbox/code-block-directive/docs/myfunction.py.tex
index d8c934c7a..689a02703 100644
--- a/sandbox/code-block-directive/docs/myfunction.py.tex
+++ b/sandbox/code-block-directive/docs/myfunction.py.tex
@@ -1,20 +1,20 @@
-\documentclass[10pt,a4paper,english]{article}
+\documentclass[10pt,a4paper,english]{scrartcl}
 \usepackage{babel}
-\usepackage{ae}
-\usepackage{aeguill}
+\usepackage[T1]{fontenc}
 \usepackage{shortvrb}
-\usepackage[latin1]{inputenc}
+\usepackage{ucs}
+\usepackage[utf8x]{inputenc}
 \usepackage{tabularx}
 \usepackage{longtable}
+\usepackage{booktabs}
 \setlength{\extrarowheight}{2pt}
 \usepackage{amsmath}
 \usepackage{graphicx}
 \usepackage{color}
 \usepackage{multirow}
 \usepackage{ifthen}
-\usepackage[colorlinks=true,linkcolor=blue,urlcolor=blue]{hyperref}
-\usepackage[DIV12]{typearea}
-%% generator Docutils: http://docutils.sourceforge.net/
+\typearea{12}
+% generated by Docutils <http://docutils.sourceforge.net/>
 \newlength{\admonitionwidth}
 \setlength{\admonitionwidth}{0.9\textwidth}
 \newlength{\docinfowidth}
@@ -55,6 +55,20 @@
 \newcommand{\rubric}[1]{\subsection*{~\hfill {\it #1} \hfill ~}}
 \newcommand{\titlereference}[1]{\textsl{#1}}
 % end of "some commands"
+% user specified packages and stylesheets:
+\usepackage{cmlgc}
+\ifthenelse{\isundefined{\hypersetup}}{
+\usepackage[colorlinks=true,linkcolor=blue,urlcolor=blue]{hyperref}
+}{}
+% Custom roles: 
+% \DUrole{NAME}{content} calls \docutilsroleNAME if it exists
+\providecommand{\DUrole}[2]{%
+  \ifcsname docutilsrole#1\endcsname%
+    \csname docutilsrole#1\endcsname{#2}%
+  \else%
+    #2%
+  \fi%
+}
 \title{}
 \author{}
 \date{}
@@ -68,17 +82,10 @@ This is a test of the new code-block directive:
 % 
 % rst2html-highlight --stylesheet=pygments-default.css --link-stylesheet 
 \begin{quote}{\ttfamily \raggedright \noindent
-\docutilsroleNone{def}~\docutilsroleNone{my{\_}function}\docutilsroleNone{():}~\\
-~~~~\docutilsroleNone{"just~a~test"}~\\
-~~~~\docutilsroleNone{print}~\docutilsroleNone{8}\docutilsroleNone{/}\docutilsroleNone{2}~\\
+\DUrole{k}{def}~\DUrole{nf}{my{\_}function}\DUrole{p}{():}~\\
+~~~~\DUrole{s}{"just~a~test"}~\\
+~~~~\DUrole{k}{print}~\DUrole{mf}{8}\DUrole{o}{/}\DUrole{mf}{2}~\\
 
 }\end{quote}
 
-\begin{center}\small
-
-Generated on: 2007-06-05.
-
-
-\end{center}
-
 \end{document}
diff --git a/sandbox/code-block-directive/docs/syntax-highlight.txt b/sandbox/code-block-directive/docs/syntax-highlight.txt
index 54f6ac794..a5d6ffc34 100644
--- a/sandbox/code-block-directive/docs/syntax-highlight.txt
+++ b/sandbox/code-block-directive/docs/syntax-highlight.txt
@@ -3,11 +3,17 @@
 Syntax Highlight
 ================
 
-.. contents::
+:Author:   Günter Milde
+:Contact:  milde@users.berlios.de
+:Date: 	   $Date$
+:Abstract: Proposal to add syntax highlight of code blocks to the
+	   capabilities of Docutils_.
+
 .. sectnum::
+.. contents::
 
 Syntax highlighting significantly enhances the readability of code. However,
-in the current version, docutils does not highlight literal blocks. 
+in the current version, docutils does not highlight literal blocks.
 
 This sandbox project aims to add syntax highlight of code blocks to the
 capabilities of docutils. To find its way into the docutils core, it should
@@ -30,38 +36,47 @@ State of the art
 
 There are already docutils extensions providing syntax colouring, e.g:
 
-SilverCity_, 
+SilverCity_,
   a C++ library and Python extension that can provide lexical
   analysis for over 20 different programming languages. A recipe__ for a
   "code-block" directive provides syntax highlight by SilverCity.
-  
+
 __ http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/252170
 
 `listings`_,
   a LaTeX package providing highly customisable and advanced syntax
-  highlight, though only for LaTeX (and LaTeX derived PS|PDF).  
-  
-  Since Docutils 0.5, the latex2e writer (and rst2latex.py) support
+  highlight, though only for LaTeX (and LaTeX derived PS|PDF).
+
+  Since Docutils 0.5, the "latex2e" writer supports
   syntax highlight of literal blocks by `listings` with the
-  ``--literal-block-env=listings`` option. You need to provide a
-  stylesheet (see also `Pylit Examples`_)
-  
-.. _Pylit Examples: http://pylit.berlios.de/examples/index.html#latex-packages
-  
-Trac_ 
+  ``--literal-block-env=lstlistings`` option. You need to provide a
+  stylesheet (see also `PyLit Examples`_).
+
+Trac_
   has `reStructuredText support`__ and offers syntax highlighting with
   a "code-block" directive using GNU Enscript_, SilverCity_, or Pygments_.
-  
+
 __ http://trac.edgewall.org/wiki/WikiRestructuredText
 
 rest2web_,
   the "site builder" provides the `colorize`__ macro (using the
-  `Moin-Moin Python colorizer`_) 
+  `Moin-Moin Python colorizer`_)
 
 __ http://www.voidspace.org.uk/python/rest2web/macros.html#colorize
 
-Pygments_ 
-  is a generic syntax highlighter written completely in Python. 
+
+Sphinx_
+  features automatic highlighting using the Pygments_ highlighter.
+  It introduces the custom directives 
+  
+  :code-block: similar to the proposal below,
+  :sourcecode: an alias to "code-block", and
+  :highlight:  configre highlight of "literal blocks".
+  
+  (see http://sphinx.pocoo.org/markup/code.html).
+
+Pygments_
+  is a generic syntax highlighter written completely in Python.
 
   * Usable as a command-line tool and as a Python package.
   * A wide range of common `languages and markup formats`_ is supported.
@@ -76,7 +91,7 @@ Pygments_
     It is used in the `Pygments enhanced docutils front-ends`_ below.
 
 Odtwriter_, experimental writer for Docutils OpenOffice export supports syntax
-  colours using Pygments_. (See (the outdated) section `Odtwriter syntax`_.)
+  colours using Pygments_. (See the (outdated) section `Odtwriter syntax`_.)
 
 Pygments_ seems to be the most promising docutils highlighter. For printed
 output, the listings_ package has its advantages too.
@@ -91,7 +106,7 @@ pygments.
    "something users [will have to] put together themselves"
 
 Advantages:
-  + Easy implementation with no changes to the stock docutils_. 
+  + Easy implementation with no changes to the stock docutils_.
   + Separation of code blocks and ordinary literal blocks.
 
 Disadvantages:
@@ -103,7 +118,7 @@ Disadvantages:
      (no "minimal" code block marker -- three additional lines per code block)
 
 
-Point 1 and 2 lead to the `code-block directive proposal`_. 
+Point 1 and 2 lead to the `code-block directive proposal`_.
 
 Point 3 becomes an issue in literate programming where a code block is
 the most used block markup. It is addressed in the proposal for a
@@ -116,8 +131,8 @@ the most used block markup. It is addressed in the proposal for a
 Syntax
 """"""
 
-This is the first draft for a reStructuredText definition, analogue to
-other directives in ``directives.txt``.
+.. note:: This is the first draft for a reStructuredText definition,
+   	  analogue to other directives in ``directives.txt``.
 
 :Directive Type: "code-block"
 :Doctree Element: literal_block
@@ -127,32 +142,38 @@ other directives in ``directives.txt``.
 
 The "code-block" directive constructs a literal block where the content is
 parsed as source code and syntax highlight rules for `language` are
-applied. If syntax rules for `language` are not known to docutils, it
+applied. If syntax rules for `language` are not known to Docutils, it
 is rendered like an ordinary literal block.
 
-For example, the following snipped is parsed and marked up as python source 
-code. The actual rendering depends on the style-sheet. ::
+Example:
+  A bit of Python code ::
 
-  .. code-block:: python
+   .. code-block:: python
 
     def my_function():
         "just a test"
         print 8/2
 
-If the language argument is missing, a (configurable) default language
-should be used. 
+  The directive content will be parsed and marked up as Python source
+  code. The actual rendering depends on the style-sheet.
+
+Remarks:
+
+* If the language argument is missing, a (configurable) default language
+  should be used.
 
-Additional arguments might be used and e.g. passed to
-the pygments_ parser or (as class arguments) the output document.
+* Additional arguments might be defined and passed to the pygments_
+  parser or the output document (as class arguments), e.g.
+
+  :number-lines: let pygments include line-numbers
 
-:number-lines: let pygments include line-numbers
 
 Include directive option
-"""""""""""""""""""""""
+""""""""""""""""""""""""
 
 The include directive should get a matching new option:
 
-code : language 
+code: language
   The entire included text is inserted into the document as if it were the
   content of a code-block directive (useful for program listings).
 
@@ -178,8 +199,8 @@ directive.
 * The ``code_block_directive`` function inserts the tokens in a "rich"
   <literal_block> element with "classified" <inline> nodes.
 
-The XML rendering of the small example file `myfunction.py.txt`_ looks like
-`myfunction.py.xml`_.
+The XML rendering of the small example file `<myfunction.py.txt>`_ looks like
+`<myfunction.py.xml>`_.
 
 Writing
 '''''''
@@ -189,84 +210,72 @@ the tokens. They should ignore the class information if they are unable to
 use it or to pass it on.
 
 HTML
-  The "html" writer works out of the box. 
+  The "html" writer works out of the box.
 
   * The rst2html-highlight_ front end registers the "code-block" directive and
-    converts an input file to html. 
+    converts an input file to html.
   * Styling is done with the adapted CSS style sheet `pygments-default.css`_
-    based on docutils' default stylesheet and the output of 
+    based on docutils' default stylesheet and the output of
     ``pygmentize -S default -f html``.
-  * The result looks like `myfunction.py.htm`_.
-  
+    
+  The conversion of `myfunction.py.txt`_ looks like
+  `<myfunction.py.htm>`_.
+
   The "s5" and "pep" writers are not tested yet.
+
+XML
+  "xml" and "pseudoxml" work out of the box. 
   
-XML 
-  "xml" and "pseudoxml" work out of the box, too. See `myfunction.py.xml`_
-  and `myfunction.py.pseudoxml`_
+  The conversion of `myfunction.py.txt`_ looks like
+  `<myfunction.py.xml>`_ and `<myfunction.py.pseudoxml>`_
 
 LaTeX
-  Latex writers must be fixed to handle the "rich" <literal_block> element
-  correct:
+  "latex2e" (SVN version) works out of the box. 
   
-  * The "latex" writer currently fails to handle "classified" <inline>
-    doctree elements. The output `myfunction.py.tex`_ contains undefined
-    control sequences ``\docutilsroleNone``.
+  * A style file is required to actually highlight the code in the
+    output. (As with HTML, the pygments-produced style file will not work
+    with docutils' output.)
   
-  * The "newlatex2e" writer produces a valid LaTeX document
-    (`myfunction.py.newlatex2e.tex`_). However there is some garbage in the
-    `pdflatex` output (`myfunction.py.newlatex2e.pdf`_). 
-    
-  The pygments-produced LaTeX style file will not work with docutils' LaTeX
-  output. The writer(s) will need to 
+  * Alternatively, the latex writer could reconstruct the original
+    content and pass it to a ``lstlistings`` environment. 
   
-  a) be extended in order to convert the "classified" <inline> doctree
-     elements into LaTeX styling instructions, or
-     
-  b) reconstruct the original content and pass it to a ``lstlistings``
-     environment.
+    TODO: This should be the default behaviour with
+    ``--literal-block-env=lstlistings``.
     
-OpenOffice 
-  The non-official "odtwriter" provides syntax highlight with
-  pygments but uses a different syntax and implementation.
+  The conversion of `myfunction.py.txt`_ looks like
+  `<myfunction.py.tex>`_ and converted with pdflatex (without style
+  sheet) like `<myfunction.py.pdf>`_
+
+  "newlatex2e" produces a valid LaTeX document
+  (`<myfunction.py.newlatex2e.tex>`_).  There is some garbage in the
+  pdflatex output (`<myfunction.py.newlatex2e.pdf>`_) but this might be
+  unrelated to the code-block directive.
+
+OpenOffice
+  The sandbox project `odtwriter` provided syntax highlight with
+  pygments but used a different syntax and implementation.
+  
+  (What is the status of the official `odtwriter` now included in the 
+  standard distribution?)
 
 
 TODO
 """"
 
-1. Let the latex2e writer ignore unknown class arguments.
-
-2. Write functional test case and sample.
-
-3. Minimal implementation: 
+1. Minimal implementation:
 
    * move the code from `pygments_code_block_directive.py`_ to "the right
      place".
-     
+
    * add the CSS rules to the default style-sheet (see pygments-default.css_)
    
-4. Enable the latex2e writers to highlight code-blocks (more precise: to
-   write LaTeX code that lets the latex engine highlight them).
-
-5. Think about an interface for pygments' options (like "encoding" or
-  "linenumbers").
+   * provide a LaTeX style.
 
+2. Write functional test case and sample.
 
 
-.. _proof of concept:
-     http://article.gmane.org/gmane.text.docutils.user/3689
-.. _pygments_code_block_directive.py: ../pygments_code_block_directive.py
-.. _pygments_code_block_directive: pygments_code_block_directive-bunt.py.htm
-.. _pygments_docutils_interface.py: pygments_docutils_interface.py
-.. _myfunction.py.txt: myfunction.py.txt
-.. _myfunction.py.xml: myfunction.py.xml
-.. _myfunction.py.htm: myfunction.py.htm
-.. _myfunction.py.pseudoxml: myfunction.py.pseudoxml
-.. _myfunction.py.tex: myfunction.py.tex
-.. _myfunction.py.newlatex2e.tex: myfunction.py.newlatex2e.tex
-.. _myfunction.py.newlatex2e.pdf: myfunction.py.newlatex2e.pdf
-.. _rst2html-highlight: ../rst2html-highlight
-.. _pygments-long.css: ../data/pygments-long.css
-
+3. Think about an interface for pygments' options (like "encoding" or
+   "linenumbers").
 
 
 Configurable literal block directive
@@ -277,7 +286,7 @@ Goal
 
 A clean and simple syntax for highlighted code blocks -- preserving the
 space saving feature of the "minimised" literal block marker (``::`` at the
-end of a text paragraph). This is especially desirable in documents with 
+end of a text paragraph). This is especially desirable in documents with
 many code blocks like tutorials or literate programs.
 
 Inline analogon
@@ -289,13 +298,13 @@ syntax for the most often used role, e.g. in a chemical paper, one could
 use::
 
   .. default-role:: subscript
-  
+
   The triple point of H\ `2`\O is at 0°C.
 
 .. default-role:: subscript
 
-to produce 
-  
+to produce
+
   The triple point of H\ `2`\O is at 0°C.
 
 This customisation is currently not possible for block markup.
@@ -327,100 +336,106 @@ used for e.g.
 Example::
 
   ordinary literal block::
-  
+
      some text typeset in monospace
 
   .. default-literal-block::  code-block python
-     
+
   this is colourful Python code::
-     
+
      def hello():
          print "hello world"
 
-  
+
 In the same line, a "default-block-quote" setting or directive could be
 considered to configure the role of a block quote.
 
 Odtwriter syntax
 ----------------
 
+.. attention::
+   The content of this section relates to an old version of the 
+   `odtwriter`. Things changed with the inclusion of the `odtwriter` into
+   standard Docutils.
+   
+   This is only kept for historical reasons.
+
 Dave Kuhlman's odtwriter_ extension can add syntax highlighting
 to ordinary literal blocks.
 
-.. attention:: 
-   The following might no longer be true for the current version of
-   the odtwriter_.
-
 The ``--add-syntax-highlighting`` command line flag activates syntax
 highlighting in literal blocks. By default, the "python" lexer is used.
 
 You can change this within your reST document with the `sourcecode`
 directive::
-  
+
   .. sourcecode:: off
-  
+
   ordinary literal block::
-  
+
      content set in teletype
 
   .. sourcecode:: on
   .. sourcecode:: python
-     
+
   colourful Python code::
-     
+
      def hello():
          print "hello world"
 
 
 The "sourcecode" directive defined by the odtwriter is principally
 different from the "code-block" directive of ``rst2html-pygments``:
-  
+
 * The odtwriter directive does not have content. It is a switch.
 
 * The syntax highlighting state and language/lexer set by this directive
   remain in effect until the next sourcecode directive is encountered in the
   reST document.
-  
-  ``.. sourcecode:: <newstate>`` 
-       make highlighting active or inactive. 
+
+  ``.. sourcecode:: <newstate>``
+       make highlighting active or inactive.
        <newstate> is either ``on`` or ``off``.
-  
-  ``.. sourcecode:: <lexer>`` 
+
+  ``.. sourcecode:: <lexer>``
        change the lexer parsing literal code blocks.
        <lexer> should be one of aliases listed at pygment's `languages and
        markup formats`_.
 
 I.e. the odtwriter implements a `configurable literal block directive`_
-(but with a slightly different syntax than my proposal below).
+(but with a slightly different syntax than the proposal above).
 
 
-.. External links 
-.. _pylit: http://pylit.berlios.de
-.. _docutils: http://docutils.sourceforge.net/
+.. External links
 .. _rest2web: http://www.voidspace.org.uk/python/rest2web/
 .. _Enscript: http://www.gnu.org/software/enscript/enscript.html
 .. _SilverCity: http://silvercity.sourceforge.net/
 .. _Trac: http://trac.edgewall.org/
-.. _Moin-Moin Python colorizer: 
+.. _Moin-Moin Python colorizer:
     http://www.standards-schmandards.com/2005/fangs-093/
 .. _odtwriter: http://www.rexx.com/~dkuhlman/odtwriter.html
-.. _pygments: http://pygments.org/
-.. _listings: 
+.. _Sphinx: http://sphinx.pocoo.org
+.. _listings:
     http://www.ctan.org/tex-archive/help/Catalogue/entries/listings.html
-.. _fancyvrb: 
-    http://www.ctan.org/tex-archive/help/Catalogue/entries/fancyvrb.html
-.. _alltt: http://www.ctan.org/tex-archive/help/Catalogue/entries/alltt.html
-.. _moreverb:
-    http://www.ctan.org/tex-archive/help/Catalogue/entries/moreverb.html
-.. _verbatim: 
-    http://www.ctan.org/tex-archive/help/Catalogue/entries/verbatim.html
+.. _PyLit: http://pylit.berlios.de
+.. _PyLit Examples: http://pylit.berlios.de/examples/index.html#latex-packages
+
+.. _Pygments: http://pygments.org/
 .. _languages and markup formats: http://pygments.org/languages
 .. _Using Pygments in ReST documents: http://pygments.org/docs/rstdirective/
-.. _Docutils Document Tree: 
+
+.. _Docutils: http://docutils.sourceforge.net/
+.. _Docutils Document Tree:
     http://docutils.sf.net/docs/ref/doctree.html#classes
 .. _latex-variants: http://docutils.sourceforge.net/sandbox/latex-variants/
+.. _proof of concept:
+    http://article.gmane.org/gmane.text.docutils.user/3689
 
 .. Internal links
 .. _front-end scripts: ../tools/pygments-enhanced-front-ends
 .. _pygments-default.css: ../data/pygments-default.css
+.. _pygments_code_block_directive.py: ../pygments_code_block_directive.py
+.. _pygments_code_block_directive: pygments_code_block_directive-bunt.py.htm
+.. _rst2html-highlight: ../rst2html-highlight
+.. _pygments-long.css: ../data/pygments-long.css