diff options
| author | milde <milde@929543f6-e4f2-0310-98a6-ba3bd3dd1d04> | 2007-06-13 10:20:42 +0000 |
|---|---|---|
| committer | milde <milde@929543f6-e4f2-0310-98a6-ba3bd3dd1d04> | 2007-06-13 10:20:42 +0000 |
| commit | 37aec3754971395f373c48f5ca2a2aca4caaed96 (patch) | |
| tree | 1f892c2dc24ccfe834071bebdda9d512c5497d18 /sandbox/code-block-directive | |
| parent | 2e5900fc8a1de5348802b8069ed65bb05e424263 (diff) | |
| download | docutils-37aec3754971395f373c48f5ca2a2aca4caaed96.tar.gz | |
Added links to the README file.
self-documenting feature: literal code with syntax highlight
Moved the front-end to the base dir. This way it works without installing.
git-svn-id: http://svn.code.sf.net/p/docutils/code/trunk@5234 929543f6-e4f2-0310-98a6-ba3bd3dd1d04
Diffstat (limited to 'sandbox/code-block-directive')
| -rw-r--r-- | sandbox/code-block-directive/README.html | 55 | ||||
| -rw-r--r-- | sandbox/code-block-directive/README.txt | 38 | ||||
| -rw-r--r-- | sandbox/code-block-directive/docs/pygments_code_block_directive-bunt.py.html | 314 | ||||
| -rw-r--r-- | sandbox/code-block-directive/docs/pygments_code_block_directive.py.html | 232 | ||||
| -rwxr-xr-x | sandbox/code-block-directive/pygments_code_block_directive.py | 9 | ||||
| -rwxr-xr-x | sandbox/code-block-directive/rst2html-highlight (renamed from sandbox/code-block-directive/tools/rst2html-highlight) | 0 |
6 files changed, 588 insertions, 60 deletions
diff --git a/sandbox/code-block-directive/README.html b/sandbox/code-block-directive/README.html new file mode 100644 index 000000000..28579cee9 --- /dev/null +++ b/sandbox/code-block-directive/README.html @@ -0,0 +1,55 @@ +<?xml version="1.0" encoding="iso-8859-1" ?> +<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> +<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> +<head> +<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" /> +<meta name="generator" content="Docutils 0.4.1: http://docutils.sourceforge.net/" /> +<title>Proposal for a code-block directive in docutils</title> +<meta name="author" content="Guenter Milde <milde@users.berlios.de>" /> +<meta name="date" content="2007-06-08" /> +<link rel="stylesheet" href="../../../../lib/python2.4/site-packages/docutils/writers/html4css1/html4css1.css" type="text/css" /> +</head> +<body> +<div class="document" id="proposal-for-a-code-block-directive-in-docutils"> +<h1 class="title">Proposal for a code-block directive in docutils</h1> +<table class="docinfo" frame="void" rules="none"> +<col class="docinfo-name" /> +<col class="docinfo-content" /> +<tbody valign="top"> +<tr><th class="docinfo-name">Author:</th> +<td>Guenter Milde <<a class="reference" href="mailto:milde@users.berlios.de">milde@users.berlios.de</a>></td></tr> +<tr><th class="docinfo-name">Date:</th> +<td>2007-06-08</td></tr> +</tbody> +</table> +<!-- -*- rst-mode -*- --> +<p>This sandbox project contains experimental code and documentation related to +the proposal for syntax highlight of source code in docutils using a +"code-block" directive.</p> +<p>See <a class="reference" href="docs/syntax-highlight.html">syntax-highlight.html</a> for the full picture.</p> +<dl class="docutils"> +<dt><a class="reference" href="pygments_code_block_directive.py">pygments_code_block_directive.py</a></dt> +<dd><p class="first">Working example: defines and registers a +code-block directive using the <a class="reference" href="http://pygments.org/">Pygments</a> syntax highlighter.</p> +<p class="last">colourful literal code (maybe outdated): +<a class="reference" href="docs/pygments_code_block_directive-bunt.py.html">pygments_code_block_directive-bunt.py.html</a></p> +</dd> +<dt><a class="reference" href="rst2html-highlight">rst2html-highlight</a></dt> +<dd>front end for reStructuredText -> HTML conversion supporting the +"code-block" directive.</dd> +<dt><a class="reference" href="docs">docs</a></dt> +<dd>Documentation, concepts, discussion, examples...</dd> +<dt><a class="reference" href="tools">tools</a></dt> +<dd>Style sheets, alternative (legacy) front ends, script for interactive +testing.</dd> +</dl> +<!-- References --> +</div> +<div class="footer"> +<hr class="footer" /> +<a class="reference" href="README.txt">View document source</a>. +Generated on: 2007-06-13. + +</div> +</body> +</html> diff --git a/sandbox/code-block-directive/README.txt b/sandbox/code-block-directive/README.txt index 2bf06ea63..61ed3614b 100644 --- a/sandbox/code-block-directive/README.txt +++ b/sandbox/code-block-directive/README.txt @@ -1,17 +1,49 @@ +.. -*- rst-mode -*- + =============================================== Proposal for a code-block directive in docutils =============================================== -# :Author: Guenter Milde <milde@users.berlios.de> -# :Date: $Date$ +:Author: Guenter Milde <milde@users.berlios.de> +:Date: $Date$ This sandbox project contains experimental code and documentation related to the proposal for syntax highlight of source code in docutils using a "code-block" directive. -See `syntax-highlight.html`_ for an overview. +See `syntax-highlight.html`_ for the full picture. + +`pygments_code_block_directive.py`_ + Working example: defines and registers a + code-block directive using the Pygments_ syntax highlighter. + + colourful literal code (maybe outdated): + `pygments_code_block_directive-bunt.py.html`_ + +`rst2html-highlight`_ + front end for reStructuredText -> HTML conversion supporting the + "code-block" directive. + +docs_ + Documentation, concepts, discussion, examples... + + +tools_ + Style sheets, alternative (legacy) front ends, script for interactive + testing. + + +.. References + +.. _pygments: http://pygments.org/ +.. _pygments_code_block_directive.py: pygments_code_block_directive.py +.. _pygments_code_block_directive-bunt.py.html: + docs/pygments_code_block_directive-bunt.py.html +.. _rst2html-highlight: rst2html-highlight +.. _docs: docs +.. _tools: tools .. _syntax-highlight.html: docs/syntax-highlight.html diff --git a/sandbox/code-block-directive/docs/pygments_code_block_directive-bunt.py.html b/sandbox/code-block-directive/docs/pygments_code_block_directive-bunt.py.html new file mode 100644 index 000000000..c44f69e9b --- /dev/null +++ b/sandbox/code-block-directive/docs/pygments_code_block_directive-bunt.py.html @@ -0,0 +1,314 @@ +<?xml version="1.0" encoding="iso-8859-1" ?> +<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> +<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> +<head> +<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" /> +<meta name="generator" content="Docutils 0.4.1: http://docutils.sourceforge.net/" /> +<title></title> +<meta name="author" content="a Pygments author|contributor; Felix Wiemann; Guenter Milde" /> +<meta name="date" content="2007-06-08" /> +<meta name="copyright" content="This module has been placed in the public domain." /> +<style type="text/css"> + +/* Stylesheet for pygments enhanced reStructured Text */ +/* ================================================== */ + +/* :Author: Guenter Milde */ +/* :Copyright: 2007 G. Milde */ +/* This stylesheet is released under the GPL v. 2 or later */ + +/* This stylesheet provides syntax highlight for documents generated with a */ +/* pygments_ enhanced reStructured Text -> html converter. */ + +/* Import the default docutils style sheet */ +/* --------------------------------------- */ +/* :: */ + +@import url("/stylesheets/html4css1.css"); + +/* Indent the code block */ +/* --------------------- */ + +/* Content copied from the `html4css1.css` rule for literal blocks. */ +/* Selector adapted to the output of Pygments_. :: */ + +div.highlight { + margin-left: 2em ; + margin-right: 2em ; + background-color: #eeeeee + } + + +/* Colour code blocks */ +/* ------------------ */ + +/* Pygments_ has an option to generate stylesheets for html and latex. */ +/* The following code is generated with the command */ +/* `pygmentize -S default -f html > pygments-default.css`:: */ + +.c { color: #008800; font-style: italic } /* Comment */ +.err { border: 1px solid #FF0000 } /* Error */ +.k { color: #AA22FF; font-weight: bold } /* Keyword */ +.o { color: #666666 } /* Operator */ +.cm { color: #008800; font-style: italic } /* Comment.Multiline */ +.cp { color: #008800 } /* Comment.Preproc */ +.c1 { color: #008800; font-style: italic } /* Comment.Single */ +.gd { color: #A00000 } /* Generic.Deleted */ +.ge { font-style: italic } /* Generic.Emph */ +.gr { color: #FF0000 } /* Generic.Error */ +.gh { color: #000080; font-weight: bold } /* Generic.Heading */ +.gi { color: #00A000 } /* Generic.Inserted */ +.go { color: #808080 } /* Generic.Output */ +.gp { color: #000080; font-weight: bold } /* Generic.Prompt */ +.gs { font-weight: bold } /* Generic.Strong */ +.gu { color: #800080; font-weight: bold } /* Generic.Subheading */ +.gt { color: #0040D0 } /* Generic.Traceback */ +.kc { color: #AA22FF; font-weight: bold } /* Keyword.Constant */ +.kd { color: #AA22FF; font-weight: bold } /* Keyword.Declaration */ +.kp { color: #AA22FF } /* Keyword.Pseudo */ +.kr { color: #AA22FF; font-weight: bold } /* Keyword.Reserved */ +.kt { color: #AA22FF; font-weight: bold } /* Keyword.Type */ +.m { color: #666666 } /* Literal.Number */ +.s { color: #BB4444 } /* Literal.String */ +.na { color: #BB4444 } /* Name.Attribute */ +.nb { color: #AA22FF } /* Name.Builtin */ +.nc { color: #0000FF } /* Name.Class */ +.no { color: #880000 } /* Name.Constant */ +.nd { color: #AA22FF } /* Name.Decorator */ +.ni { color: #999999; font-weight: bold } /* Name.Entity */ +.ne { color: #D2413A; font-weight: bold } /* Name.Exception */ +.nf { color: #00A000 } /* Name.Function */ +.nl { color: #A0A000 } /* Name.Label */ +.nn { color: #0000FF; font-weight: bold } /* Name.Namespace */ +.nt { color: #008000; font-weight: bold } /* Name.Tag */ +.nv { color: #B8860B } /* Name.Variable */ +.ow { color: #AA22FF; font-weight: bold } /* Operator.Word */ +.mf { color: #666666 } /* Literal.Number.Float */ +.mh { color: #666666 } /* Literal.Number.Hex */ +.mi { color: #666666 } /* Literal.Number.Integer */ +.mo { color: #666666 } /* Literal.Number.Oct */ +.sb { color: #BB4444 } /* Literal.String.Backtick */ +.sc { color: #BB4444 } /* Literal.String.Char */ +.sd { color: #BB4444; font-style: italic } /* Literal.String.Doc */ +.s2 { color: #BB4444 } /* Literal.String.Double */ +.se { color: #BB6622; font-weight: bold } /* Literal.String.Escape */ +.sh { color: #BB4444 } /* Literal.String.Heredoc */ +.si { color: #BB6688; font-weight: bold } /* Literal.String.Interpol */ +.sx { color: #008000 } /* Literal.String.Other */ +.sr { color: #BB6688 } /* Literal.String.Regex */ +.s1 { color: #BB4444 } /* Literal.String.Single */ +.ss { color: #B8860B } /* Literal.String.Symbol */ +.bp { color: #AA22FF } /* Name.Builtin.Pseudo */ +.vc { color: #B8860B } /* Name.Variable.Class */ +.vg { color: #B8860B } /* Name.Variable.Global */ +.vi { color: #B8860B } /* Name.Variable.Instance */ +.il { color: #666666 } /* Literal.Number.Integer.Long */ + +/* .. _pygments: http://pygments.org/ */ + +</style> +</head> +<body> +<div class="document"> +<table class="docinfo" frame="void" rules="none"> +<col class="docinfo-name" /> +<col class="docinfo-content" /> +<tbody valign="top"> +<tr><th class="docinfo-name">Author:</th> +<td>a Pygments author|contributor; Felix Wiemann; Guenter Milde</td></tr> +<tr><th class="docinfo-name">Date:</th> +<td>2007-06-08</td></tr> +<tr><th class="docinfo-name">Copyright:</th> +<td>This module has been placed in the public domain.</td></tr> +</tbody> +</table> +<!-- #!/usr/bin/python --> +<p>This is a merge of <a class="reference" href="http://pygments.org/docs/rstdirective/">Using Pygments in ReST documents</a> from the <a class="reference" href="http://pygments.org/">pygments</a> +documentation, and a <a class="reference" href="http://article.gmane.org/gmane.text.docutils.user/3689">proof of concept</a> by Felix Wiemann.</p> +<table border="1" class="docutils"> +<colgroup> +<col width="14%" /> +<col width="86%" /> +</colgroup> +<tbody valign="top"> +<tr><td>2007-06-01</td> +<td>Removed redundancy from class values.</td> +</tr> +<tr><td>2007-06-04</td> +<td>Merge of successive tokens of same type +(code taken from pygments.formatters.others).</td> +</tr> +<tr><td>2007-06-05</td> +<td>Separate docutils formatter script +Use pygments' CSS class names (like the html formatter) +allowing the use of pygments-produced style sheets.</td> +</tr> +<tr><td>2007-06-07</td> +<td>Merge in the formatting of the parsed tokens +(misnamed as docutils_formatter) as class DocutilsInterface</td> +</tr> +<tr><td>2007-06-08</td> +<td>Failsave implementation (fallback to a standard literal block +if pygments not found)</td> +</tr> +</tbody> +</table> +<!-- to get the syntax highlight in the html output of this file, convert with:: + +./rst2html-highlight - -stylesheet=tools/pygments-default.css --> +<pre class="code-block python literal-block"> +<span class="sd">"""Define and register a code-block directive using pygments +"""</span> + +</pre> +<div class="section"> +<h1><a id="requirements" name="requirements">Requirements</a></h1> +<pre class="code-block python literal-block"> +<span class="k">from</span> <span class="nn">docutils</span> <span class="k">import</span> <span class="n">nodes</span> +<span class="k">from</span> <span class="nn">docutils.parsers.rst</span> <span class="k">import</span> <span class="n">directives</span> +<span class="k">try</span><span class="p">:</span> + <span class="k">import</span> <span class="nn">pygments</span> + <span class="k">from</span> <span class="nn">pygments.lexers</span> <span class="k">import</span> <span class="n">get_lexer_by_name</span> + <span class="k">from</span> <span class="nn">pygments.formatters.html</span> <span class="k">import</span> <span class="n">_get_ttype_class</span> +<span class="k">except</span> <span class="ne">ImportError</span><span class="p">:</span> + <span class="k">pass</span> + +</pre> +</div> +<div class="section"> +<h1><a id="customisation" name="customisation">Customisation</a></h1> +<p>Do not insert inline nodes for the following tokens. +(You could add e.g. Token.Punctuation like <tt class="docutils literal"><span class="pre">['',</span> <span class="pre">'p']</span></tt>.)</p> +<pre class="code-block python literal-block"> +<span class="n">unstyled_tokens</span> <span class="o">=</span> <span class="p">[</span><span class="s">''</span><span class="p">]</span> + +</pre> +</div> +<div class="section"> +<h1><a id="docutilsinterface" name="docutilsinterface">DocutilsInterface</a></h1> +<p>This interface class combines code from +pygments.formatters.html and pygments.formatters.others.</p> +<p>It does not require anything of docutils and could also become a part of +pygments</p> +<pre class="code-block python literal-block"> +<span class="k">class</span> <span class="nc">DocutilsInterface</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span> + <span class="sd">"""Parse `code` string and yield "classified" tokens. + + Arguments + + code -- string of source code to parse + language -- formal language the code is written in. + + Merge subsequent tokens of the same token-type. + + Yields the tokens as ``(ttype_class, value)`` tuples, + where ttype_class is taken from pygments.token.STANDARD_TYPES and + corresponds to the class argument used in pygments html output. + + """</span> + + <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">code</span><span class="p">,</span> <span class="n">language</span><span class="p">):</span> + <span class="bp">self</span><span class="o">.</span><span class="n">code</span> <span class="o">=</span> <span class="n">code</span> + <span class="bp">self</span><span class="o">.</span><span class="n">language</span> <span class="o">=</span> <span class="n">language</span> + + <span class="k">def</span> <span class="nf">lex</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> + <span class="c"># Get lexer for language (use text as fallback)</span> + <span class="k">try</span><span class="p">:</span> + <span class="n">lexer</span> <span class="o">=</span> <span class="n">get_lexer_by_name</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">language</span><span class="p">)</span> + <span class="k">except</span> <span class="ne">ValueError</span><span class="p">:</span> + <span class="c"># info: "no pygments lexer for %s, using 'text'"%self.language</span> + <span class="n">lexer</span> <span class="o">=</span> <span class="n">get_lexer_by_name</span><span class="p">(</span><span class="s">'text'</span><span class="p">)</span> + <span class="k">return</span> <span class="n">pygments</span><span class="o">.</span><span class="n">lex</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">code</span><span class="p">,</span> <span class="n">lexer</span><span class="p">)</span> + + + <span class="k">def</span> <span class="nf">join</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">tokens</span><span class="p">):</span> + <span class="sd">"""join subsequent tokens of same token-type + """</span> + <span class="n">tokens</span> <span class="o">=</span> <span class="nb">iter</span><span class="p">(</span><span class="n">tokens</span><span class="p">)</span> + <span class="p">(</span><span class="n">lasttype</span><span class="p">,</span> <span class="n">lastval</span><span class="p">)</span> <span class="o">=</span> <span class="n">tokens</span><span class="o">.</span><span class="n">next</span><span class="p">()</span> + <span class="k">for</span> <span class="n">ttype</span><span class="p">,</span> <span class="n">value</span> <span class="ow">in</span> <span class="n">tokens</span><span class="p">:</span> + <span class="k">if</span> <span class="n">ttype</span> <span class="ow">is</span> <span class="n">lasttype</span><span class="p">:</span> + <span class="n">lastval</span> <span class="o">+=</span> <span class="n">value</span> + <span class="k">else</span><span class="p">:</span> + <span class="k">yield</span><span class="p">(</span><span class="n">lasttype</span><span class="p">,</span> <span class="n">lastval</span><span class="p">)</span> + <span class="p">(</span><span class="n">lasttype</span><span class="p">,</span> <span class="n">lastval</span><span class="p">)</span> <span class="o">=</span> <span class="p">(</span><span class="n">ttype</span><span class="p">,</span> <span class="n">value</span><span class="p">)</span> + <span class="k">yield</span><span class="p">(</span><span class="n">lasttype</span><span class="p">,</span> <span class="n">lastval</span><span class="p">)</span> + + <span class="k">def</span> <span class="nf">__iter__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> + <span class="sd">"""parse code string and yield "clasified" tokens + """</span> + <span class="k">try</span><span class="p">:</span> + <span class="n">tokens</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">lex</span><span class="p">()</span> + <span class="k">except</span> <span class="ne">IOError</span><span class="p">:</span> + <span class="k">print</span> <span class="s">"INFO: Pygments lexer not found, using fallback"</span> + <span class="c"># TODO: write message to INFO</span> + <span class="k">yield</span> <span class="p">(</span><span class="s">''</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">code</span><span class="p">)</span> + <span class="k">return</span> + + <span class="k">for</span> <span class="n">ttype</span><span class="p">,</span> <span class="n">value</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">tokens</span><span class="p">):</span> + <span class="k">yield</span> <span class="p">(</span><span class="n">_get_ttype_class</span><span class="p">(</span><span class="n">ttype</span><span class="p">),</span> <span class="n">value</span><span class="p">)</span> + +</pre> +</div> +<div class="section"> +<h1><a id="code-block-directive" name="code-block-directive">code_block_directive</a></h1> +<pre class="code-block python literal-block"> +<span class="k">def</span> <span class="nf">code_block_directive</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">arguments</span><span class="p">,</span> <span class="n">options</span><span class="p">,</span> <span class="n">content</span><span class="p">,</span> <span class="n">lineno</span><span class="p">,</span> + <span class="n">content_offset</span><span class="p">,</span> <span class="n">block_text</span><span class="p">,</span> <span class="n">state</span><span class="p">,</span> <span class="n">state_machine</span><span class="p">):</span> + <span class="sd">"""parse and classify content of a code_block + """</span> + <span class="n">language</span> <span class="o">=</span> <span class="n">arguments</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> + <span class="c"># create a literal block element and set class argument</span> + <span class="n">code_block</span> <span class="o">=</span> <span class="n">nodes</span><span class="o">.</span><span class="n">literal_block</span><span class="p">(</span><span class="n">classes</span><span class="o">=</span><span class="p">[</span><span class="s">"code-block"</span><span class="p">,</span> <span class="n">language</span><span class="p">])</span> + + <span class="c"># parse content with pygments and add to code_block element</span> + <span class="k">for</span> <span class="n">cls</span><span class="p">,</span> <span class="n">value</span> <span class="ow">in</span> <span class="n">DocutilsInterface</span><span class="p">(</span><span class="n">u</span><span class="s">'</span><span class="se">\n</span><span class="s">'</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">content</span><span class="p">),</span> <span class="n">language</span><span class="p">):</span> + <span class="k">if</span> <span class="n">cls</span> <span class="ow">in</span> <span class="n">unstyled_tokens</span><span class="p">:</span> + <span class="c"># insert as Text to decrease the verbosity of the output.</span> + <span class="n">code_block</span> <span class="o">+=</span> <span class="n">nodes</span><span class="o">.</span><span class="n">Text</span><span class="p">(</span><span class="n">value</span><span class="p">,</span> <span class="n">value</span><span class="p">)</span> + <span class="k">else</span><span class="p">:</span> + <span class="n">code_block</span> <span class="o">+=</span> <span class="n">nodes</span><span class="o">.</span><span class="n">inline</span><span class="p">(</span><span class="n">value</span><span class="p">,</span> <span class="n">value</span><span class="p">,</span> <span class="n">classes</span><span class="o">=</span><span class="p">[</span><span class="n">cls</span><span class="p">])</span> + + <span class="k">return</span> <span class="p">[</span><span class="n">code_block</span><span class="p">]</span> + +</pre> +</div> +<div class="section"> +<h1><a id="register-directive" name="register-directive">Register Directive</a></h1> +<pre class="code-block python literal-block"> +<span class="n">code_block_directive</span><span class="o">.</span><span class="n">arguments</span> <span class="o">=</span> <span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span> +<span class="n">code_block_directive</span><span class="o">.</span><span class="n">content</span> <span class="o">=</span> <span class="mi">1</span> +<span class="n">directives</span><span class="o">.</span><span class="n">register_directive</span><span class="p">(</span><span class="s">'code-block'</span><span class="p">,</span> <span class="n">code_block_directive</span><span class="p">)</span> + +</pre> +</div> +<div class="section"> +<h1><a id="test-output" name="test-output">Test output</a></h1> +<p>If called from the command line, call the docutils publisher to render the +input</p> +<pre class="code-block python literal-block"> +<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">'__main__'</span><span class="p">:</span> + <span class="k">from</span> <span class="nn">docutils.core</span> <span class="k">import</span> <span class="n">publish_cmdline</span><span class="p">,</span> <span class="n">default_description</span> + <span class="n">description</span> <span class="o">=</span> <span class="s">"code-block directive test output"</span> <span class="o">+</span> <span class="n">default_description</span> + <span class="k">try</span><span class="p">:</span> + <span class="k">import</span> <span class="nn">locale</span> + <span class="n">locale</span><span class="o">.</span><span class="n">setlocale</span><span class="p">(</span><span class="n">locale</span><span class="o">.</span><span class="n">LC_ALL</span><span class="p">,</span> <span class="s">''</span><span class="p">)</span> + <span class="k">except</span><span class="p">:</span> + <span class="k">pass</span> + <span class="c"># Uncomment the desired output format:</span> + <span class="n">publish_cmdline</span><span class="p">(</span><span class="n">writer_name</span><span class="o">=</span><span class="s">'pseudoxml'</span><span class="p">,</span> <span class="n">description</span><span class="o">=</span><span class="n">description</span><span class="p">)</span> + <span class="c"># publish_cmdline(writer_name='xml', description=description)</span> + <span class="c"># publish_cmdline(writer_name='html', description=description)</span> + <span class="c"># publish_cmdline(writer_name='latex', description=description)</span> + <span class="c"># publish_cmdline(writer_name='newlatex2e', description=description)</span> + +</pre> +</div> +</div> +<div class="footer"> +<hr class="footer" /> +Generated on: 2007-06-13. + +</div> +</body> +</html> diff --git a/sandbox/code-block-directive/docs/pygments_code_block_directive.py.html b/sandbox/code-block-directive/docs/pygments_code_block_directive.py.html index 15af90a8c..ba7364dbe 100644 --- a/sandbox/code-block-directive/docs/pygments_code_block_directive.py.html +++ b/sandbox/code-block-directive/docs/pygments_code_block_directive.py.html @@ -6,9 +6,9 @@ <meta name="generator" content="Docutils 0.4.1: http://docutils.sourceforge.net/" /> <title></title> <meta name="author" content="a Pygments author|contributor; Felix Wiemann; Guenter Milde" /> -<meta name="date" content="$Date$" /> +<meta name="date" content="2007-06-08" /> <meta name="copyright" content="This module has been placed in the public domain." /> -<link rel="stylesheet" href="/usr/lib/python2.4/site-packages/docutils/writers/html4css1/html4css1.css" type="text/css" /> +<link rel="stylesheet" href="../../../../lib/python2.4/site-packages/docutils/writers/html4css1/html4css1.css" type="text/css" /> </head> <body> <div class="document"> @@ -19,7 +19,7 @@ <tr><th class="docinfo-name">Author:</th> <td>a Pygments author|contributor; Felix Wiemann; Guenter Milde</td></tr> <tr><th class="docinfo-name">Date:</th> -<td>$Date$</td></tr> +<td>2007-06-08</td></tr> <tr><th class="docinfo-name">Copyright:</th> <td>This module has been placed in the public domain.</td></tr> </tbody> @@ -45,92 +45,218 @@ documentation, and a <a class="reference" href="http://article.gmane.org/gmane.t Use pygments' CSS class names (like the html formatter) allowing the use of pygments-produced style sheets.</td> </tr> +<tr><td>2007-06-07</td> +<td>Merge in the formatting of the parsed tokens +(misnamed as docutils_formatter) as class DocutilsInterface</td> +</tr> +<tr><td>2007-06-08</td> +<td>Failsave implementation (fallback to a standard literal block +if pygments not found)</td> +</tr> </tbody> </table> +<div class="system-message"> +<p class="system-message-title">System Message: ERROR/3 (<tt class="docutils">/usr/src/docutils-svn/sandbox/code-block-directive/pygments_code_block_directive.py.txt</tt>, line 23)</p> +<p>Unknown directive type "code-block".</p> <pre class="literal-block"> -"""Define and register a code-block directive using pygments -""" +.. code-block:: + + """Define and register a code-block directive using pygments + """ + </pre> +</div> <div class="section"> <h1><a id="requirements" name="requirements">Requirements</a></h1> +<div class="system-message"> +<p class="system-message-title">System Message: ERROR/3 (<tt class="docutils">/usr/src/docutils-svn/sandbox/code-block-directive/pygments_code_block_directive.py.txt</tt>, line 31)</p> +<p>Unknown directive type "code-block".</p> <pre class="literal-block"> -from docutils import nodes -from docutils.parsers.rst import directives -import pygments -from pygments.lexers import get_lexer_by_name -from pygments_docutils_formatter import DocutilsFormatter +.. code-block:: + + from docutils import nodes + from docutils.parsers.rst import directives + try: + import pygments + from pygments.lexers import get_lexer_by_name + from pygments.formatters.html import _get_ttype_class + except ImportError: + pass + + + </pre> </div> +</div> <div class="section"> <h1><a id="customisation" name="customisation">Customisation</a></h1> <p>Do not insert inline nodes for the following tokens. (You could add e.g. Token.Punctuation like <tt class="docutils literal"><span class="pre">['',</span> <span class="pre">'p']</span></tt>.)</p> +<div class="system-message"> +<p class="system-message-title">System Message: ERROR/3 (<tt class="docutils">/usr/src/docutils-svn/sandbox/code-block-directive/pygments_code_block_directive.py.txt</tt>, line 50)</p> +<p>Unknown directive type "code-block".</p> <pre class="literal-block"> -unstyled_tokens = [''] +.. code-block:: + + unstyled_tokens = [''] + </pre> </div> +</div> +<div class="section"> +<h1><a id="docutilsinterface" name="docutilsinterface">DocutilsInterface</a></h1> +<p>This interface class combines code from +pygments.formatters.html and pygments.formatters.others.</p> +<p>It does not require anything of docutils and could also become a part of +pygments</p> +<div class="system-message"> +<p class="system-message-title">System Message: ERROR/3 (<tt class="docutils">/usr/src/docutils-svn/sandbox/code-block-directive/pygments_code_block_directive.py.txt</tt>, line 63)</p> +<p>Unknown directive type "code-block".</p> +<pre class="literal-block"> +.. code-block:: + + class DocutilsInterface(object): + """Parse `code` string and yield "classified" tokens. + + Arguments + + code -- string of source code to parse + language -- formal language the code is written in. + + Merge subsequent tokens of the same token-type. + + Yields the tokens as ``(ttype_class, value)`` tuples, + where ttype_class is taken from pygments.token.STANDARD_TYPES and + corresponds to the class argument used in pygments html output. + + """ + + def __init__(self, code, language): + self.code = code + self.language = language + + def lex(self): + # Get lexer for language (use text as fallback) + try: + lexer = get_lexer_by_name(self.language) + except ValueError: + # info: "no pygments lexer for %s, using 'text'"%self.language + lexer = get_lexer_by_name('text') + return pygments.lex(self.code, lexer) + + + def join(self, tokens): + """join subsequent tokens of same token-type + """ + tokens = iter(tokens) + (lasttype, lastval) = tokens.next() + for ttype, value in tokens: + if ttype is lasttype: + lastval += value + else: + yield(lasttype, lastval) + (lasttype, lastval) = (ttype, value) + yield(lasttype, lastval) + + def __iter__(self): + """parse code string and yield "clasified" tokens + """ + try: + tokens = self.lex() + except IOError: + print "INFO: Pygments lexer not found, using fallback" + # TODO: write message to INFO + yield ('', self.code) + return + + for ttype, value in self.join(tokens): + yield (_get_ttype_class(ttype), value) + + + +</pre> +</div> +</div> <div class="section"> <h1><a id="code-block-directive" name="code-block-directive">code_block_directive</a></h1> +<div class="system-message"> +<p class="system-message-title">System Message: ERROR/3 (<tt class="docutils">/usr/src/docutils-svn/sandbox/code-block-directive/pygments_code_block_directive.py.txt</tt>, line 127)</p> +<p>Unknown directive type "code-block".</p> <pre class="literal-block"> -def code_block_directive(name, arguments, options, content, lineno, - content_offset, block_text, state, state_machine): - language = arguments[0] - # create a literal block element and set class argument - code_block = nodes.literal_block(raw_content=content, - classes=["code-block", language]) - # Get lexer for language (use text as fallback) - try: - lexer = get_lexer_by_name(language) - except ValueError: - lexer = get_lexer_by_name('text') - - # parse content with pygments - tokens = list(pygments.lex(u'\n'.join(content), lexer)) - - for cls, value in DocutilsFormatter(tokens): - if cls in unstyled_tokens: - # insert as Text to decrease the verbosity of the output. - code_block += nodes.Text(value, value) - else: - code_block += nodes.inline(value, value, classes=[cls]) - - return [code_block] +.. code-block:: + + def code_block_directive(name, arguments, options, content, lineno, + content_offset, block_text, state, state_machine): + """parse and classify content of a code_block + """ + language = arguments[0] + # create a literal block element and set class argument + code_block = nodes.literal_block(classes=["code-block", language]) + + # parse content with pygments and add to code_block element + for cls, value in DocutilsInterface(u'\n'.join(content), language): + if cls in unstyled_tokens: + # insert as Text to decrease the verbosity of the output. + code_block += nodes.Text(value, value) + else: + code_block += nodes.inline(value, value, classes=[cls]) + + return [code_block] + + </pre> </div> +</div> <div class="section"> <h1><a id="register-directive" name="register-directive">Register Directive</a></h1> +<div class="system-message"> +<p class="system-message-title">System Message: ERROR/3 (<tt class="docutils">/usr/src/docutils-svn/sandbox/code-block-directive/pygments_code_block_directive.py.txt</tt>, line 151)</p> +<p>Unknown directive type "code-block".</p> <pre class="literal-block"> -code_block_directive.arguments = (1, 0, 1) -code_block_directive.content = 1 -directives.register_directive('code-block', code_block_directive) +.. code-block:: + + code_block_directive.arguments = (1, 0, 1) + code_block_directive.content = 1 + directives.register_directive('code-block', code_block_directive) + </pre> </div> +</div> <div class="section"> <h1><a id="test-output" name="test-output">Test output</a></h1> <p>If called from the command line, call the docutils publisher to render the -input:</p> +input</p> +<div class="system-message"> +<p class="system-message-title">System Message: ERROR/3 (<tt class="docutils">/usr/src/docutils-svn/sandbox/code-block-directive/pygments_code_block_directive.py.txt</tt>, line 169)</p> +<p>Unknown directive type "code-block".</p> <pre class="literal-block"> -if __name__ == '__main__': - from docutils.core import publish_cmdline, default_description - description = "code-block directive test output" + default_description - try: - import locale - locale.setlocale(locale.LC_ALL, '') - except: - pass - # Uncomment the desired output format: - publish_cmdline(writer_name='pseudoxml', description=description) - # publish_cmdline(writer_name='xml', description=description) - # publish_cmdline(writer_name='html', description=description) - # publish_cmdline(writer_name='latex', description=description) - # publish_cmdline(writer_name='newlatex2e', description=description) +.. code-block:: + + if __name__ == '__main__': + from docutils.core import publish_cmdline, default_description + description = "code-block directive test output" + default_description + try: + import locale + locale.setlocale(locale.LC_ALL, '') + except: + pass + # Uncomment the desired output format: + publish_cmdline(writer_name='pseudoxml', description=description) + # publish_cmdline(writer_name='xml', description=description) + # publish_cmdline(writer_name='html', description=description) + # publish_cmdline(writer_name='latex', description=description) + # publish_cmdline(writer_name='newlatex2e', description=description) + + + </pre> </div> </div> +</div> <div class="footer"> <hr class="footer" /> <a class="reference" href="pygments_code_block_directive.py.txt">View document source</a>. -Generated on: 2007-06-05. +Generated on: 2007-06-13. </div> </body> diff --git a/sandbox/code-block-directive/pygments_code_block_directive.py b/sandbox/code-block-directive/pygments_code_block_directive.py index 9095e4908..a0e7d5823 100755 --- a/sandbox/code-block-directive/pygments_code_block_directive.py +++ b/sandbox/code-block-directive/pygments_code_block_directive.py @@ -14,9 +14,10 @@ # 2007-06-05 Separate docutils formatter script # Use pygments' CSS class names (like the html formatter) # allowing the use of pygments-produced style sheets. -# 2007-06-07 Re-include the formatting of the parsed tokens -# (class DocutilsInterface) -# 2007-06-08 Failsave implementation (fallback if pygments not found) +# 2007-06-07 Merge in the formatting of the parsed tokens +# (misnamed as docutils_formatter) as class DocutilsInterface +# 2007-06-08 Failsave implementation (fallback to a standard literal block +# if pygments not found) # ========== =========================================================== # # :: @@ -53,7 +54,7 @@ unstyled_tokens = [''] # This interface class combines code from # pygments.formatters.html and pygments.formatters.others. # -# It does not require anything of docutils and could alse become a part of +# It does not require anything of docutils and could also become a part of # pygments:: class DocutilsInterface(object): diff --git a/sandbox/code-block-directive/tools/rst2html-highlight b/sandbox/code-block-directive/rst2html-highlight index 737a8c802..737a8c802 100755 --- a/sandbox/code-block-directive/tools/rst2html-highlight +++ b/sandbox/code-block-directive/rst2html-highlight |