[svn] implemented filters for pygments (first approach, api might change), it's impossible at the moment to add filters by using pygmentize

7 files changed, 149 insertions, 1 deletions

diff --git a/docs/src/filterdevelopment.txt b/docs/src/filterdevelopment.txt
new file mode 100644
index 00000000..af467831
--- /dev/null
+++ b/docs/src/filterdevelopment.txt
@@ -0,0 +1,68 @@
+.. -*- mode: rst -*-
+
+=====================
+Write your own filter
+=====================
+
+*New in Pygments 0.7*
+
+Writing own filters is very easy. All you have to do is to subclass
+the `Filter` class and override the `filter` method. Additionally a
+filter is instanciated with some keyword arguments you can use to
+adjust the behavior of your filter.
+
+
+Subclassing Filters
+===================
+
+As an example we write a filter that converts all `Name.Function` tokens
+to normal `Name` tokens to make the output less colorful.
+
+.. sourcecode:: pycon
+
+    from pygments.util import get_bool_opt
+    from pygments.token import Name
+    from pygments.filter import Filter
+
+    class UncolorFilter(Filter):
+
+        def __init__(self, **options):
+            Filter.__init__(self, **options)
+            self.class_too = get_bool_opt(options, 'classtoo')
+
+        def filter(self, lexer, stream):
+            for ttype, value in stream:
+                if ttype == Name.Function or (self.class_too and
+                                              ttype == Name.Class):
+                    ttype = Name
+                yield ttype, value
+
+Some words on the `lexer` argument. That can be quite confusing since it
+must not be a lexer instance. If a filter was added by using the `add_filter`
+function of lexers that lexer is registered for the filter. In that case
+`lexer` will be point to the lexer that has registered the filter. It can
+be used (but must not) to access options passed to a lexer. Because it
+could be `None` you always have to check for that case if you access it.
+
+
+Using a Decorator
+=================
+
+You can also use the `simplefilter` decorator from the `pygments.filter`
+module:
+
+.. sourcecode:: pycon
+
+    from pygments.util import get_bool_opt
+    from pygments.token import Name
+    from pygments.filter import simplefilter
+
+
+    @simplefilter
+    def uncolor(lexer, stream, options):
+        class_too = get_bool_opt(options, 'classtoo')
+        for ttype, value in stream:
+            if ttype == Name.Function or (class_too and
+                                          ttype == Name.Class):
+                ttype = Name
+            yield ttype, value
diff --git a/docs/src/filters.txt b/docs/src/filters.txt
new file mode 100644
index 00000000..55b74faf
--- /dev/null
+++ b/docs/src/filters.txt
@@ -0,0 +1,57 @@
+.. -*- mode: rst -*-
+
+=======
+Filters
+=======
+
+Since Pygments 0.7 you can filter token streams to improve the output. For
+example you can highlight special words in comments, convert keywords
+to upper or lowercase to enforce an styleguide etc.
+
+To apply an filter you can use the `add_filter` method of a lexer:
+
+.. sourcecode:: pycon
+
+    >>> from pygments.lexers import PythonLexer
+    >>> l = PythonLexer()
+    >>> # as string
+    >>> l.add_filter("codetagify")
+    >>> l.filters
+    [<pygments.filters.CodeTagFilter object at 0xb785decc>]
+    >>> from pygments.filters import KeywordRewriteFilter
+    >>> # or class
+    >>> l.add_filter(KeywordRewriteFilter(keywordcase='lower'))
+
+The `add_filter` method also takes keyword arguments which are forwarded
+to the constructor of the filter.
+
+To get a list of all registered filters by name you can use the
+`get_all_filters` function from the `pygments.filters` module that returns
+an iterable for all known filters.
+
+If you want to write your own lexer have a look at `Write your own filter`_.
+
+.. _Write your own filter: filterdevelopment.txt
+
+
+Builtin Filters
+===============
+
+`CodeTagFilter`
+
+    Highlights special code tags in comments and docstrings. Per
+    default the list of highlighted tags is ``XXX``, ``TODO``,
+    ``BUG`` and ``NOTE``. You can override this list by specifying
+    a ``codetags`` parameter that takes a list of words.
+
+    :Name:  ``codetagify``
+
+`KeywordCaseFilter`
+
+    Converts keywords to ``lower``, ``upper`` or ``capitalize`` which
+    means first letter uppercase, rest lowercase. This can be useful
+    if you highlight pascal code and want to adapt the code to your
+    styleguide. The default is ``lower``, override that by providing
+    the `keywordcase` parameter.
+
+    :Name:  ``keywordcase``
diff --git a/docs/src/formatterdev.txt b/docs/src/formatterdevelopment.txt
index 82208aa0..82208aa0 100644
--- a/docs/src/formatterdev.txt
+++ b/docs/src/formatterdevelopment.txt
diff --git a/docs/src/index.txt b/docs/src/index.txt
index 08e77f62..88bf8591 100644
--- a/docs/src/index.txt
+++ b/docs/src/index.txt
@@ -20,6 +20,8 @@ Welcome to the Pygments documentation.
 
   - `Builtin formatters <formatters.txt>`_
 
+  - `Filters <filters.txt>`_
+
   - `Styles <styles.txt>`_
 
 - API and more
@@ -32,7 +34,9 @@ Welcome to the Pygments documentation.
 
   - `Write your own lexer <lexerdevelopment.txt>`_
 
-  - `Write your own formatter <formatterdev.txt>`_
+  - `Write your own formatter <formatterdevelopment.txt>`_
+  
+  - `Write your own filter <filterdevelopment.txt>`_
 
   - `Register Plugins <plugins.txt>`_
 
diff --git a/docs/src/lexerdevelopment.txt b/docs/src/lexerdevelopment.txt
index d7ab3923..a004df5c 100644
--- a/docs/src/lexerdevelopment.txt
+++ b/docs/src/lexerdevelopment.txt
@@ -535,3 +535,7 @@ the ``get_tokens_unprocessed()`` method. The following lexer subclasses the
                     yield index, token, value
 
 The `PhpLexer` and `LuaLexer` use this method to resolve builtin functions.
+
+**Note** Do not confuse this with the `filter`_ system.
+
+.. _filter: filters.txt
diff --git a/docs/src/plugins.txt b/docs/src/plugins.txt
index 5a302e9e..da8b33b9 100644
--- a/docs/src/plugins.txt
+++ b/docs/src/plugins.txt
@@ -61,6 +61,17 @@ Here a list of setuptools entrypoints pygments understands:
         yourstyle = yourmodule:YourStyle
 
 
+`pygments.filters`
+
+    Use this entrypoint to register a new filter. The name of the
+    entrypoint is the name of the filter:
+
+    .. sourcecode:: ini
+
+        [pygments.filters]
+        yourfilter = yourmodule:YourFilter
+
+
 How To Use Entrypoints
 ======================
 
diff --git a/docs/src/tokens.txt b/docs/src/tokens.txt
index daaf1eca..e1c4402b 100644
--- a/docs/src/tokens.txt
+++ b/docs/src/tokens.txt
@@ -245,6 +245,10 @@ Comments
 `Comment.Single`
     Token type for comments that end at the end of a line (e.g. ``# foo``).
 
+`Comment.Special`
+    Special data in comments. For example code tags, author and license
+    informations etc.
+
 
 Generic Tokens
 ==============