move module names from _XXXbuiltins to _XXX_builtins to be consistent

21 files changed, 279 insertions, 200 deletions

diff --git a/Makefile b/Makefile
index 9dcbe9d9..35814160 100644
--- a/Makefile
+++ b/Makefile
@@ -20,8 +20,7 @@ all: clean-pyc check test
 check:
 	@$(PYTHON) scripts/detect_missing_analyse_text.py || true
 	@$(PYTHON) scripts/check_sources.py -i build -i dist -i pygments/lexers/_mapping.py \
-		   -i docs/build -i pygments/formatters/_mapping.py -i pygments/unistring.py \
-		   -i pygments/lexers/_vimbuiltins.py
+		   -i docs/build -i pygments/formatters/_mapping.py -i pygments/unistring.py
 
 clean: clean-pyc
 	-rm -rf build
diff --git a/doc/docs/lexerdevelopment.rst b/doc/docs/lexerdevelopment.rst
index 48ede92c..83455d65 100644
--- a/doc/docs/lexerdevelopment.rst
+++ b/doc/docs/lexerdevelopment.rst
@@ -1,55 +1,56 @@
 .. -*- mode: rst -*-
 
+.. highlight:: python
+
 ====================
 Write your own lexer
 ====================
 
-If a lexer for your favorite language is missing in the Pygments package, you can
-easily write your own and extend Pygments.
+If a lexer for your favorite language is missing in the Pygments package, you
+can easily write your own and extend Pygments.
 
-All you need can be found inside the :mod:`pygments.lexer` module. As you can
+All you need can be found inside the :mod:`pygments.lexer` module.  As you can
 read in the :doc:`API documentation <api>`, a lexer is a class that is
 initialized with some keyword arguments (the lexer options) and that provides a
 :meth:`.get_tokens_unprocessed()` method which is given a string or unicode
-object with the data to parse.
+object with the data to lex.
 
 The :meth:`.get_tokens_unprocessed()` method must return an iterator or iterable
-containing tuples in the form ``(index, token, value)``. Normally you don't need
-to do this since there are numerous base lexers you can subclass.
+containing tuples in the form ``(index, token, value)``.  Normally you don't
+need to do this since there are base lexers that do most of the work and that
+you can subclass.
 
 
 RegexLexer
 ==========
 
-A very powerful (but quite easy to use) lexer is the :class:`RegexLexer`. This
-lexer base class allows you to define lexing rules in terms of *regular
-expressions* for different *states*.
+The lexer base class used by almost all of Pygments' lexers is the
+:class:`RegexLexer`.  This class allows you to define lexing rules in terms of
+*regular expressions* for different *states*.
 
 States are groups of regular expressions that are matched against the input
-string at the *current position*. If one of these expressions matches, a
-corresponding action is performed (normally yielding a token with a specific
-type), the current position is set to where the last match ended and the
-matching process continues with the first regex of the current state.
+string at the *current position*.  If one of these expressions matches, a
+corresponding action is performed (such as yielding a token with a specific
+type, or changing state), the current position is set to where the last match
+ended and the matching process continues with the first regex of the current
+state.
 
-Lexer states are kept in a state stack: each time a new state is entered, the
-new state is pushed onto the stack.  The most basic lexers (like the
-`DiffLexer`) just need one state.
+Lexer states are kept on a stack: each time a new state is entered, the new
+state is pushed onto the stack.  The most basic lexers (like the `DiffLexer`)
+just need one state.
 
 Each state is defined as a list of tuples in the form (`regex`, `action`,
 `new_state`) where the last item is optional.  In the most basic form, `action`
 is a token type (like `Name.Builtin`).  That means: When `regex` matches, emit a
 token with the match text and type `tokentype` and push `new_state` on the state
 stack.  If the new state is ``'#pop'``, the topmost state is popped from the
-stack instead. (To pop more than one state, use ``'#pop:2'`` and so on.)
-``'#push'`` is a synonym for pushing the current state on the
-stack.
+stack instead.  To pop more than one state, use ``'#pop:2'`` and so on.
+``'#push'`` is a synonym for pushing the current state on the stack.
 
-The following example shows the `DiffLexer` from the builtin lexers. Note that
+The following example shows the `DiffLexer` from the builtin lexers.  Note that
 it contains some additional attributes `name`, `aliases` and `filenames` which
-aren't required for a lexer. They are used by the builtin lexer lookup
-functions.
-
-.. sourcecode:: python
+aren't required for a lexer.  They are used by the builtin lexer lookup
+functions. ::
 
     from pygments.lexer import RegexLexer
     from pygments.token import *
@@ -72,15 +73,16 @@ functions.
         }
 
 As you can see this lexer only uses one state.  When the lexer starts scanning
-the text, it first checks if the current character is a space. If this is true
-it scans everything until newline and returns the parsed data as `Text` token.
+the text, it first checks if the current character is a space.  If this is true
+it scans everything until newline and returns the data as a `Text` token (which
+is the "no special highlighting" token).
 
 If this rule doesn't match, it checks if the current char is a plus sign.  And
 so on.
 
 If no rule matches at the current position, the current char is emitted as an
-`Error` token that indicates a parsing error, and the position is increased by
-1.
+`Error` token that indicates a lexing error, and the position is increased by
+one.
 
 
 Adding and testing a new lexer
@@ -91,33 +93,33 @@ steps:
 
 First, change to the current directory containing the pygments source code:
 
-.. sourcecode:: console
+.. code-block:: console
 
     $ cd .../pygments-main
 
-Next, make sure the lexer is known from outside of the module. All modules in
-the ``pygments.lexers`` specify ``__all__``. For example, ``other.py`` sets:
+Select a matching module under ``pygments/lexers``, or create a new module for
+your lexer class.
 
-.. sourcecode:: python
+Next, make sure the lexer is known from outside of the module.  All modules in
+the ``pygments.lexers`` specify ``__all__``. For example, ``esoteric.py`` sets::
 
     __all__ = ['BrainfuckLexer', 'BefungeLexer', ...]
 
 Simply add the name of your lexer class to this list.
 
-Finally the lexer can be made publically known by rebuilding the lexer
-mapping:
+Finally the lexer can be made publicly known by rebuilding the lexer mapping:
 
-.. sourcecode:: console
+.. code-block:: console
 
     $ make mapfiles
 
 To test the new lexer, store an example file with the proper extension in
-``tests/examplefiles``. For example, to test your ``DiffLexer``, add a
+``tests/examplefiles``.  For example, to test your ``DiffLexer``, add a
 ``tests/examplefiles/example.diff`` containing a sample diff output.
 
 Now you can use pygmentize to render your example to HTML:
 
-.. sourcecode:: console
+.. code-block:: console
 
     $ ./pygmentize -O full -f html -o /tmp/example.html tests/examplefiles/example.diff
 
@@ -130,29 +132,35 @@ To view the result, open ``/tmp/example.html`` in your browser.
 
 Once the example renders as expected, you should run the complete test suite:
 
-.. sourcecode:: console
+.. code-block:: console
 
     $ make test
 
+It also tests that your lexer fulfills the lexer API and certain invariants,
+such as that the concatenation of all token text is the same as the input text.
+
 
 Regex Flags
 ===========
 
-You can either define regex flags in the regex (``r'(?x)foo bar'``) or by adding
-a `flags` attribute to your lexer class. If no attribute is defined, it defaults
-to `re.MULTILINE`. For more informations about regular expression flags see the
-`regular expressions`_ help page in the python documentation.
+You can either define regex flags locally in the regex (``r'(?x)foo bar'``) or
+globally by adding a `flags` attribute to your lexer class.  If no attribute is
+defined, it defaults to `re.MULTILINE`.  For more informations about regular
+expression flags see the page about `regular expressions`_ in the Python
+documentation.
 
-.. _regular expressions: http://docs.python.org/lib/re-syntax.html
+.. _regular expressions: http://docs.python.org/library/re.html#regular-expression-syntax
 
 
 Scanning multiple tokens at once
 ================================
 
-Here is a more complex lexer that highlights INI files. INI files consist of
-sections, comments and key = value pairs:
+So far, the `action` element in the rule tuple of regex, action and state has
+been a single token type.  Now we look at the first of several other possible
+values.
 
-.. sourcecode:: python
+Here is a more complex lexer that highlights INI files.  INI files consist of
+sections, comments and ``key = value`` pairs::
 
     from pygments.lexer import RegexLexer, bygroups
     from pygments.token import *
@@ -172,43 +180,41 @@ sections, comments and key = value pairs:
             ]
         }
 
-The lexer first looks for whitespace, comments and section names. And later it
+The lexer first looks for whitespace, comments and section names.  Later it
 looks for a line that looks like a key, value pair, separated by an ``'='``
 sign, and optional whitespace.
 
-The `bygroups` helper makes sure that each group is yielded with a different
-token type. First the `Name.Attribute` token, then a `Text` token for the
+The `bygroups` helper yields each capturing group in the regex with a different
+token type.  First the `Name.Attribute` token, then a `Text` token for the
 optional whitespace, after that a `Operator` token for the equals sign. Then a
-`Text` token for the whitespace again. The rest of the line is returned as
+`Text` token for the whitespace again.  The rest of the line is returned as
 `String`.
 
 Note that for this to work, every part of the match must be inside a capturing
 group (a ``(...)``), and there must not be any nested capturing groups.  If you
 nevertheless need a group, use a non-capturing group defined using this syntax:
-``r'(?:some|words|here)'`` (note the ``?:`` after the beginning parenthesis).
+``(?:some|words|here)`` (note the ``?:`` after the beginning parenthesis).
 
-If you find yourself needing a capturing group inside the regex which
-shouldn't be part of the output but is used in the regular expressions for
-backreferencing (eg: ``r'(<(foo|bar)>)(.*?)(</\2>)'``), you can pass `None`
-to the bygroups function and that group will be skipped in the
-output.
+If you find yourself needing a capturing group inside the regex which shouldn't
+be part of the output but is used in the regular expressions for backreferencing
+(eg: ``r'(<(foo|bar)>)(.*?)(</\2>)'``), you can pass `None` to the bygroups
+function and that group will be skipped in the output.
 
 
 Changing states
 ===============
 
-Many lexers need multiple states to work as expected. For example, some
-languages allow multiline comments to be nested. Since this is a recursive
+Many lexers need multiple states to work as expected.  For example, some
+languages allow multiline comments to be nested.  Since this is a recursive
 pattern it's impossible to lex just using regular expressions.
 
-Here is the solution:
-
-.. sourcecode:: python
+Here is a lexer that recognizes C++ style comments (multi-line with ``/* */``
+and single-line with ``//`` until end of line)::
 
     from pygments.lexer import RegexLexer
     from pygments.token import *
 
-    class ExampleLexer(RegexLexer):
+    class CppCommentLexer(RegexLexer):
         name = 'Example Lexer with states'
 
         tokens = {
@@ -227,28 +233,29 @@ Here is the solution:
         }
 
 This lexer starts lexing in the ``'root'`` state. It tries to match as much as
-possible until it finds a slash (``'/'``). If the next character after the slash
-is a star (``'*'``) the `RegexLexer` sends those two characters to the output
-stream marked as `Comment.Multiline` and continues parsing with the rules
+possible until it finds a slash (``'/'``).  If the next character after the slash
+is an asterisk (``'*'``) the `RegexLexer` sends those two characters to the
+output stream marked as `Comment.Multiline` and continues lexing with the rules
 defined in the ``'comment'`` state.
 
-If there wasn't a star after the slash, the `RegexLexer` checks if it's a
-singleline comment (eg: followed by a second slash). If this also wasn't the
-case it must be a single slash (the separate regex for a single slash must also
-be given, else the slash would be marked as an error token).
+If there wasn't an asterisk after the slash, the `RegexLexer` checks if it's a
+singleline comment (i.e. followed by a second slash).  If this also wasn't the
+case it must be a single slash, which is not a comment starter (the separate
+regex for a single slash must also be given, else the slash would be marked as
+an error token).
 
-Inside the ``'comment'`` state, we do the same thing again. Scan until the lexer
-finds a star or slash. If it's the opening of a multiline comment, push the
-``'comment'`` state on the stack and continue scanning, again in the
-``'comment'`` state.  Else, check if it's the end of the multiline comment. If
+Inside the ``'comment'`` state, we do the same thing again.  Scan until the
+lexer finds a star or slash.  If it's the opening of a multiline comment, push
+the ``'comment'`` state on the stack and continue scanning, again in the
+``'comment'`` state.  Else, check if it's the end of the multiline comment.  If
 yes, pop one state from the stack.
 
-Note: If you pop from an empty stack you'll get an `IndexError`. (There is an
+Note: If you pop from an empty stack you'll get an `IndexError`.  (There is an
 easy way to prevent this from happening: don't ``'#pop'`` in the root state).
 
 If the `RegexLexer` encounters a newline that is flagged as an error token, the
-stack is emptied and the lexer continues scanning in the ``'root'`` state. This
-helps producing error-tolerant highlighting for erroneous input, e.g. when a
+stack is emptied and the lexer continues scanning in the ``'root'`` state.  This
+can help producing error-tolerant highlighting for erroneous input, e.g. when a
 single-line string is not closed.
 
 
@@ -258,14 +265,14 @@ Advanced state tricks
 There are a few more things you can do with states:
 
 - You can push multiple states onto the stack if you give a tuple instead of a
-  simple string as the third item in a rule tuple. For example, if you want to
-  match a comment containing a directive, something like::
+  simple string as the third item in a rule tuple.  For example, if you want to
+  match a comment containing a directive, something like:
 
-      /* <processing directive>    rest of comment */
+  .. code-block:: text
 
-  you can use this rule:
+      /* <processing directive>    rest of comment */
 
-  .. sourcecode:: python
+  you can use this rule::
 
       tokens = {
           'root': [
@@ -286,7 +293,7 @@ There are a few more things you can do with states:
   When this encounters the above sample, first ``'comment'`` and ``'directive'``
   are pushed onto the stack, then the lexer continues in the directive state
   until it finds the closing ``>``, then it continues in the comment state until
-  the closing ``*/``. Then, both states are popped from the stack again and
+  the closing ``*/``.  Then, both states are popped from the stack again and
   lexing continues in the root state.
 
   .. versionadded:: 0.9
@@ -295,9 +302,7 @@ There are a few more things you can do with states:
 
 
 - You can include the rules of a state in the definition of another.  This is
-  done by using `include` from `pygments.lexer`:
-
-  .. sourcecode:: python
+  done by using `include` from `pygments.lexer`::
 
       from pygments.lexer import RegexLexer, bygroups, include
       from pygments.token import *
@@ -323,15 +328,14 @@ There are a few more things you can do with states:
           }
 
   This is a hypothetical lexer for a language that consist of functions and
-  comments. Because comments can occur at toplevel and in functions, we need
-  rules for comments in both states. As you can see, the `include` helper saves
+  comments.  Because comments can occur at toplevel and in functions, we need
+  rules for comments in both states.  As you can see, the `include` helper saves
   repeating rules that occur more than once (in this example, the state
   ``'comment'`` will never be entered by the lexer, as it's only there to be
   included in ``'root'`` and ``'function'``).
 
-
 - Sometimes, you may want to "combine" a state from existing ones.  This is
-  possible with the `combine` helper from `pygments.lexer`.
+  possible with the `combined` helper from `pygments.lexer`.
 
   If you, instead of a new state, write ``combined('state1', 'state2')`` as the
   third item of a rule tuple, a new anonymous state will be formed from state1
@@ -340,14 +344,12 @@ There are a few more things you can do with states:
   This is not used very often, but can be helpful in some cases, such as the
   `PythonLexer`'s string literal processing.
 
-- If you want your lexer to start lexing in a different state you can modify
-  the stack by overloading the `get_tokens_unprocessed()` method:
-
-  .. sourcecode:: python
+- If you want your lexer to start lexing in a different state you can modify the
+  stack by overloading the `get_tokens_unprocessed()` method::
 
       from pygments.lexer import RegexLexer
 
-      class MyLexer(RegexLexer):
+      class ExampleLexer(RegexLexer):
           tokens = {...}
 
           def get_tokens_unprocessed(self, text):
@@ -356,29 +358,69 @@ There are a few more things you can do with states:
                   yield item
 
   Some lexers like the `PhpLexer` use this to make the leading ``<?php``
-  preprocessor comments optional. Note that you can crash the lexer easily
-  by putting values into the stack that don't exist in the token map. Also
+  preprocessor comments optional.  Note that you can crash the lexer easily by
+  putting values into the stack that don't exist in the token map.  Also
   removing ``'root'`` from the stack can result in strange errors!
 
-- An empty regex at the end of a state list, combined with ``'#pop'``, can
-  act as a return point from a state that doesn't have a clear end marker.
+- In some lexers, a state should be popped if anything is encountered that isn't
+  matched by a rule in the state.  You could use an empty regex at the end of
+  the state list, but Pygments provides a more obvious way of spelling that:
+  ``default('#pop')`` is equivalent to ``('', Text, '#pop')``.
+
+  .. versionadded:: 2.0
+
+
+Subclassing lexers derived from RegexLexer
+==========================================
+
+Sometimes multiple languages are very similar, but should still be lexed by
+different lexer classes.
+
+When subclassing a lexer derived from RegexLexer, the ``tokens`` dictionaries
+defined in the parent and child class are merged.  For example::
+
+      from pygments.lexer import RegexLexer, bygroups, include
+      from pygments.token import *
+
+      class BaseLexer(RegexLexer):
+          tokens = {
+              'root': [
+                  ('[a-z]+', Name),
+                  ('"', String, 'string'),
+                  ('\s+', Text),
+              ],
+              'string': [
+                  ('[^"]+', String),
+                  ('"', String, '#pop'),
+              ],
+          }
+
+      class DerivedLexer(BaseLexer):
+          tokens = {
+              'root': [
+                  inherit,
+              ],
+              'string': [
+                  ('[^"]
+              ],
+          }
+
+  .. versionadded:: 1.6
 
 
 Using multiple lexers
 =====================
 
-Using multiple lexers for the same input can be tricky. One of the easiest
-combination techniques is shown here: You can replace the token type entry in a
-rule tuple (the second item) with a lexer class. The matched text will then be
-lexed with that lexer, and the resulting tokens will be yielded.
-
-For example, look at this stripped-down HTML lexer:
+Using multiple lexers for the same input can be tricky.  One of the easiest
+combination techniques is shown here: You can replace the action entry in a rule
+tuple with a lexer class.  The matched text will then be lexed with that lexer,
+and the resulting tokens will be yielded.
 
-.. sourcecode:: python
+For example, look at this stripped-down HTML lexer::
 
     from pygments.lexer import RegexLexer, bygroups, using
     from pygments.token import *
-    from pygments.lexers.web import JavascriptLexer
+    from pygments.lexers.javascript import JavascriptLexer
 
     class HtmlLexer(RegexLexer):
         name = 'HTML'
@@ -402,26 +444,29 @@ For example, look at this stripped-down HTML lexer:
         }
 
 Here the content of a ``<script>`` tag is passed to a newly created instance of
-a `JavascriptLexer` and not processed by the `HtmlLexer`. This is done using the
-`using` helper that takes the other lexer class as its parameter.
-
-Note the combination of `bygroups` and `using`. This makes sure that the content
-up to the ``</script>`` end tag is processed by the `JavascriptLexer`, while the
-end tag is yielded as a normal token with the `Name.Tag` type.
+a `JavascriptLexer` and not processed by the `HtmlLexer`.  This is done using
+the `using` helper that takes the other lexer class as its parameter.
 
-As an additional goodie, if the lexer class is replaced by `this` (imported from
-`pygments.lexer`), the "other" lexer will be the current one (because you cannot
-refer to the current class within the code that runs at class definition time).
+Note the combination of `bygroups` and `using`.  This makes sure that the
+content up to the ``</script>`` end tag is processed by the `JavascriptLexer`,
+while the end tag is yielded as a normal token with the `Name.Tag` type.
 
 Also note the ``(r'<\s*script\s*', Name.Tag, ('script-content', 'tag'))`` rule.
 Here, two states are pushed onto the state stack, ``'script-content'`` and
-``'tag'``.  That means that first ``'tag'`` is processed, which will parse
+``'tag'``.  That means that first ``'tag'`` is processed, which will lex
 attributes and the closing ``>``, then the ``'tag'`` state is popped and the
 next state on top of the stack will be ``'script-content'``.
 
+Since you cannot refer to the class currently being defined, use `this`
+(imported from `pygments.lexer`) to refer to the current lexer class, i.e.
+``using(this)``.  This construct may seem unnecessary, but this is often the
+most obvious way of lexing arbitrary syntax between fixed delimiters without
+introducing deeply nested states.
+
 The `using()` helper has a special keyword argument, `state`, which works as
 follows: if given, the lexer to use initially is not in the ``"root"`` state,
-but in the state given by this argument. This *only* works with a `RegexLexer`.
+but in the state given by this argument.  This does not work with advanced
+`RegexLexer` subclasses such as `ExtendedRegexLexer` (see below).
 
 Any other keywords arguments passed to `using()` are added to the keyword
 arguments used to create the lexer.
@@ -430,17 +475,15 @@ arguments used to create the lexer.
 Delegating Lexer
 ================
 
-Another approach for nested lexers is the `DelegatingLexer` which is for
-example used for the template engine lexers. It takes two lexers as
-arguments on initialisation: a `root_lexer` and a `language_lexer`.
+Another approach for nested lexers is the `DelegatingLexer` which is for example
+used for the template engine lexers.  It takes two lexers as arguments on
+initialisation: a `root_lexer` and a `language_lexer`.
 
 The input is processed as follows: First, the whole text is lexed with the
-`language_lexer`. All tokens yielded with a type of ``Other`` are then
-concatenated and given to the `root_lexer`. The language tokens of the
-`language_lexer` are then inserted into the `root_lexer`'s token stream
-at the appropriate positions.
-
-.. sourcecode:: python
+`language_lexer`.  All tokens yielded with the special type of ``Other`` are
+then concatenated and given to the `root_lexer`.  The language tokens of the
+`language_lexer` are then inserted into the `root_lexer`'s token stream at the
+appropriate positions. ::
 
     from pygments.lexer import DelegatingLexer
     from pygments.lexers.web import HtmlLexer, PhpLexer
@@ -452,10 +495,8 @@ at the appropriate positions.
 This procedure ensures that e.g. HTML with template tags in it is highlighted
 correctly even if the template tags are put into HTML tags or attributes.
 
-If you want to change the needle token ``Other`` to something else, you can
-give the lexer another token type as the third parameter:
-
-.. sourcecode:: python
+If you want to change the needle token ``Other`` to something else, you can give
+the lexer another token type as the third parameter::
 
     DelegatingLexer.__init__(MyLexer, OtherLexer, Text, **options)
 
@@ -464,24 +505,22 @@ Callbacks
 =========
 
 Sometimes the grammar of a language is so complex that a lexer would be unable
-to parse it just by using regular expressions and stacks.
+to process it just by using regular expressions and stacks.
 
 For this, the `RegexLexer` allows callbacks to be given in rule tuples, instead
 of token types (`bygroups` and `using` are nothing else but preimplemented
-callbacks). The callback must be a function taking two arguments:
+callbacks).  The callback must be a function taking two arguments:
 
 * the lexer itself
 * the match object for the last matched rule
 
 The callback must then return an iterable of (or simply yield) ``(index,
 tokentype, value)`` tuples, which are then just passed through by
-`get_tokens_unprocessed()`. The ``index`` here is the position of the token in
+`get_tokens_unprocessed()`.  The ``index`` here is the position of the token in
 the input string, ``tokentype`` is the normal token type (like `Name.Builtin`),
 and ``value`` the associated part of the input string.
 
-You can see an example here:
-
-.. sourcecode:: python
+You can see an example here::
 
     from pygments.lexer import RegexLexer
     from pygments.token import Generic
@@ -499,26 +538,25 @@ You can see an example here:
             ]
         }
 
-If the regex for the `headline_callback` matches, the function is called with the
-match object. Note that after the callback is done, processing continues
-normally, that is, after the end of the previous match. The callback has no
+If the regex for the `headline_callback` matches, the function is called with
+the match object.  Note that after the callback is done, processing continues
+normally, that is, after the end of the previous match.  The callback has no
 possibility to influence the position.
 
 There are not really any simple examples for lexer callbacks, but you can see
-them in action e.g. in the `compiled.py`_ source code in the `CLexer` and
-`JavaLexer` classes.
+them in action e.g. in the `SMLLexer` class in `ml.py`_.
 
-.. _compiled.py: http://bitbucket.org/birkenfeld/pygments-main/src/tip/pygments/lexers/compiled.py
+.. _ml.py: http://bitbucket.org/birkenfeld/pygments-main/src/tip/pygments/lexers/ml.py
 
 
 The ExtendedRegexLexer class
 ============================
 
 The `RegexLexer`, even with callbacks, unfortunately isn't powerful enough for
-the funky syntax rules of some languages that will go unnamed, such as Ruby.
+the funky syntax rules of languages such as Ruby.
 
 But fear not; even then you don't have to abandon the regular expression
-approach. For Pygments has a subclass of `RegexLexer`, the `ExtendedRegexLexer`.
+approach: Pygments has a subclass of `RegexLexer`, the `ExtendedRegexLexer`.
 All features known from RegexLexers are available here too, and the tokens are
 specified in exactly the same way, *except* for one detail:
 
@@ -542,9 +580,7 @@ creating a new one for the string argument.
 Note that because you can set the current position to anything in the callback,
 it won't be automatically be set by the caller after the callback is finished.
 For example, this is how the hypothetical lexer above would be written with the
-`ExtendedRegexLexer`:
-
-.. sourcecode:: python
+`ExtendedRegexLexer`::
 
     from pygments.lexer import ExtendedRegexLexer
     from pygments.token import Generic
@@ -564,31 +600,58 @@ For example, this is how the hypothetical lexer above would be written with the
         }
 
 This might sound confusing (and it can really be). But it is needed, and for an
-example look at the Ruby lexer in `agile.py`_.
+example look at the Ruby lexer in `ruby.py`_.
+
+.. _ruby.py: https://bitbucket.org/birkenfeld/pygments-main/src/tip/pygments/lexers/ruby.py
 
-.. _agile.py: https://bitbucket.org/birkenfeld/pygments-main/src/tip/pygments/lexers/agile.py
 
+Handling Lists of Keywords
+==========================
 
-Filtering Token Streams
+For a relatively short list (hundreds) you can construct an optimized regular
+expression directly using ``words()`` (longer lists, see next section).  This
+function handles a few things for you automatically, including escaping
+metacharacters and Python's first-match rather than longest-match in
+alternations.  Feel free to put the lists themselves in
+``pygments/lexers/_$lang_builtins.py`` (see examples there), and generated by
+code if possible.
+
+An example of using ``words()`` is something like::
+
+    from pygments.lexer import RegexLexer, words, Name
+
+    class MyLexer(RegexLexer):
+
+        tokens = {
+            'root': [
+                (words(('else', 'elseif'), suffix=r'\b'), Name.Builtin),
+                (r'\w+', Name),
+            ],
+        }
+
+As you can see, you can add ``prefix`` and ``suffix`` parts to the constructed
+regex.
+
+
+Modifying Token Streams
 =======================
 
-Some languages ship a lot of builtin functions (for example PHP). The total
+Some languages ship a lot of builtin functions (for example PHP).  The total
 amount of those functions differs from system to system because not everybody
-has every extension installed. In the case of PHP there are over 3000 builtin
-functions. That's an incredible huge amount of functions, much more than you
-can put into a regular expression.
-
-But because only `Name` tokens can be function names it's solvable by overriding
-the ``get_tokens_unprocessed()`` method. The following lexer subclasses the
-`PythonLexer` so that it highlights some additional names as pseudo keywords:
+has every extension installed.  In the case of PHP there are over 3000 builtin
+functions.  That's an incredibly huge amount of functions, much more than you
+want to put into a regular expression.
 
-.. sourcecode:: python
+But because only `Name` tokens can be function names this is solvable by
+overriding the ``get_tokens_unprocessed()`` method.  The following lexer
+subclasses the `PythonLexer` so that it highlights some additional names as
+pseudo keywords::
 
     from pygments.lexers.python import PythonLexer
     from pygments.token import Name, Keyword
 
     class MyPythonLexer(PythonLexer):
-        EXTRA_KEYWORDS = ['foo', 'bar', 'foobar', 'barfoo', 'spam', 'eggs']
+        EXTRA_KEYWORDS = set(('foo', 'bar', 'foobar', 'barfoo', 'spam', 'eggs'))
 
         def get_tokens_unprocessed(self, text):
             for index, token, value in PythonLexer.get_tokens_unprocessed(self, text):
@@ -598,5 +661,3 @@ the ``get_tokens_unprocessed()`` method. The following lexer subclasses the
                     yield index, token, value
 
 The `PhpLexer` and `LuaLexer` use this method to resolve builtin functions.
-
-.. note:: Do not confuse this with the :doc:`filter <filters>` system.
diff --git a/pygments/lexers/_asybuiltins.py b/pygments/lexers/_asy_builtins.py
index b1c65890..2dcd60d7 100644
--- a/pygments/lexers/_asybuiltins.py
+++ b/pygments/lexers/_asy_builtins.py
@@ -1,7 +1,7 @@
 # -*- coding: utf-8 -*-
 """
-    pygments.lexers._asybuiltins
-    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+    pygments.lexers._asy_builtins
+    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
     This file contains the asy-function names and asy-variable names of
     Asymptote.
diff --git a/pygments/lexers/_clbuiltins.py b/pygments/lexers/_cl_builtins.py
index 81e58234..9ed13b4a 100644
--- a/pygments/lexers/_clbuiltins.py
+++ b/pygments/lexers/_cl_builtins.py
@@ -1,7 +1,7 @@
 # -*- coding: utf-8 -*-
 """
-    pygments.lexers._clbuiltins
-    ~~~~~~~~~~~~~~~~~~~~~~~~~~~
+    pygments.lexers._cl_builtins
+    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
     ANSI Common Lisp builtins.
 
@@ -9,7 +9,7 @@
     :license: BSD, see LICENSE for details.
 """
 
-BUILTIN_FUNCTIONS = set(( # 638 functions
+BUILTIN_FUNCTIONS = set((  # 638 functions
     '<', '<=', '=', '>', '>=', '-', '/', '/=', '*', '+', '1-', '1+',
     'abort', 'abs', 'acons', 'acos', 'acosh', 'add-method', 'adjoin',
     'adjustable-array-p', 'adjust-array', 'allocate-instance',
diff --git a/pygments/lexers/_cocoabuiltins.py b/pygments/lexers/_cocoa_builtins.py
index 5550b1a6..7829c48f 100644
--- a/pygments/lexers/_cocoabuiltins.py
+++ b/pygments/lexers/_cocoa_builtins.py
@@ -1,7 +1,7 @@
 # -*- coding: utf-8 -*-
 """
-    pygments.lexers._cocoabuiltins
-    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+    pygments.lexers._cocoa_builtins
+    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
     This file defines a set of types used across Cocoa frameworks from Apple.
     There is a list of @interfaces, @protocols and some other (structs, unions)
diff --git a/pygments/lexers/_lassobuiltins.py b/pygments/lexers/_lasso_builtins.py
index aa9d2343..f7413fce 100644
--- a/pygments/lexers/_lassobuiltins.py
+++ b/pygments/lexers/_lasso_builtins.py
@@ -1,7 +1,7 @@
 # -*- coding: utf-8 -*-
 """
-    pygments.lexers._lassobuiltins
-    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+    pygments.lexers._lasso_builtins
+    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
     Built-in Lasso types, traits, methods, and members.
 
diff --git a/pygments/lexers/_luabuiltins.py b/pygments/lexers/_lua_builtins.py
index 87bfb26e..0519c6cb 100644
--- a/pygments/lexers/_luabuiltins.py
+++ b/pygments/lexers/_lua_builtins.py
@@ -1,7 +1,7 @@
 # -*- coding: utf-8 -*-
 """
-    pygments.lexers._luabuiltins
-    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+    pygments.lexers._lua_builtins
+    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
     This file contains the names and modules of lua functions
     It is able to re-generate itself, but for adding new functions you
diff --git a/pygments/lexers/_openedgebuiltins.py b/pygments/lexers/_openedge_builtins.py
index b76c340d..1bd97ca8 100644
--- a/pygments/lexers/_openedgebuiltins.py
+++ b/pygments/lexers/_openedge_builtins.py
@@ -1,7 +1,7 @@
 # -*- coding: utf-8 -*-
 """
-    pygments.lexers._openedgebuiltins
-    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+    pygments.lexers._openedge_builtins
+    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
     Builtin list for the OpenEdgeLexer.
 
diff --git a/pygments/lexers/_phpbuiltins.py b/pygments/lexers/_php_builtins.py
index 6ee91eb0..1e04cfc0 100644
--- a/pygments/lexers/_phpbuiltins.py
+++ b/pygments/lexers/_php_builtins.py
@@ -1,7 +1,7 @@
 # -*- coding: utf-8 -*-
 """
-    pygments.lexers._phpbuiltins
-    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+    pygments.lexers._php_builtins
+    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
     This file loads the function names and their modules from the
     php webpage and generates itself.
diff --git a/pygments/lexers/_sourcemodbuiltins.py b/pygments/lexers/_sourcemod_builtins.py
index 80695648..8ef6cbef 100644
--- a/pygments/lexers/_sourcemodbuiltins.py
+++ b/pygments/lexers/_sourcemod_builtins.py
@@ -1,7 +1,7 @@
 # -*- coding: utf-8 -*-
 """
-    pygments.lexers._sourcemodbuiltins
-    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+    pygments.lexers._sourcemod_builtins
+    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
     This file contains the names of SourceMod functions.
     It is able to re-generate itself.
diff --git a/pygments/lexers/_vimbuiltins.py b/pygments/lexers/_vim_builtins.py
index ee4cc311..e8e2b248 100644
--- a/pygments/lexers/_vimbuiltins.py
+++ b/pygments/lexers/_vim_builtins.py
@@ -1,4 +1,13 @@
-# This file is autogenerated by scripts/get_vimkw.py
+# -*- coding: utf-8 -*-
+"""
+    pygments.lexers._vim_builtins
+    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+    This file is autogenerated by scripts/get_vimkw.py
+
+    :copyright: Copyright 2006-2014 by the Pygments team, see AUTHORS.
+    :license: BSD, see LICENSE for details.
+"""
 
 # Split up in multiple functions so it's importable by jython, which has a
 # per-method size limit.
diff --git a/pygments/lexers/business.py b/pygments/lexers/business.py
index 399a4639..e2806664 100644
--- a/pygments/lexers/business.py
+++ b/pygments/lexers/business.py
@@ -15,7 +15,7 @@ from pygments.lexer import RegexLexer, include, words, bygroups
 from pygments.token import Text, Comment, Operator, Keyword, Name, String, \
     Number, Punctuation, Error
 
-from pygments.lexers._openedgebuiltins import OPENEDGEKEYWORDS
+from pygments.lexers._openedge_builtins import OPENEDGEKEYWORDS
 
 __all__ = ['CobolLexer', 'CobolFreeformatLexer', 'ABAPLexer', 'OpenEdgeLexer',
            'GoodDataCLLexer', 'MaqlLexer']
@@ -445,9 +445,10 @@ class OpenEdgeLexer(RegexLexer):
              r'INT64|INTEGER|INT|INTE|INTEG|INTEGE|'
              r'LOGICAL|LONGCHAR|MEMPTR|RAW|RECID|ROWID)\s*($|(?=[^0-9a-z_\-]))')
 
-    keywords = (r'(?i)(^|(?<=[^0-9a-z_\-]))(' +
-                r'|'.join(OPENEDGEKEYWORDS) +
-                r')\s*($|(?=[^0-9a-z_\-]))')
+    keywords = words(OPENEDGEKEYWORDS,
+                     prefix=r'(?i)(^|(?<=[^0-9a-z_\-]))',
+                     suffix=r'\s*($|(?=[^0-9a-z_\-]))')
+
     tokens = {
         'root': [
             (r'/\*', Comment.Multiline, 'comment'),
diff --git a/pygments/lexers/graphics.py b/pygments/lexers/graphics.py
index 7591cf55..9376f905 100644
--- a/pygments/lexers/graphics.py
+++ b/pygments/lexers/graphics.py
@@ -278,7 +278,7 @@ class AsymptoteLexer(RegexLexer):
     }
 
     def get_tokens_unprocessed(self, text):
-        from pygments.lexers._asybuiltins import ASYFUNCNAME, ASYVARNAME
+        from pygments.lexers._asy_builtins import ASYFUNCNAME, ASYVARNAME
         for index, token, value in \
                 RegexLexer.get_tokens_unprocessed(self, text):
             if token is Name and value in ASYFUNCNAME:
diff --git a/pygments/lexers/javascript.py b/pygments/lexers/javascript.py
index 6e4e07e8..968436ed 100644
--- a/pygments/lexers/javascript.py
+++ b/pygments/lexers/javascript.py
@@ -713,7 +713,7 @@ class LassoLexer(RegexLexer):
         self._builtins = set()
         self._members = set()
         if self.builtinshighlighting:
-            from pygments.lexers._lassobuiltins import BUILTINS, MEMBERS
+            from pygments.lexers._lasso_builtins import BUILTINS, MEMBERS
             for key, value in iteritems(BUILTINS):
                 self._builtins.update(value)
             for key, value in iteritems(MEMBERS):
diff --git a/pygments/lexers/lisp.py b/pygments/lexers/lisp.py
index a1a8cbef..a0a08efe 100644
--- a/pygments/lexers/lisp.py
+++ b/pygments/lexers/lisp.py
@@ -189,7 +189,7 @@ class CommonLispLexer(RegexLexer):
     symbol = r'(\|[^|]+\||(?:%s)(?:%s)*)' % (nonmacro, constituent)
 
     def __init__(self, **options):
-        from pygments.lexers._clbuiltins import BUILTIN_FUNCTIONS, \
+        from pygments.lexers._cl_builtins import BUILTIN_FUNCTIONS, \
             SPECIAL_FORMS, MACROS, LAMBDA_LIST_KEYWORDS, DECLARATIONS, \
             BUILTIN_TYPES, BUILTIN_CLASSES
         self.builtin_function = BUILTIN_FUNCTIONS
diff --git a/pygments/lexers/objective.py b/pygments/lexers/objective.py
index a9c60187..df564cf1 100644
--- a/pygments/lexers/objective.py
+++ b/pygments/lexers/objective.py
@@ -176,7 +176,7 @@ def objective(baselexer):
             return 0
 
         def get_tokens_unprocessed(self, text):
-            from pygments.lexers._cocoabuiltins import COCOA_INTERFACES, \
+            from pygments.lexers._cocoa_builtins import COCOA_INTERFACES, \
                 COCOA_PROTOCOLS, COCOA_PRIMITIVES
 
             for index, token, value in \
diff --git a/pygments/lexers/pawn.py b/pygments/lexers/pawn.py
index d506072e..d55e2cc6 100644
--- a/pygments/lexers/pawn.py
+++ b/pygments/lexers/pawn.py
@@ -112,7 +112,7 @@ class SourcePawnLexer(RegexLexer):
 
         self._functions = set()
         if self.smhighlighting:
-            from pygments.lexers._sourcemodbuiltins import FUNCTIONS
+            from pygments.lexers._sourcemod_builtins import FUNCTIONS
             self._functions.update(FUNCTIONS)
         RegexLexer.__init__(self, **options)
 
diff --git a/pygments/lexers/php.py b/pygments/lexers/php.py
index 632ac9e0..9bee6d2e 100644
--- a/pygments/lexers/php.py
+++ b/pygments/lexers/php.py
@@ -104,11 +104,11 @@ class PhpLexer(RegexLexer):
         that are known to php but are undocumented.
 
         To get a list of allowed modules have a look into the
-        `_phpbuiltins` module:
+        `_php_builtins` module:
 
         .. sourcecode:: pycon
 
-            >>> from pygments.lexers._phpbuiltins import MODULES
+            >>> from pygments.lexers._php_builtins import MODULES
             >>> MODULES.keys()
             ['PHP Options/Info', 'Zip', 'dba', ...]
 
@@ -219,7 +219,7 @@ class PhpLexer(RegexLexer):
         # collect activated functions in a set
         self._functions = set()
         if self.funcnamehighlighting:
-            from pygments.lexers._phpbuiltins import MODULES
+            from pygments.lexers._php_builtins import MODULES
             for key, value in iteritems(MODULES):
                 if key not in self.disabledmodules:
                     self._functions.update(value)
diff --git a/pygments/lexers/scripting.py b/pygments/lexers/scripting.py
index 3b3ca8f3..602877f4 100644
--- a/pygments/lexers/scripting.py
+++ b/pygments/lexers/scripting.py
@@ -35,11 +35,11 @@ class LuaLexer(RegexLexer):
         should not be highlighted. By default all modules are highlighted.
 
         To get a list of allowed modules have a look into the
-        `_luabuiltins` module:
+        `_lua_builtins` module:
 
         .. sourcecode:: pycon
 
-            >>> from pygments.lexers._luabuiltins import MODULES
+            >>> from pygments.lexers._lua_builtins import MODULES
             >>> MODULES.keys()
             ['string', 'coroutine', 'modules', 'io', 'basic', ...]
     """
@@ -122,7 +122,7 @@ class LuaLexer(RegexLexer):
 
         self._functions = set()
         if self.func_name_highlighting:
-            from pygments.lexers._luabuiltins import MODULES
+            from pygments.lexers._lua_builtins import MODULES
             for mod, func in iteritems(MODULES):
                 if mod not in self.disabled_modules:
                     self._functions.update(func)
diff --git a/pygments/lexers/textedit.py b/pygments/lexers/textedit.py
index 1f6d3fee..04dd879a 100644
--- a/pygments/lexers/textedit.py
+++ b/pygments/lexers/textedit.py
@@ -121,7 +121,7 @@ class VimLexer(RegexLexer):
     }
 
     def __init__(self, **options):
-        from pygments.lexers._vimbuiltins import command, option, auto
+        from pygments.lexers._vim_builtins import command, option, auto
         self._cmd = command
         self._opt = option
         self._aut = auto
diff --git a/scripts/get_vimkw.py b/scripts/get_vimkw.py
index 7bf2f1c5..fc4d5ec6 100644
--- a/scripts/get_vimkw.py
+++ b/scripts/get_vimkw.py
@@ -9,7 +9,16 @@ r_line = re.compile(r"^(syn keyword vimCommand contained|syn keyword vimOption "
 r_item = re.compile(r"(\w+)(?:\[(\w+)\])?")
 
 HEADER = '''\
-# This file is autogenerated by scripts/get_vimkw.py
+# -*- coding: utf-8 -*-
+"""
+    pygments.lexers._vim_builtins
+    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+    This file is autogenerated by scripts/get_vimkw.py
+
+    :copyright: Copyright 2006-2014 by the Pygments team, see AUTHORS.
+    :license: BSD, see LICENSE for details.
+"""
 
 # Split up in multiple functions so it's importable by jython, which has a
 # per-method size limit.
@@ -62,4 +71,4 @@ def is_keyword(w, keywords):
 
 if __name__ == "__main__":
     getkw("/usr/share/vim/vim74/syntax/vim.vim",
-          "pygments/lexers/_vimbuiltins.py")
+          "pygments/lexers/_vim_builtins.py")