path: root/tests/examplefiles/wdiff/wdiff_example1.wdiff.output

'.. '         Text
'-'           Text
'*'           Text
'-'           Text
' mode: rst ' Text
'-'           Text
'*'           Text
'-'           Text
'\n\n'        Text

'{+'          Generic.Inserted
'.. highlight:: python' Generic.Inserted
'+}'          Generic.Inserted
'\n\n====================\nWrite your own lexer\n====================\n\nIf a lexer for your favorite language is missing in the Pygments package, you\ncan easily write your own and extend Pygments.\n\nAll you need can be found inside the :mod:`pygments.lexer` module.  As you can\nread in the :doc:`API documentation <api>`, a lexer is a class that is\ninitialized with some keyword arguments (the lexer options) and that provides a\n:meth:`.get_tokens_unprocessed()` method which is given a string or unicode\nobject with the data to ' Text
'[-'          Generic.Deleted
'parse.'      Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
'lex.'        Generic.Inserted
'+}'          Generic.Inserted
"\n\nThe :meth:`.get_tokens_unprocessed()` method must return an iterator or iterable\ncontaining tuples in the form ``(index, token, value)``.  Normally you don't\nneed to do this since there are " Text
'[-'          Generic.Deleted
'numerous'    Generic.Deleted
'-]'          Generic.Deleted
' base lexers ' Text
'{+'          Generic.Inserted
'that do most of the work and that' Generic.Inserted
'+}'          Generic.Inserted
'\nyou can subclass.\n\n\nRegexLexer\n==========\n\n' Text

'[-'          Generic.Deleted
'A very powerful (but quite easy to use)' Generic.Deleted
'-]'          Generic.Deleted
'\n\n'        Text

'{+'          Generic.Inserted
'The'         Generic.Inserted
'+}'          Generic.Inserted
' lexer '     Text
'{+'          Generic.Inserted
"base class used by almost all of Pygments' lexers" Generic.Inserted
'+}'          Generic.Inserted
' is the\n:class:`RegexLexer`.  This\n' Text

'[-'          Generic.Deleted
'lexer base'  Generic.Deleted
'-]'          Generic.Deleted
' class allows you to define lexing rules in terms of\n*regular expressions* for different *states*.\n\nStates are groups of regular expressions that are matched against the input\nstring at the *current position*.  If one of these expressions matches, a\ncorresponding action is performed ' Text
'[-'          Generic.Deleted
'(normally'   Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
'(such as'    Generic.Inserted
'+}'          Generic.Inserted
' yielding a token with a specific\n' Text

'[-'          Generic.Deleted
'type),'      Generic.Deleted
'-]'          Generic.Deleted
'\n'          Text

'{+'          Generic.Inserted
'type, or changing state),' Generic.Inserted
'+}'          Generic.Inserted
' the current position is set to where the last match\nended and the matching process continues with the first regex of the current\nstate.\n\nLexer states are kept ' Text
'[-'          Generic.Deleted
'in'          Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
'on'          Generic.Inserted
'+}'          Generic.Inserted
' a '         Text
'[-'          Generic.Deleted
'state'       Generic.Deleted
'-]'          Generic.Deleted
" stack: each time a new state is entered, the new\nstate is pushed onto the stack.  The most basic lexers (like the `DiffLexer`)\njust need one state.\n\nEach state is defined as a list of tuples in the form (`regex`, `action`,\n`new_state`) where the last item is optional.  In the most basic form, `action`\nis a token type (like `Name.Builtin`).  That means: When `regex` matches, emit a\ntoken with the match text and type `tokentype` and push `new_state` on the state\nstack.  If the new state is ``'#pop'``, the topmost state is popped from the\nstack instead. " Text
'[-'          Generic.Deleted
'(To'         Generic.Deleted
'-]'          Generic.Deleted
'  '          Text
'{+'          Generic.Inserted
'To'          Generic.Inserted
'+}'          Generic.Inserted
" pop more than one state, use ``'#pop:2'`` and so " Text
'[-'          Generic.Deleted
'on.)'        Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
'on.'         Generic.Inserted
'+}'          Generic.Inserted
"\n``'#push'`` is a synonym for pushing the current state on the stack.\n\nThe following example shows the `DiffLexer` from the builtin lexers.  Note that\nit contains some additional attributes `name`, `aliases` and `filenames` which\naren't required for a lexer.  They are used by the builtin lexer lookup\nfunctions.\n\n" Text

'[-'          Generic.Deleted
'.. sourcecode:: python' Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
'::'          Generic.Inserted
'+}'          Generic.Inserted
"\n\n    from pygments.lexer import RegexLexer\n    from pygments.token import *\n\n    class DiffLexer(RegexLexer):\n        name = 'Diff'\n        aliases = " Text
'['           Text
"'diff'"      Text
']'           Text
'\n        filenames = ' Text
'['           Text
"'*.diff'"    Text
']'           Text
'\n\n        tokens = ' Text
'{'           Text
"\n            'root': " Text
'['           Text
"\n                (r' .*\\n', Text),\n                (r'\\" Text
'+'           Text
".*\\n', Generic.Inserted),\n                (r'" Text
'-'           Text
".*\\n', Generic.Deleted),\n                (r'@.*\\n', Generic.Subheading),\n                (r'Index.*\\n', Generic.Heading),\n                (r'=.*\\n', Generic.Heading),\n                (r'.*\\n', Text),\n            " Text
']'           Text
'\n        '  Text
'}'           Text
'\n\nAs you can see this lexer only uses one state.  When the lexer starts scanning\nthe text, it first checks if the current character is a space.  If this is true\nit scans everything until newline and returns the ' Text
'[-'          Generic.Deleted
'parsed'      Generic.Deleted
'-]'          Generic.Deleted
' data as '   Text
'{+'          Generic.Inserted
'a'           Generic.Inserted
'+}'          Generic.Inserted
' `Text` '    Text
'[-'          Generic.Deleted
'token.'      Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
'token (which\nis the "no special highlighting" token).' Generic.Inserted
'+}'          Generic.Inserted
"\n\nIf this rule doesn't match, it checks if the current char is a plus sign.  And\nso on.\n\nIf no rule matches at the current position, the current char is emitted as an\n`Error` token that indicates a " Text
'[-'          Generic.Deleted
'parsing'     Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
'lexing'      Generic.Inserted
'+}'          Generic.Inserted
' error, and the position is increased by\n' Text

'[-'          Generic.Deleted
'1.'          Generic.Deleted
'-]'          Generic.Deleted
'\n'          Text

'{+'          Generic.Inserted
'one.'        Generic.Inserted
'+}'          Generic.Inserted
'\n\n\nAdding and testing a new lexer\n==============================\n\nTo make ' Text
'[-'          Generic.Deleted
'pygments'    Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
'Pygments'    Generic.Inserted
'+}'          Generic.Inserted
' aware of your new lexer, you have to perform the following\nsteps:\n\nFirst, change to the current directory containing the ' Text
'[-'          Generic.Deleted
'pygments'    Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
'Pygments'    Generic.Inserted
'+}'          Generic.Inserted
' source code:\n\n.. ' Text
'[-'          Generic.Deleted
'sourcecode::' Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
'code'        Generic.Inserted
'-'           Generic.Inserted
'block::'     Generic.Inserted
'+}'          Generic.Inserted
' console\n\n    $ cd .../pygments' Text
'-'           Text
'main\n\n'    Text

'{+'          Generic.Inserted
'Select a matching module under ``pygments/lexers``, or create a new module for\nyour lexer class.' Generic.Inserted
'+}'          Generic.Inserted
'\n\nNext, make sure the lexer is known from outside of the module.  All modules in\nthe ``pygments.lexers`` specify ``__all__``. For example, ' Text
'[-'          Generic.Deleted
'``other.py`` sets:\n\n.. sourcecode:: python' Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
'``esoteric.py`` sets::' Generic.Inserted
'+}'          Generic.Inserted
'\n\n    __all__ = ' Text
'['           Text
"'BrainfuckLexer', 'BefungeLexer', ..." Text
']'           Text
'\n\nSimply add the name of your lexer class to this list.\n\nFinally the lexer can be made ' Text
'[-'          Generic.Deleted
'publically'  Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
'publicly'    Generic.Inserted
'+}'          Generic.Inserted
' known by rebuilding the lexer mapping:\n\n.. ' Text
'[-'          Generic.Deleted
'sourcecode::' Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
'code'        Generic.Inserted
'-'           Generic.Inserted
'block::'     Generic.Inserted
'+}'          Generic.Inserted
' console\n\n    $ make mapfiles\n\nTo test the new lexer, store an example file with the proper extension in\n``tests/examplefiles``.  For example, to test your ``DiffLexer``, add a\n``tests/examplefiles/example.diff`` containing a sample diff output.\n\nNow you can use pygmentize to render your example to HTML:\n\n.. ' Text
'[-'          Generic.Deleted
'sourcecode::' Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
'code'        Generic.Inserted
'-'           Generic.Inserted
'block::'     Generic.Inserted
'+}'          Generic.Inserted
' console\n\n    $ ./pygmentize ' Text
'-'           Text
'O full '     Text
'-'           Text
'f html '     Text
'-'           Text
'o /tmp/example.html tests/examplefiles/example.diff\n\nNote that this ' Text
'[-'          Generic.Deleted
'explicitely' Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
'explicitly'  Generic.Inserted
'+}'          Generic.Inserted
' calls the ``pygmentize`` in the current directory\nby preceding it with ``./``. This ensures your modifications are used.\nOtherwise a possibly already installed, unmodified version without your new\nlexer would have been called from the system search path (``$PATH``).\n\nTo view the result, open ``/tmp/example.html`` in your browser.\n\nOnce the example renders as expected, you should run the complete test suite:\n\n.. ' Text
'[-'          Generic.Deleted
'sourcecode::' Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
'code'        Generic.Inserted
'-'           Generic.Inserted
'block::'     Generic.Inserted
'+}'          Generic.Inserted
' console\n\n    $ make test\n\n' Text

'{+'          Generic.Inserted
'It also tests that your lexer fulfills the lexer API and certain invariants,\nsuch as that the concatenation of all token text is the same as the input text.' Generic.Inserted
'+}'          Generic.Inserted
'\n\n\nRegex Flags\n===========\n\nYou can either define regex flags ' Text
'{+'          Generic.Inserted
'locally'     Generic.Inserted
'+}'          Generic.Inserted
" in the regex (``r'(?x)foo bar'``) or\n" Text

'{+'          Generic.Inserted
'globally'    Generic.Inserted
'+}'          Generic.Inserted
' by adding a `flags` attribute to your lexer class.  If no attribute is\ndefined, it defaults to `re.MULTILINE`.  For more ' Text
'[-'          Generic.Deleted
'informations' Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
'information' Generic.Inserted
'+}'          Generic.Inserted
' about regular\nexpression flags see the ' Text
'{+'          Generic.Inserted
'page about'  Generic.Inserted
'+}'          Generic.Inserted
' `regular expressions`_ ' Text
'[-'          Generic.Deleted
'help page'   Generic.Deleted
'-]'          Generic.Deleted
' in the '    Text
'[-'          Generic.Deleted
'python'      Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
'Python'      Generic.Inserted
'+}'          Generic.Inserted
'\ndocumentation.\n\n.. _regular expressions: ' Text
'[-'          Generic.Deleted
'http://docs.python.org/lib/re' Generic.Deleted
'-'           Generic.Deleted
'syntax.html' Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
'http://docs.python.org/library/re.html#regular' Generic.Inserted
'-'           Generic.Inserted
'expression'  Generic.Inserted
'-'           Generic.Inserted
'syntax'      Generic.Inserted
'+}'          Generic.Inserted
'\n\n\nScanning multiple tokens at once\n================================\n\n' Text

'{+'          Generic.Inserted
'So far, the `action` element in the rule tuple of regex, action and state has\nbeen a single token type.  Now we look at the first of several other possible\nvalues.' Generic.Inserted
'+}'          Generic.Inserted
'\n\nHere is a more complex lexer that highlights INI files.  INI files consist of\nsections, comments and ' Text
'[-'          Generic.Deleted
'key'         Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
'``key'       Generic.Inserted
'+}'          Generic.Inserted
' = '         Text
'[-'          Generic.Deleted
'value pairs:\n\n.. sourcecode:: python' Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
'value`` pairs::' Generic.Inserted
'+}'          Generic.Inserted
"\n\n    from pygments.lexer import RegexLexer, bygroups\n    from pygments.token import *\n\n    class IniLexer(RegexLexer):\n        name = 'INI'\n        aliases = " Text
'['           Text
"'ini', 'cfg'" Text
']'           Text
'\n        filenames = ' Text
'['           Text
"'*.ini', '*.cfg'" Text
']'           Text
'\n\n        tokens = ' Text
'{'           Text
"\n            'root': " Text
'['           Text
"\n                (r'\\s" Text
'+'           Text
"', Text),\n                (r';.*?$', Comment),\n                (r'\\" Text
'['           Text
'.*?\\'       Text
']'           Text
"$', Keyword),\n                (r'(.*?)(\\s*)(=)(\\s*)(.*?)$',\n                 bygroups(Name.Attribute, Text, Operator, Text, String))\n            " Text
']'           Text
'\n        '  Text
'}'           Text
'\n\nThe lexer first looks for whitespace, comments and section names. ' Text
'[-'          Generic.Deleted
'And later'   Generic.Deleted
'-]'          Generic.Deleted
'  '          Text
'{+'          Generic.Inserted
'Later'       Generic.Inserted
'+}'          Generic.Inserted
" it\nlooks for a line that looks like a key, value pair, separated by an ``'='``\nsign, and optional whitespace.\n\nThe `bygroups` helper " Text
'[-'          Generic.Deleted
'makes sure that' Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
'yields'      Generic.Inserted
'+}'          Generic.Inserted
' each '      Text
'{+'          Generic.Inserted
'capturing'   Generic.Inserted
'+}'          Generic.Inserted
' group '     Text
'[-'          Generic.Deleted
'is yielded'  Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
'in the regex' Generic.Inserted
'+}'          Generic.Inserted
' with a different\ntoken type.  First the `Name.Attribute` token, then a `Text` token for the\noptional whitespace, after that a `Operator` token for the equals sign. Then a\n`Text` token for the whitespace again.  The rest of the line is returned as\n`String`.\n\nNote that for this to work, every part of the match must be inside a capturing\ngroup (a ``(...)``), and there must not be any nested capturing groups.  If you\nnevertheless need a group, use a non' Text
'-'           Text
'capturing group defined using this syntax:\n' Text

'[-'          Generic.Deleted
"``r'(?:some|words|here)'``" Generic.Deleted
'-]'          Generic.Deleted
'\n'          Text

'{+'          Generic.Inserted
'``(?:some|words|here)``' Generic.Inserted
'+}'          Generic.Inserted
" (note the ``?:`` after the beginning parenthesis).\n\nIf you find yourself needing a capturing group inside the regex which shouldn't\nbe part of the output but is used in the regular expressions for backreferencing\n(eg: ``r'(<(foo|bar)>)(.*?)(</\\2>)'``), you can pass `None` to the bygroups\nfunction and " Text
'[-'          Generic.Deleted
'it will skip' Generic.Deleted
'-]'          Generic.Deleted
" that group will be skipped in the output.\n\n\nChanging states\n===============\n\nMany lexers need multiple states to work as expected.  For example, some\nlanguages allow multiline comments to be nested.  Since this is a recursive\npattern it's impossible to lex just using regular expressions.\n\nHere is " Text
'[-'          Generic.Deleted
'the solution:\n\n.. sourcecode:: python' Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
'a lexer that recognizes C' Generic.Inserted
'+'           Generic.Inserted
'+'           Generic.Inserted
' style comments (multi' Generic.Inserted
'-'           Generic.Inserted
'line with ``/* */``\nand single' Generic.Inserted
'-'           Generic.Inserted
'line with ``//`` until end of line)::' Generic.Inserted
'+}'          Generic.Inserted
'\n\n    from pygments.lexer import RegexLexer\n    from pygments.token import *\n\n    class ' Text
'[-'          Generic.Deleted
'ExampleLexer(RegexLexer):' Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
'CppCommentLexer(RegexLexer):' Generic.Inserted
'+}'          Generic.Inserted
"\n        name = 'Example Lexer with states'\n\n        tokens = " Text
'{'           Text
"\n            'root': " Text
'['           Text
"\n                (r'" Text
'['           Text
'^/'          Text
']'           Text
'+'           Text
"', Text),\n                (r'/\\*', Comment.Multiline, 'comment'),\n                (r'//.*?$', Comment.Singleline),\n                (r'/', Text)\n            " Text
']'           Text
",\n            'comment': " Text
'['           Text
"\n                (r'" Text
'['           Text
'^*/'         Text
']'           Text
"', Comment.Multiline),\n                (r'/\\*', Comment.Multiline, '#push'),\n                (r'\\*/', Comment.Multiline, '#pop'),\n                (r'" Text
'['           Text
'*/'          Text
']'           Text
"', Comment.Multiline)\n            " Text
']'           Text
'\n        '  Text
'}'           Text
"\n\nThis lexer starts lexing in the ``'root'`` state. It tries to match as much as\npossible until it finds a slash (``'/'``).  If the next character after the slash\nis " Text
'[-'          Generic.Deleted
'a star'      Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
'an asterisk' Generic.Inserted
'+}'          Generic.Inserted
" (``'*'``) the `RegexLexer` sends those two characters to the\noutput stream marked as `Comment.Multiline` and continues " Text
'[-'          Generic.Deleted
'parsing'     Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
'lexing'      Generic.Inserted
'+}'          Generic.Inserted
" with the rules\ndefined in the ``'comment'`` state.\n\nIf there wasn't " Text
'[-'          Generic.Deleted
'a star'      Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
'an asterisk' Generic.Inserted
'+}'          Generic.Inserted
" after the slash, the `RegexLexer` checks if it's a\n" Text

'[-'          Generic.Deleted
'singleline'  Generic.Deleted
'-]'          Generic.Deleted
'\n'          Text

'{+'          Generic.Inserted
'Singleline'  Generic.Inserted
'+}'          Generic.Inserted
' comment '   Text
'[-'          Generic.Deleted
'(eg:'        Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
'(i.e.'       Generic.Inserted
'+}'          Generic.Inserted
" followed by a second slash).  If this also wasn't the\ncase it must be a single " Text
'[-'          Generic.Deleted
'slash'       Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
'slash, which is not a comment starter' Generic.Inserted
'+}'          Generic.Inserted
" (the separate\nregex for a single slash must also be given, else the slash would be marked as\nan error token).\n\nInside the ``'comment'`` state, we do the same thing again.  Scan until the\nlexer finds a star or slash.  If it's the opening of a multiline comment, push\nthe ``'comment'`` state on the stack and continue scanning, again in the\n``'comment'`` state.  Else, check if it's the end of the multiline comment.  If\nyes, pop one state from the stack.\n\nNote: If you pop from an empty stack you'll get an `IndexError`.  (There is an\neasy way to prevent this from happening: don't ``'#pop'`` in the root state).\n\nIf the `RegexLexer` encounters a newline that is flagged as an error token, the\nstack is emptied and the lexer continues scanning in the ``'root'`` state.  This\n" Text

'[-'          Generic.Deleted
'helps'       Generic.Deleted
'-]'          Generic.Deleted
'\n'          Text

'{+'          Generic.Inserted
'can help'    Generic.Inserted
'+}'          Generic.Inserted
' producing error' Text
'-'           Text
'tolerant highlighting for erroneous input, e.g. when a\nsingle' Text
'-'           Text
'line string is not closed.\n\n\nAdvanced state tricks\n=====================\n\nThere are a few more things you can do with states:\n\n' Text

'-'           Text
' You can push multiple states onto the stack if you give a tuple instead of a\n  simple string as the third item in a rule tuple.  For example, if you want to\n  match a comment containing a directive, something ' Text
'[-'          Generic.Deleted
'like::'      Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
'like:\n\n  .. code' Generic.Inserted
'-'           Generic.Inserted
'block:: text' Generic.Inserted
'+}'          Generic.Inserted
'\n\n      /* <processing directive>    rest of comment */\n\n  you can use this ' Text
'[-'          Generic.Deleted
'rule:\n\n  .. sourcecode:: python' Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
'rule::'      Generic.Inserted
'+}'          Generic.Inserted
'\n\n      tokens = ' Text
'{'           Text
"\n          'root': " Text
'['           Text
"\n              (r'/\\* <', Comment, ('comment', 'directive')),\n              ...\n          " Text
']'           Text
",\n          'directive': " Text
'['           Text
"\n              (r'" Text
'['           Text
'^>'          Text
']'           Text
"*', Comment.Directive),\n              (r'>', Comment, '#pop'),\n          " Text
']'           Text
",\n          'comment': " Text
'['           Text
"\n              (r'" Text
'['           Text
'^*'          Text
']'           Text
'+'           Text
"', Comment),\n              (r'\\*/', Comment, '#pop'),\n              (r'\\*', Comment),\n          " Text
']'           Text
'\n      '    Text
'}'           Text
"\n\n  When this encounters the above sample, first ``'comment'`` and ``'directive'``\n  are pushed onto the stack, then the lexer continues in the directive state\n  until it finds the closing ``>``, then it continues in the comment state until\n  the closing ``*/``.  Then, both states are popped from the stack again and\n  lexing continues in the root state.\n\n  .. versionadded:: 0.9\n     The tuple can contain the special ``'#push'`` and ``'#pop'`` (but not\n     ``'#pop:n'``) directives.\n\n\n" Text

'-'           Text
' You can include the rules of a state in the definition of another.  This is\n  done by using `include` from ' Text
'[-'          Generic.Deleted
'`pygments.lexer`:\n\n  .. sourcecode:: python' Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
'`pygments.lexer`::' Generic.Inserted
'+}'          Generic.Inserted
'\n\n      from pygments.lexer import RegexLexer, bygroups, include\n      from pygments.token import *\n\n      class ExampleLexer(RegexLexer):\n          tokens = ' Text
'{'           Text
"\n              'comments': " Text
'['           Text
"\n                  (r'/\\*.*?\\*/', Comment),\n                  (r'//.*?\\n', Comment),\n              " Text
']'           Text
",\n              'root': " Text
'['           Text
"\n                  include('comments'),\n                  (r'(function )(\\w" Text
'+'           Text
')( '         Text
'{'           Text
")',\n                   bygroups(Keyword, Name, Keyword), 'function'),\n                  (r'.', Text),\n              " Text
']'           Text
",\n              'function': " Text
'['           Text
"\n                  (r'" Text
'['           Text
'^'           Text
'}'           Text
'/'           Text
']'           Text
'+'           Text
"', Text),\n                  include('comments'),\n                  (r'/', Text),\n                  " Text
'[-'          Generic.Deleted
"(r'"         Generic.Deleted
'}'           Generic.Deleted
"',"          Generic.Deleted
'-]'          Generic.Deleted
'\n                  ' Text
'{+'          Generic.Inserted
"(r'\\"       Generic.Inserted
'}'           Generic.Inserted
"',"          Generic.Inserted
'+}'          Generic.Inserted
" Keyword, '#pop'),\n              " Text
']'           Text
'\n          ' Text
'}'           Text
"\n\n  This is a hypothetical lexer for a language that consist of functions and\n  comments.  Because comments can occur at toplevel and in functions, we need\n  rules for comments in both states.  As you can see, the `include` helper saves\n  repeating rules that occur more than once (in this example, the state\n  ``'comment'`` will never be entered by the lexer, as it's only there to be\n  included in ``'root'`` and ``'function'``).\n\n" Text

'-'           Text
' Sometimes, you may want to "combine" a state from existing ones.  This is\n  possible with the ' Text
'[-'          Generic.Deleted
'`combine`'   Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
'`combined`'  Generic.Inserted
'+}'          Generic.Inserted
" helper from `pygments.lexer`.\n\n  If you, instead of a new state, write ``combined('state1', 'state2')`` as the\n  third item of a rule tuple, a new anonymous state will be formed from state1\n  and state2 and if the rule matches, the lexer will enter this state.\n\n  This is not used very often, but can be helpful in some cases, such as the\n  `PythonLexer`'s string literal processing.\n\n" Text

'-'           Text
' If you want your lexer to start lexing in a different state you can modify the\n  stack by ' Text
'[-'          Generic.Deleted
'overloading' Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
'overriding'  Generic.Inserted
'+}'          Generic.Inserted
' the `get_tokens_unprocessed()` ' Text
'[-'          Generic.Deleted
'method:\n\n  .. sourcecode:: python' Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
'method::'    Generic.Inserted
'+}'          Generic.Inserted
'\n\n      from pygments.lexer import RegexLexer\n\n      class ' Text
'[-'          Generic.Deleted
'MyLexer(RegexLexer):' Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
'ExampleLexer(RegexLexer):' Generic.Inserted
'+}'          Generic.Inserted
'\n          tokens = ' Text
'{'           Text
'...'         Text
'}'           Text
'\n\n          def get_tokens_unprocessed(self, ' Text
'[-'          Generic.Deleted
'text):\n              stack = ' Generic.Deleted
'['           Generic.Deleted
"'root', 'otherstate'" Generic.Deleted
']'           Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
"text, stack=('root', 'otherstate')):" Generic.Inserted
'+}'          Generic.Inserted
"\n              for item in RegexLexer.get_tokens_unprocessed(text, stack):\n                  yield item\n\n  Some lexers like the `PhpLexer` use this to make the leading ``<?php``\n  preprocessor comments optional.  Note that you can crash the lexer easily by\n  putting values into the stack that don't exist in the token map.  Also\n  removing ``'root'`` from the stack can result in strange errors!\n\n" Text

'-'           Text
' '           Text
'[-'          Generic.Deleted
'An'          Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
"In some lexers, a state should be popped if anything is encountered that isn't\n  matched by a rule in the state.  You could use an" Generic.Inserted
'+}'          Generic.Inserted
' empty regex at the end of ' Text
'[-'          Generic.Deleted
'a'           Generic.Deleted
'-]'          Generic.Deleted
'\n  '        Text
'{+'          Generic.Inserted
'the'         Generic.Inserted
'+}'          Generic.Inserted
' state list, ' Text
'[-'          Generic.Deleted
"combined with ``'#pop'``, can\n  act as" Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
'but Pygments provides' Generic.Inserted
'+}'          Generic.Inserted
' a '         Text
'[-'          Generic.Deleted
'return point' Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
"more obvious way of spelling that:\n  ``default('#pop')`` is equivalent to ``('', Text, '#pop')``.\n\n  .. versionadded:: 2.0\n\n\nSubclassing lexers derived" Generic.Inserted
'+}'          Generic.Inserted
' from '      Text
'{+'          Generic.Inserted
'RegexLexer\n==========================================\n\n.. versionadded:: 1.6\n\nSometimes multiple languages are very similar, but should still be lexed by\ndifferent lexer classes.\n\nWhen subclassing' Generic.Inserted
'+}'          Generic.Inserted
' a '         Text
'{+'          Generic.Inserted
'lexer derived from RegexLexer, the ``tokens`` dictionaries\ndefined in the parent and child class are merged.  For example::\n\n      from pygments.lexer import RegexLexer, inherit\n      from pygments.token import *\n\n      class BaseLexer(RegexLexer):\n          tokens = ' Generic.Inserted
'{'           Generic.Inserted
"\n              'root': " Generic.Inserted
'['           Generic.Inserted
"\n                  ('" Generic.Inserted
'['           Generic.Inserted
'a'           Generic.Inserted
'-'           Generic.Inserted
'z'           Generic.Inserted
']'           Generic.Inserted
'+'           Generic.Inserted
'\', Name),\n                  (r\'/\\*\', Comment, \'comment\'),\n                  (\'"\', String, \'string\'),\n                  (\'\\s' Generic.Inserted
'+'           Generic.Inserted
"', Text),\n              " Generic.Inserted
']'           Generic.Inserted
",\n              'string': " Generic.Inserted
'['           Generic.Inserted
"\n                  ('" Generic.Inserted
'['           Generic.Inserted
'^"'          Generic.Inserted
']'           Generic.Inserted
'+'           Generic.Inserted
'\', String),\n                  (\'"\', String, \'#pop\'),\n              ' Generic.Inserted
']'           Generic.Inserted
",\n              'comment': " Generic.Inserted
'['           Generic.Inserted
'\n                  ...\n              ' Generic.Inserted
']'           Generic.Inserted
',\n          ' Generic.Inserted
'}'           Generic.Inserted
'\n\n      class DerivedLexer(BaseLexer):\n          tokens = ' Generic.Inserted
'{'           Generic.Inserted
"\n              'root': " Generic.Inserted
'['           Generic.Inserted
"\n                  ('" Generic.Inserted
'['           Generic.Inserted
'0'           Generic.Inserted
'-'           Generic.Inserted
'9'           Generic.Inserted
']'           Generic.Inserted
'+'           Generic.Inserted
"', Number),\n                  inherit,\n              " Generic.Inserted
']'           Generic.Inserted
",\n              'string': " Generic.Inserted
'['           Generic.Inserted
"\n                  (r'" Generic.Inserted
'['           Generic.Inserted
'^"\\\\'      Generic.Inserted
']'           Generic.Inserted
'+'           Generic.Inserted
'\', String),\n                  (r\'\\\\.\', String.Escape),\n                  (\'"\', String, \'#pop\'),\n              ' Generic.Inserted
']'           Generic.Inserted
',\n          ' Generic.Inserted
'}'           Generic.Inserted
'\n\nThe `BaseLexer` defines two states, lexing names and strings.  The\n`DerivedLexer` defines its own tokens dictionary, which extends the definitions\nof the base lexer:\n\n* The "root"' Generic.Inserted
'+}'          Generic.Inserted
' state '     Text
'{+'          Generic.Inserted
'has an additional rule and then the special object `inherit`,\n  which tells Pygments to insert the token definitions of the parent class at' Generic.Inserted
'+}'          Generic.Inserted
'\n  that '   Text
'[-'          Generic.Deleted
"doesn't have a clear end marker." Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
'point.\n\n* The "string" state is replaced entirely, since there is not `inherit` rule.\n\n* The "comment" state is inherited entirely.' Generic.Inserted
'+}'          Generic.Inserted
'\n\n\nUsing multiple lexers\n=====================\n\nUsing multiple lexers for the same input can be tricky.  One of the easiest\ncombination techniques is shown here: You can replace the ' Text
'[-'          Generic.Deleted
'token type'  Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
'action'      Generic.Inserted
'+}'          Generic.Inserted
' entry in a rule\ntuple ' Text
'[-'          Generic.Deleted
'(the second item)' Generic.Deleted
'-]'          Generic.Deleted
' with a lexer class.  The matched text will then be lexed with that lexer,\nand the resulting tokens will be yielded.\n\nFor example, look at this stripped' Text
'-'           Text
'down HTML '  Text
'[-'          Generic.Deleted
'lexer:\n\n.. sourcecode:: python' Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
'lexer::'     Generic.Inserted
'+}'          Generic.Inserted
'\n\n    from pygments.lexer import RegexLexer, bygroups, using\n    from pygments.token import *\n    from ' Text
'[-'          Generic.Deleted
'pygments.lexers.web' Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
'pygments.lexers.javascript' Generic.Inserted
'+}'          Generic.Inserted
" import JavascriptLexer\n\n    class HtmlLexer(RegexLexer):\n        name = 'HTML'\n        aliases = " Text
'['           Text
"'html'"      Text
']'           Text
'\n        filenames = ' Text
'['           Text
"'*.html', '*.htm'" Text
']'           Text
'\n\n        flags = re.IGNORECASE | re.DOTALL\n        tokens = ' Text
'{'           Text
"\n            'root': " Text
'['           Text
"\n                ('" Text
'['           Text
'^<&'         Text
']'           Text
'+'           Text
"', Text),\n                ('&.*?;', Name.Entity),\n                (r'<\\s*script\\s*', Name.Tag, ('script" Text
'-'           Text
"content', 'tag')),\n                (r'<\\s*" Text
'['           Text
'a'           Text
'-'           Text
'zA'          Text
'-'           Text
'Z0'          Text
'-'           Text
'9:'          Text
']'           Text
'+'           Text
"', Name.Tag, 'tag'),\n                (r'<\\s*/\\s*" Text
'['           Text
'a'           Text
'-'           Text
'zA'          Text
'-'           Text
'Z0'          Text
'-'           Text
'9:'          Text
']'           Text
'+'           Text
"\\s*>', Name.Tag),\n            " Text
']'           Text
",\n            'script" Text
'-'           Text
"content': "  Text
'['           Text
"\n                (r'(." Text
'+'           Text
"?)(<\\s*/\\s*script\\s*>)',\n                 bygroups(using(JavascriptLexer), Name.Tag),\n                 '#pop'),\n            " Text
']'           Text
'\n        '  Text
'}'           Text
'\n\nHere the content of a ``<script>`` tag is passed to a newly created instance of\na `JavascriptLexer` and not processed by the `HtmlLexer`.  This is done using\nthe `using` helper that takes the other lexer class as its parameter.\n\nNote the combination of `bygroups` and `using`.  This makes sure that the\ncontent up to the ``</script>`` end tag is processed by the `JavascriptLexer`,\nwhile the end tag is yielded as a normal token with the `Name.Tag` type.\n\n' Text

'[-'          Generic.Deleted
'As an additional goodie, if the lexer class is replaced by `this` (imported from\n`pygments.lexer`), the "other" lexer will be the current one (because you cannot\nrefer to the current class within the code that runs at class definition time).' Generic.Deleted
'-]'          Generic.Deleted
"\n\nAlso note the ``(r'<\\s*script\\s*', Name.Tag, ('script" Text
'-'           Text
"content', 'tag'))`` rule.\nHere, two states are pushed onto the state stack, ``'script" Text
'-'           Text
"content'`` and\n``'tag'``.  That means that first ``'tag'`` is processed, which will " Text
'[-'          Generic.Deleted
'parse'       Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
'lex'         Generic.Inserted
'+}'          Generic.Inserted
"\nattributes and the closing ``>``, then the ``'tag'`` state is popped and the\nnext state on top of the stack will be ``'script" Text
'-'           Text
"content'``.\n\n" Text

'{+'          Generic.Inserted
'Since you cannot refer to the class currently being defined, use `this`\n(imported from `pygments.lexer`) to refer to the current lexer class, i.e.\n``using(this)``.  This construct may seem unnecessary, but this is often the\nmost obvious way of lexing arbitrary syntax between fixed delimiters without\nintroducing deeply nested states.' Generic.Inserted
'+}'          Generic.Inserted
'\n\nThe `using()` helper has a special keyword argument, `state`, which works as\nfollows: if given, the lexer to use initially is not in the ``"root"`` state,\nbut in the state given by this argument.  This ' Text
'[-'          Generic.Deleted
'*only* works' Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
'does not work' Generic.Inserted
'+}'          Generic.Inserted
' with '      Text
'[-'          Generic.Deleted
'a `RegexLexer`.' Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
'advanced\n`RegexLexer` subclasses such as `ExtendedRegexLexer` (see below).' Generic.Inserted
'+}'          Generic.Inserted
'\n\nAny other keywords arguments passed to `using()` are added to the keyword\narguments used to create the lexer.\n\n\nDelegating Lexer\n================\n\nAnother approach for nested lexers is the `DelegatingLexer` which is for example\nused for the template engine lexers.  It takes two lexers as arguments on\ninitialisation: a `root_lexer` and a `language_lexer`.\n\nThe input is processed as follows: First, the whole text is lexed with the\n`language_lexer`.  All tokens yielded with ' Text
'[-'          Generic.Deleted
'a'           Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
'the special' Generic.Inserted
'+}'          Generic.Inserted
" type of ``Other`` are\nthen concatenated and given to the `root_lexer`.  The language tokens of the\n`language_lexer` are then inserted into the `root_lexer`'s token stream at the\nappropriate positions.\n\n" Text

'[-'          Generic.Deleted
'.. sourcecode:: python' Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
'::'          Generic.Inserted
'+}'          Generic.Inserted
'\n\n    from pygments.lexer import DelegatingLexer\n    from pygments.lexers.web import HtmlLexer, PhpLexer\n\n    class HtmlPhpLexer(DelegatingLexer):\n        def __init__(self, **options):\n            super(HtmlPhpLexer, self).__init__(HtmlLexer, PhpLexer, **options)\n\nThis procedure ensures that e.g. HTML with template tags in it is highlighted\ncorrectly even if the template tags are put into HTML tags or attributes.\n\nIf you want to change the needle token ``Other`` to something else, you can give\nthe lexer another token type as the third ' Text
'[-'          Generic.Deleted
'parameter:\n\n.. sourcecode:: python' Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
'parameter::' Generic.Inserted
'+}'          Generic.Inserted
'\n\n    DelegatingLexer.__init__(MyLexer, OtherLexer, Text, **options)\n\n\nCallbacks\n=========\n\nSometimes the grammar of a language is so complex that a lexer would be unable\nto ' Text
'[-'          Generic.Deleted
'parse'       Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
'process'     Generic.Inserted
'+}'          Generic.Inserted
' it just by using regular expressions and stacks.\n\nFor this, the `RegexLexer` allows callbacks to be given in rule tuples, instead\nof token types (`bygroups` and `using` are nothing else but preimplemented\ncallbacks).  The callback must be a function taking two arguments:\n\n* the lexer itself\n* the match object for the last matched rule\n\nThe callback must then return an iterable of (or simply yield) ``(index,\ntokentype, value)`` tuples, which are then just passed through by\n`get_tokens_unprocessed()`.  The ``index`` here is the position of the token in\nthe input string, ``tokentype`` is the normal token type (like `Name.Builtin`),\nand ``value`` the associated part of the input string.\n\nYou can see an example ' Text
'[-'          Generic.Deleted
'here:\n\n.. sourcecode:: python' Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
'here::'      Generic.Inserted
'+}'          Generic.Inserted
'\n\n    from pygments.lexer import RegexLexer\n    from pygments.token import Generic\n\n    class HypotheticLexer(RegexLexer):\n\n        def headline_callback(lexer, match):\n            equal_signs = match.group(1)\n            text = match.group(2)\n            yield match.start(), Generic.Headline, equal_signs ' Text
'+'           Text
' text '      Text
'+'           Text
' equal_signs\n\n        tokens = ' Text
'{'           Text
"\n            'root': " Text
'['           Text
"\n                (r'(=" Text
'+'           Text
")(.*?)(\\1)', headline_callback)\n            " Text
']'           Text
'\n        '  Text
'}'           Text
'\n\nIf the regex for the `headline_callback` matches, the function is called with\nthe match object.  Note that after the callback is done, processing continues\nnormally, that is, after the end of the previous match.  The callback has no\npossibility to influence the position.\n\nThere are not really any simple examples for lexer callbacks, but you can see\nthem in action e.g. in the ' Text
'[-'          Generic.Deleted
'`compiled.py`_ source code' Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
'`SMLLexer` class' Generic.Inserted
'+}'          Generic.Inserted
' in '        Text
'[-'          Generic.Deleted
'the `CLexer` and\n`JavaLexer` classes.' Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
'`ml.py`_.'   Generic.Inserted
'+}'          Generic.Inserted
'\n\n.. '     Text
'[-'          Generic.Deleted
'_compiled.py: http://bitbucket.org/birkenfeld/pygments' Generic.Deleted
'-'           Generic.Deleted
'main/src/tip/pygments/lexers/compiled.py' Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
'_ml.py: http://bitbucket.org/birkenfeld/pygments' Generic.Inserted
'-'           Generic.Inserted
'main/src/tip/pygments/lexers/ml.py' Generic.Inserted
'+}'          Generic.Inserted
"\n\n\nThe ExtendedRegexLexer class\n============================\n\nThe `RegexLexer`, even with callbacks, unfortunately isn't powerful enough for\nthe funky syntax rules of " Text
'[-'          Generic.Deleted
'some'        Generic.Deleted
'-]'          Generic.Deleted
' languages ' Text
'[-'          Generic.Deleted
'that will go unnamed,' Generic.Deleted
'-]'          Generic.Deleted
" such as Ruby.\n\nBut fear not; even then you don't have to abandon the regular expression\n" Text

'[-'          Generic.Deleted
'approach. For' Generic.Deleted
'-]'          Generic.Deleted
'\n'          Text

'{+'          Generic.Inserted
'approach:'   Generic.Inserted
'+}'          Generic.Inserted
' Pygments has a subclass of `RegexLexer`, the `ExtendedRegexLexer`.\nAll features known from RegexLexers are available here too, and the tokens are\nspecified in exactly the same way, *except* for one detail:\n\nThe `get_tokens_unprocessed()` method holds its internal state data not as local\nvariables, but in an instance of the `pygments.lexer.LexerContext` class, and\nthat instance is passed to callbacks as a third argument. This means that you\ncan modify the lexer state in callbacks.\n\nThe `LexerContext` class has the following members:\n\n* `text` ' Text
'-'           Text
'-'           Text
' the input text\n* `pos` ' Text
'-'           Text
'-'           Text
' the current starting position that is used for matching regexes\n* `stack` ' Text
'-'           Text
'-'           Text
' a list containing the state stack\n* `end` ' Text
'-'           Text
'-'           Text
" the maximum position to which regexes are matched, this defaults to\n  the length of `text`\n\nAdditionally, the `get_tokens_unprocessed()` method can be given a\n`LexerContext` instead of a string and will then process this context instead of\ncreating a new one for the string argument.\n\nNote that because you can set the current position to anything in the callback,\nit won't be automatically be set by the caller after the callback is finished.\nFor example, this is how the hypothetical lexer above would be written with the\n" Text

'[-'          Generic.Deleted
'`ExtendedRegexLexer`:\n\n.. sourcecode:: python' Generic.Deleted
'-]'          Generic.Deleted
'\n'          Text

'{+'          Generic.Inserted
'`ExtendedRegexLexer`::' Generic.Inserted
'+}'          Generic.Inserted
'\n\n    from pygments.lexer import ExtendedRegexLexer\n    from pygments.token import Generic\n\n    class ExHypotheticLexer(ExtendedRegexLexer):\n\n        def headline_callback(lexer, match, ctx):\n            equal_signs = match.group(1)\n            text = match.group(2)\n            yield match.start(), Generic.Headline, equal_signs ' Text
'+'           Text
' text '      Text
'+'           Text
' equal_signs\n            ctx.pos = match.end()\n\n        tokens = ' Text
'{'           Text
"\n            'root': " Text
'['           Text
"\n                (r'(=" Text
'+'           Text
")(.*?)(\\1)', headline_callback)\n            " Text
']'           Text
'\n        '  Text
'}'           Text
'\n\nThis might sound confusing (and it can really be). But it is needed, and for an\nexample look at the Ruby lexer in ' Text
'[-'          Generic.Deleted
'`agile.py`_.' Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
'`ruby.py`_.' Generic.Inserted
'+}'          Generic.Inserted
'\n\n.. '     Text
'[-'          Generic.Deleted
'_agile.py: https://bitbucket.org/birkenfeld/pygments' Generic.Deleted
'-'           Generic.Deleted
'main/src/tip/pygments/lexers/agile.py\n\n\nFiltering' Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
'_ruby.py: https://bitbucket.org/birkenfeld/pygments' Generic.Inserted
'-'           Generic.Inserted
"main/src/tip/pygments/lexers/ruby.py\n\n\nHandling Lists of Keywords\n==========================\n\nFor a relatively short list (hundreds) you can construct an optimized regular\nexpression directly using ``words()`` (longer lists, see next section).  This\nfunction handles a few things for you automatically, including escaping\nmetacharacters and Python's first" Generic.Inserted
'-'           Generic.Inserted
'match rather than longest' Generic.Inserted
'-'           Generic.Inserted
'match in\nalternations.  Feel free to put the lists themselves in\n``pygments/lexers/_$lang_builtins.py`` (see examples there), and generated by\ncode if possible.\n\nAn example of using ``words()`` is something like::\n\n    from pygments.lexer import RegexLexer, words, Name\n\n    class MyLexer(RegexLexer):\n\n        tokens = ' Generic.Inserted
'{'           Generic.Inserted
"\n            'root': " Generic.Inserted
'['           Generic.Inserted
"\n                (words(('else', 'elseif'), suffix=r'\\b'), Name.Builtin),\n                (r'\\w" Generic.Inserted
'+'           Generic.Inserted
"', Name),\n            " Generic.Inserted
']'           Generic.Inserted
',\n        ' Generic.Inserted
'}'           Generic.Inserted
'\n\nAs you can see, you can add ``prefix`` and ``suffix`` parts to the constructed\nregex.\n\n\nModifying' Generic.Inserted
'+}'          Generic.Inserted
" Token Streams\n=======================\n\nSome languages ship a lot of builtin functions (for example PHP).  The total\namount of those functions differs from system to system because not everybody\nhas every extension installed.  In the case of PHP there are over 3000 builtin\nfunctions.  That's an " Text
'[-'          Generic.Deleted
'incredible'  Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
'incredibly'  Generic.Inserted
'+}'          Generic.Inserted
' huge amount of functions, much more than you\n' Text

'[-'          Generic.Deleted
'can'         Generic.Deleted
'-]'          Generic.Deleted
'\n'          Text

'{+'          Generic.Inserted
'want to'     Generic.Inserted
'+}'          Generic.Inserted
' put into a regular expression.\n\nBut because only `Name` tokens can be function names ' Text
'[-'          Generic.Deleted
"it's"        Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
'this is'     Generic.Inserted
'+}'          Generic.Inserted
' solvable by\noverriding the ``get_tokens_unprocessed()`` method.  The following lexer\nsubclasses the `PythonLexer` so that it highlights some additional names as\npseudo ' Text
'[-'          Generic.Deleted
'keywords:\n\n.. sourcecode:: python' Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
'keywords::'  Generic.Inserted
'+}'          Generic.Inserted
'\n\n    from ' Text
'[-'          Generic.Deleted
'pygments.lexers.agile' Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
'pygments.lexers.python' Generic.Inserted
'+}'          Generic.Inserted
' import PythonLexer\n    from pygments.token import Name, Keyword\n\n    class MyPythonLexer(PythonLexer):\n        EXTRA_KEYWORDS = ' Text
'[-'          Generic.Deleted
'['           Generic.Deleted
"'foo',"      Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
"set(('foo'," Generic.Inserted
'+}'          Generic.Inserted
" 'bar', 'foobar', 'barfoo', 'spam', " Text
'[-'          Generic.Deleted
"'eggs'"      Generic.Deleted
']'           Generic.Deleted
'-]'          Generic.Deleted
' '           Text
'{+'          Generic.Inserted
"'eggs'))"    Generic.Inserted
'+}'          Generic.Inserted
'\n\n        def get_tokens_unprocessed(self, text):\n            for index, token, value in PythonLexer.get_tokens_unprocessed(self, text):\n                if token is Name and value in self.EXTRA_KEYWORDS:\n                    yield index, Keyword.Pseudo, value\n                else:\n                    yield index, token, value\n\nThe `PhpLexer` and `LuaLexer` use this method to resolve builtin functions.\n\n' Text

'[-'          Generic.Deleted
'.. note:: Do not confuse this with the :doc:`filter <filters>` system.' Generic.Deleted
'-]'          Generic.Deleted
'\n'          Text