summaryrefslogtreecommitdiff
path: root/tests/test_raw_token.py
Commit message (Collapse)AuthorAgeFilesLines
* RawToken{Formatter,Lexer}: support Python 3 and handle exceptions (#1602)Anders Kaseorg2021-02-141-0/+68
In Python 3, RawTokenFormatter would output non-ASCII for non-ASCII input, and RawTokenLexer would throw Unicode-related exceptions for ASCII or non-ASCII input; fix them. Also, handle all exceptions, so that callers who find RawTokenLexer via get_lexer_by_name on user input don’t unexpectedly get a lexer that throws exceptions. Signed-off-by: Anders Kaseorg <andersk@mit.edu>