RawToken{Formatter,Lexer}: support Python 3 and handle exceptions (#1602)

In Python 3, RawTokenFormatter would output non-ASCII for non-ASCII
input, and RawTokenLexer would throw Unicode-related exceptions for
ASCII or non-ASCII input; fix them.  Also, handle all exceptions, so
that callers who find RawTokenLexer via get_lexer_by_name on user
input don’t unexpectedly get a lexer that throws exceptions.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>

1 files changed, 6 insertions, 8 deletions

diff --git a/pygments/formatters/other.py b/pygments/formatters/other.py
index 16c2fceb..1a12c42b 100644
--- a/pygments/formatters/other.py
+++ b/pygments/formatters/other.py
@@ -87,34 +87,32 @@ class RawTokenFormatter(Formatter):
             import gzip
             outfile = gzip.GzipFile('', 'wb', 9, outfile)
 
-            def write(text):
-                outfile.write(text.encode())
-            flush = outfile.flush
+            write = outfile.write
+            flush = outfile.close
         elif self.compress == 'bz2':
             import bz2
             compressor = bz2.BZ2Compressor(9)
 
             def write(text):
-                outfile.write(compressor.compress(text.encode()))
+                outfile.write(compressor.compress(text))
 
             def flush():
                 outfile.write(compressor.flush())
                 outfile.flush()
         else:
-            def write(text):
-                outfile.write(text.encode())
+            write = outfile.write
             flush = outfile.flush
 
         if self.error_color:
             for ttype, value in tokensource:
-                line = "%s\t%r\n" % (ttype, value)
+                line = b"%r\t%r\n" % (ttype, value)
                 if ttype is Token.Error:
                     write(colorize(self.error_color, line))
                 else:
                     write(line)
         else:
             for ttype, value in tokensource:
-                write("%s\t%r\n" % (ttype, value))
+                write(b"%r\t%r\n" % (ttype, value))
         flush()