diff options
author | Ezio Melotti <ezio.melotti@gmail.com> | 2009-12-20 12:24:57 +0000 |
---|---|---|
committer | Ezio Melotti <ezio.melotti@gmail.com> | 2009-12-20 12:24:57 +0000 |
commit | db8f5a9b004e738fbf52b5ad99843b0bb926eab8 (patch) | |
tree | dee05c50efbe616e0be34ab5fcf4891994aeacab /Doc/library/tokenize.rst | |
parent | 84b731c7eb96f16fa546f907a8b15d4d5834774c (diff) | |
download | cpython-db8f5a9b004e738fbf52b5ad99843b0bb926eab8.tar.gz |
Fixed capitalization and markup; added imports in the example
Diffstat (limited to 'Doc/library/tokenize.rst')
-rw-r--r-- | Doc/library/tokenize.rst | 9 |
1 files changed, 6 insertions, 3 deletions
diff --git a/Doc/library/tokenize.rst b/Doc/library/tokenize.rst index deb7826393..d66b07c62d 100644 --- a/Doc/library/tokenize.rst +++ b/Doc/library/tokenize.rst @@ -94,16 +94,19 @@ function it uses to do this is available: (as a string) and a list of any lines (not decoded from bytes) it has read in. - It detects the encoding from the presence of a utf-8 bom or an encoding - cookie as specified in pep-0263. If both a bom and a cookie are present, + It detects the encoding from the presence of a UTF-8 BOM or an encoding + cookie as specified in :pep:`263`. If both a BOM and a cookie are present, but disagree, a SyntaxError will be raised. - If no encoding is specified, then the default of 'utf-8' will be returned. + If no encoding is specified, then the default of ``'utf-8'`` will be returned. Example of a script re-writer that transforms float literals into Decimal objects:: + from tokenize import tokenize, untokenize, NUMBER, STRING, NAME, OP + from io import BytesIO + def decistmt(s): """Substitute Decimals for floats in a string of statements. |