Issue #23731: Implement PEP 488.

The concept of .pyo files no longer exists. Now .pyc files have an
optional `opt-` tag which specifies if any extra optimizations beyond
the peepholer were applied.

16 files changed, 125 insertions, 57 deletions

diff --git a/Doc/c-api/import.rst b/Doc/c-api/import.rst
index 3641fc69b1..01e0deb304 100644
--- a/Doc/c-api/import.rst
+++ b/Doc/c-api/import.rst
@@ -183,9 +183,9 @@ Importing Modules
 
 .. c:function:: long PyImport_GetMagicNumber()
 
-   Return the magic number for Python bytecode files (a.k.a. :file:`.pyc` and
-   :file:`.pyo` files).  The magic number should be present in the first four bytes
-   of the bytecode file, in little-endian byte order. Returns -1 on error.
+   Return the magic number for Python bytecode files (a.k.a. :file:`.pyc` file).
+   The magic number should be present in the first four bytes of the bytecode
+   file, in little-endian byte order. Returns -1 on error.
 
    .. versionchanged:: 3.3
       Return value of -1 upon failure.
diff --git a/Doc/distutils/apiref.rst b/Doc/distutils/apiref.rst
index 53d7527c65..087c6715ce 100644
--- a/Doc/distutils/apiref.rst
+++ b/Doc/distutils/apiref.rst
@@ -1193,12 +1193,12 @@ other utility module.
 
 .. function:: byte_compile(py_files[, optimize=0, force=0, prefix=None, base_dir=None, verbose=1, dry_run=0, direct=None])
 
-   Byte-compile a collection of Python source files to either :file:`.pyc` or
-   :file:`.pyo` files in a :file:`__pycache__` subdirectory (see :pep:`3147`).
+   Byte-compile a collection of Python source files to :file:`.pyc` files in a
+   :file:`__pycache__` subdirectory (see :pep:`3147` and :pep:`488`).
    *py_files* is a list of files to compile; any files that don't end in
    :file:`.py` are silently skipped.  *optimize* must be one of the following:
 
-   * ``0`` - don't optimize (generate :file:`.pyc`)
+   * ``0`` - don't optimize
    * ``1`` - normal optimization (like ``python -O``)
    * ``2`` - extra optimization (like ``python -OO``)
 
@@ -1222,10 +1222,13 @@ other utility module.
    doing, leave it set to ``None``.
 
    .. versionchanged:: 3.2.3
-      Create ``.pyc`` or ``.pyo`` files with an :func:`import magic tag
+      Create ``.pyc`` files with an :func:`import magic tag
       <imp.get_tag>` in their name, in a :file:`__pycache__` subdirectory
       instead of files without tag in the current directory.
 
+   .. versionchanged: 3.5
+      Create ``.pyc`` files according to :pep:`488`.
+
 
 .. function:: rfc822_escape(header)
 
diff --git a/Doc/distutils/introduction.rst b/Doc/distutils/introduction.rst
index 0ece646c34..8f46bd74c5 100644
--- a/Doc/distutils/introduction.rst
+++ b/Doc/distutils/introduction.rst
@@ -156,8 +156,8 @@ module
 
 pure Python module
    a module written in Python and contained in a single :file:`.py` file (and
-   possibly associated :file:`.pyc` and/or :file:`.pyo` files).  Sometimes referred
-   to as a "pure module."
+   possibly associated :file:`.pyc` files).  Sometimes referred to as a
+   "pure module."
 
 extension module
    a module written in the low-level language of the Python implementation: C/C++
@@ -210,5 +210,3 @@ distribution root
    the top-level directory of your source tree (or  source distribution); the
    directory where :file:`setup.py` exists.  Generally  :file:`setup.py` will be
    run from this directory.
-
-
diff --git a/Doc/library/compileall.rst b/Doc/library/compileall.rst
index 430362ebab..57f4804979 100644
--- a/Doc/library/compileall.rst
+++ b/Doc/library/compileall.rst
@@ -93,6 +93,10 @@ compile Python sources.
 .. versionchanged:: 3.5
    ``-q`` option was changed to a multilevel value.
 
+.. versionchanged:: 3.5
+   ``-b`` will always produce a byte-code file ending in ``.pyc``, never
+   ``.pyo``.
+
 
 There is no command-line option to control the optimization level used by the
 :func:`compile` function, because the Python interpreter itself already
@@ -150,6 +154,10 @@ Public functions
    .. versionchanged:: 3.5
       *quiet* parameter was changed to a multilevel value.
 
+   .. versionchanged:: 3.5
+      The *legacy* parameter only writes out ``.pyc`` files, not ``.pyo`` files
+      no matter what the value of *optimize* is.
+
 .. function:: compile_file(fullname, ddir=None, force=False, rx=None, quiet=0, legacy=False, optimize=-1)
 
    Compile the file with path *fullname*.
@@ -182,6 +190,10 @@ Public functions
    .. versionchanged:: 3.5
       *quiet* parameter was changed to a multilevel value.
 
+   .. versionchanged:: 3.5
+      The *legacy* parameter only writes out ``.pyc`` files, not ``.pyo`` files
+      no matter what the value of *optimize* is.
+
 .. function:: compile_path(skip_curdir=True, maxlevels=0, force=False, quiet=0, legacy=False, optimize=-1)
 
    Byte-compile all the :file:`.py` files found along ``sys.path``. If
@@ -196,6 +208,10 @@ Public functions
    .. versionchanged:: 3.5
       *quiet* parameter was changed to a multilevel value.
 
+   .. versionchanged:: 3.5
+      The *legacy* parameter only writes out ``.pyc`` files, not ``.pyo`` files
+      no matter what the value of *optimize* is.
+
 To force a recompile of all the :file:`.py` files in the :file:`Lib/`
 subdirectory and all its subdirectories::
 
diff --git a/Doc/library/imp.rst b/Doc/library/imp.rst
index c2dbdc5cf9..55573460e1 100644
--- a/Doc/library/imp.rst
+++ b/Doc/library/imp.rst
@@ -203,11 +203,9 @@ file paths.
    value would be ``/foo/bar/__pycache__/baz.cpython-32.pyc`` for Python 3.2.
    The ``cpython-32`` string comes from the current magic tag (see
    :func:`get_tag`; if :attr:`sys.implementation.cache_tag` is not defined then
-   :exc:`NotImplementedError` will be raised).  The returned path will end in
-   ``.pyc`` when ``__debug__`` is ``True`` or ``.pyo`` for an optimized Python
-   (i.e. ``__debug__`` is ``False``).  By passing in ``True`` or ``False`` for
-   *debug_override* you can override the system's value for ``__debug__`` for
-   extension selection.
+   :exc:`NotImplementedError` will be raised). By passing in ``True`` or
+   ``False`` for *debug_override* you can override the system's value for
+   ``__debug__``, leading to optimized bytecode.
 
    *path* need not exist.
 
@@ -218,6 +216,9 @@ file paths.
    .. deprecated:: 3.4
       Use :func:`importlib.util.cache_from_source` instead.
 
+   .. versionchanged:: 3.5
+      The *debug_override* parameter no longer creates a ``.pyo`` file.
+
 
 .. function:: source_from_cache(path)
 
diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst
index e61ac351cd..bd9c5ebc79 100644
--- a/Doc/library/importlib.rst
+++ b/Doc/library/importlib.rst
@@ -711,6 +711,9 @@ find and load modules.
 
    .. versionadded:: 3.3
 
+   .. deprecated:: 3.5
+      Use :attr:`BYTECODE_SUFFIXES` instead.
+
 .. attribute:: OPTIMIZED_BYTECODE_SUFFIXES
 
    A list of strings representing the file suffixes for optimized bytecode
@@ -718,14 +721,19 @@ find and load modules.
 
    .. versionadded:: 3.3
 
+   .. deprecated:: 3.5
+      Use :attr:`BYTECODE_SUFFIXES` instead.
+
 .. attribute:: BYTECODE_SUFFIXES
 
    A list of strings representing the recognized file suffixes for bytecode
-   modules. Set to either :attr:`DEBUG_BYTECODE_SUFFIXES` or
-   :attr:`OPTIMIZED_BYTECODE_SUFFIXES` based on whether ``__debug__`` is true.
+   modules (including the leading dot).
 
    .. versionadded:: 3.3
 
+   .. versionchanged:: 3.5
+      The value is no longer dependent on ``__debug__``.
+
 .. attribute:: EXTENSION_SUFFIXES
 
    A list of strings representing the recognized file suffixes for
@@ -1074,23 +1082,37 @@ an :term:`importer`.
 
    .. versionadded:: 3.4
 
-.. function:: cache_from_source(path, debug_override=None)
+.. function:: cache_from_source(path, debug_override=None, *, optimization=None)
 
-   Return the :pep:`3147` path to the byte-compiled file associated with the
-   source *path*.  For example, if *path* is ``/foo/bar/baz.py`` the return
+   Return the :pep:`3147`/:pep:`488` path to the byte-compiled file associated
+   with the source *path*.  For example, if *path* is ``/foo/bar/baz.py`` the return
    value would be ``/foo/bar/__pycache__/baz.cpython-32.pyc`` for Python 3.2.
    The ``cpython-32`` string comes from the current magic tag (see
    :func:`get_tag`; if :attr:`sys.implementation.cache_tag` is not defined then
-   :exc:`NotImplementedError` will be raised).  The returned path will end in
-   ``.pyc`` when ``__debug__`` is ``True`` or ``.pyo`` for an optimized Python
-   (i.e. ``__debug__`` is ``False``).  By passing in ``True`` or ``False`` for
-   *debug_override* you can override the system's value for ``__debug__`` for
-   extension selection.
-
-   *path* need not exist.
+   :exc:`NotImplementedError` will be raised).
+
+   The *optimization* parameter is used to specify the optimization level of the
+   bytecode file. An empty string represents no optimization, so
+   ``/foo/bar/baz.py`` with an *optimization* of ``''`` will result in a
+   bytecode path of ``/foo/bar/__pycache__/baz.cpython-32.pyc``. ``None`` causes
+   the interpter's optimization level to be used. Any other value's string
+   representation being used, so ``/foo/bar/baz.py`` with an *optimization* of
+   ``2`` will lead to the bytecode path of
+   ``/foo/bar/__pycache__/baz.cpython-32.opt-2.pyc``. The string representation
+   of *optimization* can only be alphanumeric, else :exc:`ValueError` is raised.
+
+   The *debug_override* parameter is deprecated and can be used to override
+   the system's value for ``__debug__``. A ``True`` value is the equivalent of
+   setting *optimization* to the empty string. A ``False`` value is the same as
+   setting *optimization* to ``1``. If both *debug_override* an *optimization*
+   are not ``None`` then :exc:`TypeError` is raised.
 
    .. versionadded:: 3.4
 
+   .. versionchanged ::3.5
+      The *optimization* parameter was added and the *debug_override* parameter
+      was deprecated.
+
 
 .. function:: source_from_cache(path)
 
@@ -1098,7 +1120,7 @@ an :term:`importer`.
    file path.  For example, if *path* is
    ``/foo/bar/__pycache__/baz.cpython-32.pyc`` the returned path would be
    ``/foo/bar/baz.py``.  *path* need not exist, however if it does not conform
-   to :pep:`3147` format, a ``ValueError`` is raised. If
+   to :pep:`3147` or :pep`488` format, a ``ValueError`` is raised. If
    :attr:`sys.implementation.cache_tag` is not defined,
    :exc:`NotImplementedError` is raised.
 
diff --git a/Doc/library/py_compile.rst b/Doc/library/py_compile.rst
index bae8450b7c..3647571090 100644
--- a/Doc/library/py_compile.rst
+++ b/Doc/library/py_compile.rst
@@ -29,9 +29,9 @@ byte-code cache files in the directory containing the source code.
 .. function:: compile(file, cfile=None, dfile=None, doraise=False, optimize=-1)
 
    Compile a source file to byte-code and write out the byte-code cache file.
-   The source code is loaded from the file name *file*.  The  byte-code is
-   written to *cfile*, which defaults to the :PEP:`3147` path, ending in
-   ``.pyc`` (``.pyo`` if optimization is enabled in the current interpreter).
+   The source code is loaded from the file name *file*.  The byte-code is
+   written to *cfile*, which defaults to the :pep:`3147`/:pep`488` path, ending
+   in ``.pyc``.
    For example, if *file* is ``/foo/bar/baz.py`` *cfile* will default to
    ``/foo/bar/__pycache__/baz.cpython-32.pyc`` for Python 3.2.  If *dfile* is
    specified, it is used as the name of the source file in error messages when
@@ -68,7 +68,7 @@ byte-code cache files in the directory containing the source code.
 .. function:: main(args=None)
 
    Compile several source files.  The files named in *args* (or on the command
-   line, if *args* is ``None``) are compiled and the resulting bytecode is
+   line, if *args* is ``None``) are compiled and the resulting byte-code is
    cached in the normal manner.  This function does not search a directory
    structure to locate source files; it only compiles files named explicitly.
    If ``'-'`` is the only parameter in args, the list of files is taken from
@@ -86,4 +86,3 @@ could not be compiled.
 
    Module :mod:`compileall`
       Utilities to compile all Python source files in a directory tree.
-
diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst
index fa125a0c96..3080b69445 100644
--- a/Doc/library/sys.rst
+++ b/Doc/library/sys.rst
@@ -167,7 +167,7 @@ always available.
 
 .. data:: dont_write_bytecode
 
-   If this is true, Python won't try to write ``.pyc`` or ``.pyo`` files on the
+   If this is true, Python won't try to write ``.pyc`` files on the
    import of source modules.  This value is initially set to ``True`` or
    ``False`` depending on the :option:`-B` command line option and the
    :envvar:`PYTHONDONTWRITEBYTECODE` environment variable, but you can set it
@@ -1231,4 +1231,3 @@ always available.
 .. rubric:: Citations
 
 .. [C99] ISO/IEC 9899:1999.  "Programming languages -- C."  A public draft of this standard is available at http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1256.pdf\ .
-
diff --git a/Doc/library/tracemalloc.rst b/Doc/library/tracemalloc.rst
index a04a432716..13c81a7fed 100644
--- a/Doc/library/tracemalloc.rst
+++ b/Doc/library/tracemalloc.rst
@@ -363,7 +363,7 @@ Filter
    Filter on traces of memory blocks.
 
    See the :func:`fnmatch.fnmatch` function for the syntax of
-   *filename_pattern*. The ``'.pyc'`` and ``'.pyo'`` file extensions are
+   *filename_pattern*. The ``'.pyc'`` file extension is
    replaced with ``'.py'``.
 
    Examples:
@@ -374,6 +374,10 @@ Filter
      :mod:`tracemalloc` module
    * ``Filter(False, "<unknown>")`` excludes empty tracebacks
 
+
+   .. versionchanged:: 3.5
+      The ``'.pyo'`` file extension is no longer replaced with ``'.py'``.
+
    .. attribute:: inclusive
 
       If *inclusive* is ``True`` (include), only trace memory blocks allocated
@@ -631,4 +635,3 @@ Traceback
               obj = Object()
             File "test.py", line 12
               tb = tracemalloc.get_object_traceback(f())
-
diff --git a/Doc/library/zipfile.rst b/Doc/library/zipfile.rst
index a15c461d9e..6192cd0d38 100644
--- a/Doc/library/zipfile.rst
+++ b/Doc/library/zipfile.rst
@@ -405,8 +405,7 @@ The :class:`PyZipFile` constructor takes the same parameters as the
       archive.
 
       If the *optimize* parameter to :class:`PyZipFile` was not given or ``-1``,
-      the corresponding file is a :file:`\*.pyo` file if available, else a
-      :file:`\*.pyc` file, compiling if necessary.
+      the corresponding file is a :file:`\*.pyc` file, compiling if necessary.
 
       If the *optimize* parameter to :class:`PyZipFile` was ``0``, ``1`` or
       ``2``, only files with that optimization level (see :func:`compile`) are
@@ -569,4 +568,3 @@ Instances have the following attributes:
 .. attribute:: ZipInfo.file_size
 
    Size of the uncompressed file.
-
diff --git a/Doc/library/zipimport.rst b/Doc/library/zipimport.rst
index 2cf508b85c..8a5d5d19df 100644
--- a/Doc/library/zipimport.rst
+++ b/Doc/library/zipimport.rst
@@ -20,10 +20,10 @@ subdirectory.  For example, the path :file:`example.zip/lib/` would only
 import from the :file:`lib/` subdirectory within the archive.
 
 Any files may be present in the ZIP archive, but only files :file:`.py` and
-:file:`.py[co]` are available for import.  ZIP import of dynamic modules
+:file:`.pyc` are available for import.  ZIP import of dynamic modules
 (:file:`.pyd`, :file:`.so`) is disallowed. Note that if an archive only contains
 :file:`.py` files, Python will not attempt to modify the archive by adding the
-corresponding :file:`.pyc` or :file:`.pyo` file, meaning that if a ZIP archive
+corresponding :file:`.pyc` file, meaning that if a ZIP archive
 doesn't contain :file:`.pyc` files, importing may be rather slow.
 
 ZIP archives with an archive comment are currently not supported.
@@ -161,4 +161,3 @@ Here is an example that imports a module from a ZIP archive - note that the
    >>> import jwzthreading
    >>> jwzthreading.__file__
    'example.zip/jwzthreading.py'
-
diff --git a/Doc/reference/import.rst b/Doc/reference/import.rst
index 2d60e62f2b..dec5f8f14b 100644
--- a/Doc/reference/import.rst
+++ b/Doc/reference/import.rst
@@ -646,7 +646,7 @@ path entry finder that knows how to handle that particular kind of path.
 
 The default set of path entry finders implement all the semantics for finding
 modules on the file system, handling special file types such as Python source
-code (``.py`` files), Python byte code (``.pyc`` and ``.pyo`` files) and
+code (``.py`` files), Python byte code (``.pyc`` files) and
 shared libraries (e.g. ``.so`` files). When supported by the :mod:`zipimport`
 module in the standard library, the default path entry finders also handle
 loading all of these file types (other than shared libraries) from zipfiles.
diff --git a/Doc/tutorial/modules.rst b/Doc/tutorial/modules.rst
index fd361aed69..9ae64b0cf7 100644
--- a/Doc/tutorial/modules.rst
+++ b/Doc/tutorial/modules.rst
@@ -216,15 +216,15 @@ Some tips for experts:
   statements, the ``-OO`` switch removes both assert statements and __doc__
   strings.  Since some programs may rely on having these available, you should
   only use this option if you know what you're doing.  "Optimized" modules have
-  a .pyo rather than a .pyc suffix and are usually smaller.  Future releases may
+  an ``opt-`` tag and are usually smaller.  Future releases may
   change the effects of optimization.
 
-* A program doesn't run any faster when it is read from a ``.pyc`` or ``.pyo``
+* A program doesn't run any faster when it is read from a ``.pyc``
   file than when it is read from a ``.py`` file; the only thing that's faster
-  about ``.pyc`` or ``.pyo`` files is the speed with which they are loaded.
+  about ``.pyc`` files is the speed with which they are loaded.
 
-* The module :mod:`compileall` can create .pyc files (or .pyo files when
-  :option:`-O` is used) for all modules in a directory.
+* The module :mod:`compileall` can create .pyc files for all modules in a
+  directory.
 
 * There is more detail on this process, including a flow chart of the
   decisions, in PEP 3147.
@@ -548,4 +548,3 @@ modules found in a package.
 .. [#] In fact function definitions are also 'statements' that are 'executed'; the
    execution of a module-level function definition enters the function name in
    the module's global symbol table.
-
diff --git a/Doc/using/cmdline.rst b/Doc/using/cmdline.rst
index f52afa0229..81cd38a533 100644
--- a/Doc/using/cmdline.rst
+++ b/Doc/using/cmdline.rst
@@ -199,7 +199,7 @@ Miscellaneous options
 
 .. cmdoption:: -B
 
-   If given, Python won't try to write ``.pyc`` or ``.pyo`` files on the
+   If given, Python won't try to write ``.pyc``` files on the
    import of source modules.  See also :envvar:`PYTHONDONTWRITEBYTECODE`.
 
 
diff --git a/Doc/using/windows.rst b/Doc/using/windows.rst
index f194802493..7113dfa118 100644
--- a/Doc/using/windows.rst
+++ b/Doc/using/windows.rst
@@ -111,7 +111,7 @@ of available options is shown below.
 |                           | launcher is also installed.          |                          |
 +---------------------------+--------------------------------------+--------------------------+
 | CompileAll                | Compile all ``.py`` files to         | 0                        |
-|                           | ``.pyc`` and ``.pyo``.               |                          |
+|                           | ``.pyc``.                            |                          |
 +---------------------------+--------------------------------------+--------------------------+
 | PrependPath               | Add install and Scripts directories  | 0                        |
 |                           | tho :envvar:`PATH` and ``.PY`` to    |                          |
@@ -451,7 +451,7 @@ From file associations
 ^^^^^^^^^^^^^^^^^^^^^^
 
 The launcher should have been associated with Python files (i.e. ``.py``,
-``.pyw``, ``.pyc``, ``.pyo`` files) when it was installed.  This means that
+``.pyw``, ``.pyc`` files) when it was installed.  This means that
 when you double-click on one of these files from Windows explorer the launcher
 will be used, and therefore you can use the same facilities described above to
 have the script specify the version which should be used.
@@ -796,5 +796,3 @@ Other resources
 
    :pep:`397` - Python launcher for Windows
       The proposal for the launcher to be included in the Python distribution.
-
-
diff --git a/Doc/whatsnew/3.5.rst b/Doc/whatsnew/3.5.rst
index 28914ef76d..5c21054db7 100644
--- a/Doc/whatsnew/3.5.rst
+++ b/Doc/whatsnew/3.5.rst
@@ -88,6 +88,8 @@ Implementation improvements:
   ``surrogateescape`` error handler, instead of the ``strict`` error handler
   (:issue:`19977`).
 
+* :pep:`488`, the elimination of ``.pyo`` files.
+
 Significantly Improved Library Modules:
 
 * None yet.
@@ -195,6 +197,24 @@ environment will be used.
 
     :pep:`486` -- Make the Python Launcher aware of virtual environments
 
+
+PEP 488: Elimination of PYO files
+---------------------------------
+
+:pep:`488` does away with the concept of ``.pyo`` files. This means that
+``.pyc`` files represent both unoptimized and optimized bytecode. To prevent
+the need to constantly regenerate bytecode files, ``.pyc`` files now have an
+optional ``opt-`` tag in their name when the bytecode is optimized. This has
+the side-effect of no more bytecode file name clashes when running under either
+``-O`` or ``-OO``, thus allowing unoptimized, ``-O``, and ``-OO`` bytecode files
+to all exist simultaneously. :func:`importlib.util.cache_from_source` has an
+updated API to help with this change.
+
+.. seealso::
+
+   :pep:`488` -- Elimination of PYO files
+
+
 Other Language Changes
 ======================
 
@@ -535,7 +555,7 @@ The following performance enhancements have been added:
   ``FindFirstFile``/``FindNextFile`` system calls. (Contributed by
   Ben Hoyt with help from Victor Stinner in :issue:`23605`.)
 
-* Construction of ``bytes(int)`` (filled by zero bytes) is faster and use less
+* Construction of ``bytes(int)`` (filled by zero bytes) is faster and uses less
   memory for large objects. ``calloc()`` is used instead of ``malloc()`` to
   allocate memory for these objects.
 
@@ -630,6 +650,8 @@ removed:
   3.4, and has now been removed.
   (Contributed by Matt Chaput in :issue:`6623`.)
 
+* The concept of ``.pyo`` files has been removed.
+
 * The JoinableQueue class in the provisional asyncio module was deprecated
   in 3.4.4 and is now removed (:issue:`23464`).
 
@@ -759,6 +781,17 @@ Changes in the Python API
   *LegalChars* parameter of :func:`~http.cookies.Morsel.set` is deprecated and
   is now ignored.  (:issue:`2211`)
 
+* :pep:`488` has removed ``.pyo`` files from Python and introduced the optional
+  ``opt-`` tag in ``.pyc`` file names. The
+  :func:`importlib.util.cache_from_source` has gained an *optimization*
+  parameter to help control the ``opt-`` tag. Because of this, the
+  *debug_override* parameter of the function is now deprecated. `.pyo` files
+  are also no longer supported as a file argument to the Python interpreter and
+  thus serve no purpose when distributed on their own (i.e. sourcless code
+  distribution). Due to the fact that the magic number for bytecode has changed
+  in Python 3.5, all old `.pyo` files from previous versions of Python are
+  invalid regardless of this PEP.
+
 Changes in the C API
 --------------------