diff options
Diffstat (limited to 'Doc/whatsnew/3.5.rst')
| -rw-r--r-- | Doc/whatsnew/3.5.rst | 1249 |
1 files changed, 1249 insertions, 0 deletions
diff --git a/Doc/whatsnew/3.5.rst b/Doc/whatsnew/3.5.rst new file mode 100644 index 0000000000..11f871b786 --- /dev/null +++ b/Doc/whatsnew/3.5.rst @@ -0,0 +1,1249 @@ +**************************** + What's New In Python 3.5 +**************************** + +:Release: |release| +:Date: |today| + +.. Rules for maintenance: + + * Anyone can add text to this document. Do not spend very much time + on the wording of your changes, because your text will probably + get rewritten to some degree. + + * The maintainer will go through Misc/NEWS periodically and add + changes; it's therefore more important to add your changes to + Misc/NEWS than to this file. + + * This is not a complete list of every single change; completeness + is the purpose of Misc/NEWS. Some changes I consider too small + or esoteric to include. If such a change is added to the text, + I'll just remove it. (This is another reason you shouldn't spend + too much time on writing your addition.) + + * If you want to draw your new text to the attention of the + maintainer, add 'XXX' to the beginning of the paragraph or + section. + + * It's OK to just add a fragmentary note about a change. For + example: "XXX Describe the transmogrify() function added to the + socket module." The maintainer will research the change and + write the necessary text. + + * You can comment out your additions if you like, but it's not + necessary (especially when a final release is some months away). + + * Credit the author of a patch or bugfix. Just the name is + sufficient; the e-mail address isn't necessary. + + * It's helpful to add the bug/patch number as a comment: + + XXX Describe the transmogrify() function added to the socket + module. + (Contributed by P.Y. Developer in :issue:`12345`.) + + This saves the maintainer the effort of going through the Mercurial log + when researching a change. + +This article explains the new features in Python 3.5, compared to 3.4. + +For full details, see the :source:`Misc/NEWS` file. + +.. note:: + + Prerelease users should be aware that this document is currently in draft + form. It will be updated substantially as Python 3.5 moves towards release, + so it's worth checking back even after reading earlier versions. + + +.. seealso:: + + :pep:`478` - Python 3.5 Release Schedule + + +Summary -- Release highlights +============================= + +.. This section singles out the most important changes in Python 3.5. + Brevity is key. + +New syntax features: + +* :pep:`465`, a new matrix multiplication operator: ``a @ b``. +* :pep:`492`, coroutines with async and await syntax. +* :pep:`448`, additional unpacking generalizations. + +New library modules: + +* :mod:`zipapp`: :ref:`Improving Python ZIP Application Support + <whatsnew-zipapp>` (:pep:`441`). + +New built-in features: + +* ``bytes % args``, ``bytearray % args``: :pep:`461` - Adding ``%`` formatting + to bytes and bytearray +* ``b'\xf0\x9f\x90\x8d'.hex()``, ``bytearray(b'\xf0\x9f\x90\x8d').hex()``, + ``memoryview(b'\xf0\x9f\x90\x8d').hex()``: :issue:`9951` - A ``hex`` method + has been added to bytes, bytearray, and memoryview. +* Generators have new ``gi_yieldfrom`` attribute, which returns the + object being iterated by ``yield from`` expressions. (Contributed + by Benno Leslie and Yury Selivanov in :issue:`24450`.) +* New :exc:`RecursionError` exception. (Contributed by Georg Brandl + in :issue:`19235`.) + +Implementation improvements: + +* When the ``LC_TYPE`` locale is the POSIX locale (``C`` locale), + :py:data:`sys.stdin` and :py:data:`sys.stdout` are now using the + ``surrogateescape`` error handler, instead of the ``strict`` error handler + (:issue:`19977`). + +* :pep:`488`, the elimination of ``.pyo`` files. +* :pep:`489`, multi-phase initialization of extension modules. + +Significantly Improved Library Modules: + +* :class:`collections.OrderedDict` is now implemented in C, which improves + its performance between 4x to 100x times. Contributed by Eric Snow in + :issue:`16991`. + +* You may now pass bytes to the :mod:`tempfile` module's APIs and it will + return the temporary pathname as bytes instead of str. It also accepts + a value of ``None`` on parameters where only str was accepted in the past to + do the right thing based on the types of the other inputs. Two functions, + :func:`gettempdirb` and :func:`gettempprefixb`, have been added to go along + with this. This behavior matches that of the :mod:`os` APIs. + +* :mod:`ssl` module gained support for Memory BIO, which decouples SSL + protocol handling from network IO. (Contributed by Geert Jansen in + :issue:`21965`.) + +Security improvements: + +* None yet. + +Windows improvements: + +* A new installer for Windows has replaced the old MSI. See :ref:`using-on-windows` + for more information. +* Windows builds now use Microsoft Visual C++ 14.0, and extension modules + should use the same. + + +Please read on for a comprehensive list of user-facing changes. + + +.. PEP-sized items next. + +.. _pep-4XX: + +.. PEP 4XX: Virtual Environments +.. ============================= + + +.. (Implemented by Foo Bar.) + +.. .. seealso:: + + :pep:`4XX` - Python Virtual Environments + PEP written by Carl Meyer + + +PEP 492 - Coroutines with async and await syntax +------------------------------------------------ + +The PEP added dedicated syntax for declaring :term:`coroutines <coroutine>`, +:keyword:`await` expressions, new asynchronous :keyword:`async for` +and :keyword:`async with` statements. + +Example:: + + async def read_data(db): + async with db.transaction(): + data = await db.fetch('SELECT ...') + +PEP written and implemented by Yury Selivanov. + +.. seealso:: + + :pep:`492` -- Coroutines with async and await syntax + + +PEP 461 - Formatting support for bytes and bytearray +---------------------------------------------------- + +This PEP proposes adding % formatting operations similar to Python 2's ``str`` +type to :class:`bytes` and :class:`bytearray`. + +Examples:: + + >>> b'Hello %s!' % b'World' + b'Hello World!' + >>> b'x=%i y=%f' % (1, 2.5) + b'x=1 y=2.500000' + +Unicode is not allowed for ``%s``, but it is accepted by ``%a`` (equivalent of +``repr(obj).encode('ascii', 'backslashreplace')``):: + + >>> b'Hello %s!' % 'World' + Traceback (most recent call last): + File "<stdin>", line 1, in <module> + TypeError: %b requires bytes, or an object that implements __bytes__, not 'str' + >>> b'price: %a' % '10€' + b"price: '10\\u20ac'" + +.. seealso:: + + :pep:`461` -- Adding % formatting to bytes and bytearray + + +PEP 465 - A dedicated infix operator for matrix multiplication +-------------------------------------------------------------- + +This PEP proposes a new binary operator to be used for matrix multiplication, +called ``@``. (Mnemonic: ``@`` is ``*`` for mATrices.) + +.. seealso:: + + :pep:`465` -- A dedicated infix operator for matrix multiplication + + +PEP 448 - Additional Unpacking Generalizations +---------------------------------------------- + +This PEP proposes extended usages of the ``*`` iterable unpacking +operator and ``**`` dictionary unpacking operators +to allow unpacking in more positions, an arbitrary number of +times, and in additional circumstances. Specifically, +in function calls, in comprehensions and generator expressions, and +in displays. + +Function calls are proposed to support an arbitrary number of +unpackings rather than just one:: + + >>> print(*[1], *[2], 3) + 1 2 3 + >>> dict(**{'x': 1}, y=2, **{'z': 3}) + {'x': 1, 'y': 2, 'z': 3} + +Unpacking is proposed to be allowed inside tuple, list, set, +and dictionary displays:: + + >>> *range(4), 4 + (0, 1, 2, 3, 4) + >>> [*range(4), 4] + [0, 1, 2, 3, 4] + >>> {*range(4), 4} + {0, 1, 2, 3, 4} + >>> {'x': 1, **{'y': 2}} + {'x': 1, 'y': 2} + +In dictionaries, later values will always override earlier ones:: + + >>> {'x': 1, **{'x': 2}} + {'x': 2} + + >>> {**{'x': 2}, 'x': 1} + {'x': 1} + +.. seealso:: + + :pep:`448` -- Additional Unpacking Generalizations + + +PEP 484 - Type Hints +-------------------- + +This PEP introduces a provisional module to provide these standard +definitions and tools, along with some conventions for situations +where annotations are not available. + +For example, here is a simple function whose argument and return type +are declared in the annotations:: + + def greeting(name: str) -> str: + return 'Hello ' + name + +The type system supports unions, generic types, and a special type +named ``Any`` which is consistent with (i.e. assignable to and from) all +types. + +.. seealso:: + + * :pep:`484` -- Type Hints + * :mod:`typing` module documentation + + +PEP 471 - os.scandir() function -- a better and faster directory iterator +------------------------------------------------------------------------- + +:pep:`471` adds a new directory iteration function, :func:`os.scandir`, +to the standard library. Additionally, :func:`os.walk` is now +implemented using :func:`os.scandir`, which speeds it up by 3-5 times +on POSIX systems and by 7-20 times on Windows systems. + +PEP and implementation written by Ben Hoyt with the help of Victor Stinner. + +.. seealso:: + + :pep:`471` -- os.scandir() function -- a better and faster directory + iterator + + +PEP 475: Retry system calls failing with EINTR +---------------------------------------------- + +:pep:`475` adds support for automatic retry of system calls failing with +:py:data:`~errno.EINTR`: this means that user code doesn't have to deal with +EINTR or :exc:`InterruptedError` manually, and should make it more robust +against asynchronous signal reception. + +Examples of functions which are now retried when interrupted by a signal +instead of raising :exc:`InterruptedError` if the Python signal handler does +not raise an exception: + +* :func:`open`, :func:`os.open`, :func:`io.open` +* functions of the :mod:`faulthandler` module +* :mod:`os` functions: + + - :func:`os.fchdir` + - :func:`os.fchmod` + - :func:`os.fchown` + - :func:`os.fdatasync` + - :func:`os.fstat` + - :func:`os.fstatvfs` + - :func:`os.fsync` + - :func:`os.ftruncate` + - :func:`os.mkfifo` + - :func:`os.mknod` + - :func:`os.posix_fadvise` + - :func:`os.posix_fallocate` + - :func:`os.pread` + - :func:`os.pwrite` + - :func:`os.read` + - :func:`os.readv` + - :func:`os.sendfile` + - :func:`os.wait3` + - :func:`os.wait4` + - :func:`os.wait` + - :func:`os.waitid` + - :func:`os.waitpid` + - :func:`os.write` + - :func:`os.writev` + - special cases: :func:`os.close` and :func:`os.dup2` now ignore + :py:data:`~errno.EINTR` error, the syscall is not retried (see the PEP + for the rationale) + +* :mod:`select` functions: + + - :func:`select.devpoll.poll` + - :func:`select.epoll.poll` + - :func:`select.kqueue.control` + - :func:`select.poll.poll` + - :func:`select.select` + +* :func:`socket.socket` methods: + + - :meth:`~socket.socket.accept` + - :meth:`~socket.socket.connect` (except for non-blocking sockets) + - :meth:`~socket.socket.recv` + - :meth:`~socket.socket.recvfrom` + - :meth:`~socket.socket.recvmsg` + - :meth:`~socket.socket.send` + - :meth:`~socket.socket.sendall` + - :meth:`~socket.socket.sendmsg` + - :meth:`~socket.socket.sendto` + +* :func:`signal.sigtimedwait`, :func:`signal.sigwaitinfo` +* :func:`time.sleep` + +PEP and implementation written by Charles-François Natali and Victor Stinner, +with the help of Antoine Pitrou (the french connection). + +.. seealso:: + + :pep:`475` -- Retry system calls failing with EINTR + + +PEP 479: Change StopIteration handling inside generators +-------------------------------------------------------- + +:pep:`479` changes the behavior of generators: when a :exc:`StopIteration` +exception is raised inside a generator, it is replaced with a +:exc:`RuntimeError`. To enable the feature a ``__future__`` import should +be used:: + + from __future__ import generator_stop + +Without a ``__future__`` import, a :exc:`PendingDeprecationWarning` will be +raised. + +PEP written by Chris Angelico and Guido van Rossum. Implemented by +Chris Angelico, Yury Selivanov and Nick Coghlan. + +.. seealso:: + + :pep:`479` -- Change StopIteration handling inside generators + + +PEP 486: Make the Python Launcher aware of virtual environments +--------------------------------------------------------------- + +:pep:`486` makes the Windows launcher (see :pep:`397`) aware of an active +virtual environment. When the default interpreter would be used and the +``VIRTUAL_ENV`` environment variable is set, the interpreter in the virtual +environment will be used. + +.. seealso:: + + :pep:`486` -- Make the Python Launcher aware of virtual environments + + +PEP 488: Elimination of PYO files +--------------------------------- + +:pep:`488` does away with the concept of ``.pyo`` files. This means that +``.pyc`` files represent both unoptimized and optimized bytecode. To prevent the +need to constantly regenerate bytecode files, ``.pyc`` files now have an +optional ``opt-`` tag in their name when the bytecode is optimized. This has the +side-effect of no more bytecode file name clashes when running under either +``-O`` or ``-OO``. Consequently, bytecode files generated from ``-O``, and +``-OO`` may now exist simultaneously. :func:`importlib.util.cache_from_source` +has an updated API to help with this change. + +.. seealso:: + + :pep:`488` -- Elimination of PYO files + + +PEP 489: Multi-phase extension module initialization +---------------------------------------------------- + +:pep:`489` updates extension module initialization to take advantage of the +two step module loading mechanism introduced by :pep:`451` in Python 3.4. + +This change brings the import semantics of extension modules that opt-in to +using the new mechanism much closer to those of Python source and bytecode +modules, including the ability to use any valid identifier as a module name, +rather than being restricted to ASCII. + +.. seealso:: + + :pep:`488` -- Multi-phase extension module initialization + +PEP 485: A function for testing approximate equality +---------------------------------------------------- + +:pep:`485` adds the :func:`math.isclose` and :func:`cmath.isclose` +functions which tell whether two values are approximately equal or +"close" to each other. Whether or not two values are considered +close is determined according to given absolute and relative tolerances. + +.. seealso:: + + :pep:`485` -- A function for testing approximate equality + +Other Language Changes +====================== + +Some smaller changes made to the core Python language are: + +* Added the ``'namereplace'`` error handlers. The ``'backslashreplace'`` + error handlers now works with decoding and translating. + (Contributed by Serhiy Storchaka in :issue:`19676` and :issue:`22286`.) + +* The :option:`-b` option now affects comparisons of :class:`bytes` with + :class:`int`. (Contributed by Serhiy Storchaka in :issue:`23681`) + +* New Kazakh :ref:`codec <standard-encodings>` ``kz1048``. (Contributed by + Serhiy Storchaka in :issue:`22682`.) + +* Property docstrings are now writable. This is especially useful for + :func:`collections.namedtuple` docstrings. + (Contributed by Berker Peksag in :issue:`24064`.) + +* New Tajik :ref:`codec <standard-encodings>` ``koi8_t``. (Contributed by + Serhiy Storchaka in :issue:`22681`.) + + +New Modules +=========== + +.. _whatsnew-zipapp: + +zipapp +------ + +The new :mod:`zipapp` module (specified in :pep:`441`) provides an API and +command line tool for creating executable Python Zip Applications, which +were introduced in Python 2.6 in :issue:`1739468` but which were not well +publicised, either at the time or since. + +With the new module, bundling your application is as simple as putting all +the files, including a ``__main__.py`` file, into a directory ``myapp`` +and running:: + + $ python -m zipapp myapp + $ python myapp.pyz + + +Improved Modules +================ + +argparse +-------- + +* :class:`~argparse.ArgumentParser` now allows to disable + :ref:`abbreviated usage <prefix-matching>` of long options by setting + :ref:`allow_abbrev` to ``False``. + (Contributed by Jonathan Paugh, Steven Bethard, paul j3 and Daniel Eriksson.) + +cgi +--- + +* :class:`~cgi.FieldStorage` now supports the context management protocol. + (Contributed by Berker Peksag in :issue:`20289`.) + +cmath +----- + +* :func:`cmath.isclose` function added. + (Contributed by Chris Barker and Tal Einat in :issue:`24270`.) + + +code +---- + +* The :func:`code.InteractiveInterpreter.showtraceback` method now prints + the full chained traceback, just like the interactive interpreter. + (Contributed by Claudiu Popa in :issue:`17442`.) + +collections +----------- + +* You can now update docstrings produced by :func:`collections.namedtuple`:: + + Point = namedtuple('Point', ['x', 'y']) + Point.__doc__ = 'ordered pair' + Point.x.__doc__ = 'abscissa' + Point.y.__doc__ = 'ordinate' + + (Contributed by Berker Peksag in :issue:`24064`.) + +compileall +---------- + +* :func:`compileall.compile_dir` and :mod:`compileall`'s command-line interface + can now do parallel bytecode compilation. + (Contributed by Claudiu Popa in :issue:`16104`.) + +contextlib +---------- + +* The new :func:`contextlib.redirect_stderr` context manager(similar to + :func:`contextlib.redirect_stdout`) makes it easier for utility scripts to + handle inflexible APIs that write their output to :data:`sys.stderr` and + don't provide any options to redirect it. + (Contributed by Berker Peksag in :issue:`22389`.) + +curses +------ +* The new :func:`curses.update_lines_cols` function updates the variables + :envvar:`curses.LINES` and :envvar:`curses.COLS`. + +difflib +------- + +* The charset of the HTML document generated by :meth:`difflib.HtmlDiff.make_file` + can now be customized by using *charset* keyword-only parameter. The default + charset of HTML document changed from ``'ISO-8859-1'`` to ``'utf-8'``. + (Contributed by Berker Peksag in :issue:`2052`.) + +* It's now possible to compare lists of byte strings with + :func:`difflib.diff_bytes` (fixes a regression from Python 2). + +distutils +--------- + +* The ``build`` and ``build_ext`` commands now accept a ``-j`` + option to enable parallel building of extension modules. + (Contributed by Antoine Pitrou in :issue:`5309`.) + +* Added support for the LZMA compression. + (Contributed by Serhiy Storchaka in :issue:`16314`.) + +doctest +------- + +* :func:`doctest.DocTestSuite` returns an empty :class:`unittest.TestSuite` if + *module* contains no docstrings instead of raising :exc:`ValueError`. + (Contributed by Glenn Jones in :issue:`15916`.) + +email +----- + +* A new policy option :attr:`~email.policy.Policy.mangle_from_` controls + whether or not lines that start with "From " in email bodies are prefixed with + a '>' character by generators. The default is ``True`` for + :attr:`~email.policy.compat32` and ``False`` for all other policies. + (Contributed by Milan Oberkirch in :issue:`20098`.) + +* A new method :meth:`~email.message.Message.get_content_disposition` provides + easy access to a canonical value for the :mailheader:`Content-Disposition` + header (``None`` if there is no such header). (Contributed by Abhilash Raj + in :issue:`21083`.) + +* A new policy option :attr:`~email.policy.EmailPolicy.utf8` can be set + ``True`` to encode email headers using the utf8 charset instead of using + encoded words. This allows ``Messages`` to be formatted according to + :rfc:`6532` and used with an SMTP server that supports the :rfc:`6531` + ``SMTPUTF8`` extension. (Contributed by R. David Murray in :issue:`24211`.) + +glob +---- + +* :func:`~glob.iglob` and :func:`~glob.glob` now support recursive search in + subdirectories using the "``**``" pattern. + (Contributed by Serhiy Storchaka in :issue:`13968`.) + +idlelib and IDLE +---------------- + +Since idlelib implements the IDLE shell and editor and is not intended for +import by other programs, it gets improvements with every release. See +:file:`Lib/idlelib/NEWS.txt` for a cumulative list of changes since 3.4.0, +as well as changes made in future 3.5.x releases. This file is also available +from the IDLE Help -> About Idle dialog. + +imaplib +------- + +* :class:`IMAP4` now supports the context management protocol. When used in a + :keyword:`with` statement, the IMAP4 ``LOGOUT`` command will be called + automatically at the end of the block. (Contributed by Tarek Ziadé and + Serhiy Storchaka in :issue:`4972`.) + +* :mod:`imaplib` now supports :rfc:`5161`: the :meth:`~imaplib.IMAP4.enable` + extension), and :rfc:`6855`: utf-8 support (internationalized email, via the + ``UTF8=ACCEPT`` argument to :meth:`~imaplib.IMAP4.enable`). A new attribute, + :attr:`~imaplib.IMAP4.utf8_enabled`, tracks whether or not :rfc:`6855` + support is enabled. Milan Oberkirch, R. David Murray, and Maciej Szulik in + :issue:`21800`.) + +* :mod:`imaplib` now automatically encodes non-ASCII string usernames and + passwords using ``UTF8``, as recommended by the RFCs. (Contributed by Milan + Oberkirch in :issue:`21800`.) + +imghdr +------ + +* :func:`~imghdr.what` now recognizes the `OpenEXR <http://www.openexr.com>`_ + format. (Contributed by Martin Vignali and Claudiu Popa in :issue:`20295`.) + +importlib +--------- + +* :class:`importlib.util.LazyLoader` allows for the lazy loading of modules in + applications where startup time is paramount. + (Contributed by Brett Cannon in :issue:`17621`.) + +* :func:`importlib.abc.InspectLoader.source_to_code` is now a + static method to make it easier to work with source code in a string. + With a module object that you want to initialize you can then use + ``exec(code, module.__dict__)`` to execute the code in the module. + +* :func:`importlib.util.module_from_spec` is now the preferred way to create a + new module. Compared to :class:`types.ModuleType`, this new function will set + the various import-controlled attributes based on the passed-in spec object. + +inspect +------- + +* :class:`inspect.Signature` and :class:`inspect.Parameter` are now + picklable and hashable. (Contributed by Yury Selivanov in :issue:`20726` + and :issue:`20334`.) + +* New method :meth:`inspect.BoundArguments.apply_defaults`. (Contributed + by Yury Selivanov in :issue:`24190`.) + +* New class method :meth:`inspect.Signature.from_callable`, which makes + subclassing of :class:`~inspect.Signature` easier. (Contributed + by Yury Selivanov and Eric Snow in :issue:`17373`.) + +* New argument ``follow_wrapped`` for :func:`inspect.signature`. + (Contributed by Yury Selivanov in :issue:`20691`.) + +* New :func:`~inspect.iscoroutine`, :func:`~inspect.iscoroutinefunction` + and :func:`~inspect.isawaitable` functions. (Contributed by + Yury Selivanov in :issue:`24017`.) + +* New :func:`~inspect.getcoroutinelocals` and :func:`~inspect.getcoroutinestate` + functions. (Contributed by Yury Selivanov in :issue:`24400`.) + +ipaddress +--------- + +* :class:`ipaddress.IPv4Network` and :class:`ipaddress.IPv6Network` now + accept an ``(address, netmask)`` tuple argument, so as to easily construct + network objects from existing addresses. (Contributed by Peter Moody + and Antoine Pitrou in :issue:`16531`.) + +json +---- + +* The output of :mod:`json.tool` command line interface is now in the same + order as the input. Use the :option:`--sort-keys` option to sort the output + of dictionaries alphabetically by key. (Contributed by Berker Peksag in + :issue:`21650`.) + +* JSON decoder now raises :exc:`json.JSONDecodeError` instead of + :exc:`ValueError`. (Contributed by Serhiy Storchaka in :issue:`19361`.) + +math +---- + +* :data:`math.inf` and :data:`math.nan` constants added. (Contributed by Mark + Dickinson in :issue:`23185`.) +* :func:`math.isclose` function added. + (Contributed by Chris Barker and Tal Einat in :issue:`24270`.) + +os +-- + +* New :func:`os.scandir` function that exposes file information from + the operating system when listing a directory. :func:`os.scandir` + returns an iterator of :class:`os.DirEntry` objects corresponding to + the entries in the directory given by *path*. (Contributed by Ben + Hoyt with the help of Victor Stinner in :issue:`22524`.) + +* :class:`os.stat_result` now has a :attr:`~os.stat_result.st_file_attributes` + attribute on Windows. (Contributed by Ben Hoyt in :issue:`21719`.) + +* :func:`os.urandom`: On Linux 3.17 and newer, the ``getrandom()`` syscall is + now used when available. On OpenBSD 5.6 and newer, the C ``getentropy()`` + function is now used. These functions avoid the usage of an internal file + descriptor. + +os.path +------- + +* New :func:`~os.path.commonpath` function that extracts common path prefix. + Unlike the :func:`~os.path.commonprefix` function, it always returns a valid + path. (Contributed by Rafik Draoui and Serhiy Storchaka in :issue:`10395`.) + +pickle +------ + +* Serializing more "lookupable" objects (such as unbound methods or nested + classes) now are supported with pickle protocols < 4. + (Contributed by Serhiy Storchaka in :issue:`23611`.) + +poplib +------ + +* A new command :meth:`~poplib.POP3.utf8` enables :rfc:`6856` + (internationalized email) support if the POP server supports it. (Contributed + by Milan OberKirch in :issue:`21804`.) + +re +-- + +* Number of capturing groups in regular expression is no longer limited by 100. + (Contributed by Serhiy Storchaka in :issue:`22437`.) + +* Now unmatched groups are replaced with empty strings in :func:`re.sub` + and :func:`re.subn`. (Contributed by Serhiy Storchaka in :issue:`1519638`.) + +shutil +------ + +* :func:`~shutil.move` now accepts a *copy_function* argument, allowing, + for example, :func:`~shutil.copy` to be used instead of the default + :func:`~shutil.copy2` if there is a need to ignore metadata. (Contributed by + Claudiu Popa in :issue:`19840`.) + +signal +------ + +* On Windows, :func:`signal.set_wakeup_fd` now also supports socket handles. + (Contributed by Victor Stinner in :issue:`22018`.) + +* Different constants of :mod:`signal` module are now enumeration values using + the :mod:`enum` module. This allows meaningful names to be printed during + debugging, instead of integer “magic numbers”. (Contributed by Giampaolo + Rodola' in :issue:`21076`.) + +smtpd +----- + +* Both :class:`~smtpd.SMTPServer` and :class:`smtpd.SMTPChannel` now accept a + *decode_data* keyword to determine if the DATA portion of the SMTP + transaction is decoded using the ``utf-8`` codec or is instead provided to + :meth:`~smtpd.SMTPServer.process_message` as a byte string. The default + is ``True`` for backward compatibility reasons, but will change to ``False`` + in Python 3.6. If *decode_data* is set to ``False``, the + :meth:`~smtpd.SMTPServer.process_message` method must be prepared to accept + keyword arguments. (Contributed by Maciej Szulik in :issue:`19662`.) + +* :class:`~smtpd.SMTPServer` now advertises the ``8BITMIME`` extension + (:rfc:`6152`) if if *decode_data* has been set ``True``. If the client + specifies ``BODY=8BITMIME`` on the ``MAIL`` command, it is passed to + :meth:`~smtpd.SMTPServer.process_message` via the ``mail_options`` keyword. + (Contributed by Milan Oberkirch and R. David Murray in :issue:`21795`.) + +* :class:`~smtpd.SMTPServer` now supports the ``SMTPUTF8`` extension + (:rfc:`6531`: Internationalized Email). If the client specified ``SMTPUTF8 + BODY=8BITMIME`` on the ``MAIL`` command, they are passed to + :meth:`~smtpd.SMTPServer.process_message` via the ``mail_options`` keyword. + It is the responsibility of the :meth:`~smtpd.SMTPServer.process_message` + method to correctly handle the ``SMTPUTF8`` data. (Contributed by Milan + Oberkirch in :issue:`21725`.) + +* It is now possible to provide, directly or via name resolution, IPv6 + addresses in the :class:`~smtpd.SMTPServer` constructor, and have it + successfully connect. (Contributed by Milan Oberkirch in :issue:`14758`.) + +smtplib +------- + +* A new :meth:`~smtplib.SMTP.auth` method provides a convenient way to + implement custom authentication mechanisms. + (Contributed by Milan Oberkirch in :issue:`15014`.) + +* Additional debuglevel (2) shows timestamps for debug messages in + :class:`smtplib.SMTP`. (Contributed by Gavin Chappell and Maciej Szulik in + :issue:`16914`.) + +* :mod:`smtplib` now supports :rfc:`6531` (SMTPUTF8) in both the + :meth:`~smtplib.SMTP.sendmail` and :meth:`~smtplib.SMTP.send_message` + commands. (Contributed by Milan Oberkirch and R. David Murray in + :issue:`22027`.) + +sndhdr +------ + +* :func:`~sndhdr.what` and :func:`~sndhdr.whathdr` now return + :func:`~collections.namedtuple`. + (Contributed by Claudiu Popa in :issue:`18615`.) + +ssl +--- + +* The :meth:`~ssl.SSLSocket.do_handshake`, :meth:`~ssl.SSLSocket.read`, + :meth:`~ssl.SSLSocket.shutdown`, and :meth:`~ssl.SSLSocket.write` methods of + :class:`ssl.SSLSocket` don't reset the socket timeout anymore each time bytes + are received or sent. The socket timeout is now the maximum total duration of + the method. + +* Memory BIO Support: new classes :class:`~ssl.SSLObject`, + :class:`~ssl.MemoryBIO`, and new + :meth:`SSLContext.wrap_bio <ssl.SSLContext.wrap_bio>` method. + (Contributed by Geert Jansen in :issue:`21965`.) + +socket +------ + +* New :meth:`socket.socket.sendfile` method allows to send a file over a socket + by using high-performance :func:`os.sendfile` function on UNIX resulting in + uploads being from 2x to 3x faster than when using plain + :meth:`socket.socket.send`. + (Contributed by Giampaolo Rodola' in :issue:`17552`.) + +* The :meth:`socket.socket.sendall` method don't reset the socket timeout + anymore each time bytes are received or sent. The socket timeout is now the + maximum total duration to send all data. + +subprocess +---------- + +* The new :func:`subprocess.run` function runs subprocesses and returns a + :class:`subprocess.CompletedProcess` object. It Provides a more consistent + API than :func:`~subprocess.call`, :func:`~subprocess.check_call` and + :func:`~subprocess.check_output`. + +sys +--- + +* New :func:`~sys.set_coroutine_wrapper` and :func:`~sys.get_coroutine_wrapper` + functions. (Contributed by Yury Selivanov in :issue:`24017`.) + +sysconfig +--------- + +* The user scripts directory on Windows is now versioned. + (Contributed by Paul Moore in :issue:`23437`.) + +tarfile +------- + +* The :func:`tarfile.open` function now supports ``'x'`` (exclusive creation) + mode. (Contributed by Berker Peksag in :issue:`21717`.) + +* The :meth:`~tarfile.TarFile.extractall` and :meth:`~tarfile.TarFile.extract` + methods now take a keyword parameter *numeric_only*. If set to ``True``, + the extracted files and directories will be owned by the numeric uid and gid + from the tarfile. If set to ``False`` (the default, and the behavior in + versions prior to 3.5), they will be owned bythe named user and group in the + tarfile. (Contributed by Michael Vogt and Eric Smith in :issue:`23193`.) + +time +---- + +* The :func:`time.monotonic` function is now always available. (Contributed by + Victor Stinner in :issue:`22043`.) + +tkinter +------- + +* The :mod:`tkinter._fix` module used for setting up the Tcl/Tk environment + on Windows has been replaced by a private function in the :mod:`_tkinter` + module which makes no permanent changes to environment variables. + (Contributed by Zachary Ware in :issue:`20035`.) + +types +----- + +* New :func:`~types.coroutine` function. (Contributed by Yury Selivanov + in :issue:`24017`.) + +* New :class:`~types.CoroutineType`. (Contributed by Yury Selivanov + in :issue:`24400`.) + +urllib +------ + +* A new :class:`~urllib.request.HTTPPasswordMgrWithPriorAuth` allows HTTP Basic + Authentication credentials to be managed so as to eliminate unnecessary + ``401`` response handling, or to unconditionally send credentials + on the first request in order to communicate with servers that return a + ``404`` response instead of a ``401`` if the ``Authorization`` header is not + sent. (Contributed by Matej Cepl in :issue:`19494` and Akshit Khurana in + :issue:`7159`.) + +* A new :func:`~urllib.parse.urlencode` parameter *quote_via* provides a way to + control the encoding of query parts if needed. (Contributed by Samwyse and + Arnon Yaari in :issue:`13866`.) + +unicodedata +----------- + +* The :mod:`unicodedata` module now uses data from `Unicode 8.0.0 + <http://unicode.org/versions/Unicode8.0.0/>`_. + + +wsgiref +------- + +* *headers* parameter of :class:`wsgiref.headers.Headers` is now optional. + (Contributed by Pablo Torres Navarrete and SilentGhost in :issue:`5800`.) + +xmlrpc +------ + +* :class:`xmlrpc.client.ServerProxy` is now a :term:`context manager`. + (Contributed by Claudiu Popa in :issue:`20627`.) + +xml.sax +------- + +* SAX parsers now support a character stream of + :class:`~xml.sax.xmlreader.InputSource` object. + (Contributed by Serhiy Storchaka in :issue:`2175`.) + +faulthandler +------------ + +* :func:`~faulthandler.enable`, :func:`~faulthandler.register`, + :func:`~faulthandler.dump_traceback` and + :func:`~faulthandler.dump_traceback_later` functions now accept file + descriptors. (Contributed by Wei Wu in :issue:`23566`.) + +zipfile +------- + +* Added support for writing ZIP files to unseekable streams. + (Contributed by Serhiy Storchaka in :issue:`23252`.) + +* The :func:`zipfile.ZipFile.open` function now supports ``'x'`` (exclusive + creation) mode. (Contributed by Serhiy Storchaka in :issue:`21717`.) + + +Optimizations +============= + +The following performance enhancements have been added: + +* :func:`os.walk` has been sped up by 3-5x on POSIX systems and 7-20x + on Windows. This was done using the new :func:`os.scandir` function, + which exposes file information from the underlying ``readdir`` and + ``FindFirstFile``/``FindNextFile`` system calls. (Contributed by + Ben Hoyt with help from Victor Stinner in :issue:`23605`.) + +* Construction of ``bytes(int)`` (filled by zero bytes) is faster and uses less + memory for large objects. ``calloc()`` is used instead of ``malloc()`` to + allocate memory for these objects. + +* Some operations on :class:`~ipaddress.IPv4Network` and + :class:`~ipaddress.IPv6Network` have been massively sped up, such as + :meth:`~ipaddress.IPv4Network.subnets`, :meth:`~ipaddress.IPv4Network.supernet`, + :func:`~ipaddress.summarize_address_range`, :func:`~ipaddress.collapse_addresses`. + The speed up can range from 3x to 15x. + (:issue:`21486`, :issue:`21487`, :issue:`20826`) + +* Many operations on :class:`io.BytesIO` are now 50% to 100% faster. + (Contributed by Serhiy Storchaka in :issue:`15381` and David Wilson in + :issue:`22003`.) + +* :func:`marshal.dumps` is now faster (65%-85% with versions 3--4, 20-25% with + versions 0--2 on typical data, and up to 5x in best cases). + (Contributed by Serhiy Storchaka in :issue:`20416` and :issue:`23344`.) + +* The UTF-32 encoder is now 3x to 7x faster. (Contributed by Serhiy Storchaka + in :issue:`15027`.) + + +Build and C API Changes +======================= + +Changes to Python's build process and to the C API include: + +* New ``calloc`` functions: + + * :c:func:`PyMem_RawCalloc` + * :c:func:`PyMem_Calloc` + * :c:func:`PyObject_Calloc` + * :c:func:`_PyObject_GC_Calloc` + +* Windows builds now require Microsoft Visual C++ 14.0, which + is available as part of `Visual Studio 2015 <http://www.visualstudio.com>`_. + +Deprecated +========== + +New Keywords +------------ + +``async`` and ``await`` are not recommended to be used as variable, class, +function or module names. Introduced by :pep:`492` in Python 3.5, they will +become proper keywords in Python 3.7. + + +Unsupported Operating Systems +----------------------------- + +* Windows XP - Per :PEP:`11`, Microsoft support of Windows XP has ended. + + +Deprecated Python modules, functions and methods +------------------------------------------------ + +* The :mod:`formatter` module has now graduated to full deprecation and is still + slated for removal in Python 3.6. + +* :mod:`smtpd` has in the past always decoded the DATA portion of email + messages using the ``utf-8`` codec. This can now be controlled by the new + *decode_data* keyword to :class:`~smtpd.SMTPServer`. The default value is + ``True``, but this default is deprecated. Specify the *decode_data* keyword + with an appropriate value to avoid the deprecation warning. + +* Directly assigning values to the :attr:`~http.cookies.Morsel.key`, + :attr:`~http.cookies.Morsel.value` and + :attr:`~http.cookies.Morsel.coded_value` of :class:`~http.cookies.Morsel` + objects is deprecated. Use the :func:`~http.cookies.Morsel.set` method + instead. In addition, the undocumented *LegalChars* parameter of + :func:`~http.cookies.Morsel.set` is deprecated, and is now ignored. + +* Passing a format string as keyword argument *format_string* to the + :meth:`~string.Formatter.format` method of the :class:`string.Formatter` + class has been deprecated. + +* :func:`platform.dist` and :func:`platform.linux_distribution` functions are + now deprecated and will be removed in Python 3.7. Linux distributions use + too many different ways of describing themselves, so the functionality is + left to a package. + (Contributed by Vajrasky Kok and Berker Peksag in :issue:`1322`.) + +* The previously undocumented ``from_function`` and ``from_builtin`` methods of + :class:`inspect.Signature` are deprecated. Use new + :meth:`inspect.Signature.from_callable` instead. (Contributed by Yury + Selivanov in :issue:`24248`.) + +* :func:`inspect.getargspec` is deprecated and scheduled to be removed in + Python 3.6. (See :issue:`20438` for details.) + +* :func:`~inspect.getfullargspec`, :func:`~inspect.getargvalues`, + :func:`~inspect.getcallargs`, :func:`~inspect.getargvalues`, + :func:`~inspect.formatargspec`, and :func:`~inspect.formatargvalues` are + deprecated in favor of :func:`inspect.signature` API. (See :issue:`20438` + for details.) + + +Deprecated functions and types of the C API +------------------------------------------- + +* None yet. + + +Deprecated features +------------------- + +* None yet. + + +Removed +======= + +API and Feature Removals +------------------------ + +The following obsolete and previously deprecated APIs and features have been +removed: + +* The ``__version__`` attribute has been dropped from the email package. The + email code hasn't been shipped separately from the stdlib for a long time, + and the ``__version__`` string was not updated in the last few releases. + +* The internal ``Netrc`` class in the :mod:`ftplib` module was deprecated in + 3.4, and has now been removed. + (Contributed by Matt Chaput in :issue:`6623`.) + +* The concept of ``.pyo`` files has been removed. + +* The JoinableQueue class in the provisional asyncio module was deprecated + in 3.4.4 and is now removed (:issue:`23464`). + + +Porting to Python 3.5 +===================== + +This section lists previously described changes and other bugfixes +that may require changes to your code. + +Changes in the Python API +------------------------- + +* :pep:`475`: System calls are now retried when interrupted by a signal instead + of raising :exc:`InterruptedError` if the Python signal handler does not + raise an exception. + +* Before Python 3.5, a :class:`datetime.time` object was considered to be false + if it represented midnight in UTC. This behavior was considered obscure and + error-prone and has been removed in Python 3.5. See :issue:`13936` for full + details. + +* :meth:`ssl.SSLSocket.send()` now raises either :exc:`ssl.SSLWantReadError` + or :exc:`ssl.SSLWantWriteError` on a non-blocking socket if the operation + would block. Previously, it would return 0. See :issue:`20951`. + +* The ``__name__`` attribute of generator is now set from the function name, + instead of being set from the code name. Use ``gen.gi_code.co_name`` to + retrieve the code name. Generators also have a new ``__qualname__`` + attribute, the qualified name, which is now used for the representation + of a generator (``repr(gen)``). See :issue:`21205`. + +* The deprecated "strict" mode and argument of :class:`~html.parser.HTMLParser`, + :meth:`HTMLParser.error`, and the :exc:`HTMLParserError` exception have been + removed. (Contributed by Ezio Melotti in :issue:`15114`.) + The *convert_charrefs* argument of :class:`~html.parser.HTMLParser` is + now ``True`` by default. (Contributed by Berker Peksag in :issue:`21047`.) + +* Although it is not formally part of the API, it is worth noting for porting + purposes (ie: fixing tests) that error messages that were previously of the + form "'sometype' does not support the buffer protocol" are now of the form "a + bytes-like object is required, not 'sometype'". (Contributed by Ezio Melotti + in :issue:`16518`.) + +* If the current directory is set to a directory that no longer exists then + :exc:`FileNotFoundError` will no longer be raised and instead + :meth:`~importlib.machinery.FileFinder.find_spec` will return ``None`` + **without** caching ``None`` in :data:`sys.path_importer_cache` which is + different than the typical case (:issue:`22834`). + +* HTTP status code and messages from :mod:`http.client` and :mod:`http.server` + were refactored into a common :class:`~http.HTTPStatus` enum. The values in + :mod:`http.client` and :mod:`http.server` remain available for backwards + compatibility. (Contributed by Demian Brecht in :issue:`21793`.) + +* When an import loader defines :meth:`~importlib.machinery.Loader.exec_module` + it is now expected to also define + :meth:`~importlib.machinery.Loader.create_module` (raises a + :exc:`DeprecationWarning` now, will be an error in Python 3.6). If the loader + inherits from :class:`importlib.abc.Loader` then there is nothing to do, else + simply define :meth:`~importlib.machinery.Loader.create_module` to return + ``None`` (:issue:`23014`). + +* :func:`re.split` always ignored empty pattern matches, so the ``'x*'`` + pattern worked the same as ``'x+'``, and the ``'\b'`` pattern never worked. + Now :func:`re.split` raises a warning if the pattern could match + an empty string. For compatibility use patterns that never match an empty + string (e.g. ``'x+'`` instead of ``'x*'``). Patterns that could only match + an empty string (such as ``'\b'``) now raise an error. + +* The :class:`~http.cookies.Morsel` dict-like interface has been made self + consistent: morsel comparison now takes the :attr:`~http.cookies.Morsel.key` + and :attr:`~http.cookies.Morsel.value` into account, + :meth:`~http.cookies.Morsel.copy` now results in a + :class:`~http.cookies.Morsel` instance rather than a :class:`dict`, and + :meth:`~http.cookies.Morsel.update` will now raise an exception if any of the + keys in the update dictionary are invalid. In addition, the undocumented + *LegalChars* parameter of :func:`~http.cookies.Morsel.set` is deprecated and + is now ignored. (:issue:`2211`) + +* :pep:`488` has removed ``.pyo`` files from Python and introduced the optional + ``opt-`` tag in ``.pyc`` file names. The + :func:`importlib.util.cache_from_source` has gained an *optimization* + parameter to help control the ``opt-`` tag. Because of this, the + *debug_override* parameter of the function is now deprecated. `.pyo` files + are also no longer supported as a file argument to the Python interpreter and + thus serve no purpose when distributed on their own (i.e. sourcless code + distribution). Due to the fact that the magic number for bytecode has changed + in Python 3.5, all old `.pyo` files from previous versions of Python are + invalid regardless of this PEP. + +* The :mod:`socket` module now exports the CAN_RAW_FD_FRAMES constant on linux + 3.6 and greater. + +* The ``pygettext.py`` Tool now uses the standard +NNNN format for timezones in + the POT-Creation-Date header. + +* The :mod:`smtplib` module now uses :data:`sys.stderr` instead of previous + module level :data:`stderr` variable for debug output. If your (test) + program depends on patching the module level variable to capture the debug + output, you will need to update it to capture sys.stderr instead. + +* The :meth:`str.startswith` and :meth:`str.endswith` methods no longer return + ``True`` when finding the empty string and the indexes are completely out of + range. See :issue:`24284`. + +* The :func:`inspect.getdoc` function now returns documentation strings + inherited from base classes. Documentation strings no longer need to be + duplicated if the inherited documentation is appropriate. To suppress an + inherited string, an empty string must be specified (or the documentation + may be filled in). This change affects the output of the :mod:`pydoc` + module and the :func:`help` function. See :issue:`15582`. + +Changes in the C API +-------------------- + +* The undocumented :c:member:`~PyMemoryViewObject.format` member of the + (non-public) :c:type:`PyMemoryViewObject` structure has been removed. + + All extensions relying on the relevant parts in ``memoryobject.h`` + must be rebuilt. + +* The :c:type:`PyMemAllocator` structure was renamed to + :c:type:`PyMemAllocatorEx` and a new ``calloc`` field was added. + +* Removed non-documented macro :c:macro:`PyObject_REPR` which leaked references. + Use format character ``%R`` in :c:func:`PyUnicode_FromFormat`-like functions + to format the :func:`repr` of the object. + +* Because the lack of the :attr:`__module__` attribute breaks pickling and + introspection, a deprecation warning now is raised for builtin type without + the :attr:`__module__` attribute. Would be an AttributeError in future. + (:issue:`20204`) + +* As part of :pep:`492` implementation, ``tp_reserved`` slot of + :c:type:`PyTypeObject` was replaced with a + :c:member:`tp_as_async` slot. Refer to :ref:`coro-objects` for + new types, structures and functions. |