diff options
| author | Terry Jan Reedy <tjreedy@udel.edu> | 2015-09-29 01:56:35 -0400 |
|---|---|---|
| committer | Terry Jan Reedy <tjreedy@udel.edu> | 2015-09-29 01:56:35 -0400 |
| commit | 76a6df2cf858afd07c1c7ca471a733a2ef37611a (patch) | |
| tree | eac5bdcd34c641b0440bb7fff8a08d1aaec0e0a3 /Doc/library | |
| parent | c36fc7217eae7355215d1695670dddab9c9bb1a0 (diff) | |
| parent | e33af433e6951fcfd411c8ddc6b78aafcc1b9cd4 (diff) | |
| download | cpython-76a6df2cf858afd07c1c7ca471a733a2ef37611a.tar.gz | |
Merge with 3.4, Issue #24028: Add subsection about Idle calltips.
Diffstat (limited to 'Doc/library')
128 files changed, 5270 insertions, 1326 deletions
diff --git a/Doc/library/__future__.rst b/Doc/library/__future__.rst index 72f2963a2c..73d8b6b7e8 100644 --- a/Doc/library/__future__.rst +++ b/Doc/library/__future__.rst @@ -87,6 +87,9 @@ language using this mechanism: | unicode_literals | 2.6.0a2 | 3.0 | :pep:`3112`: | | | | | *Bytes literals in Python 3000* | +------------------+-------------+--------------+---------------------------------------------+ +| generator_stop | 3.5.0b1 | 3.7 | :pep:`479`: | +| | | | *StopIteration handling inside generators* | ++------------------+-------------+--------------+---------------------------------------------+ .. seealso:: diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst index 067fb8f748..72095a825f 100644 --- a/Doc/library/argparse.rst +++ b/Doc/library/argparse.rst @@ -135,7 +135,7 @@ ArgumentParser objects formatter_class=argparse.HelpFormatter, \ prefix_chars='-', fromfile_prefix_chars=None, \ argument_default=None, conflict_handler='error', \ - add_help=True) + add_help=True, allow_abbrev=True) Create a new :class:`ArgumentParser` object. All parameters should be passed as keyword arguments. Each parameter has its own more detailed description @@ -169,6 +169,12 @@ ArgumentParser objects * add_help_ - Add a -h/--help option to the parser (default: ``True``) + * allow_abbrev_ - Allows long options to be abbreviated if the + abbreviation is unambiguous. (default: ``True``) + + .. versionchanged:: 3.5 + *allow_abbrev* parameter was added. + The following sections describe how each of these are used. @@ -518,6 +524,26 @@ calls, we supply ``argument_default=SUPPRESS``:: >>> parser.parse_args([]) Namespace() +.. _allow_abbrev: + +allow_abbrev +^^^^^^^^^^^^ + +Normally, when you pass an argument list to the +:meth:`~ArgumentParser.parse_args` method of a :class:`ArgumentParser`, +it :ref:`recognizes abbreviations <prefix-matching>` of long options. + +This feature can be disabled by setting ``allow_abbrev`` to ``False``:: + + >>> parser = argparse.ArgumentParser(prog='PROG', allow_abbrev=False) + >>> parser.add_argument('--foobar', action='store_true') + >>> parser.add_argument('--foonley', action='store_false') + >>> parser.parse_args(['--foon']) + usage: PROG [-h] [--foobar] [--foonley] + PROG: error: unrecognized arguments: --foon + +.. versionadded:: 3.5 + conflict_handler ^^^^^^^^^^^^^^^^ @@ -1410,9 +1436,9 @@ argument:: Argument abbreviations (prefix matching) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -The :meth:`~ArgumentParser.parse_args` method allows long options to be -abbreviated to a prefix, if the abbreviation is unambiguous (the prefix matches -a unique option):: +The :meth:`~ArgumentParser.parse_args` method :ref:`by default <allow_abbrev>` +allows long options to be abbreviated to a prefix, if the abbreviation is +unambiguous (the prefix matches a unique option):: >>> parser = argparse.ArgumentParser(prog='PROG') >>> parser.add_argument('-bacon') @@ -1426,6 +1452,7 @@ a unique option):: PROG: error: ambiguous option: -ba could match -badger, -bacon An error is produced for arguments that could produce more than one options. +This feature can be disabled by setting :ref:`allow_abbrev` to ``False``. Beyond ``sys.argv`` diff --git a/Doc/library/asynchat.rst b/Doc/library/asynchat.rst index c6fa06162e..794da8cced 100644 --- a/Doc/library/asynchat.rst +++ b/Doc/library/asynchat.rst @@ -147,40 +147,6 @@ connection requests. by the channel after :meth:`found_terminator` is called. -asynchat - Auxiliary Classes ------------------------------------------- - -.. class:: fifo(list=None) - - A :class:`fifo` holding data which has been pushed by the application but - not yet popped for writing to the channel. A :class:`fifo` is a list used - to hold data and/or producers until they are required. If the *list* - argument is provided then it should contain producers or data items to be - written to the channel. - - - .. method:: is_empty() - - Returns ``True`` if and only if the fifo is empty. - - - .. method:: first() - - Returns the least-recently :meth:`push`\ ed item from the fifo. - - - .. method:: push(data) - - Adds the given data (which may be a string or a producer object) to the - producer fifo. - - - .. method:: pop() - - If the fifo is not empty, returns ``True, first()``, deleting the popped - item. Returns ``False, None`` for an empty fifo. - - .. _asynchat-example: asynchat Example diff --git a/Doc/library/asyncio-dev.rst b/Doc/library/asyncio-dev.rst index 4812c2f832..48e7666bbb 100644 --- a/Doc/library/asyncio-dev.rst +++ b/Doc/library/asyncio-dev.rst @@ -99,7 +99,7 @@ To schedule a callback from a different thread, the :meth:`BaseEventLoop.call_soon_threadsafe` method should be used. Example to schedule a coroutine from a different thread:: - loop.call_soon_threadsafe(asyncio.async, coro_func()) + loop.call_soon_threadsafe(asyncio.ensure_future, coro_func()) Most asyncio objects are not thread safe. You should only worry if you access objects outside the event loop. For example, to cancel a future, don't call @@ -162,10 +162,10 @@ Detect coroutine objects never scheduled ---------------------------------------- When a coroutine function is called and its result is not passed to -:func:`async` or to the :meth:`BaseEventLoop.create_task` method, the execution -of the coroutine object will never be scheduled which is probably a bug. -:ref:`Enable the debug mode of asyncio <asyncio-debug-mode>` to :ref:`log a -warning <asyncio-logger>` to detect it. +:func:`ensure_future` or to the :meth:`BaseEventLoop.create_task` method, +the execution of the coroutine object will never be scheduled which is +probably a bug. :ref:`Enable the debug mode of asyncio <asyncio-debug-mode>` +to :ref:`log a warning <asyncio-logger>` to detect it. Example with the bug:: @@ -184,7 +184,7 @@ Output in debug mode:: File "test.py", line 7, in <module> test() -The fix is to call the :func:`async` function or the +The fix is to call the :func:`ensure_future` function or the :meth:`BaseEventLoop.create_task` method with the coroutine object. .. seealso:: diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst index 78a07a00dd..0792a83556 100644 --- a/Doc/library/asyncio-eventloop.rst +++ b/Doc/library/asyncio-eventloop.rst @@ -36,7 +36,7 @@ Run an event loop Run until the :class:`Future` is done. If the argument is a :ref:`coroutine object <coroutine>`, it is wrapped by - :func:`async`. + :func:`ensure_future`. Return the Future's result, or raise its exception. @@ -275,7 +275,9 @@ Creating connections to bind the socket to locally. The *local_host* and *local_port* are looked up using getaddrinfo(), similarly to *host* and *port*. - On Windows with :class:`ProactorEventLoop`, SSL/TLS is not supported. + .. versionchanged:: 3.5 + + On Windows with :class:`ProactorEventLoop`, SSL/TLS is now supported. .. seealso:: @@ -361,14 +363,16 @@ Creating listening connections This method is a :ref:`coroutine <coroutine>`. - On Windows with :class:`ProactorEventLoop`, SSL/TLS is not supported. + .. versionchanged:: 3.5 + + On Windows with :class:`ProactorEventLoop`, SSL/TLS is now supported. .. seealso:: The function :func:`start_server` creates a (:class:`StreamReader`, :class:`StreamWriter`) pair and calls back a function with this pair. - .. versionchanged:: 3.4.4 + .. versionchanged:: 3.5.1 The *host* parameter can now be a sequence of strings. diff --git a/Doc/library/asyncio-eventloops.rst b/Doc/library/asyncio-eventloops.rst index afb8b9f35b..ae3bf90298 100644 --- a/Doc/library/asyncio-eventloops.rst +++ b/Doc/library/asyncio-eventloops.rst @@ -100,8 +100,6 @@ Common limits of Windows event loops: :class:`ProactorEventLoop` specific limits: -- SSL is not supported: :meth:`~BaseEventLoop.create_connection` and - :meth:`~BaseEventLoop.create_server` cannot be used with SSL for example - :meth:`~BaseEventLoop.create_datagram_endpoint` (UDP) is not supported - :meth:`~BaseEventLoop.add_reader` and :meth:`~BaseEventLoop.add_writer` are not supported @@ -112,6 +110,10 @@ The best resolution is 0.5 msec. The resolution depends on the hardware <http://fr.wikipedia.org/wiki/High_Precision_Event_Timer>`_) and on the Windows configuration. See :ref:`asyncio delayed calls <asyncio-delayed-calls>`. +.. versionchanged:: 3.5 + + :class:`ProactorEventLoop` now supports SSL. + Mac OS X ^^^^^^^^ diff --git a/Doc/library/asyncio-protocol.rst b/Doc/library/asyncio-protocol.rst index 656816f75b..18aeafa81f 100644 --- a/Doc/library/asyncio-protocol.rst +++ b/Doc/library/asyncio-protocol.rst @@ -82,7 +82,7 @@ BaseTransport - ``'subprocess'``: :class:`subprocess.Popen` instance - .. versionchanged:: 3.4.4 + .. versionchanged:: 3.5.1 ``'ssl_object'`` info was added to SSL sockets. @@ -453,9 +453,9 @@ buffer size reaches the low-water mark. Coroutines and protocols ------------------------ -Coroutines can be scheduled in a protocol method using :func:`async`, but there -is no guarantee made about the execution order. Protocols are not aware of -coroutines created in protocol methods and so will not wait for them. +Coroutines can be scheduled in a protocol method using :func:`ensure_future`, +but there is no guarantee made about the execution order. Protocols are not +aware of coroutines created in protocol methods and so will not wait for them. To have a reliable execution order, use :ref:`stream objects <asyncio-streams>` in a coroutine with ``yield from``. For example, the :meth:`StreamWriter.drain` diff --git a/Doc/library/asyncio-queue.rst b/Doc/library/asyncio-queue.rst index 33706726a3..f11c09ac29 100644 --- a/Doc/library/asyncio-queue.rst +++ b/Doc/library/asyncio-queue.rst @@ -8,7 +8,6 @@ Queues: * :class:`Queue` * :class:`PriorityQueue` * :class:`LifoQueue` -* :class:`JoinableQueue` asyncio queue API was designed to be close to classes of the :mod:`queue` module (:class:`~queue.Queue`, :class:`~queue.PriorityQueue`, @@ -144,16 +143,6 @@ LifoQueue first. -JoinableQueue -^^^^^^^^^^^^^ - -.. class:: JoinableQueue - - Deprecated alias for :class:`Queue`. - - .. deprecated:: 3.4.4 - - Exceptions ^^^^^^^^^^ diff --git a/Doc/library/asyncio-task.rst b/Doc/library/asyncio-task.rst index e7ff7d2e2d..166ab73878 100644 --- a/Doc/library/asyncio-task.rst +++ b/Doc/library/asyncio-task.rst @@ -8,17 +8,23 @@ Tasks and coroutines Coroutines ---------- -A coroutine is a generator that follows certain conventions. For -documentation purposes, all coroutines should be decorated with -``@asyncio.coroutine``, but this cannot be strictly enforced. - -Coroutines use the ``yield from`` syntax introduced in :pep:`380`, +Coroutines used with :mod:`asyncio` may be implemented using the +:keyword:`async def` statement, or by using :term:`generators <generator>`. +The :keyword:`async def` type of coroutine was added in Python 3.5, and +is recommended if there is no need to support older Python versions. + +Generator-based coroutines should be decorated with :func:`@asyncio.coroutine +<asyncio.coroutine>`, although this is not strictly enforced. +The decorator enables compatibility with :keyword:`async def` coroutines, +and also serves as documentation. Generator-based +coroutines use the ``yield from`` syntax introduced in :pep:`380`, instead of the original ``yield`` syntax. The word "coroutine", like the word "generator", is used for two different (though related) concepts: -- The function that defines a coroutine (a function definition +- The function that defines a coroutine + (a function definition using :keyword:`async def` or decorated with ``@asyncio.coroutine``). If disambiguation is needed we will call this a *coroutine function* (:func:`iscoroutinefunction` returns ``True``). @@ -30,29 +36,30 @@ different (though related) concepts: Things a coroutine can do: -- ``result = yield from future`` -- suspends the coroutine until the +- ``result = await future`` or ``result = yield from future`` -- + suspends the coroutine until the future is done, then returns the future's result, or raises an exception, which will be propagated. (If the future is cancelled, it will raise a ``CancelledError`` exception.) Note that tasks are futures, and everything said about futures also applies to tasks. -- ``result = yield from coroutine`` -- wait for another coroutine to +- ``result = await coroutine`` or ``result = yield from coroutine`` -- + wait for another coroutine to produce a result (or raise an exception, which will be propagated). The ``coroutine`` expression must be a *call* to another coroutine. - ``return expression`` -- produce a result to the coroutine that is - waiting for this one using ``yield from``. + waiting for this one using :keyword:`await` or ``yield from``. - ``raise exception`` -- raise an exception in the coroutine that is - waiting for this one using ``yield from``. + waiting for this one using :keyword:`await` or ``yield from``. -Calling a coroutine does not start its code running -- it is just a -generator, and the coroutine object returned by the call is really a -generator object, which doesn't do anything until you iterate over it. -In the case of a coroutine object, there are two basic ways to start -it running: call ``yield from coroutine`` from another coroutine +Calling a coroutine does not start its code running -- +the coroutine object returned by the call doesn't do anything until you +schedule its execution. There are two basic ways to start it running: +call ``await coroutine`` or ``yield from coroutine`` from another coroutine (assuming the other coroutine is already running!), or schedule its execution -using the :func:`async` function or the :meth:`BaseEventLoop.create_task` +using the :func:`ensure_future` function or the :meth:`BaseEventLoop.create_task` method. @@ -60,9 +67,15 @@ Coroutines (and tasks) can only run when the event loop is running. .. decorator:: coroutine - Decorator to mark coroutines. + Decorator to mark generator-based coroutines. This enables + the generator use :keyword:`!yield from` to call :keyword:`async + def` coroutines, and also enables the generator to be called by + :keyword:`async def` coroutines, for instance using an + :keyword:`await` expression. + + There is no need to decorate :keyword:`async def` coroutines themselves. - If the coroutine is not yielded from before it is destroyed, an error + If the generator is not yielded from before it is destroyed, an error message is logged. See :ref:`Detect coroutines never scheduled <asyncio-coroutine-not-scheduled>`. @@ -72,7 +85,7 @@ Coroutines (and tasks) can only run when the event loop is running. even if they are plain Python functions returning a :class:`Future`. This is intentional to have a freedom of tweaking the implementation of these functions in the future. If such a function is needed to be - used in a callback-style code, wrap its result with :func:`async`. + used in a callback-style code, wrap its result with :func:`ensure_future`. .. _asyncio-hello-world-coroutine: @@ -84,8 +97,7 @@ Example of coroutine displaying ``"Hello World"``:: import asyncio - @asyncio.coroutine - def hello_world(): + async def hello_world(): print("Hello World!") loop = asyncio.get_event_loop() @@ -111,20 +123,30 @@ using the :meth:`sleep` function:: import asyncio import datetime - @asyncio.coroutine - def display_date(loop): + async def display_date(loop): end_time = loop.time() + 5.0 while True: print(datetime.datetime.now()) if (loop.time() + 1.0) >= end_time: break - yield from asyncio.sleep(1) + await asyncio.sleep(1) loop = asyncio.get_event_loop() # Blocking call which returns when the display_date() coroutine is done loop.run_until_complete(display_date(loop)) loop.close() +The same coroutine implemented using a generator:: + + @asyncio.coroutine + def display_date(loop): + end_time = loop.time() + 5.0 + while True: + print(datetime.datetime.now()) + if (loop.time() + 1.0) >= end_time: + break + yield from asyncio.sleep(1) + .. seealso:: The :ref:`display the current date with call_later() @@ -139,15 +161,13 @@ Example chaining coroutines:: import asyncio - @asyncio.coroutine - def compute(x, y): + async def compute(x, y): print("Compute %s + %s ..." % (x, y)) - yield from asyncio.sleep(1.0) + await asyncio.sleep(1.0) return x + y - @asyncio.coroutine - def print_sum(x, y): - result = yield from compute(x, y) + async def print_sum(x, y): + result = await compute(x, y) print("%s + %s = %s" % (x, y, result)) loop = asyncio.get_event_loop() @@ -374,7 +394,7 @@ Task <coroutine>` did not complete. It is probably a bug and a warning is logged: see :ref:`Pending task destroyed <asyncio-pending-task-destroyed>`. - Don't directly create :class:`Task` instances: use the :func:`async` + Don't directly create :class:`Task` instances: use the :func:`ensure_future` function or the :meth:`BaseEventLoop.create_task` method. This class is :ref:`not thread safe <asyncio-multithreading>`. @@ -550,12 +570,14 @@ Task functions .. function:: iscoroutine(obj) - Return ``True`` if *obj* is a :ref:`coroutine object <coroutine>`. + Return ``True`` if *obj* is a :ref:`coroutine object <coroutine>`, + which may be based on a generator or an :keyword:`async def` coroutine. -.. function:: iscoroutinefunction(obj) +.. function:: iscoroutinefunction(func) - Return ``True`` if *func* is a decorated :ref:`coroutine function - <coroutine>`. + Return ``True`` if *func* is determined to be a :ref:`coroutine function + <coroutine>`, which may be a decorated generator function or an + :keyword:`async def` function. .. coroutinefunction:: sleep(delay, result=None, \*, loop=None) diff --git a/Doc/library/bz2.rst b/Doc/library/bz2.rst index 488cda5968..1b8d9cffc6 100644 --- a/Doc/library/bz2.rst +++ b/Doc/library/bz2.rst @@ -120,6 +120,10 @@ All of the classes in this module may safely be accessed from multiple threads. .. versionchanged:: 3.4 The ``'x'`` (exclusive creation) mode was added. + .. versionchanged:: 3.5 + The :meth:`~io.BufferedIOBase.read` method now accepts an argument of + ``None``. + Incremental (de)compression --------------------------- @@ -162,15 +166,32 @@ Incremental (de)compression you need to decompress a multi-stream input with :class:`BZ2Decompressor`, you must use a new decompressor for each stream. - .. method:: decompress(data) + .. method:: decompress(data, max_length=-1) + + Decompress *data* (a :term:`bytes-like object`), returning + uncompressed data as bytes. Some of *data* may be buffered + internally, for use in later calls to :meth:`decompress`. The + returned data should be concatenated with the output of any + previous calls to :meth:`decompress`. + + If *max_length* is nonnegative, returns at most *max_length* + bytes of decompressed data. If this limit is reached and further + output can be produced, the :attr:`~.needs_input` attribute will + be set to ``False``. In this case, the next call to + :meth:`~.decompress` may provide *data* as ``b''`` to obtain + more of the output. - Provide data to the decompressor object. Returns a chunk of decompressed - data if possible, or an empty byte string otherwise. + If all of the input data was decompressed and returned (either + because this was less than *max_length* bytes, or because + *max_length* was negative), the :attr:`~.needs_input` attribute + will be set to ``True``. - Attempting to decompress data after the end of the current stream is - reached raises an :exc:`EOFError`. If any data is found after the end of - the stream, it is ignored and saved in the :attr:`unused_data` attribute. + Attempting to decompress data after the end of stream is reached + raises an `EOFError`. Any data found after the end of the + stream is ignored and saved in the :attr:`~.unused_data` attribute. + .. versionchanged:: 3.5 + Added the *max_length* parameter. .. attribute:: eof @@ -186,6 +207,13 @@ Incremental (de)compression If this attribute is accessed before the end of the stream has been reached, its value will be ``b''``. + .. attribute:: needs_input + + ``False`` if the :meth:`.decompress` method can provide more + decompressed data before requiring new uncompressed input. + + .. versionadded:: 3.5 + One-shot (de)compression ------------------------ diff --git a/Doc/library/cgi.rst b/Doc/library/cgi.rst index 74abed5c55..7e496ca361 100644 --- a/Doc/library/cgi.rst +++ b/Doc/library/cgi.rst @@ -157,6 +157,9 @@ return bytes):: if not line: break linecount = linecount + 1 +:class:`FieldStorage` objects also support being used in a :keyword:`with` +statement, which will automatically close them when done. + If an error is encountered when obtaining the contents of an uploaded file (for example, when the user interrupts the form submission by clicking on a Back or Cancel button) the :attr:`~FieldStorage.done` attribute of the @@ -182,6 +185,10 @@ A form submitted via POST that also has a query string will contain both The :attr:`~FieldStorage.file` attribute is automatically closed upon the garbage collection of the creating :class:`FieldStorage` instance. +.. versionchanged:: 3.5 + Added support for the context management protocol to the + :class:`FieldStorage` class. + Higher Level Interface ---------------------- diff --git a/Doc/library/cmath.rst b/Doc/library/cmath.rst index a981d94a11..ab619a082a 100644 --- a/Doc/library/cmath.rst +++ b/Doc/library/cmath.rst @@ -207,6 +207,38 @@ Classification functions and ``False`` otherwise. +.. function:: isclose(a, b, *, rel_tol=1e-09, abs_tol=0.0) + + Return ``True`` if the values *a* and *b* are close to each other and + ``False`` otherwise. + + Whether or not two values are considered close is determined according to + given absolute and relative tolerances. + + *rel_tol* is the relative tolerance -- it is the maximum allowed difference + between *a* and *b*, relative to the larger absolute value of *a* or *b*. + For example, to set a tolerance of 5%, pass ``rel_tol=0.05``. The default + tolerance is ``1e-09``, which assures that the two values are the same + within about 9 decimal digits. *rel_tol* must be greater than zero. + + *abs_tol* is the minimum absolute tolerance -- useful for comparisons near + zero. *abs_tol* must be at least zero. + + If no errors occur, the result will be: + ``abs(a-b) <= max(rel_tol * max(abs(a), abs(b)), abs_tol)``. + + The IEEE 754 special values of ``NaN``, ``inf``, and ``-inf`` will be + handled according to IEEE rules. Specifically, ``NaN`` is not considered + close to any other value, including ``NaN``. ``inf`` and ``-inf`` are only + considered close to themselves. + + .. versionadded:: 3.5 + + .. seealso:: + + :pep:`485` -- A function for testing approximate equality + + Constants --------- diff --git a/Doc/library/code.rst b/Doc/library/code.rst index 5b5d7cc8c1..275201c69b 100644 --- a/Doc/library/code.rst +++ b/Doc/library/code.rst @@ -4,6 +4,7 @@ .. module:: code :synopsis: Facilities to implement read-eval-print loops. +**Source code:** :source:`Lib/code.py` The ``code`` module provides facilities to implement read-eval-print loops in Python. Two classes and convenience functions are included which can be used to @@ -113,6 +114,9 @@ Interactive Interpreter Objects because it is within the interpreter object implementation. The output is written by the :meth:`write` method. + .. versionchanged:: 3.5 The full chained traceback is displayed instead + of just the primary traceback. + .. method:: InteractiveInterpreter.write(data) @@ -165,4 +169,3 @@ interpreter objects as well as the following additions. newline. When the user enters the EOF key sequence, :exc:`EOFError` is raised. The base implementation reads from ``sys.stdin``; a subclass may replace this with a different implementation. - diff --git a/Doc/library/codecs.rst b/Doc/library/codecs.rst index 628969c65d..46d72b5e9b 100644 --- a/Doc/library/codecs.rst +++ b/Doc/library/codecs.rst @@ -7,6 +7,7 @@ .. sectionauthor:: Marc-André Lemburg <mal@lemburg.com> .. sectionauthor:: Martin v. Löwis <martin@v.loewis.de> +**Source code:** :source:`Lib/codecs.py` .. index:: single: Unicode @@ -29,10 +30,9 @@ module features are restricted to use specifically with The module defines the following functions for encoding and decoding with any codec: -.. function:: encode(obj, [encoding[, errors]]) +.. function:: encode(obj, encoding='utf-8', errors='strict') - Encodes *obj* using the codec registered for *encoding*. The default - encoding is ``utf-8``. + Encodes *obj* using the codec registered for *encoding*. *Errors* may be given to set the desired error handling scheme. The default error handler is ``'strict'`` meaning that encoding errors raise @@ -40,10 +40,9 @@ any codec: :exc:`UnicodeEncodeError`). Refer to :ref:`codec-base-classes` for more information on codec error handling. -.. function:: decode(obj, [encoding[, errors]]) +.. function:: decode(obj, encoding='utf-8', errors='strict') - Decodes *obj* using the codec registered for *encoding*. The default - encoding is ``utf-8``. + Decodes *obj* using the codec registered for *encoding*. *Errors* may be given to set the desired error handling scheme. The default error handler is ``'strict'`` meaning that decoding errors raise @@ -104,7 +103,6 @@ The full details for each codec can also be looked up directly: To simplify access to the various codec components, the module provides these additional functions which use :func:`lookup` for the codec lookup: - .. function:: getencoder(encoding) Look up the codec for the given encoding and return its encoder function. @@ -274,6 +272,7 @@ implement the file protocols. Codec authors also need to define how the codec will handle encoding and decoding errors. +.. _surrogateescape: .. _error-handlers: Error Handlers @@ -315,10 +314,14 @@ The following error handlers are only applicable to | | reference (only for encoding). Implemented | | | in :func:`xmlcharrefreplace_errors`. | +-------------------------+-----------------------------------------------+ -| ``'backslashreplace'`` | Replace with backslashed escape sequences | -| | (only for encoding). Implemented in | +| ``'backslashreplace'`` | Replace with backslashed escape sequences. | +| | Implemented in | | | :func:`backslashreplace_errors`. | +-------------------------+-----------------------------------------------+ +| ``'namereplace'`` | Replace with ``\N{...}`` escape sequences | +| | (only for encoding). Implemented in | +| | :func:`namereplace_errors`. | ++-------------------------+-----------------------------------------------+ | ``'surrogateescape'`` | On decoding, replace byte with individual | | | surrogate code ranging from ``U+DC80`` to | | | ``U+DCFF``. This code will then be turned | @@ -344,6 +347,13 @@ In addition, the following error handler is specific to the given codecs: .. versionchanged:: 3.4 The ``'surrogatepass'`` error handlers now works with utf-16\* and utf-32\* codecs. +.. versionadded:: 3.5 + The ``'namereplace'`` error handler. + +.. versionchanged:: 3.5 + The ``'backslashreplace'`` error handlers now works with decoding and + translating. + The set of allowed values can be extended by registering a new named error handler: @@ -411,9 +421,17 @@ functions: .. function:: backslashreplace_errors(exception) - Implements the ``'backslashreplace'`` error handling (for encoding with + Implements the ``'backslashreplace'`` error handling (for + :term:`text encodings <text encoding>` only): malformed data is + replaced by a backslashed escape sequence. + +.. function:: namereplace_errors(exception) + + Implements the ``'namereplace'`` error handling (for encoding with :term:`text encodings <text encoding>` only): the - unencodable character is replaced by a backslashed escape sequence. + unencodable character is replaced by a ``\N{...}`` escape sequence. + + .. versionadded:: 3.5 .. _codec-objects: @@ -1144,8 +1162,16 @@ particular, the following variants typically exist: +-----------------+--------------------------------+--------------------------------+ | koi8_r | | Russian | +-----------------+--------------------------------+--------------------------------+ +| koi8_t | | Tajik | +| | | | +| | | .. versionadded:: 3.5 | ++-----------------+--------------------------------+--------------------------------+ | koi8_u | | Ukrainian | +-----------------+--------------------------------+--------------------------------+ +| kz1048 | kz_1048, strk1048_2002, rk1048 | Kazakh | +| | | | +| | | .. versionadded:: 3.5 | ++-----------------+--------------------------------+--------------------------------+ | mac_cyrillic | maccyrillic | Bulgarian, Byelorussian, | | | | Macedonian, Russian, Serbian | +-----------------+--------------------------------+--------------------------------+ @@ -1446,4 +1472,3 @@ This module implements a variant of the UTF-8 codec: On encoding a UTF-8 encoded BOM will be prepended to the UTF-8 encoded bytes. For the stateful encoder this is only done once (on the first write to the byte stream). For decoding an optional UTF-8 encoded BOM at the start of the data will be skipped. - diff --git a/Doc/library/collections.abc.rst b/Doc/library/collections.abc.rst index 99c43113e3..563c1bc4c5 100644 --- a/Doc/library/collections.abc.rst +++ b/Doc/library/collections.abc.rst @@ -33,13 +33,14 @@ The collections module offers the following :term:`ABCs <abstract base class>`: .. tabularcolumns:: |l|L|L|L| -========================= ===================== ====================== ==================================================== +========================== ====================== ======================= ==================================================== ABC Inherits from Abstract Methods Mixin Methods -========================= ===================== ====================== ==================================================== +========================== ====================== ======================= ==================================================== :class:`Container` ``__contains__`` :class:`Hashable` ``__hash__`` :class:`Iterable` ``__iter__`` :class:`Iterator` :class:`Iterable` ``__next__`` ``__iter__`` +:class:`Generator` :class:`Iterator` ``send``, ``throw`` ``close``, ``__iter__``, ``__next__`` :class:`Sized` ``__len__`` :class:`Callable` ``__call__`` @@ -80,7 +81,11 @@ ABC Inherits from Abstract Methods Mixin :class:`KeysView` :class:`MappingView`, ``__contains__``, :class:`Set` ``__iter__`` :class:`ValuesView` :class:`MappingView` ``__contains__``, ``__iter__`` -========================= ===================== ====================== ==================================================== +:class:`Awaitable` ``__await__`` +:class:`Coroutine` :class:`Awaitable` ``send``, ``throw`` ``close`` +:class:`AsyncIterable` ``__aiter__`` +:class:`AsyncIterator` :class:`AsyncIterable` ``__anext__`` ``__aiter__`` +========================== ====================== ======================= ==================================================== .. class:: Container @@ -102,11 +107,34 @@ ABC Inherits from Abstract Methods Mixin :meth:`~iterator.__next__` methods. See also the definition of :term:`iterator`. +.. class:: Generator + + ABC for generator classes that implement the protocol defined in + :pep:`342` that extends iterators with the :meth:`~generator.send`, + :meth:`~generator.throw` and :meth:`~generator.close` methods. + See also the definition of :term:`generator`. + + .. versionadded:: 3.5 + .. class:: Sequence MutableSequence ABCs for read-only and mutable :term:`sequences <sequence>`. + Implementation note: Some of the mixin methods, such as + :meth:`__iter__`, :meth:`__reversed__` and :meth:`index`, make + repeated calls to the underlying :meth:`__getitem__` method. + Consequently, if :meth:`__getitem__` is implemented with constant + access speed, the mixin methods will have linear performance; + however, if the underlying method is linear (as it would be with a + linked list), the mixins will have quadratic performance and will + likely need to be overridden. + + .. versionchanged:: 3.5 + The index() method added support for *stop* and *start* + arguments. + + .. class:: Set MutableSet @@ -124,6 +152,56 @@ ABC Inherits from Abstract Methods Mixin ABCs for mapping, items, keys, and values :term:`views <view>`. +.. class:: Awaitable + + ABC for :term:`awaitable` objects, which can be used in :keyword:`await` + expressions. Custom implementations must provide the :meth:`__await__` + method. + + :term:`Coroutine` objects and instances of the + :class:`~collections.abc.Coroutine` ABC are all instances of this ABC. + + .. note:: + In CPython, generator-based coroutines (generators decorated with + :func:`types.coroutine` or :func:`asyncio.coroutine`) are + *awaitables*, even though they do not have an :meth:`__await__` method. + Using ``isinstance(gencoro, Awaitable)`` for them will return ``False``. + Use :func:`inspect.isawaitable` to detect them. + + .. versionadded:: 3.5 + +.. class:: Coroutine + + ABC for coroutine compatible classes. These implement the + following methods, defined in :ref:`coroutine-objects`: + :meth:`~coroutine.send`, :meth:`~coroutine.throw`, and + :meth:`~coroutine.close`. Custom implementations must also implement + :meth:`__await__`. All :class:`Coroutine` instances are also instances of + :class:`Awaitable`. See also the definition of :term:`coroutine`. + + .. note:: + In CPython, generator-based coroutines (generators decorated with + :func:`types.coroutine` or :func:`asyncio.coroutine`) are + *awaitables*, even though they do not have an :meth:`__await__` method. + Using ``isinstance(gencoro, Coroutine)`` for them will return ``False``. + Use :func:`inspect.isawaitable` to detect them. + + .. versionadded:: 3.5 + +.. class:: AsyncIterable + + ABC for classes that provide ``__aiter__`` method. See also the + definition of :term:`asynchronous iterable`. + + .. versionadded:: 3.5 + +.. class:: AsyncIterator + + ABC for classes that provide ``__aiter__`` and ``__anext__`` + methods. See also the definition of :term:`asynchronous iterator`. + + .. versionadded:: 3.5 + These ABCs allow us to ask classes or instances if they provide particular functionality, for example:: diff --git a/Doc/library/collections.rst b/Doc/library/collections.rst index 347cf1fca5..8121cc411a 100644 --- a/Doc/library/collections.rst +++ b/Doc/library/collections.rst @@ -387,7 +387,7 @@ or subtracting from an empty counter. Section 4.6.3, Exercise 19*. * To enumerate all distinct multisets of a given size over a given set of - elements, see :func:`itertools.combinations_with_replacement`. + elements, see :func:`itertools.combinations_with_replacement`: map(Counter, combinations_with_replacement('ABC', 2)) --> AA AB AC BB BC CC @@ -437,6 +437,13 @@ or subtracting from an empty counter. Remove all elements from the deque leaving it with length 0. + .. method:: copy() + + Create a shallow copy of the deque. + + .. versionadded:: 3.5 + + .. method:: count(x) Count the number of deque elements equal to *x*. @@ -457,6 +464,22 @@ or subtracting from an empty counter. elements in the iterable argument. + .. method:: index(x[, start[, stop]]) + + Return the position of *x* in the deque (at or after index *start* + and before index *stop*). Returns the first match or raises + :exc:`ValueError` if not found. + + .. versionadded:: 3.5 + + + .. method:: insert(i, x) + + Insert *x* into the deque at position *i*. + + .. versionadded:: 3.5 + + .. method:: pop() Remove and return an element from the right side of the deque. If no @@ -471,7 +494,7 @@ or subtracting from an empty counter. .. method:: remove(value) - Removed the first occurrence of *value*. If not found, raises a + Remove the first occurrence of *value*. If not found, raises a :exc:`ValueError`. @@ -504,6 +527,9 @@ the :keyword:`in` operator, and subscript references such as ``d[-1]``. Indexed access is O(1) at both ends but slows to O(n) in the middle. For fast random access, use lists instead. +Starting in version 3.5, deques support ``__add__()``, ``__mul__()``, +and ``__imul__()``. + Example: .. doctest:: @@ -899,6 +925,15 @@ create a new named tuple type from the :attr:`_fields` attribute: >>> Point3D = namedtuple('Point3D', Point._fields + ('z',)) +Docstrings can be customized by making direct assignments to the ``__doc__`` +fields: + + >>> Book = namedtuple('Book', ['id', 'title', 'authors']) + >>> Book.__doc__ = 'Hardcover book in active collection' + >>> Book.id.__doc__ = '13-digit ISBN' + >>> Book.title.__doc__ = 'Title of first printing' + >>> Book.author.__doc__ = 'List of authors sorted by last name' + Default values can be implemented by using :meth:`_replace` to customize a prototype instance: @@ -982,6 +1017,9 @@ The :class:`OrderedDict` constructor and :meth:`update` method both accept keyword arguments, but their order is lost because Python's function call semantics pass-in keyword arguments using a regular unordered dictionary. +.. versionchanged:: 3.5 + The items, keys, and values :term:`views <view>` of :class:`OrderedDict` now + support reverse iteration using :func:`reversed`. :class:`OrderedDict` Examples and Recipes ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -1119,3 +1157,7 @@ attribute. be an instance of :class:`bytes`, :class:`str`, :class:`UserString` (or a subclass) or an arbitrary sequence which can be converted into a string using the built-in :func:`str` function. + + .. versionchanged:: 3.5 + New methods ``__getnewargs__``, ``__rmod__``, ``casefold``, + ``format_map``, ``isprintable``, and ``maketrans``. diff --git a/Doc/library/compileall.rst b/Doc/library/compileall.rst index 9712de2fef..c5736f2043 100644 --- a/Doc/library/compileall.rst +++ b/Doc/library/compileall.rst @@ -42,7 +42,8 @@ compile Python sources. .. cmdoption:: -q - Do not print the list of files compiled, print only error messages. + Do not print the list of files compiled. If passed once, error messages will + still be printed. If passed twice (``-qq``), all output is suppressed. .. cmdoption:: -d destdir @@ -70,9 +71,28 @@ compile Python sources. is to write files to their :pep:`3147` locations and names, which allows byte-code files from multiple versions of Python to coexist. +.. cmdoption:: -r + + Control the maximum recursion level for subdirectories. + If this is given, then ``-l`` option will not be taken into account. + :program:`python -m compileall <directory> -r 0` is equivalent to + :program:`python -m compileall <directory> -l`. + +.. cmdoption:: -j N + + Use *N* workers to compile the files within the given directory. + If ``0`` is used, then the result of :func:`os.cpu_count()` + will be used. + .. versionchanged:: 3.2 Added the ``-i``, ``-b`` and ``-h`` options. +.. versionchanged:: 3.5 + Added the ``-j``, ``-r``, and ``-qq`` options. ``-q`` option + was changed to a multilevel value. ``-b`` will always produce a + byte-code file ending in ``.pyc``, never ``.pyo``. + + There is no command-line option to control the optimization level used by the :func:`compile` function, because the Python interpreter itself already provides the option: :program:`python -O -m compileall`. @@ -80,7 +100,7 @@ provides the option: :program:`python -O -m compileall`. Public functions ---------------- -.. function:: compile_dir(dir, maxlevels=10, ddir=None, force=False, rx=None, quiet=False, legacy=False, optimize=-1) +.. function:: compile_dir(dir, maxlevels=10, ddir=None, force=False, rx=None, quiet=0, legacy=False, optimize=-1, workers=1) Recursively descend the directory tree named by *dir*, compiling all :file:`.py` files along the way. @@ -101,8 +121,9 @@ Public functions file considered for compilation, and if it returns a true value, the file is skipped. - If *quiet* is true, nothing is printed to the standard output unless errors - occur. + If *quiet* is ``False`` or ``0`` (the default), the filenames and other + information are printed to standard out. Set to ``1``, only errors are + printed. Set to ``2``, all output is suppressed. If *legacy* is true, byte-code files are written to their legacy locations and names, which may overwrite byte-code files created by another version of @@ -113,11 +134,26 @@ Public functions *optimize* specifies the optimization level for the compiler. It is passed to the built-in :func:`compile` function. + The argument *workers* specifies how many workers are used to + compile files in parallel. The default is to not use multiple workers. + If the platform can't use multiple workers and *workers* argument is given, + then sequential compilation will be used as a fallback. If *workers* is + lower than ``0``, a :exc:`ValueError` will be raised. + .. versionchanged:: 3.2 Added the *legacy* and *optimize* parameter. + .. versionchanged:: 3.5 + Added the *workers* parameter. + + .. versionchanged:: 3.5 + *quiet* parameter was changed to a multilevel value. + + .. versionchanged:: 3.5 + The *legacy* parameter only writes out ``.pyc`` files, not ``.pyo`` files + no matter what the value of *optimize* is. -.. function:: compile_file(fullname, ddir=None, force=False, rx=None, quiet=False, legacy=False, optimize=-1) +.. function:: compile_file(fullname, ddir=None, force=False, rx=None, quiet=0, legacy=False, optimize=-1) Compile the file with path *fullname*. @@ -131,8 +167,9 @@ Public functions file being compiled, and if it returns a true value, the file is not compiled and ``True`` is returned. - If *quiet* is true, nothing is printed to the standard output unless errors - occur. + If *quiet* is ``False`` or ``0`` (the default), the filenames and other + information are printed to standard out. Set to ``1``, only errors are + printed. Set to ``2``, all output is suppressed. If *legacy* is true, byte-code files are written to their legacy locations and names, which may overwrite byte-code files created by another version of @@ -145,8 +182,14 @@ Public functions .. versionadded:: 3.2 + .. versionchanged:: 3.5 + *quiet* parameter was changed to a multilevel value. -.. function:: compile_path(skip_curdir=True, maxlevels=0, force=False, legacy=False, optimize=-1) + .. versionchanged:: 3.5 + The *legacy* parameter only writes out ``.pyc`` files, not ``.pyo`` files + no matter what the value of *optimize* is. + +.. function:: compile_path(skip_curdir=True, maxlevels=0, force=False, quiet=0, legacy=False, optimize=-1) Byte-compile all the :file:`.py` files found along ``sys.path``. If *skip_curdir* is true (the default), the current directory is not included @@ -157,6 +200,12 @@ Public functions .. versionchanged:: 3.2 Added the *legacy* and *optimize* parameter. + .. versionchanged:: 3.5 + *quiet* parameter was changed to a multilevel value. + + .. versionchanged:: 3.5 + The *legacy* parameter only writes out ``.pyc`` files, not ``.pyo`` files + no matter what the value of *optimize* is. To force a recompile of all the :file:`.py` files in the :file:`Lib/` subdirectory and all its subdirectories:: diff --git a/Doc/library/concurrent.futures.rst b/Doc/library/concurrent.futures.rst index cc8b335b8f..9ac617117f 100644 --- a/Doc/library/concurrent.futures.rst +++ b/Doc/library/concurrent.futures.rst @@ -38,7 +38,7 @@ Executor Objects future = executor.submit(pow, 323, 1235) print(future.result()) - .. method:: map(func, *iterables, timeout=None) + .. method:: map(func, *iterables, timeout=None, chunksize=1) Equivalent to :func:`map(func, *iterables) <map>` except *func* is executed asynchronously and several calls to *func* may be made concurrently. The @@ -48,7 +48,16 @@ Executor Objects *timeout* can be an int or a float. If *timeout* is not specified or ``None``, there is no limit to the wait time. If a call raises an exception, then that exception will be raised when its value is - retrieved from the iterator. + retrieved from the iterator. When using :class:`ProcessPoolExecutor`, this + method chops *iterables* into a number of chunks which it submits to the + pool as separate tasks. The (approximate) size of these chunks can be + specified by setting *chunksize* to a positive integer. For very long + iterables, using a large value for *chunksize* can significantly improve + performance compared to the default size of 1. With :class:`ThreadPoolExecutor`, + *chunksize* has no effect. + + .. versionchanged:: 3.5 + Added the *chunksize* argument. .. method:: shutdown(wait=True) @@ -115,11 +124,19 @@ And:: executor.submit(wait_on_future) -.. class:: ThreadPoolExecutor(max_workers) +.. class:: ThreadPoolExecutor(max_workers=None) An :class:`Executor` subclass that uses a pool of at most *max_workers* threads to execute calls asynchronously. + .. versionchanged:: 3.5 + If *max_workers* is ``None`` or + not given, it will default to the number of processors on the machine, + multiplied by ``5``, assuming that :class:`ThreadPoolExecutor` is often + used to overlap I/O instead of CPU work and the number of workers + should be higher than the number of workers + for :class:`ProcessPoolExecutor`. + .. _threadpoolexecutor-example: @@ -175,6 +192,8 @@ to a :class:`ProcessPoolExecutor` will result in deadlock. An :class:`Executor` subclass that executes calls asynchronously using a pool of at most *max_workers* processes. If *max_workers* is ``None`` or not given, it will default to the number of processors on the machine. + If *max_workers* is lower or equal to ``0``, then a :exc:`ValueError` + will be raised. .. versionchanged:: 3.3 When one of the worker processes terminates abruptly, a diff --git a/Doc/library/configparser.rst b/Doc/library/configparser.rst index 92551bc3bd..c9187a3441 100644 --- a/Doc/library/configparser.rst +++ b/Doc/library/configparser.rst @@ -11,6 +11,8 @@ .. sectionauthor:: Christopher G. Petrilli <petrilli@amber.org> .. sectionauthor:: Łukasz Langa <lukasz@langa.pl> +**Source code:** :source:`Lib/configparser.py` + .. index:: pair: .ini; file pair: configuration; file @@ -142,12 +144,13 @@ datatypes, you should convert on your own: >>> float(topsecret['CompressionLevel']) 9.0 -Extracting Boolean values is not that simple, though. Passing the value -to ``bool()`` would do no good since ``bool('False')`` is still -``True``. This is why config parsers also provide :meth:`getboolean`. -This method is case-insensitive and recognizes Boolean values from -``'yes'``/``'no'``, ``'on'``/``'off'`` and ``'1'``/``'0'`` [1]_. -For example: +Since this task is so common, config parsers provide a range of handy getter +methods to handle integers, floats and booleans. The last one is the most +interesting because simply passing the value to ``bool()`` would do no good +since ``bool('False')`` is still ``True``. This is why config parsers also +provide :meth:`getboolean`. This method is case-insensitive and recognizes +Boolean values from ``'yes'``/``'no'``, ``'on'``/``'off'``, +``'true'``/``'false'`` and ``'1'``/``'0'`` [1]_. For example: .. doctest:: @@ -159,10 +162,8 @@ For example: True Apart from :meth:`getboolean`, config parsers also provide equivalent -:meth:`getint` and :meth:`getfloat` methods, but these are far less -useful since conversion using :func:`int` and :func:`float` is -sufficient for these types. - +:meth:`getint` and :meth:`getfloat` methods. You can register your own +converters and customize the provided ones. [1]_ Fallback Values --------------- @@ -317,11 +318,11 @@ from ``get()`` calls. .. class:: ExtendedInterpolation() An alternative handler for interpolation which implements a more advanced - syntax, used for instance in ``zc.buildout``. Extended interpolation is + syntax, used for instance in ``zc.buildout``. Extended interpolation is using ``${section:option}`` to denote a value from a foreign section. - Interpolation can span multiple levels. For convenience, if the ``section:`` - part is omitted, interpolation defaults to the current section (and possibly - the default values from the special section). + Interpolation can span multiple levels. For convenience, if the + ``section:`` part is omitted, interpolation defaults to the current section + (and possibly the default values from the special section). For example, the configuration specified above with basic interpolation, would look like this with extended interpolation: @@ -399,13 +400,13 @@ However, there are a few differences that should be taken into account: * ``parser.popitem()`` never returns it. * ``parser.get(section, option, **kwargs)`` - the second argument is **not** - a fallback value. Note however that the section-level ``get()`` methods are + a fallback value. Note however that the section-level ``get()`` methods are compatible both with the mapping protocol and the classic configparser API. * ``parser.items()`` is compatible with the mapping protocol (returns a list of *section_name*, *section_proxy* pairs including the DEFAULTSECT). However, this method can also be invoked with arguments: ``parser.items(section, raw, - vars)``. The latter call returns a list of *option*, *value* pairs for + vars)``. The latter call returns a list of *option*, *value* pairs for a specified ``section``, with all interpolations expanded (unless ``raw=True`` is provided). @@ -539,9 +540,9 @@ the :meth:`__init__` options: * *delimiters*, default value: ``('=', ':')`` - Delimiters are substrings that delimit keys from values within a section. The - first occurrence of a delimiting substring on a line is considered a delimiter. - This means values (but not keys) can contain the delimiters. + Delimiters are substrings that delimit keys from values within a section. + The first occurrence of a delimiting substring on a line is considered + a delimiter. This means values (but not keys) can contain the delimiters. See also the *space_around_delimiters* argument to :meth:`ConfigParser.write`. @@ -553,7 +554,7 @@ the :meth:`__init__` options: Comment prefixes are strings that indicate the start of a valid comment within a config file. *comment_prefixes* are used only on otherwise empty lines (optionally indented) whereas *inline_comment_prefixes* can be used after - every valid value (e.g. section names, options and empty lines as well). By + every valid value (e.g. section names, options and empty lines as well). By default inline comments are disabled and ``'#'`` and ``';'`` are used as prefixes for whole line comments. @@ -563,10 +564,10 @@ the :meth:`__init__` options: Please note that config parsers don't support escaping of comment prefixes so using *inline_comment_prefixes* may prevent users from specifying option - values with characters used as comment prefixes. When in doubt, avoid setting - *inline_comment_prefixes*. In any circumstances, the only way of storing - comment prefix characters at the beginning of a line in multiline values is to - interpolate the prefix, for example:: + values with characters used as comment prefixes. When in doubt, avoid + setting *inline_comment_prefixes*. In any circumstances, the only way of + storing comment prefix characters at the beginning of a line in multiline + values is to interpolate the prefix, for example:: >>> from configparser import ConfigParser, ExtendedInterpolation >>> parser = ConfigParser(interpolation=ExtendedInterpolation()) @@ -611,7 +612,7 @@ the :meth:`__init__` options: When set to ``True``, the parser will not allow for any section or option duplicates while reading from a single source (using :meth:`read_file`, - :meth:`read_string` or :meth:`read_dict`). It is recommended to use strict + :meth:`read_string` or :meth:`read_dict`). It is recommended to use strict parsers in new applications. .. versionchanged:: 3.2 @@ -646,12 +647,12 @@ the :meth:`__init__` options: The convention of allowing a special section of default values for other sections or interpolation purposes is a powerful concept of this library, - letting users create complex declarative configurations. This section is + letting users create complex declarative configurations. This section is normally called ``"DEFAULT"`` but this can be customized to point to any - other valid section name. Some typical values include: ``"general"`` or - ``"common"``. The name provided is used for recognizing default sections when - reading from any source and is used when writing configuration back to - a file. Its current value can be retrieved using the + other valid section name. Some typical values include: ``"general"`` or + ``"common"``. The name provided is used for recognizing default sections + when reading from any source and is used when writing configuration back to + a file. Its current value can be retrieved using the ``parser_instance.default_section`` attribute and may be modified at runtime (i.e. to convert files from one format to another). @@ -660,14 +661,30 @@ the :meth:`__init__` options: Interpolation behaviour may be customized by providing a custom handler through the *interpolation* argument. ``None`` can be used to turn off interpolation completely, ``ExtendedInterpolation()`` provides a more - advanced variant inspired by ``zc.buildout``. More on the subject in the + advanced variant inspired by ``zc.buildout``. More on the subject in the `dedicated documentation section <#interpolation-of-values>`_. :class:`RawConfigParser` has a default value of ``None``. +* *converters*, default value: not set + + Config parsers provide option value getters that perform type conversion. By + default :meth:`getint`, :meth:`getfloat`, and :meth:`getboolean` are + implemented. Should other getters be desirable, users may define them in + a subclass or pass a dictionary where each key is a name of the converter and + each value is a callable implementing said conversion. For instance, passing + ``{'decimal': decimal.Decimal}`` would add :meth:`getdecimal` on both the + parser object and all section proxies. In other words, it will be possible + to write both ``parser_instance.getdecimal('section', 'key', fallback=0)`` + and ``parser_instance['section'].getdecimal('key', 0)``. + + If the converter needs to access the state of the parser, it can be + implemented as a method on a config parser subclass. If the name of this + method starts with ``get``, it will be available on all section proxies, in + the dict-compatible form (see the ``getdecimal()`` example above). More advanced customization may be achieved by overriding default values of -these parser attributes. The defaults are defined on the classes, so they -may be overridden by subclasses or by attribute assignment. +these parser attributes. The defaults are defined on the classes, so they may +be overridden by subclasses or by attribute assignment. .. attribute:: BOOLEAN_STATES @@ -725,10 +742,11 @@ may be overridden by subclasses or by attribute assignment. .. attribute:: SECTCRE - A compiled regular expression used to parse section headers. The default - matches ``[section]`` to the name ``"section"``. Whitespace is considered part - of the section name, thus ``[ larch ]`` will be read as a section of name - ``" larch "``. Override this attribute if that's unsuitable. For example: + A compiled regular expression used to parse section headers. The default + matches ``[section]`` to the name ``"section"``. Whitespace is considered + part of the section name, thus ``[ larch ]`` will be read as a section of + name ``" larch "``. Override this attribute if that's unsuitable. For + example: .. doctest:: @@ -859,7 +877,7 @@ interpolation if an option used is not defined elsewhere. :: ConfigParser Objects -------------------- -.. class:: ConfigParser(defaults=None, dict_type=collections.OrderedDict, allow_no_value=False, delimiters=('=', ':'), comment_prefixes=('#', ';'), inline_comment_prefixes=None, strict=True, empty_lines_in_values=True, default_section=configparser.DEFAULTSECT, interpolation=BasicInterpolation()) +.. class:: ConfigParser(defaults=None, dict_type=collections.OrderedDict, allow_no_value=False, delimiters=('=', ':'), comment_prefixes=('#', ';'), inline_comment_prefixes=None, strict=True, empty_lines_in_values=True, default_section=configparser.DEFAULTSECT, interpolation=BasicInterpolation(), converters={}) The main configuration parser. When *defaults* is given, it is initialized into the dictionary of intrinsic defaults. When *dict_type* is given, it @@ -869,8 +887,8 @@ ConfigParser Objects When *delimiters* is given, it is used as the set of substrings that divide keys from values. When *comment_prefixes* is given, it will be used as the set of substrings that prefix comments in otherwise empty lines. - Comments can be indented. When *inline_comment_prefixes* is given, it will be - used as the set of substrings that prefix comments in non-empty lines. + Comments can be indented. When *inline_comment_prefixes* is given, it will + be used as the set of substrings that prefix comments in non-empty lines. When *strict* is ``True`` (the default), the parser won't allow for any section or option duplicates while reading from a single source (file, @@ -884,13 +902,13 @@ ConfigParser Objects When *default_section* is given, it specifies the name for the special section holding default values for other sections and interpolation purposes - (normally named ``"DEFAULT"``). This value can be retrieved and changed on + (normally named ``"DEFAULT"``). This value can be retrieved and changed on runtime using the ``default_section`` instance attribute. Interpolation behaviour may be customized by providing a custom handler through the *interpolation* argument. ``None`` can be used to turn off interpolation completely, ``ExtendedInterpolation()`` provides a more - advanced variant inspired by ``zc.buildout``. More on the subject in the + advanced variant inspired by ``zc.buildout``. More on the subject in the `dedicated documentation section <#interpolation-of-values>`_. All option names used in interpolation will be passed through the @@ -899,6 +917,12 @@ ConfigParser Objects converts option names to lower case), the values ``foo %(bar)s`` and ``foo %(BAR)s`` are equivalent. + When *converters* is given, it should be a dictionary where each key + represents the name of a type converter and each value is a callable + implementing the conversion from string to the desired datatype. Every + converter gets its own corresponding :meth:`get*()` method on the parser + object and section proxies. + .. versionchanged:: 3.1 The default *dict_type* is :class:`collections.OrderedDict`. @@ -907,6 +931,9 @@ ConfigParser Objects *empty_lines_in_values*, *default_section* and *interpolation* were added. + .. versionchanged:: 3.5 + The *converters* argument was added. + .. method:: defaults() @@ -944,7 +971,7 @@ ConfigParser Objects .. method:: has_option(section, option) If the given *section* exists, and contains the given *option*, return - :const:`True`; otherwise return :const:`False`. If the specified + :const:`True`; otherwise return :const:`False`. If the specified *section* is :const:`None` or an empty string, DEFAULT is assumed. @@ -1069,7 +1096,7 @@ ConfigParser Objects :meth:`get` method. .. versionchanged:: 3.2 - Items present in *vars* no longer appear in the result. The previous + Items present in *vars* no longer appear in the result. The previous behaviour mixed actual parser options with variables provided for interpolation. @@ -1170,7 +1197,7 @@ RawConfigParser Objects .. note:: Consider using :class:`ConfigParser` instead which checks types of - the values to be stored internally. If you don't want interpolation, you + the values to be stored internally. If you don't want interpolation, you can use ``ConfigParser(interpolation=None)``. @@ -1181,7 +1208,7 @@ RawConfigParser Objects *default section* name is passed, :exc:`ValueError` is raised. Type of *section* is not checked which lets users create non-string named - sections. This behaviour is unsupported and may cause internal errors. + sections. This behaviour is unsupported and may cause internal errors. .. method:: set(section, option, value) @@ -1282,3 +1309,4 @@ Exceptions .. [1] Config parsers allow for heavy customization. If you are interested in changing the behaviour outlined by the footnote reference, consult the `Customizing Parser Behaviour`_ section. + diff --git a/Doc/library/constants.rst b/Doc/library/constants.rst index 42b5af23a5..d5a0f09173 100644 --- a/Doc/library/constants.rst +++ b/Doc/library/constants.rst @@ -45,7 +45,6 @@ A small number of constants live in the built-in namespace. They are: for more details. - .. data:: Ellipsis The same as ``...``. Special value used mostly in conjunction with extended diff --git a/Doc/library/contextlib.rst b/Doc/library/contextlib.rst index 6f36864ac2..550b3476bd 100644 --- a/Doc/library/contextlib.rst +++ b/Doc/library/contextlib.rst @@ -172,6 +172,16 @@ Functions and classes provided: .. versionadded:: 3.4 +.. function:: redirect_stderr(new_target) + + Similar to :func:`~contextlib.redirect_stdout` but redirecting + :data:`sys.stderr` to another file or file-like object. + + This context manager is :ref:`reentrant <reentrant-cms>`. + + .. versionadded:: 3.5 + + .. class:: ContextDecorator() A base class that enables a context manager to also be used as a decorator. diff --git a/Doc/library/csv.rst b/Doc/library/csv.rst index e9a9cb5ddb..4fcfaef258 100644 --- a/Doc/library/csv.rst +++ b/Doc/library/csv.rst @@ -5,6 +5,7 @@ :synopsis: Write and read tabular data to and from delimited files. .. sectionauthor:: Skip Montanaro <skip@pobox.com> +**Source code:** :source:`Lib/csv.py` .. index:: single: csv @@ -418,7 +419,7 @@ Writer Objects :class:`Writer` objects (:class:`DictWriter` instances and objects returned by the :func:`writer` function) have the following public methods. A *row* must be -a sequence of strings or numbers for :class:`Writer` objects and a dictionary +an iterable of strings or numbers for :class:`Writer` objects and a dictionary mapping fieldnames to strings or numbers (by passing them through :func:`str` first) for :class:`DictWriter` objects. Note that complex numbers are written out surrounded by parens. This may cause some problems for other programs which @@ -430,6 +431,8 @@ read CSV files (assuming they support complex numbers at all). Write the *row* parameter to the writer's file object, formatted according to the current dialect. + .. versionchanged:: 3.5 + Added support of arbitrary iterables. .. method:: csvwriter.writerows(rows) diff --git a/Doc/library/curses.rst b/Doc/library/curses.rst index f3e60b4be9..e8dfd833e4 100644 --- a/Doc/library/curses.rst +++ b/Doc/library/curses.rst @@ -599,6 +599,13 @@ The module :mod:`curses` defines the following functions: Only one *ch* can be pushed before :meth:`getch` is called. +.. function:: update_lines_cols() + + Update :envvar:`LINES` and :envvar:`COLS`. Useful for detecting manual screen resize. + + .. versionadded:: 3.5 + + .. function:: unget_wch(ch) Push *ch* so the next :meth:`get_wch` will return it. diff --git a/Doc/library/datetime.rst b/Doc/library/datetime.rst index 88f3f6eea7..976cd49000 100644 --- a/Doc/library/datetime.rst +++ b/Doc/library/datetime.rst @@ -7,6 +7,8 @@ .. sectionauthor:: Tim Peters <tim@zope.com> .. sectionauthor:: A.M. Kuchling <amk@amk.ca> +**Source code:** :source:`Lib/datetime.py` + .. XXX what order should the types be discussed in? The :mod:`datetime` module supplies classes for manipulating dates and times in @@ -757,13 +759,19 @@ Other constructors, all class methods: :attr:`tzinfo` ``None``. This may raise :exc:`OverflowError`, if the timestamp is out of the range of values supported by the platform C :c:func:`gmtime` function, and :exc:`OSError` on :c:func:`gmtime` failure. - It's common for this to be restricted to years in 1970 through 2038. See also - :meth:`fromtimestamp`. + It's common for this to be restricted to years in 1970 through 2038. + + To get an aware :class:`.datetime` object, call :meth:`fromtimestamp`:: + + datetime.fromtimestamp(timestamp, timezone.utc) + + On the POSIX compliant platforms, it is equivalent to the following + expression:: - On the POSIX compliant platforms, ``utcfromtimestamp(timestamp)`` - is equivalent to the following expression:: + datetime(1970, 1, 1, tzinfo=timezone.utc) + timedelta(seconds=timestamp) - datetime(1970, 1, 1) + timedelta(seconds=timestamp) + except the latter formula always supports the full years range: between + :const:`MINYEAR` and :const:`MAXYEAR` inclusive. .. versionchanged:: 3.3 Raise :exc:`OverflowError` instead of :exc:`ValueError` if the timestamp @@ -1376,10 +1384,13 @@ Supported operations: * efficient pickling -* in Boolean contexts, a :class:`.time` object is considered to be true if and - only if, after converting it to minutes and subtracting :meth:`utcoffset` (or - ``0`` if that's ``None``), the result is non-zero. +In boolean contexts, a :class:`.time` object is always considered to be true. +.. versionchanged:: 3.5 + Before Python 3.5, a :class:`.time` object was considered to be false if it + represented midnight in UTC. This behavior was considered obscure and + error-prone and has been removed in Python 3.5. See :issue:`13936` for full + details. Instance methods: diff --git a/Doc/library/dbm.rst b/Doc/library/dbm.rst index e6a82d67ce..3f3c43d438 100644 --- a/Doc/library/dbm.rst +++ b/Doc/library/dbm.rst @@ -325,13 +325,18 @@ The module defines the following: dumbdbm database is created, files with :file:`.dat` and :file:`.dir` extensions are created. - The optional *flag* argument is currently ignored; the database is always opened - for update, and will be created if it does not exist. + The optional *flag* argument supports only the semantics of ``'c'`` + and ``'n'`` values. Other values will default to database being always + opened for update, and will be created if it does not exist. The optional *mode* argument is the Unix mode of the file, used only when the database has to be created. It defaults to octal ``0o666`` (and will be modified by the prevailing umask). + .. versionchanged:: 3.5 + :func:`.open` always creates a new database when the flag has the value + ``'n'``. + In addition to the methods provided by the :class:`collections.abc.MutableMapping` class, :class:`dumbdbm` objects provide the following methods: diff --git a/Doc/library/decimal.rst b/Doc/library/decimal.rst index 759be70c9f..5d7ffffbe9 100644 --- a/Doc/library/decimal.rst +++ b/Doc/library/decimal.rst @@ -12,6 +12,8 @@ .. moduleauthor:: Stefan Krah <skrah at bytereef.org> .. sectionauthor:: Raymond D. Hettinger <python at rcn.com> +**Source code:** :source:`Lib/decimal.py` + .. import modules for testing inline doctests with the Sphinx doctest builder .. testsetup:: * @@ -742,7 +744,7 @@ Decimal objects * ``"NaN"``, indicating that the operand is a quiet NaN (Not a Number). * ``"sNaN"``, indicating that the operand is a signaling NaN. - .. method:: quantize(exp, rounding=None, context=None, watchexp=True) + .. method:: quantize(exp, rounding=None, context=None) Return a value equal to the first operand after rounding and having the exponent of the second operand. @@ -765,14 +767,8 @@ Decimal objects ``context`` argument; if neither argument is given the rounding mode of the current thread's context is used. - If *watchexp* is set (default), then an error is returned whenever the - resulting exponent is greater than :attr:`Emax` or less than - :attr:`Etiny`. - - .. deprecated:: 3.3 - *watchexp* is an implementation detail from the pure Python version - and is not present in the C version. It will be removed in version - 3.4, where it defaults to ``True``. + An error is returned whenever the resulting exponent is greater than + :attr:`Emax` or less than :attr:`Etiny`. .. method:: radix() @@ -2092,4 +2088,3 @@ Alternatively, inputs can be rounded upon creation using the >>> Context(prec=5, rounding=ROUND_DOWN).create_decimal('1.2345678') Decimal('1.2345') - diff --git a/Doc/library/development.rst b/Doc/library/development.rst index 06e7048a04..d2b5fa2aa4 100644 --- a/Doc/library/development.rst +++ b/Doc/library/development.rst @@ -16,6 +16,7 @@ The list of modules described in this chapter is: .. toctree:: + typing.rst pydoc.rst doctest.rst unittest.rst diff --git a/Doc/library/difflib.rst b/Doc/library/difflib.rst index 5f72ea6233..38245e776a 100644 --- a/Doc/library/difflib.rst +++ b/Doc/library/difflib.rst @@ -7,6 +7,8 @@ .. sectionauthor:: Tim Peters <tim_one@users.sourceforge.net> .. Markup by Fred L. Drake, Jr. <fdrake@acm.org> +**Source code:** :source:`Lib/difflib.py` + .. testsetup:: import sys @@ -25,7 +27,9 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module. little fancier than, an algorithm published in the late 1980's by Ratcliff and Obershelp under the hyperbolic name "gestalt pattern matching." The idea is to find the longest contiguous matching subsequence that contains no "junk" - elements (the Ratcliff and Obershelp algorithm doesn't address junk). The same + elements; these "junk" elements are ones that are uninteresting in some + sense, such as blank lines or whitespace. (Handling junk is an + extension to the Ratcliff and Obershelp algorithm.) The same idea is then applied recursively to the pieces of the sequences to the left and to the right of the matching subsequence. This does not yield minimal edit sequences, but does tend to yield matches that "look right" to people. @@ -100,7 +104,8 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module. The following methods are public: - .. method:: make_file(fromlines, tolines, fromdesc='', todesc='', context=False, numlines=5) + .. method:: make_file(fromlines, tolines, fromdesc='', todesc='', context=False, \ + numlines=5, *, charset='utf-8') Compares *fromlines* and *tolines* (lists of strings) and returns a string which is a complete HTML file containing a table showing line by line differences with @@ -119,6 +124,10 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module. the next difference highlight at the top of the browser without any leading context). + .. versionchanged:: 3.5 + *charset* keyword-only argument was added. The default charset of + HTML document changed from ``'ISO-8859-1'`` to ``'utf-8'``. + .. method:: make_table(fromlines, tolines, fromdesc='', todesc='', context=False, numlines=5) Compares *fromlines* and *tolines* (lists of strings) and returns a string which @@ -208,7 +217,7 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module. Compare *a* and *b* (lists of strings); return a :class:`Differ`\ -style delta (a :term:`generator` generating the delta lines). - Optional keyword parameters *linejunk* and *charjunk* are for filter functions + Optional keyword parameters *linejunk* and *charjunk* are filtering functions (or ``None``): *linejunk*: A function that accepts a single string argument, and returns @@ -222,7 +231,7 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module. *charjunk*: A function that accepts a character (a string of length 1), and returns if the character is junk, or false if not. The default is module-level function :func:`IS_CHARACTER_JUNK`, which filters out whitespace characters (a - blank or tab; note: bad idea to include newline in this!). + blank or tab; it's a bad idea to include newline in this!). :file:`Tools/scripts/ndiff.py` is a command-line front-end to this function. @@ -306,6 +315,21 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module. See :ref:`difflib-interface` for a more detailed example. +.. function:: diff_bytes(dfunc, a, b, fromfile=b'', tofile=b'', fromfiledate=b'', tofiledate=b'', n=3, lineterm=b'\\n') + + Compare *a* and *b* (lists of bytes objects) using *dfunc*; yield a + sequence of delta lines (also bytes) in the format returned by *dfunc*. + *dfunc* must be a callable, typically either :func:`unified_diff` or + :func:`context_diff`. + + Allows you to compare data with unknown or inconsistent encoding. All + inputs except *n* must be bytes objects, not str. Works by losslessly + converting all inputs (except *n*) to str, and calling ``dfunc(a, b, + fromfile, tofile, fromfiledate, tofiledate, n, lineterm)``. The output of + *dfunc* is then converted back to bytes, so the delta lines that you + receive have the same unknown/inconsistent encodings as *a* and *b*. + + .. versionadded:: 3.5 .. function:: IS_LINE_JUNK(line) @@ -622,6 +646,12 @@ The :class:`Differ` class has this constructor: length 1), and returns true if the character is junk. The default is ``None``, meaning that no character is considered junk. + These junk-filtering functions speed up matching to find + differences and do not cause any differing lines or characters to + be ignored. Read the description of the + :meth:`~SequenceMatcher.find_longest_match` method's *isjunk* + parameter for an explanation. + :class:`Differ` objects are used (deltas generated) via a single method: @@ -713,65 +743,4 @@ This example shows how to use difflib to create a ``diff``-like utility. It is also contained in the Python source distribution, as :file:`Tools/scripts/diff.py`. -.. testcode:: - - """ Command line interface to difflib.py providing diffs in four formats: - - * ndiff: lists every line and highlights interline changes. - * context: highlights clusters of changes in a before/after format. - * unified: highlights clusters of changes in an inline format. - * html: generates side by side comparison with change highlights. - - """ - - import sys, os, time, difflib, optparse - - def main(): - # Configure the option parser - usage = "usage: %prog [options] fromfile tofile" - parser = optparse.OptionParser(usage) - parser.add_option("-c", action="store_true", default=False, - help='Produce a context format diff (default)') - parser.add_option("-u", action="store_true", default=False, - help='Produce a unified format diff') - hlp = 'Produce HTML side by side diff (can use -c and -l in conjunction)' - parser.add_option("-m", action="store_true", default=False, help=hlp) - parser.add_option("-n", action="store_true", default=False, - help='Produce a ndiff format diff') - parser.add_option("-l", "--lines", type="int", default=3, - help='Set number of context lines (default 3)') - (options, args) = parser.parse_args() - - if len(args) == 0: - parser.print_help() - sys.exit(1) - if len(args) != 2: - parser.error("need to specify both a fromfile and tofile") - - n = options.lines - fromfile, tofile = args # as specified in the usage string - - # we're passing these as arguments to the diff function - fromdate = time.ctime(os.stat(fromfile).st_mtime) - todate = time.ctime(os.stat(tofile).st_mtime) - with open(fromfile) as fromf, open(tofile) as tof: - fromlines, tolines = list(fromf), list(tof) - - if options.u: - diff = difflib.unified_diff(fromlines, tolines, fromfile, tofile, - fromdate, todate, n=n) - elif options.n: - diff = difflib.ndiff(fromlines, tolines) - elif options.m: - diff = difflib.HtmlDiff().make_file(fromlines, tolines, fromfile, - tofile, context=options.c, - numlines=n) - else: - diff = difflib.context_diff(fromlines, tolines, fromfile, tofile, - fromdate, todate, n=n) - - # we're using writelines because diff is a generator - sys.stdout.writelines(diff) - - if __name__ == '__main__': - main() +.. literalinclude:: ../../Tools/scripts/diff.py diff --git a/Doc/library/dis.rst b/Doc/library/dis.rst index 273fb20634..1bcb3a4a07 100644 --- a/Doc/library/dis.rst +++ b/Doc/library/dis.rst @@ -48,8 +48,9 @@ code. .. class:: Bytecode(x, *, first_line=None, current_offset=None) - Analyse the bytecode corresponding to a function, method, string of source - code, or a code object (as returned by :func:`compile`). + + Analyse the bytecode corresponding to a function, generator, method, string + of source code, or a code object (as returned by :func:`compile`). This is a convenience wrapper around many of the functions listed below, most notably :func:`get_instructions`, as iterating over a :class:`Bytecode` @@ -109,7 +110,7 @@ operation is being performed, so the intermediate analysis object isn't useful: .. function:: code_info(x) Return a formatted multi-line string with detailed code object information - for the supplied function, method, source code string or code object. + for the supplied function, generator, method, source code string or code object. Note that the exact contents of code info strings are highly implementation dependent and they may change arbitrarily across Python VMs or Python @@ -136,11 +137,11 @@ operation is being performed, so the intermediate analysis object isn't useful: .. function:: dis(x=None, *, file=None) Disassemble the *x* object. *x* can denote either a module, a class, a - method, a function, a code object, a string of source code or a byte sequence - of raw bytecode. For a module, it disassembles all functions. For a class, - it disassembles all methods. For a code object or sequence of raw bytecode, - it prints one line per bytecode instruction. Strings are first compiled to - code objects with the :func:`compile` built-in function before being + method, a function, a generator, a code object, a string of source code or + a byte sequence of raw bytecode. For a module, it disassembles all functions. + For a class, it disassembles all methods. For a code object or sequence of + raw bytecode, it prints one line per bytecode instruction. Strings are first + compiled to code objects with the :func:`compile` built-in function before being disassembled. If no object is provided, this function disassembles the last traceback. @@ -345,6 +346,14 @@ result back on the stack. Implements ``TOS = iter(TOS)``. +.. opcode:: GET_YIELD_FROM_ITER + + If ``TOS`` is a :term:`generator iterator` or :term:`coroutine` object + it is left as is. Otherwise, implements ``TOS = iter(TOS)``. + + .. versionadded:: 3.5 + + **Binary operations** Binary operations remove the top of the stack (TOS) and the second top-most @@ -361,6 +370,13 @@ result back on the stack. Implements ``TOS = TOS1 * TOS``. +.. opcode:: BINARY_MATRIX_MULTIPLY + + Implements ``TOS = TOS1 @ TOS``. + + .. versionadded:: 3.5 + + .. opcode:: BINARY_FLOOR_DIVIDE Implements ``TOS = TOS1 // TOS``. @@ -433,6 +449,13 @@ the original TOS1. Implements in-place ``TOS = TOS1 * TOS``. +.. opcode:: INPLACE_MATRIX_MULTIPLY + + Implements in-place ``TOS = TOS1 @ TOS``. + + .. versionadded:: 3.5 + + .. opcode:: INPLACE_FLOOR_DIVIDE Implements in-place ``TOS = TOS1 // TOS``. @@ -493,6 +516,40 @@ the original TOS1. Implements ``del TOS1[TOS]``. +**Coroutine opcodes** + +.. opcode:: GET_AWAITABLE + + Implements ``TOS = get_awaitable(TOS)``, where ``get_awaitable(o)`` + returns ``o`` if ``o`` is a coroutine object or a generator object with + the CO_ITERABLE_COROUTINE flag, or resolves + ``o.__await__``. + + +.. opcode:: GET_AITER + + Implements ``TOS = get_awaitable(TOS.__aiter__())``. See ``GET_AWAITABLE`` + for details about ``get_awaitable`` + + +.. opcode:: GET_ANEXT + + Implements ``PUSH(get_awaitable(TOS.__anext__()))``. See ``GET_AWAITABLE`` + for details about ``get_awaitable`` + + +.. opcode:: BEFORE_ASYNC_WITH + + Resolves ``__aenter__`` and ``__aexit__`` from the object on top of the + stack. Pushes ``__aexit__`` and result of ``__aenter__()`` to the stack. + + +.. opcode:: SETUP_ASYNC_WITH + + Creates a new frame object. + + + **Miscellaneous opcodes** .. opcode:: PRINT_EXPR @@ -597,7 +654,7 @@ iterations of the loop. :opcode:`UNPACK_SEQUENCE`). -.. opcode:: WITH_CLEANUP +.. opcode:: WITH_CLEANUP_START Cleans up the stack when a :keyword:`with` statement block exits. TOS is the context manager's :meth:`__exit__` bound method. Below TOS are 1--3 values @@ -609,7 +666,13 @@ iterations of the loop. * (SECOND, THIRD, FOURTH) = exc_info() In the last case, ``TOS(SECOND, THIRD, FOURTH)`` is called, otherwise - ``TOS(None, None, None)``. In addition, TOS is removed from the stack. + ``TOS(None, None, None)``. Pushes SECOND and result of the call + to the stack. + + +.. opcode:: WITH_CLEANUP_FINISH + + Pops exception type and result of 'exit' function call from the stack. If the stack represents an exception, *and* the function call returns a 'true' value, this information is "zapped" and replaced with a single @@ -795,10 +858,6 @@ the more significant byte last. Pushes a try block from a try-except clause onto the block stack. *delta* points to the finally block. -.. opcode:: STORE_MAP - - Store a key and value pair in a dictionary. Pops the key and value while - leaving the dictionary on the stack. .. opcode:: LOAD_FAST (var_num) diff --git a/Doc/library/distribution.rst b/Doc/library/distribution.rst index c4954d1b4a..3e6e84b42a 100644 --- a/Doc/library/distribution.rst +++ b/Doc/library/distribution.rst @@ -12,3 +12,4 @@ with a local index server, or without any index server at all. distutils.rst ensurepip.rst venv.rst + zipapp.rst diff --git a/Doc/library/doctest.rst b/Doc/library/doctest.rst index ea6e1b5481..9f7d12c626 100644 --- a/Doc/library/doctest.rst +++ b/Doc/library/doctest.rst @@ -1053,15 +1053,9 @@ from text files and modules with doctests: This function uses the same search technique as :func:`testmod`. - .. note:: - Unlike :func:`testmod` and :class:`DocTestFinder`, this function raises - a :exc:`ValueError` if *module* contains no docstrings. You can prevent - this error by passing a :class:`DocTestFinder` instance as the - *test_finder* argument with its *exclude_empty* keyword argument set - to ``False``:: - - >>> finder = doctest.DocTestFinder(exclude_empty=False) - >>> suite = doctest.DocTestSuite(test_finder=finder) + .. versionchanged:: 3.5 + :func:`DocTestSuite` returns an empty :class:`unittest.TestSuite` if *module* + contains no docstrings instead of raising :exc:`ValueError`. Under the covers, :func:`DocTestSuite` creates a :class:`unittest.TestSuite` out diff --git a/Doc/library/email.message.rst b/Doc/library/email.message.rst index aeea94221d..b91f26d120 100644 --- a/Doc/library/email.message.rst +++ b/Doc/library/email.message.rst @@ -578,6 +578,15 @@ Here are the methods of the :class:`Message` class: will be *failobj*. + .. method:: get_content_disposition() + + Return the lowercased value (without parameters) of the message's + :mailheader:`Content-Disposition` header if it has one, or ``None``. The + possible values for this method are *inline*, *attachment* or ``None`` + if the message follows :rfc:`2183`. + + .. versionadded:: 3.5 + .. method:: walk() The :meth:`walk` method is an all-purpose generator which can be used to diff --git a/Doc/library/email.mime.rst b/Doc/library/email.mime.rst index 1d70225fc8..67d0a67954 100644 --- a/Doc/library/email.mime.rst +++ b/Doc/library/email.mime.rst @@ -195,7 +195,8 @@ Here are the classes: set of the text and is passed as an argument to the :class:`~email.mime.nonmultipart.MIMENonMultipart` constructor; it defaults to ``us-ascii`` if the string contains only ``ascii`` code points, and - ``utf-8`` otherwise. + ``utf-8`` otherwise. The *_charset* parameter accepts either a string or a + :class:`~email.charset.Charset` instance. Unless the *_charset* argument is explicitly set to ``None``, the MIMEText object created will have both a :mailheader:`Content-Type` header @@ -206,3 +207,6 @@ Here are the classes: ``Content-Transfer-Encoding`` header, after which a ``set_payload`` call will automatically encode the new payload (and add a new :mailheader:`Content-Transfer-Encoding` header). + + .. versionchanged:: 3.5 + *_charset* also accepts :class:`~email.charset.Charset` instances. diff --git a/Doc/library/email.policy.rst b/Doc/library/email.policy.rst index d4e3fc186a..045b119303 100644 --- a/Doc/library/email.policy.rst +++ b/Doc/library/email.policy.rst @@ -187,6 +187,18 @@ added matters. To illustrate:: :const:`False` (the default), defects will be passed to the :meth:`register_defect` method. + + + .. attribute:: mangle_from\_ + + If :const:`True`, lines starting with *"From "* in the body are + escaped by putting a ``>`` in front of them. This parameter is used when + the message is being serialized by a generator. + Default: :const:`False`. + + .. versionadded:: 3.5 + The *mangle_from_* parameter. + The following :class:`Policy` method is intended to be called by code using the email library to create policy instances with custom settings: @@ -319,6 +331,13 @@ added matters. To illustrate:: :const:`compat32`, that is used as the default policy. Thus the default behavior of the email package is to maintain compatibility with Python 3.2. + The following attributes have values that are different from the + :class:`Policy` default: + + .. attribute:: mangle_from_ + + The default is ``True``. + The class provides the following concrete implementations of the abstract methods of :class:`Policy`: @@ -356,6 +375,14 @@ added matters. To illustrate:: line breaks and any (RFC invalid) binary data it may contain. +An instance of :class:`Compat32` is provided as a module constant: + +.. data:: compat32 + + An instance of :class:`Compat32`, providing backward compatibility with the + behavior of the email package in Python 3.2. + + .. note:: The documentation below describes new policies that are included in the @@ -378,6 +405,14 @@ added matters. To illustrate:: In addition to the settable attributes listed above that apply to all policies, this policy adds the following additional attributes: + .. attribute:: utf8 + + If ``False``, follow :rfc:`5322`, supporting non-ASCII characters in + headers by encoding them as "encoded words". If ``True``, follow + :rfc:`6532` and use ``utf-8`` encoding for headers. Messages + formatted in this way may be passed to SMTP servers that support + the ``SMTPUTF8`` extension (:rfc:`6531`). + .. attribute:: refold_source If the value for a header in the ``Message`` object originated from a @@ -499,6 +534,14 @@ more closely to the RFCs relevant to their domains. Like ``default``, but with ``linesep`` set to ``\r\n``, which is RFC compliant. +.. data:: SMTPUTF8 + + The same as ``SMTP`` except that :attr:`~EmailPolicy.utf8` is ``True``. + Useful for serializing messages to a message store without using encoded + words in the headers. Should only be used for SMTP trasmission if the + sender or recipient addresses have non-ASCII characters (the + :meth:`smtplib.SMTP.send_message` method handles this automatically). + .. data:: HTTP Suitable for serializing headers with for use in HTTP traffic. Like diff --git a/Doc/library/enum.rst b/Doc/library/enum.rst index cf095592f6..9b4f9b419f 100644 --- a/Doc/library/enum.rst +++ b/Doc/library/enum.rst @@ -314,11 +314,11 @@ Then:: >>> str(Mood.funky) 'my custom str! 1' -The rules for what is allowed are as follows: _sunder_ names (starting and -ending with a single underscore) are reserved by enum and cannot be used; -all other attributes defined within an enumeration will become members of this -enumeration, with the exception of *__dunder__* names and descriptors (methods -are also descriptors). +The rules for what is allowed are as follows: names that start and end with a +with a single underscore are reserved by enum and cannot be used; all other +attributes defined within an enumeration will become members of this +enumeration, with the exception of special methods (:meth:`__str__`, +:meth:`__add__`, etc.) and descriptors (methods are also descriptors). Note: if your enumeration defines :meth:`__new__` and/or :meth:`__init__` then whatever value(s) were given to the enum member will be passed into those @@ -400,7 +400,8 @@ The second argument is the *source* of enumeration member names. It can be a whitespace-separated string of names, a sequence of names, a sequence of 2-tuples with key/value pairs, or a mapping (e.g. dictionary) of names to values. The last two options enable assigning arbitrary values to -enumerations; the others auto-assign increasing integers starting with 1. A +enumerations; the others auto-assign increasing integers starting with 1 (use +the ``start`` parameter to specify a different starting value). A new class derived from :class:`Enum` is returned. In other words, the above assignment to :class:`Animal` is equivalent to:: @@ -438,12 +439,12 @@ SomeData in the global scope:: The complete signature is:: - Enum(value='NewEnumName', names=<...>, *, module='...', qualname='...', type=<mixed-in class>) + Enum(value='NewEnumName', names=<...>, *, module='...', qualname='...', type=<mixed-in class>, start=1) :value: What the new Enum class will record as its name. :names: The Enum members. This can be a whitespace or comma separated string - (values will start at 1):: + (values will start at 1 unless otherwise specified):: 'red green blue' | 'red,green,blue' | 'red, green, blue' @@ -465,6 +466,11 @@ The complete signature is:: :type: type to mix in to new Enum class. +:start: number to start counting at if only names are passed in. + +.. versionchanged:: 3.5 + The *start* parameter was added. + Derived Enumerations -------------------- diff --git a/Doc/library/errno.rst b/Doc/library/errno.rst index d2163b6258..22a5cbc450 100644 --- a/Doc/library/errno.rst +++ b/Doc/library/errno.rst @@ -41,7 +41,10 @@ defined by the module. The specific list of defined symbols is available as .. data:: EINTR - Interrupted system call + Interrupted system call. + + .. seealso:: + This error is mapped to the exception :exc:`InterruptedError`. .. data:: EIO diff --git a/Doc/library/exceptions.rst b/Doc/library/exceptions.rst index 271a5c8219..0a422b238d 100644 --- a/Doc/library/exceptions.rst +++ b/Doc/library/exceptions.rst @@ -162,7 +162,8 @@ The following exceptions are the exceptions that are usually raised. .. exception:: GeneratorExit - Raised when a :term:`generator`\'s :meth:`close` method is called. It + Raised when a :term:`generator` or :term:`coroutine` is closed; + see :meth:`generator.close` and :meth:`coroutine.close`. It directly inherits from :exc:`BaseException` instead of :exc:`Exception` since it is technically not an error. @@ -281,6 +282,16 @@ The following exceptions are the exceptions that are usually raised. handling in C, most floating point operations are not checked. +.. exception:: RecursionError + + This exception is derived from :exc:`RuntimeError`. It is raised when the + interpreter detects that the maximum recursion depth (see + :func:`sys.getrecursionlimit`) is exceeded. + + .. versionadded:: 3.5 + Previously, a plain :exc:`RuntimeError` was raised. + + .. exception:: ReferenceError This exception is raised when a weak reference proxy, created by the @@ -306,14 +317,30 @@ The following exceptions are the exceptions that are usually raised. given as an argument when constructing the exception, and defaults to :const:`None`. - When a generator function returns, a new :exc:`StopIteration` instance is + When a :term:`generator` or :term:`coroutine` function + returns, a new :exc:`StopIteration` instance is raised, and the value returned by the function is used as the :attr:`value` parameter to the constructor of the exception. + If a generator function defined in the presence of a ``from __future__ + import generator_stop`` directive raises :exc:`StopIteration`, it will be + converted into a :exc:`RuntimeError` (retaining the :exc:`StopIteration` + as the new exception's cause). + .. versionchanged:: 3.3 Added ``value`` attribute and the ability for generator functions to use it to return a value. + .. versionchanged:: 3.5 + Introduced the RuntimeError transformation. + +.. exception:: StopAsyncIteration + + Must be raised by :meth:`__anext__` method of an + :term:`asynchronous iterator` object to stop the iteration. + + .. versionadded:: 3.5 + .. exception:: SyntaxError Raised when the parser encounters a syntax error. This may occur in an @@ -536,7 +563,12 @@ depending on the system error code. .. exception:: InterruptedError Raised when a system call is interrupted by an incoming signal. - Corresponds to :c:data:`errno` ``EINTR``. + Corresponds to :c:data:`errno` :py:data:`~errno.EINTR`. + + .. versionchanged:: 3.5 + Python now retries system calls when a syscall is interrupted by a + signal, except if the signal handler raises an exception (see :pep:`475` + for the rationale), instead of raising :exc:`InterruptedError`. .. exception:: IsADirectoryError diff --git a/Doc/library/faulthandler.rst b/Doc/library/faulthandler.rst index eb2016a7b9..3a5badd0ff 100644 --- a/Doc/library/faulthandler.rst +++ b/Doc/library/faulthandler.rst @@ -47,6 +47,9 @@ Dumping the traceback Dump the tracebacks of all threads into *file*. If *all_threads* is ``False``, dump only the current thread. + .. versionchanged:: 3.5 + Added support for passing file descriptor to this function. + Fault handler state ------------------- @@ -59,6 +62,12 @@ Fault handler state produce tracebacks for every running thread. Otherwise, dump only the current thread. + The *file* must be kept open until the fault handler is disabled: see + :ref:`issue with file descriptors <faulthandler-fd>`. + + .. versionchanged:: 3.5 + Added support for passing file descriptor to this function. + .. function:: disable() Disable the fault handler: uninstall the signal handlers installed by @@ -82,9 +91,16 @@ Dumping the tracebacks after a timeout call replaces previous parameters and resets the timeout. The timer has a sub-second resolution. + The *file* must be kept open until the traceback is dumped or + :func:`cancel_dump_traceback_later` is called: see :ref:`issue with file + descriptors <faulthandler-fd>`. + This function is implemented using a watchdog thread and therefore is not available if Python is compiled with threads disabled. + .. versionchanged:: 3.5 + Added support for passing file descriptor to this function. + .. function:: cancel_dump_traceback_later() Cancel the last call to :func:`dump_traceback_later`. @@ -99,8 +115,14 @@ Dumping the traceback on a user signal the traceback of all threads, or of the current thread if *all_threads* is ``False``, into *file*. Call the previous handler if chain is ``True``. + The *file* must be kept open until the signal is unregistered by + :func:`unregister`: see :ref:`issue with file descriptors <faulthandler-fd>`. + Not available on Windows. + .. versionchanged:: 3.5 + Added support for passing file descriptor to this function. + .. function:: unregister(signum) Unregister a user signal: uninstall the handler of the *signum* signal @@ -110,6 +132,8 @@ Dumping the traceback on a user signal Not available on Windows. +.. _faulthandler-fd: + Issue with file descriptors --------------------------- diff --git a/Doc/library/fcntl.rst b/Doc/library/fcntl.rst index 8e932fb954..432140f279 100644 --- a/Doc/library/fcntl.rst +++ b/Doc/library/fcntl.rst @@ -28,41 +28,41 @@ descriptor. The module defines the following functions: -.. function:: fcntl(fd, op[, arg]) +.. function:: fcntl(fd, cmd, arg=0) - Perform the operation *op* on file descriptor *fd* (file objects providing + Perform the operation *cmd* on file descriptor *fd* (file objects providing a :meth:`~io.IOBase.fileno` method are accepted as well). The values used - for *op* are operating system dependent, and are available as constants + for *cmd* are operating system dependent, and are available as constants in the :mod:`fcntl` module, using the same names as used in the relevant C - header files. The argument *arg* is optional, and defaults to the integer - value ``0``. When present, it can either be an integer value, or a string. - With the argument missing or an integer value, the return value of this function - is the integer return value of the C :c:func:`fcntl` call. When the argument is - a string it represents a binary structure, e.g. created by :func:`struct.pack`. - The binary data is copied to a buffer whose address is passed to the C - :c:func:`fcntl` call. The return value after a successful call is the contents - of the buffer, converted to a string object. The length of the returned string - will be the same as the length of the *arg* argument. This is limited to 1024 - bytes. If the information returned in the buffer by the operating system is - larger than 1024 bytes, this is most likely to result in a segmentation - violation or a more subtle data corruption. + header files. The argument *arg* can either be an integer value, or a + :class:`bytes` object. With an integer value, the return value of this + function is the integer return value of the C :c:func:`fcntl` call. When + the argument is bytes it represents a binary structure, e.g. created by + :func:`struct.pack`. The binary data is copied to a buffer whose address is + passed to the C :c:func:`fcntl` call. The return value after a successful + call is the contents of the buffer, converted to a :class:`bytes` object. + The length of the returned object will be the same as the length of the + *arg* argument. This is limited to 1024 bytes. If the information returned + in the buffer by the operating system is larger than 1024 bytes, this is + most likely to result in a segmentation violation or a more subtle data + corruption. If the :c:func:`fcntl` fails, an :exc:`OSError` is raised. -.. function:: ioctl(fd, op[, arg[, mutate_flag]]) +.. function:: ioctl(fd, request, arg=0, mutate_flag=True) This function is identical to the :func:`~fcntl.fcntl` function, except that the argument handling is even more complicated. - The op parameter is limited to values that can fit in 32-bits. - Additional constants of interest for use as the *op* argument can be + The *request* parameter is limited to values that can fit in 32-bits. + Additional constants of interest for use as the *request* argument can be found in the :mod:`termios` module, under the same names as used in the relevant C header files. - The parameter *arg* can be one of an integer, absent (treated identically to the - integer ``0``), an object supporting the read-only buffer interface (most likely - a plain Python string) or an object supporting the read-write buffer interface. + The parameter *arg* can be one of an integer, an object supporting the + read-only buffer interface (like :class:`bytes`) or an object supporting + the read-write buffer interface (like :class:`bytearray`). In all but the last case, behaviour is as for the :func:`~fcntl.fcntl` function. @@ -72,7 +72,7 @@ The module defines the following functions: If it is false, the buffer's mutability is ignored and behaviour is as for a read-only buffer, except that the 1024 byte limit mentioned above is avoided -- - so long as the buffer you pass is as least as long as what the operating system + so long as the buffer you pass is at least as long as what the operating system wants to put there, things should work. If *mutate_flag* is true (the default), then the buffer is (in effect) passed @@ -97,25 +97,25 @@ The module defines the following functions: array('h', [13341]) -.. function:: flock(fd, op) +.. function:: flock(fd, operation) - Perform the lock operation *op* on file descriptor *fd* (file objects providing + Perform the lock operation *operation* on file descriptor *fd* (file objects providing a :meth:`~io.IOBase.fileno` method are accepted as well). See the Unix manual :manpage:`flock(2)` for details. (On some systems, this function is emulated using :c:func:`fcntl`.) -.. function:: lockf(fd, operation, [length, [start, [whence]]]) +.. function:: lockf(fd, cmd, len=0, start=0, whence=0) This is essentially a wrapper around the :func:`~fcntl.fcntl` locking calls. - *fd* is the file descriptor of the file to lock or unlock, and *operation* + *fd* is the file descriptor of the file to lock or unlock, and *cmd* is one of the following values: * :const:`LOCK_UN` -- unlock * :const:`LOCK_SH` -- acquire a shared lock * :const:`LOCK_EX` -- acquire an exclusive lock - When *operation* is :const:`LOCK_SH` or :const:`LOCK_EX`, it can also be + When *cmd* is :const:`LOCK_SH` or :const:`LOCK_EX`, it can also be bitwise ORed with :const:`LOCK_NB` to avoid blocking on lock acquisition. If :const:`LOCK_NB` is used and the lock cannot be acquired, an :exc:`OSError` will be raised and the exception will have an *errno* @@ -124,7 +124,7 @@ The module defines the following functions: systems, :const:`LOCK_EX` can only be used if the file descriptor refers to a file opened for writing. - *length* is the number of bytes to lock, *start* is the byte offset at + *len* is the number of bytes to lock, *start* is the byte offset at which the lock starts, relative to *whence*, and *whence* is as with :func:`io.IOBase.seek`, specifically: @@ -133,7 +133,7 @@ The module defines the following functions: * :const:`2` -- relative to the end of the file (:data:`os.SEEK_END`) The default for *start* is 0, which means to start at the beginning of the file. - The default for *length* is 0 which means to lock to the end of the file. The + The default for *len* is 0 which means to lock to the end of the file. The default for *whence* is also 0. Examples (all on a SVR4 compliant system):: @@ -147,9 +147,9 @@ Examples (all on a SVR4 compliant system):: rv = fcntl.fcntl(f, fcntl.F_SETLKW, lockdata) Note that in the first example the return value variable *rv* will hold an -integer value; in the second example it will hold a string value. The structure -lay-out for the *lockdata* variable is system dependent --- therefore using the -:func:`flock` call may be better. +integer value; in the second example it will hold a :class:`bytes` object. The +structure lay-out for the *lockdata* variable is system dependent --- therefore +using the :func:`flock` call may be better. .. seealso:: diff --git a/Doc/library/formatter.rst b/Doc/library/formatter.rst index 1847a8094b..a515f74f64 100644 --- a/Doc/library/formatter.rst +++ b/Doc/library/formatter.rst @@ -5,7 +5,7 @@ :synopsis: Generic output formatter and device interface. :deprecated: -.. deprecated:: 3.4 +.. deprecated-removed:: 3.4 3.6 Due to lack of usage, the formatter module has been deprecated and is slated for removal in Python 3.6. diff --git a/Doc/library/fractions.rst b/Doc/library/fractions.rst index c2c74013f0..e0f0682489 100644 --- a/Doc/library/fractions.rst +++ b/Doc/library/fractions.rst @@ -172,6 +172,9 @@ another rational number, or from a string. sign as *b* if *b* is nonzero; otherwise it takes the sign of *a*. ``gcd(0, 0)`` returns ``0``. + .. deprecated:: 3.5 + Use :func:`math.gcd` instead. + .. seealso:: diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst index 5377335a6e..409f6c4fe1 100644 --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -156,11 +156,12 @@ are always available. They are listed here in alphabetical order. .. function:: chr(i) - Return the string representing a character whose Unicode code point is the integer - *i*. For example, ``chr(97)`` returns the string ``'a'``. This is the - inverse of :func:`ord`. The valid range for the argument is from 0 through - 1,114,111 (0x10FFFF in base 16). :exc:`ValueError` will be raised if *i* is - outside that range. + Return the string representing a character whose Unicode code point is the + integer *i*. For example, ``chr(97)`` returns the string ``'a'``, while + ``chr(957)`` returns the string ``'ν'``. This is the inverse of :func:`ord`. + + The valid range for the argument is from 0 through 1,114,111 (0x10FFFF in + base 16). :exc:`ValueError` will be raised if *i* is outside that range. .. function:: classmethod(function) @@ -972,9 +973,11 @@ are always available. They are listed here in alphabetical order. Characters not supported by the encoding are replaced with the appropriate XML character reference ``&#nnn;``. - * ``'backslashreplace'`` (also only supported when writing) - replaces unsupported characters with Python's backslashed escape - sequences. + * ``'backslashreplace'`` replaces malformed data by Python's backslashed + escape sequences. + + * ``'namereplace'`` (also only supported when writing) + replaces unsupported characters with ``\N{...}`` escape sequences. .. index:: single: universal newlines; open() built-in function @@ -999,8 +1002,8 @@ are always available. They are listed here in alphabetical order. If *closefd* is ``False`` and a file descriptor rather than a filename was given, the underlying file descriptor will be kept open when the file is - closed. If a filename is given *closefd* has no effect and must be ``True`` - (the default). + closed. If a filename is given *closefd* must be ``True`` (the default) + otherwise an error will be raised. A custom opener can be used by passing a callable as *opener*. The underlying file descriptor for the file object is then obtained by calling *opener* with @@ -1062,14 +1065,18 @@ are always available. They are listed here in alphabetical order. The ``'U'`` mode. + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise an + exception, the function now retries the system call instead of raising an + :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + -.. XXX works for bytes too, but should it? .. function:: ord(c) Given a string representing one Unicode character, return an integer - representing the Unicode code - point of that character. For example, ``ord('a')`` returns the integer ``97`` - and ``ord('\u2020')`` returns ``8224``. This is the inverse of :func:`chr`. + representing the Unicode code point of that character. For example, + ``ord('a')`` returns the integer ``97`` and ``ord('ν')`` returns ``957``. + This is the inverse of :func:`chr`. .. function:: pow(x, y[, z]) @@ -1186,6 +1193,9 @@ are always available. They are listed here in alphabetical order. The returned property object also has the attributes ``fget``, ``fset``, and ``fdel`` corresponding to the constructor arguments. + .. versionchanged:: 3.5 + The docstrings of property objects are now writeable. + .. _func-range: .. function:: range(stop) @@ -1218,8 +1228,8 @@ are always available. They are listed here in alphabetical order. .. function:: round(number[, ndigits]) Return the floating point value *number* rounded to *ndigits* digits after - the decimal point. If *ndigits* is omitted, it defaults to zero. Delegates - to ``number.__round__(ndigits)``. + the decimal point. If *ndigits* is omitted, it returns the nearest integer + to its input. Delegates to ``number.__round__(ndigits)``. For the built-in types supporting :func:`round`, values are rounded to the closest multiple of 10 to the power minus *ndigits*; if two multiples are @@ -1485,7 +1495,9 @@ are always available. They are listed here in alphabetical order. The left-to-right evaluation order of the iterables is guaranteed. This makes possible an idiom for clustering a data series into n-length groups - using ``zip(*[iter(s)]*n)``. + using ``zip(*[iter(s)]*n)``. This repeats the *same* iterator ``n`` times + so that each output tuple has the result of ``n`` calls to the iterator. + This has the effect of dividing the input into n-length chunks. :func:`zip` should only be used with unequal length inputs when you don't care about trailing, unmatched values from the longer iterables. If those diff --git a/Doc/library/gc.rst b/Doc/library/gc.rst index 813554206a..d11c2e1282 100644 --- a/Doc/library/gc.rst +++ b/Doc/library/gc.rst @@ -186,7 +186,7 @@ values but should not rebind them): added to this list rather than freed. .. versionchanged:: 3.2 - If this list is non-empty at interpreter shutdown, a + If this list is non-empty at :term:`interpreter shutdown`, a :exc:`ResourceWarning` is emitted, which is silent by default. If :const:`DEBUG_UNCOLLECTABLE` is set, in addition all uncollectable objects are printed. @@ -252,8 +252,8 @@ The following constants are provided for use with :func:`set_debug`: to the ``garbage`` list. .. versionchanged:: 3.2 - Also print the contents of the :data:`garbage` list at interpreter - shutdown, if it isn't empty. + Also print the contents of the :data:`garbage` list at + :term:`interpreter shutdown`, if it isn't empty. .. data:: DEBUG_SAVEALL diff --git a/Doc/library/gettext.rst b/Doc/library/gettext.rst index ff23b59156..514cc5a989 100644 --- a/Doc/library/gettext.rst +++ b/Doc/library/gettext.rst @@ -344,9 +344,9 @@ will assume message ids as Unicode strings, not byte strings. The entire set of key/value pairs are placed into a dictionary and set as the "protected" :attr:`_info` instance variable. -If the :file:`.mo` file's magic number is invalid, or if other problems occur -while reading the file, instantiating a :class:`GNUTranslations` class can raise -:exc:`OSError`. +If the :file:`.mo` file's magic number is invalid, the major version number is +unexpected, or if other problems occur while reading the file, instantiating a +:class:`GNUTranslations` class can raise :exc:`OSError`. The following methods are overridden from the base class implementation: diff --git a/Doc/library/glob.rst b/Doc/library/glob.rst index abcbf380d9..50f38a4d36 100644 --- a/Doc/library/glob.rst +++ b/Doc/library/glob.rst @@ -29,7 +29,7 @@ For example, ``'[?]'`` matches the character ``'?'``. The :mod:`pathlib` module offers high-level path objects. -.. function:: glob(pathname) +.. function:: glob(pathname, *, recursive=False) Return a possibly-empty list of path names that match *pathname*, which must be a string containing a path specification. *pathname* can be either absolute @@ -37,8 +37,19 @@ For example, ``'[?]'`` matches the character ``'?'``. :file:`../../Tools/\*/\*.gif`), and can contain shell-style wildcards. Broken symlinks are included in the results (as in the shell). + If *recursive* is true, the pattern "``**``" will match any files and zero or + more directories and subdirectories. If the pattern is followed by a + ``os.sep``, only directories and subdirectories match. -.. function:: iglob(pathname) + .. note:: + Using the "``**``" pattern in large directory trees may consume + an inordinate amount of time. + + .. versionchanged:: 3.5 + Support for recursive globs using "``**``". + + +.. function:: iglob(pathname, recursive=False) Return an :term:`iterator` which yields the same values as :func:`glob` without actually storing them all simultaneously. @@ -55,8 +66,9 @@ For example, ``'[?]'`` matches the character ``'?'``. .. versionadded:: 3.4 -For example, consider a directory containing only the following files: -:file:`1.gif`, :file:`2.txt`, and :file:`card.gif`. :func:`glob` will produce +For example, consider a directory containing the following files: +:file:`1.gif`, :file:`2.txt`, :file:`card.gif` and a subdirectory :file:`sub` +which contains only the file :file:`3.txt`. :func:`glob` will produce the following results. Notice how any leading components of the path are preserved. :: @@ -67,6 +79,10 @@ preserved. :: ['1.gif', 'card.gif'] >>> glob.glob('?.gif') ['1.gif'] + >>> glob.glob('**/*.txt', recursive=True) + ['2.txt', 'sub/3.txt'] + >>> glob.glob('./**/', recursive=True) + ['./', './sub/'] If the directory contains files starting with ``.`` they won't be matched by default. For example, consider a directory containing :file:`card.gif` and diff --git a/Doc/library/gzip.rst b/Doc/library/gzip.rst index 78536fab51..04c41d585c 100644 --- a/Doc/library/gzip.rst +++ b/Doc/library/gzip.rst @@ -90,13 +90,9 @@ The module defines the following items: is no compression. The default is ``9``. The *mtime* argument is an optional numeric timestamp to be written to - the stream when compressing. All :program:`gzip` compressed streams are - required to contain a timestamp. If omitted or ``None``, the current - time is used. This module ignores the timestamp when decompressing; - however, some programs, such as :program:`gunzip`\ , make use of it. - The format of the timestamp is the same as that of the return value of - ``time.time()`` and of the ``st_mtime`` attribute of the object returned - by ``os.stat()``. + the last modification time field in the stream when compressing. It + should only be provided in compression mode. If omitted or ``None``, the + current time is used. See the :attr:`mtime` attribute for more details. Calling a :class:`GzipFile` object's :meth:`close` method does not close *fileobj*, since you might wish to append more material after the compressed @@ -108,9 +104,9 @@ The module defines the following items: including iteration and the :keyword:`with` statement. Only the :meth:`truncate` method isn't implemented. - :class:`GzipFile` also provides the following method: + :class:`GzipFile` also provides the following method and attribute: - .. method:: peek([n]) + .. method:: peek(n) Read *n* uncompressed bytes without advancing the file position. At most one single read on the compressed stream is done to satisfy @@ -124,9 +120,21 @@ The module defines the following items: .. versionadded:: 3.2 + .. attribute:: mtime + + When decompressing, the value of the last modification time field in + the most recently read header may be read from this attribute, as an + integer. The initial value before reading any headers is ``None``. + + All :program:`gzip` compressed streams are required to contain this + timestamp field. Some programs, such as :program:`gunzip`\ , make use + of the timestamp. The format is the same as the return value of + :func:`time.time` and the :attr:`~os.stat_result.st_mtime` attribute of + the object returned by :func:`os.stat`. + .. versionchanged:: 3.1 Support for the :keyword:`with` statement was added, along with the - *mtime* argument. + *mtime* constructor argument and :attr:`mtime` attribute. .. versionchanged:: 3.2 Support for zero-padded and unseekable files was added. @@ -137,6 +145,12 @@ The module defines the following items: .. versionchanged:: 3.4 Added support for the ``'x'`` and ``'xb'`` modes. + .. versionchanged:: 3.5 + Added support for writing arbitrary + :term:`bytes-like objects <bytes-like object>`. + The :meth:`~io.BufferedIOBase.read` method now accepts an argument of + ``None``. + .. function:: compress(data, compresslevel=9) @@ -175,9 +189,10 @@ Example of how to create a compressed GZIP file:: Example of how to GZIP compress an existing file:: import gzip + import shutil with open('/home/joe/file.txt', 'rb') as f_in: with gzip.open('/home/joe/file.txt.gz', 'wb') as f_out: - f_out.writelines(f_in) + shutil.copyfileobj(f_in, f_out) Example of how to GZIP compress a binary string:: diff --git a/Doc/library/heapq.rst b/Doc/library/heapq.rst index f8970bed73..9fbbcc6904 100644 --- a/Doc/library/heapq.rst +++ b/Doc/library/heapq.rst @@ -82,7 +82,7 @@ The following functions are provided: The module also offers three general purpose functions based on heaps. -.. function:: merge(*iterables) +.. function:: merge(*iterables, key=None, reverse=False) Merge multiple sorted inputs into a single sorted output (for example, merge timestamped entries from multiple log files). Returns an :term:`iterator` @@ -92,6 +92,18 @@ The module also offers three general purpose functions based on heaps. not pull the data into memory all at once, and assumes that each of the input streams is already sorted (smallest to largest). + Has two optional arguments which must be specified as keyword arguments. + + *key* specifies a :term:`key function` of one argument that is used to + extract a comparison key from each input element. The default value is + ``None`` (compare the elements directly). + + *reverse* is a boolean value. If set to ``True``, then the input elements + are merged as if each comparison were reversed. + + .. versionchanged:: 3.5 + Added the optional *key* and *reverse* parameters. + .. function:: nlargest(n, iterable, key=None) diff --git a/Doc/library/html.parser.rst b/Doc/library/html.parser.rst index fef9c38411..824995eddc 100644 --- a/Doc/library/html.parser.rst +++ b/Doc/library/html.parser.rst @@ -16,21 +16,13 @@ This module defines a class :class:`HTMLParser` which serves as the basis for parsing text files formatted in HTML (HyperText Mark-up Language) and XHTML. -.. class:: HTMLParser(strict=False, *, convert_charrefs=False) +.. class:: HTMLParser(*, convert_charrefs=True) - Create a parser instance. + Create a parser instance able to parse invalid markup. - If *convert_charrefs* is ``True`` (default: ``False``), all character + If *convert_charrefs* is ``True`` (the default), all character references (except the ones in ``script``/``style`` elements) are automatically converted to the corresponding Unicode characters. - The use of ``convert_charrefs=True`` is encouraged and will become - the default in Python 3.5. - - If *strict* is ``False`` (the default), the parser will accept and parse - invalid markup. If *strict* is ``True`` the parser will raise an - :exc:`~html.parser.HTMLParseError` exception instead [#]_ when it's not - able to parse the markup. The use of ``strict=True`` is discouraged and - the *strict* argument is deprecated. An :class:`.HTMLParser` instance is fed HTML data and calls handler methods when start tags, end tags, text, comments, and other markup elements are @@ -40,31 +32,11 @@ parsing text files formatted in HTML (HyperText Mark-up Language) and XHTML. This parser does not check that end tags match start tags or call the end-tag handler for elements which are closed implicitly by closing an outer element. - .. versionchanged:: 3.2 - *strict* argument added. - - .. deprecated-removed:: 3.3 3.5 - The *strict* argument and the strict mode have been deprecated. - The parser is now able to accept and parse invalid markup too. - .. versionchanged:: 3.4 *convert_charrefs* keyword argument added. -An exception is defined as well: - - -.. exception:: HTMLParseError - - Exception raised by the :class:`HTMLParser` class when it encounters an error - while parsing and *strict* is ``True``. This exception provides three - attributes: :attr:`msg` is a brief message explaining the error, - :attr:`lineno` is the number of the line on which the broken construct was - detected, and :attr:`offset` is the number of characters into the line at - which the construct starts. - - .. deprecated-removed:: 3.3 3.5 - This exception has been deprecated because it's never raised by the parser - (when the default non-strict mode is used). + .. versionchanged:: 3.5 + The default value for argument *convert_charrefs* is now ``True``. Example HTML Parser Application @@ -246,8 +218,7 @@ implementations do nothing (except for :meth:`~HTMLParser.handle_startendtag`): The *data* parameter will be the entire contents of the declaration inside the ``<![...]>`` markup. It is sometimes useful to be overridden by a - derived class. The base class implementation raises an :exc:`HTMLParseError` - when *strict* is ``True``. + derived class. The base class implementation does nothing. .. _htmlparser-examples: @@ -358,9 +329,3 @@ Parsing invalid HTML (e.g. unquoted attributes) also works:: Data : tag soup End tag : p End tag : a - -.. rubric:: Footnotes - -.. [#] For backward compatibility reasons *strict* mode does not raise - exceptions for all non-compliant HTML. That is, some invalid HTML - is tolerated even in *strict* mode. diff --git a/Doc/library/http.client.rst b/Doc/library/http.client.rst index 807f685248..d57649c4aa 100644 --- a/Doc/library/http.client.rst +++ b/Doc/library/http.client.rst @@ -180,6 +180,17 @@ The following exceptions are raised as appropriate: is received in the HTTP protocol from the server. +.. exception:: RemoteDisconnected + + A subclass of :exc:`ConnectionResetError` and :exc:`BadStatusLine`. Raised + by :meth:`HTTPConnection.getresponse` when the attempt to read the response + results in no data read from the connection, indicating that the remote end + has closed the connection. + + .. versionadded:: 3.5 + Previously, :exc:`BadStatusLine`\ ``('')`` was raised. + + The constants defined in this module are: .. data:: HTTP_PORT @@ -191,221 +202,15 @@ The constants defined in this module are: The default port for the HTTPS protocol (always ``443``). -and also the following constants for integer status codes: - -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| Constant | Value | Definition | -+==========================================+=========+=======================================================================+ -| :const:`CONTINUE` | ``100`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.1.1 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.1.1>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`SWITCHING_PROTOCOLS` | ``101`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.1.2 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.1.2>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`PROCESSING` | ``102`` | WEBDAV, `RFC 2518, Section 10.1 | -| | | <http://www.webdav.org/specs/rfc2518.html#STATUS_102>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`OK` | ``200`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.2.1 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.2.1>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`CREATED` | ``201`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.2.2 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.2.2>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`ACCEPTED` | ``202`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.2.3 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.2.3>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`NON_AUTHORITATIVE_INFORMATION` | ``203`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.2.4 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.2.4>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`NO_CONTENT` | ``204`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.2.5 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.2.5>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`RESET_CONTENT` | ``205`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.2.6 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.2.6>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`PARTIAL_CONTENT` | ``206`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.2.7 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.2.7>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`MULTI_STATUS` | ``207`` | WEBDAV `RFC 2518, Section 10.2 | -| | | <http://www.webdav.org/specs/rfc2518.html#STATUS_207>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`IM_USED` | ``226`` | Delta encoding in HTTP, | -| | | :rfc:`3229`, Section 10.4.1 | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`MULTIPLE_CHOICES` | ``300`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.3.1 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3.1>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`MOVED_PERMANENTLY` | ``301`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.3.2 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3.2>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`FOUND` | ``302`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.3.3 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3.3>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`SEE_OTHER` | ``303`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.3.4 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3.4>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`NOT_MODIFIED` | ``304`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.3.5 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3.5>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`USE_PROXY` | ``305`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.3.6 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3.6>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`TEMPORARY_REDIRECT` | ``307`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.3.8 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3.8>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`BAD_REQUEST` | ``400`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.1 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.1>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`UNAUTHORIZED` | ``401`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.2 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.2>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`PAYMENT_REQUIRED` | ``402`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.3 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.3>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`FORBIDDEN` | ``403`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.4 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.4>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`NOT_FOUND` | ``404`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.5 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.5>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`METHOD_NOT_ALLOWED` | ``405`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.6 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.6>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`NOT_ACCEPTABLE` | ``406`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.7 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.7>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`PROXY_AUTHENTICATION_REQUIRED` | ``407`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.8 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.8>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`REQUEST_TIMEOUT` | ``408`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.9 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.9>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`CONFLICT` | ``409`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.10 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.10>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`GONE` | ``410`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.11 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.11>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`LENGTH_REQUIRED` | ``411`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.12 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.12>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`PRECONDITION_FAILED` | ``412`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.13 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.13>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`REQUEST_ENTITY_TOO_LARGE` | ``413`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.14 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.14>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`REQUEST_URI_TOO_LONG` | ``414`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.15 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.15>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`UNSUPPORTED_MEDIA_TYPE` | ``415`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.16 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.16>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`REQUESTED_RANGE_NOT_SATISFIABLE` | ``416`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.17 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.17>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`EXPECTATION_FAILED` | ``417`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.18 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.18>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`UNPROCESSABLE_ENTITY` | ``422`` | WEBDAV, `RFC 2518, Section 10.3 | -| | | <http://www.webdav.org/specs/rfc2518.html#STATUS_422>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`LOCKED` | ``423`` | WEBDAV `RFC 2518, Section 10.4 | -| | | <http://www.webdav.org/specs/rfc2518.html#STATUS_423>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`FAILED_DEPENDENCY` | ``424`` | WEBDAV, `RFC 2518, Section 10.5 | -| | | <http://www.webdav.org/specs/rfc2518.html#STATUS_424>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`UPGRADE_REQUIRED` | ``426`` | HTTP Upgrade to TLS, | -| | | :rfc:`2817`, Section 6 | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`PRECONDITION_REQUIRED` | ``428`` | Additional HTTP Status Codes, | -| | | :rfc:`6585`, Section 3 | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`TOO_MANY_REQUESTS` | ``429`` | Additional HTTP Status Codes, | -| | | :rfc:`6585`, Section 4 | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`REQUEST_HEADER_FIELDS_TOO_LARGE` | ``431`` | Additional HTTP Status Codes, | -| | | :rfc:`6585`, Section 5 | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`INTERNAL_SERVER_ERROR` | ``500`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.5.1 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.5.1>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`NOT_IMPLEMENTED` | ``501`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.5.2 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.5.2>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`BAD_GATEWAY` | ``502`` | HTTP/1.1 `RFC 2616, Section | -| | | 10.5.3 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.5.3>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`SERVICE_UNAVAILABLE` | ``503`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.5.4 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.5.4>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`GATEWAY_TIMEOUT` | ``504`` | HTTP/1.1 `RFC 2616, Section | -| | | 10.5.5 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.5.5>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`HTTP_VERSION_NOT_SUPPORTED` | ``505`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.5.6 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.5.6>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`INSUFFICIENT_STORAGE` | ``507`` | WEBDAV, `RFC 2518, Section 10.6 | -| | | <http://www.webdav.org/specs/rfc2518.html#STATUS_507>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`NOT_EXTENDED` | ``510`` | An HTTP Extension Framework, | -| | | :rfc:`2774`, Section 7 | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`NETWORK_AUTHENTICATION_REQUIRED` | ``511`` | Additional HTTP Status Codes, | -| | | :rfc:`6585`, Section 6 | -+------------------------------------------+---------+-----------------------------------------------------------------------+ - -.. versionchanged:: 3.3 - Added codes ``428``, ``429``, ``431`` and ``511`` from :rfc:`6585`. - - .. data:: responses This dictionary maps the HTTP 1.1 status codes to the W3C names. Example: ``http.client.responses[http.client.NOT_FOUND]`` is ``'Not Found'``. +See :ref:`http-status-codes` for a list of HTTP status codes that are +available in this module as constants. + .. _httpconnection-objects: @@ -458,6 +263,11 @@ HTTPConnection Objects Note that you must have read the whole response before you can send a new request to the server. + .. versionchanged:: 3.5 + If a :exc:`ConnectionError` or subclass is raised, the + :class:`HTTPConnection` object will be ready to reconnect when + a new request is sent. + .. method:: HTTPConnection.set_debuglevel(level) @@ -496,7 +306,9 @@ HTTPConnection Objects .. method:: HTTPConnection.connect() - Connect to the server specified when the object was created. + Connect to the server specified when the object was created. By default, + this is called automatically when making a request if the client does not + already have a connection. .. method:: HTTPConnection.close() diff --git a/Doc/library/http.cookies.rst b/Doc/library/http.cookies.rst index 646f2e8860..7c85d0954d 100644 --- a/Doc/library/http.cookies.rst +++ b/Doc/library/http.cookies.rst @@ -143,26 +143,43 @@ Morsel Objects The keys are case-insensitive. + .. versionchanged:: 3.5 + :meth:`~Morsel.__eq__` now takes :attr:`~Morsel.key` and :attr:`~Morsel.value` + into account. + .. attribute:: Morsel.value The value of the cookie. + .. deprecated:: 3.5 + assigning to ``value``; use :meth:`~Morsel.set` instead. + .. attribute:: Morsel.coded_value The encoded value of the cookie --- this is what should be sent. + .. deprecated:: 3.5 + assigning to ``coded_value``; use :meth:`~Morsel.set` instead. + .. attribute:: Morsel.key The name of the cookie. + .. deprecated:: 3.5 + assigning to ``key``; use :meth:`~Morsel.set` instead. + .. method:: Morsel.set(key, value, coded_value) Set the *key*, *value* and *coded_value* attributes. + .. deprecated:: 3.5 + The undocumented *LegalChars* parameter is ignored and will be removed in + a future version. + .. method:: Morsel.isReservedKey(K) @@ -193,6 +210,30 @@ Morsel Objects The meaning for *attrs* is the same as in :meth:`output`. +.. method:: Morsel.update(values) + + Update the values in the Morsel dictionary with the values in the dictionary + *values*. Raise an error if any of the keys in the *values* dict is not a + valid :rfc:`2109` attribute. + + .. versionchanged:: 3.5 + an error is raised for invalid keys. + + +.. method:: Morsel.copy(value) + + Return a shallow copy of the Morsel object. + + .. versionchanged:: 3.5 + return a Morsel object instead of a dict. + + +.. method:: Morsel.setdefault(key, value=None) + + Raise an error if key is not a valid :rfc:`2109` attribute, otherwise + behave the same as :meth:`dict.setdefault`. + + .. _cookie-example: Example diff --git a/Doc/library/http.rst b/Doc/library/http.rst index a387a37ddd..b6f2c58218 100644 --- a/Doc/library/http.rst +++ b/Doc/library/http.rst @@ -1,7 +1,16 @@ :mod:`http` --- HTTP modules ============================ -``http`` is a package that collects several modules for working with the +.. module:: http + :synopsis: HTTP status codes and messages + +.. index:: + pair: HTTP; protocol + single: HTTP; http (standard module) + +**Source code:** :source:`Lib/http/__init__.py` + +:mod:`http` is a package that collects several modules for working with the HyperText Transfer Protocol: * :mod:`http.client` is a low-level HTTP protocol client; for high-level URL @@ -9,3 +18,105 @@ HyperText Transfer Protocol: * :mod:`http.server` contains basic HTTP server classes based on :mod:`socketserver` * :mod:`http.cookies` has utilities for implementing state management with cookies * :mod:`http.cookiejar` provides persistence of cookies + +:mod:`http` is also a module that defines a number of HTTP status codes and +associated messages through the :class:`http.HTTPStatus` enum: + +.. class:: HTTPStatus + + .. versionadded:: 3.5 + + A subclass of :class:`enum.IntEnum` that defines a set of HTTP status codes, + reason phrases and long descriptions written in English. + + Usage:: + + >>> from http import HTTPStatus + >>> HTTPStatus.OK + <HTTPStatus.OK: 200> + >>> HTTPStatus.OK == 200 + True + >>> http.HTTPStatus.OK.value + 200 + >>> HTTPStatus.OK.phrase + 'OK' + >>> HTTPStatus.OK.description + 'Request fulfilled, document follows' + >>> list(HTTPStatus) + [<HTTPStatus.CONTINUE: 100>, <HTTPStatus.SWITCHING_PROTOCOLS: 101>, ...] + +.. _http-status-codes: + +HTTP status codes +----------------- + +Supported, +`IANA-registered <http://www.iana.org/assignments/http-status-codes/http-status-codes.xhtml>`_ +status codes available in :class:`http.HTTPStatus` are: + +======= =================================== ================================================================== +Code Enum Name Details +======= =================================== ================================================================== +``100`` ``CONTINUE`` HTTP/1.1 :rfc:`7231`, Section 6.2.1 +``101`` ``SWITCHING_PROTOCOLS`` HTTP/1.1 :rfc:`7231`, Section 6.2.2 +``102`` ``PROCESSING`` WebDAV :rfc:`2518`, Section 10.1 +``200`` ``OK`` HTTP/1.1 :rfc:`7231`, Section 6.3.1 +``201`` ``CREATED`` HTTP/1.1 :rfc:`7231`, Section 6.3.2 +``202`` ``ACCEPTED`` HTTP/1.1 :rfc:`7231`, Section 6.3.3 +``203`` ``NON_AUTHORITATIVE_INFORMATION`` HTTP/1.1 :rfc:`7231`, Section 6.3.4 +``204`` ``NO_CONTENT`` HTTP/1.1 :rfc:`7231`, Section 6.3.5 +``205`` ``RESET_CONTENT`` HTTP/1.1 :rfc:`7231`, Section 6.3.6 +``206`` ``PARTIAL_CONTENT`` HTTP/1.1 :rfc:`7233`, Section 4.1 +``207`` ``MULTI_STATUS`` WebDAV :rfc:`4918`, Section 11.1 +``208`` ``ALREADY_REPORTED`` WebDAV Binding Extensions :rfc:`5842`, Section 7.1 (Experimental) +``226`` ``IM_USED`` Delta Encoding in HTTP :rfc:`3229`, Section 10.4.1 +``300`` ``MULTIPLE_CHOICES`` HTTP/1.1 :rfc:`7231`, Section 6.4.1 +``301`` ``MOVED_PERMANENTLY`` HTTP/1.1 :rfc:`7231`, Section 6.4.2 +``302`` ``FOUND`` HTTP/1.1 :rfc:`7231`, Section 6.4.3 +``303`` ``SEE_OTHER`` HTTP/1.1 :rfc:`7231`, Section 6.4.4 +``304`` ``NOT_MODIFIED`` HTTP/1.1 :rfc:`7232`, Section 4.1 +``305`` ``USE_PROXY`` HTTP/1.1 :rfc:`7231`, Section 6.4.5 +``307`` ``TEMPORARY_REDIRECT`` HTTP/1.1 :rfc:`7231`, Section 6.4.7 +``308`` ``PERMANENT_REDIRECT`` Permanent Redirect :rfc:`7238`, Section 3 (Experimental) +``400`` ``BAD_REQUEST`` HTTP/1.1 :rfc:`7231`, Section 6.5.1 +``401`` ``UNAUTHORIZED`` HTTP/1.1 Authentication :rfc:`7235`, Section 3.1 +``402`` ``PAYMENT_REQUIRED`` HTTP/1.1 :rfc:`7231`, Section 6.5.2 +``403`` ``FORBIDDEN`` HTTP/1.1 :rfc:`7231`, Section 6.5.3 +``404`` ``NOT_FOUND`` HTTP/1.1 :rfc:`7231`, Section 6.5.4 +``405`` ``METHOD_NOT_ALLOWED`` HTTP/1.1 :rfc:`7231`, Section 6.5.5 +``406`` ``NOT_ACCEPTABLE`` HTTP/1.1 :rfc:`7231`, Section 6.5.6 +``407`` ``PROXY_AUTHENTICATION_REQUIRED`` HTTP/1.1 Authentication :rfc:`7235`, Section 3.2 +``408`` ``REQUEST_TIMEOUT`` HTTP/1.1 :rfc:`7231`, Section 6.5.7 +``409`` ``CONFLICT`` HTTP/1.1 :rfc:`7231`, Section 6.5.8 +``410`` ``GONE`` HTTP/1.1 :rfc:`7231`, Section 6.5.9 +``411`` ``LENGTH_REQUIRED`` HTTP/1.1 :rfc:`7231`, Section 6.5.10 +``412`` ``PRECONDITION_FAILED`` HTTP/1.1 :rfc:`7232`, Section 4.2 +``413`` ``REQUEST_ENTITY_TOO_LARGE`` HTTP/1.1 :rfc:`7231`, Section 6.5.11 +``414`` ``REQUEST_URI_TOO_LONG`` HTTP/1.1 :rfc:`7231`, Section 6.5.12 +``415`` ``UNSUPPORTED_MEDIA_TYPE`` HTTP/1.1 :rfc:`7231`, Section 6.5.13 +``416`` ``REQUEST_RANGE_NOT_SATISFIABLE`` HTTP/1.1 Range Requests :rfc:`7233`, Section 4.4 +``417`` ``EXPECTATION_FAILED`` HTTP/1.1 :rfc:`7231`, Section 6.5.14 +``422`` ``UNPROCESSABLE_ENTITY`` WebDAV :rfc:`4918`, Section 11.2 +``423`` ``LOCKED`` WebDAV :rfc:`4918`, Section 11.3 +``424`` ``FAILED_DEPENDENCY`` WebDAV :rfc:`4918`, Section 11.4 +``426`` ``UPGRADE_REQUIRED`` HTTP/1.1 :rfc:`7231`, Section 6.5.15 +``428`` ``PRECONDITION_REQUIRED`` Additional HTTP Status Codes :rfc:`6585` +``429`` ``TOO_MANY_REQUESTS`` Additional HTTP Status Codes :rfc:`6585` +``431`` ``REQUEST_HEADER_FIELDS_TOO_LARGE`` Additional HTTP Status Codes :rfc:`6585` +``500`` ``INTERNAL_SERVER_ERROR`` HTTP/1.1 :rfc:`7231`, Section 6.6.1 +``501`` ``NOT_IMPLEMENTED`` HTTP/1.1 :rfc:`7231`, Section 6.6.2 +``502`` ``BAD_GATEWAY`` HTTP/1.1 :rfc:`7231`, Section 6.6.3 +``503`` ``SERVICE_UNAVAILABLE`` HTTP/1.1 :rfc:`7231`, Section 6.6.4 +``504`` ``GATEWAY_TIMEOUT`` HTTP/1.1 :rfc:`7231`, Section 6.6.5 +``505`` ``HTTP_VERSION_NOT_SUPPORTED`` HTTP/1.1 :rfc:`7231`, Section 6.6.6 +``506`` ``VARIANT_ALSO_NEGOTIATES`` Transparent Content Negotiation in HTTP :rfc:`2295`, Section 8.1 (Experimental) +``507`` ``INSUFFICIENT_STORAGE`` WebDAV :rfc:`4918`, Section 11.5 +``508`` ``LOOP_DETECTED`` WebDAV Binding Extensions :rfc:`5842`, Section 7.2 (Experimental) +``510`` ``NOT_EXTENDED`` An HTTP Extension Framework :rfc:`2774`, Section 7 (Experimental) +``511`` ``NETWORK_AUTHENTICATION_REQUIRED`` Additional HTTP Status Codes :rfc:`6585`, Section 6 +======= =================================== ================================================================== + +In order to preserve backwards compatibility, enum values are also present +in the :mod:`http.client` module in the form of constants. The enum name is +equal to the constant name (i.e. ``http.HTTPStatus.OK`` is also available as +``http.client.OK``). diff --git a/Doc/library/imaplib.rst b/Doc/library/imaplib.rst index fa736fe3af..15b0932973 100644 --- a/Doc/library/imaplib.rst +++ b/Doc/library/imaplib.rst @@ -37,6 +37,19 @@ base class: initialized. If *host* is not specified, ``''`` (the local host) is used. If *port* is omitted, the standard IMAP4 port (143) is used. + The :class:`IMAP4` class supports the :keyword:`with` statement. When used + like this, the IMAP4 ``LOGOUT`` command is issued automatically when the + :keyword:`with` statement exits. E.g.:: + + >>> from imaplib import IMAP4 + >>> with IMAP4("domain.org") as M: + ... M.noop() + ... + ('OK', [b'Nothing Accomplished. d25if65hy903weo.87']) + + .. versionchanged:: 3.5 + Support for the :keyword:`with` statement was added. + Three exceptions are defined as attributes of the :class:`IMAP4` class: @@ -64,7 +77,8 @@ Three exceptions are defined as attributes of the :class:`IMAP4` class: There's also a subclass for secure connections: -.. class:: IMAP4_SSL(host='', port=IMAP4_SSL_PORT, keyfile=None, certfile=None, ssl_context=None) +.. class:: IMAP4_SSL(host='', port=IMAP4_SSL_PORT, keyfile=None, \ + certfile=None, ssl_context=None) This is a subclass derived from :class:`IMAP4` that connects over an SSL encrypted socket (to use this class you need a socket module that was compiled @@ -198,6 +212,10 @@ An :class:`IMAP4` instance has the following methods: that will be base64 encoded and sent to the server. It should return ``None`` if the client abort response ``*`` should be sent instead. + .. versionchanged:: 3.5 + string usernames and passwords are now encoded to ``utf-8`` instead of + being limited to ASCII. + .. method:: IMAP4.check() @@ -230,6 +248,16 @@ An :class:`IMAP4` instance has the following methods: Delete the ACLs (remove any rights) set for who on mailbox. +.. method:: IMAP4.enable(capability) + + Enable *capability* (see :rfc:`5161`). Most capabilities do not need to be + enabled. Currently only the ``UTF8=ACCEPT`` capability is supported + (see :RFC:`6855`). + + .. versionadded:: 3.5 + The :meth:`enable` method itself, and :RFC:`6855` support. + + .. method:: IMAP4.expunge() Permanently remove deleted items from selected mailbox. Generates an ``EXPUNGE`` @@ -367,7 +395,9 @@ An :class:`IMAP4` instance has the following methods: Search mailbox for matching messages. *charset* may be ``None``, in which case no ``CHARSET`` will be specified in the request to the server. The IMAP protocol requires that at least one criterion be specified; an exception will be - raised when the server returns an error. + raised when the server returns an error. *charset* must be ``None`` if + the ``UTF8=ACCEPT`` capability was enabled using the :meth:`enable` + command. Example:: @@ -529,6 +559,15 @@ The following attributes are defined on instances of :class:`IMAP4`: the module variable ``Debug``. Values greater than three trace each command. +.. attribute:: IMAP4.utf8_enabled + + Boolean value that is normally ``False``, but is set to ``True`` if an + :meth:`enable` command is successfully issued for the ``UTF8=ACCEPT`` + capability. + + .. versionadded:: 3.5 + + .. _imap4-example: IMAP4 Example diff --git a/Doc/library/imghdr.rst b/Doc/library/imghdr.rst index 9e8952339c..f11f6dcf8e 100644 --- a/Doc/library/imghdr.rst +++ b/Doc/library/imghdr.rst @@ -48,6 +48,14 @@ from :func:`what`: +------------+-----------------------------------+ | ``'png'`` | Portable Network Graphics | +------------+-----------------------------------+ +| ``'webp'`` | WebP files | ++------------+-----------------------------------+ +| ``'exr'`` | OpenEXR Files | ++------------+-----------------------------------+ + +.. versionadded:: 3.5 + The *exr* and *webp* formats were added. + You can extend the list of file types :mod:`imghdr` can recognize by appending to this variable: diff --git a/Doc/library/imp.rst b/Doc/library/imp.rst index 83a52e4e14..68a6b681ef 100644 --- a/Doc/library/imp.rst +++ b/Doc/library/imp.rst @@ -197,11 +197,9 @@ file paths. value would be ``/foo/bar/__pycache__/baz.cpython-32.pyc`` for Python 3.2. The ``cpython-32`` string comes from the current magic tag (see :func:`get_tag`; if :attr:`sys.implementation.cache_tag` is not defined then - :exc:`NotImplementedError` will be raised). The returned path will end in - ``.pyc`` when ``__debug__`` is ``True`` or ``.pyo`` for an optimized Python - (i.e. ``__debug__`` is ``False``). By passing in ``True`` or ``False`` for - *debug_override* you can override the system's value for ``__debug__`` for - extension selection. + :exc:`NotImplementedError` will be raised). By passing in ``True`` or + ``False`` for *debug_override* you can override the system's value for + ``__debug__``, leading to optimized bytecode. *path* need not exist. @@ -212,6 +210,9 @@ file paths. .. deprecated:: 3.4 Use :func:`importlib.util.cache_from_source` instead. + .. versionchanged:: 3.5 + The *debug_override* parameter no longer creates a ``.pyo`` file. + .. function:: source_from_cache(path) diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst index c84d2dfbb4..632df75d57 100644 --- a/Doc/library/importlib.rst +++ b/Doc/library/importlib.rst @@ -55,6 +55,12 @@ generically as an :term:`importer`) to participate in the import process. :pep:`451` A ModuleSpec Type for the Import System + :pep:`488` + Elimination of PYO files + + :pep:`489` + Multi-phase extension module initialization + :pep:`3120` Using UTF-8 as the Default Source Encoding @@ -69,6 +75,10 @@ Functions An implementation of the built-in :func:`__import__` function. + .. note:: + Programmatic importing of modules should use :func:`import_module` + instead of this function. + .. function:: import_module(name, package=None) Import a module. The *name* argument specifies what module to @@ -81,12 +91,15 @@ Functions The :func:`import_module` function acts as a simplifying wrapper around :func:`importlib.__import__`. This means all semantics of the function are - derived from :func:`importlib.__import__`, including requiring the package - from which an import is occurring to have been previously imported - (i.e., *package* must already be imported). The most important difference - is that :func:`import_module` returns the specified package or module - (e.g. ``pkg.mod``), while :func:`__import__` returns the - top-level package or module (e.g. ``pkg``). + derived from :func:`importlib.__import__`. The most important difference + between these two functions is that :func:`import_module` returns the + specified package or module (e.g. ``pkg.mod``), while :func:`__import__` + returns the top-level package or module (e.g. ``pkg``). + + If you are dynamically importing a module that was created since the + interpreter began execution (e.g., created a Python source file), you may + need to call :func:`invalidate_caches` in order for the new module to be + noticed by the import system. .. versionchanged:: 3.3 Parent packages are automatically imported. @@ -341,13 +354,16 @@ ABC hierarchy:: .. method:: create_module(spec) - An optional method that returns the module object to use when - importing a module. create_module() may also return ``None``, - indicating that the default module creation should take place - instead. + A method that returns the module object to use when + importing a module. This method may return ``None``, + indicating that default module creation semantics should take place. .. versionadded:: 3.4 + .. versionchanged:: 3.5 + Starting in Python 3.6, this method will not be optional when + :meth:`exec_module` is defined. + .. method:: exec_module(module) An abstract method that executes the module in its own namespace @@ -411,7 +427,7 @@ ABC hierarchy:: .. deprecated:: 3.4 The recommended API for loading a module is :meth:`exec_module` - (and optionally :meth:`create_module`). Loaders should implement + (and :meth:`create_module`). Loaders should implement it instead of load_module(). The import machinery takes care of all the other responsibilities of load_module() when exec_module() is implemented. @@ -493,7 +509,7 @@ ABC hierarchy:: .. versionchanged:: 3.4 Raises :exc:`ImportError` instead of :exc:`NotImplementedError`. - .. method:: source_to_code(data, path='<string>') + .. staticmethod:: source_to_code(data, path='<string>') Create a code object from Python source. @@ -502,8 +518,14 @@ ABC hierarchy:: the "path" to where the source code originated from, which can be an abstract concept (e.g. location in a zip file). + With the subsequent code object one can execute it in a module by + running ``exec(code, module.__dict__)``. + .. versionadded:: 3.4 + .. versionchanged:: 3.5 + Made the method static. + .. method:: exec_module(module) Implementation of :meth:`Loader.exec_module`. @@ -689,6 +711,9 @@ find and load modules. .. versionadded:: 3.3 + .. deprecated:: 3.5 + Use :attr:`BYTECODE_SUFFIXES` instead. + .. attribute:: OPTIMIZED_BYTECODE_SUFFIXES A list of strings representing the file suffixes for optimized bytecode @@ -696,14 +721,19 @@ find and load modules. .. versionadded:: 3.3 + .. deprecated:: 3.5 + Use :attr:`BYTECODE_SUFFIXES` instead. + .. attribute:: BYTECODE_SUFFIXES A list of strings representing the recognized file suffixes for bytecode - modules. Set to either :attr:`DEBUG_BYTECODE_SUFFIXES` or - :attr:`OPTIMIZED_BYTECODE_SUFFIXES` based on whether ``__debug__`` is true. + modules (including the leading dot). .. versionadded:: 3.3 + .. versionchanged:: 3.5 + The value is no longer dependent on ``__debug__``. + .. attribute:: EXTENSION_SUFFIXES A list of strings representing the recognized file suffixes for @@ -732,9 +762,9 @@ find and load modules. Only class methods are defined by this class to alleviate the need for instantiation. - .. note:: - Due to limitations in the extension module C-API, for now - BuiltinImporter does not implement :meth:`Loader.exec_module`. + .. versionchanged:: 3.5 + As part of :pep:`489`, the builtin importer now implements + :meth:`Loader.create_module` and :meth:`Loader.exec_module` .. class:: FrozenImporter @@ -782,6 +812,11 @@ find and load modules. .. versionadded:: 3.4 + .. versionchanged:: 3.5 + If the current working directory -- represented by an empty string -- + is no longer valid then ``None`` is returned but no value is cached + in :data:`sys.path_importer_cache`. + .. classmethod:: find_module(fullname, path=None) A legacy wrapper around :meth:`find_spec`. @@ -944,14 +979,18 @@ find and load modules. Path to the extension module. - .. method:: load_module(name=None) + .. method:: create_module(spec) + + Creates the module object from the given specification in accordance + with :pep:`489`. + + .. versionadded:: 3.5 - Loads the extension module if and only if *fullname* is the same as - :attr:`name` or is ``None``. + .. method:: exec_module(module) - .. note:: - Due to limitations in the extension module C-API, for now - ExtensionFileLoader does not implement :meth:`Loader.exec_module`. + Initializes the given module object in accordance with :pep:`489`. + + .. versionadded:: 3.5 .. method:: is_package(fullname) @@ -1047,23 +1086,37 @@ an :term:`importer`. .. versionadded:: 3.4 -.. function:: cache_from_source(path, debug_override=None) +.. function:: cache_from_source(path, debug_override=None, *, optimization=None) - Return the :pep:`3147` path to the byte-compiled file associated with the - source *path*. For example, if *path* is ``/foo/bar/baz.py`` the return + Return the :pep:`3147`/:pep:`488` path to the byte-compiled file associated + with the source *path*. For example, if *path* is ``/foo/bar/baz.py`` the return value would be ``/foo/bar/__pycache__/baz.cpython-32.pyc`` for Python 3.2. The ``cpython-32`` string comes from the current magic tag (see :func:`get_tag`; if :attr:`sys.implementation.cache_tag` is not defined then - :exc:`NotImplementedError` will be raised). The returned path will end in - ``.pyc`` when ``__debug__`` is ``True`` or ``.pyo`` for an optimized Python - (i.e. ``__debug__`` is ``False``). By passing in ``True`` or ``False`` for - *debug_override* you can override the system's value for ``__debug__`` for - extension selection. - - *path* need not exist. + :exc:`NotImplementedError` will be raised). + + The *optimization* parameter is used to specify the optimization level of the + bytecode file. An empty string represents no optimization, so + ``/foo/bar/baz.py`` with an *optimization* of ``''`` will result in a + bytecode path of ``/foo/bar/__pycache__/baz.cpython-32.pyc``. ``None`` causes + the interpter's optimization level to be used. Any other value's string + representation being used, so ``/foo/bar/baz.py`` with an *optimization* of + ``2`` will lead to the bytecode path of + ``/foo/bar/__pycache__/baz.cpython-32.opt-2.pyc``. The string representation + of *optimization* can only be alphanumeric, else :exc:`ValueError` is raised. + + The *debug_override* parameter is deprecated and can be used to override + the system's value for ``__debug__``. A ``True`` value is the equivalent of + setting *optimization* to the empty string. A ``False`` value is the same as + setting *optimization* to ``1``. If both *debug_override* an *optimization* + are not ``None`` then :exc:`TypeError` is raised. .. versionadded:: 3.4 + .. versionchanged ::3.5 + The *optimization* parameter was added and the *debug_override* parameter + was deprecated. + .. function:: source_from_cache(path) @@ -1071,7 +1124,7 @@ an :term:`importer`. file path. For example, if *path* is ``/foo/bar/__pycache__/baz.cpython-32.pyc`` the returned path would be ``/foo/bar/baz.py``. *path* need not exist, however if it does not conform - to :pep:`3147` format, a ``ValueError`` is raised. If + to :pep:`3147` or :pep:`488` format, a ``ValueError`` is raised. If :attr:`sys.implementation.cache_tag` is not defined, :exc:`NotImplementedError` is raised. @@ -1117,6 +1170,21 @@ an :term:`importer`. .. versionadded:: 3.4 +.. function:: module_from_spec(spec) + + Create a new module based on **spec** and ``spec.loader.create_module()``. + + If ``spec.loader.create_module()`` does not return ``None``, then any + pre-existing attributes will not be reset. Also, no :exc:`AttributeError` + will be raised if triggered while accessing **spec** or setting an attribute + on the module. + + This function is preferred over using :class:`types.ModuleType` to create a + new module as **spec** is used to set as many import-controlled attributes on + the module as possible. + + .. versionadded:: 3.5 + .. decorator:: module_for_loader A :term:`decorator` for :meth:`importlib.abc.Loader.load_module` @@ -1195,3 +1263,39 @@ an :term:`importer`. module will be file-based. .. versionadded:: 3.4 + +.. class:: LazyLoader(loader) + + A class which postpones the execution of the loader of a module until the + module has an attribute accessed. + + This class **only** works with loaders that define + :meth:`~importlib.abc.Loader.exec_module` as control over what module type + is used for the module is required. For those same reasons, the loader's + :meth:`~importlib.abc.Loader.create_module` method will be ignored (i.e., the + loader's method should only return ``None``). Finally, + modules which substitute the object placed into :attr:`sys.modules` will + not work as there is no way to properly replace the module references + throughout the interpreter safely; :exc:`ValueError` is raised if such a + substitution is detected. + + .. note:: + For projects where startup time is critical, this class allows for + potentially minimizing the cost of loading a module if it is never used. + For projects where startup time is not essential then use of this class is + **heavily** discouraged due to error messages created during loading being + postponed and thus occurring out of context. + + .. versionadded:: 3.5 + + .. classmethod:: factory(loader) + + A static method which returns a callable that creates a lazy loader. This + is meant to be used in situations where the loader is passed by class + instead of by instance. + :: + + suffixes = importlib.machinery.SOURCE_SUFFIXES + loader = importlib.machinery.SourceFileLoader + lazy_loader = importlib.util.LazyLoader.factory(loader) + finder = importlib.machinery.FileFinder(path, [(lazy_loader, suffixes)]) diff --git a/Doc/library/inspect.rst b/Doc/library/inspect.rst index 57eb4fff82..26a2913d79 100644 --- a/Doc/library/inspect.rst +++ b/Doc/library/inspect.rst @@ -28,7 +28,7 @@ Types and members ----------------- The :func:`getmembers` function retrieves the members of an object such as a -class or module. The sixteen functions whose names begin with "is" are mainly +class or module. The functions whose names begin with "is" are mainly provided as convenient choices for the second argument to :func:`getmembers`. They also help you determine when you can expect to find the following special attributes: @@ -168,6 +168,33 @@ attributes: | | | arguments and local | | | | variables | +-----------+-----------------+---------------------------+ +| generator | __name__ | name | ++-----------+-----------------+---------------------------+ +| | __qualname__ | qualified name | ++-----------+-----------------+---------------------------+ +| | gi_frame | frame | ++-----------+-----------------+---------------------------+ +| | gi_running | is the generator running? | ++-----------+-----------------+---------------------------+ +| | gi_code | code | ++-----------+-----------------+---------------------------+ +| | gi_yieldfrom | object being iterated by | +| | | ``yield from``, or | +| | | ``None`` | ++-----------+-----------------+---------------------------+ +| coroutine | __name__ | name | ++-----------+-----------------+---------------------------+ +| | __qualname__ | qualified name | ++-----------+-----------------+---------------------------+ +| | cr_await | object being awaited on, | +| | | or ``None`` | ++-----------+-----------------+---------------------------+ +| | cr_frame | frame | ++-----------+-----------------+---------------------------+ +| | cr_running | is the coroutine running? | ++-----------+-----------------+---------------------------+ +| | cr_code | code | ++-----------+-----------------+---------------------------+ | builtin | __doc__ | documentation string | +-----------+-----------------+---------------------------+ | | __name__ | original name of this | @@ -180,6 +207,13 @@ attributes: | | | ``None`` | +-----------+-----------------+---------------------------+ +.. versionchanged:: 3.5 + + Add ``__qualname__`` and ``gi_yieldfrom`` attributes to generators. + + The ``__name__`` attribute of generators is now set from the function + name, instead of the code name, and it can now be modified. + .. function:: getmembers(object[, predicate]) @@ -261,6 +295,41 @@ attributes: Return true if the object is a generator. +.. function:: iscoroutinefunction(object) + + Return true if the object is a :term:`coroutine function` + (a function defined with an :keyword:`async def` syntax). + + .. versionadded:: 3.5 + + +.. function:: iscoroutine(object) + + Return true if the object is a :term:`coroutine` created by an + :keyword:`async def` function. + + .. versionadded:: 3.5 + + +.. function:: isawaitable(object) + + Return true if the object can be used in :keyword:`await` expression. + + Can also be used to distinguish generator-based coroutines from regular + generators:: + + def gen(): + yield + @types.coroutine + def gen_coro(): + yield + + assert not isawaitable(gen()) + assert isawaitable(gen_coro()) + + .. versionadded:: 3.5 + + .. function:: istraceback(object) Return true if the object is a traceback. @@ -351,6 +420,12 @@ Retrieving source code .. function:: getdoc(object) Get the documentation string for an object, cleaned up with :func:`cleandoc`. + If the documentation string for an object is not provided and the object is + a class, a method, a property or a descriptor, retrieve the documentation + string from the inheritance hierarchy. + + .. versionchanged:: 3.5 + Documentation strings are now inherited if not overridden. .. function:: getcomments(object) @@ -423,7 +498,7 @@ The Signature object represents the call signature of a callable object and its return annotation. To retrieve a Signature object, use the :func:`signature` function. -.. function:: signature(callable) +.. function:: signature(callable, \*, follow_wrapped=True) Return a :class:`Signature` object for the given ``callable``:: @@ -448,6 +523,11 @@ function. Raises :exc:`ValueError` if no signature can be provided, and :exc:`TypeError` if that type of object is not supported. + .. versionadded:: 3.5 + ``follow_wrapped`` parameter. Pass ``False`` to get a signature of + ``callable`` specifically (``callable.__wrapped__`` will not be used to + unwrap decorated callables.) + .. note:: Some callables may not be introspectable in certain implementations of @@ -473,6 +553,9 @@ function. Signature objects are *immutable*. Use :meth:`Signature.replace` to make a modified copy. + .. versionchanged:: 3.5 + Signature objects are picklable and hashable. + .. attribute:: Signature.empty A special class-level marker to specify absence of a return annotation. @@ -517,12 +600,30 @@ function. >>> str(new_sig) "(a, b) -> 'new return anno'" + .. classmethod:: Signature.from_callable(obj, \*, follow_wrapped=True) + + Return a :class:`Signature` (or its subclass) object for a given callable + ``obj``. Pass ``follow_wrapped=False`` to get a signature of ``obj`` + without unwrapping its ``__wrapped__`` chain. + + This method simplifies subclassing of :class:`Signature`:: + + class MySignature(Signature): + pass + sig = MySignature.from_callable(min) + assert isinstance(sig, MySignature) + + .. versionadded:: 3.5 + .. class:: Parameter(name, kind, \*, default=Parameter.empty, annotation=Parameter.empty) Parameter objects are *immutable*. Instead of modifying a Parameter object, you can use :meth:`Parameter.replace` to create a modified copy. + .. versionchanged:: 3.5 + Parameter objects are picklable and hashable. + .. attribute:: Parameter.empty A special class-level marker to specify absence of default values and @@ -639,27 +740,8 @@ function. Arguments for which :meth:`Signature.bind` or :meth:`Signature.bind_partial` relied on a default value are skipped. - However, if needed, it is easy to include them. - - :: - - >>> def foo(a, b=10): - ... pass - - >>> sig = signature(foo) - >>> ba = sig.bind(5) - - >>> ba.args, ba.kwargs - ((5,), {}) - - >>> for param in sig.parameters.values(): - ... if (param.name not in ba.arguments - ... and param.default is not param.empty): - ... ba.arguments[param.name] = param.default - - >>> ba.args, ba.kwargs - ((5, 10), {}) - + However, if needed, use :meth:`BoundArguments.apply_defaults` to add + them. .. attribute:: BoundArguments.args @@ -675,6 +757,26 @@ function. A reference to the parent :class:`Signature` object. + .. method:: BoundArguments.apply_defaults() + + Set default values for missing arguments. + + For variable-positional arguments (``*args``) the default is an + empty tuple. + + For variable-keyword arguments (``**kwargs``) the default is an + empty dict. + + :: + + >>> def foo(a, b='ham', *args): pass + >>> ba = inspect.signature(foo).bind('spam') + >>> ba.apply_defaults() + >>> ba.arguments + OrderedDict([('a', 'spam'), ('b', 'ham'), ('args', ())]) + + .. versionadded:: 3.5 + The :attr:`args` and :attr:`kwargs` properties can be used to invoke functions:: @@ -719,8 +821,10 @@ Classes and functions *n* elements listed in *args*. .. deprecated:: 3.0 - Use :func:`getfullargspec` instead, which provides information about - keyword-only arguments and annotations. + Use :func:`signature` and + :ref:`Signature Object <inspect-signature-object>`, which provide a + better introspecting API for callables. This function will be removed + in Python 3.6. .. function:: getfullargspec(func) @@ -741,15 +845,16 @@ Classes and functions The first four items in the tuple correspond to :func:`getargspec`. - .. note:: - Consider using the new :ref:`Signature Object <inspect-signature-object>` - interface, which provides a better way of introspecting functions. - .. versionchanged:: 3.4 This function is now based on :func:`signature`, but still ignores ``__wrapped__`` attributes and includes the already bound first parameter in the signature output for bound methods. + .. deprecated:: 3.5 + Use :func:`signature` and + :ref:`Signature Object <inspect-signature-object>`, which provide a + better introspecting API for callables. + .. function:: getargvalues(frame) @@ -759,6 +864,11 @@ Classes and functions are the names of the ``*`` and ``**`` arguments or ``None``. *locals* is the locals dictionary of the given frame. + .. deprecated:: 3.5 + Use :func:`signature` and + :ref:`Signature Object <inspect-signature-object>`, which provide a + better introspecting API for callables. + .. function:: formatargspec(args[, varargs, varkw, defaults, kwonlyargs, kwonlydefaults, annotations[, formatarg, formatvarargs, formatvarkw, formatvalue, formatreturns, formatannotations]]) @@ -781,6 +891,11 @@ Classes and functions >>> formatargspec(*getfullargspec(f)) '(a: int, b: float)' + .. deprecated:: 3.5 + Use :func:`signature` and + :ref:`Signature Object <inspect-signature-object>`, which provide a + better introspecting API for callables. + .. function:: formatargvalues(args[, varargs, varkw, locals, formatarg, formatvarargs, formatvarkw, formatvalue]) @@ -788,6 +903,11 @@ Classes and functions :func:`getargvalues`. The format\* arguments are the corresponding optional formatting functions that are called to turn names and values into strings. + .. deprecated:: 3.5 + Use :func:`signature` and + :ref:`Signature Object <inspect-signature-object>`, which provide a + better introspecting API for callables. + .. function:: getmro(cls) @@ -822,8 +942,8 @@ Classes and functions .. versionadded:: 3.2 - .. note:: - Consider using the new :meth:`Signature.bind` instead. + .. deprecated:: 3.5 + Use :meth:`Signature.bind` and :meth:`Signature.bind_partial` instead. .. function:: getclosurevars(func) @@ -864,11 +984,17 @@ Classes and functions The interpreter stack --------------------- -When the following functions return "frame records," each record is a tuple of -six items: the frame object, the filename, the line number of the current line, +When the following functions return "frame records," each record is a +:term:`named tuple` +``FrameInfo(frame, filename, lineno, function, code_context, index)``. +The tuple contains the frame object, the filename, the line number of the +current line, the function name, a list of lines of context from the source code, and the index of the current line within that list. +.. versionchanged:: 3.5 + Return a named tuple instead of a tuple. + .. note:: Keeping references to frame objects, as found in the first element of the frame @@ -913,6 +1039,11 @@ line. returned list represents *frame*; the last entry represents the outermost call on *frame*'s stack. + .. versionchanged:: 3.5 + A list of :term:`named tuples <named tuple>` + ``FrameInfo(frame, filename, lineno, function, code_context, index)`` + is returned. + .. function:: getinnerframes(traceback, context=1) @@ -921,6 +1052,11 @@ line. list represents *traceback*; the last entry represents where the exception was raised. + .. versionchanged:: 3.5 + A list of :term:`named tuples <named tuple>` + ``FrameInfo(frame, filename, lineno, function, code_context, index)`` + is returned. + .. function:: currentframe() @@ -940,6 +1076,11 @@ line. returned list represents the caller; the last entry represents the outermost call on the stack. + .. versionchanged:: 3.5 + A list of :term:`named tuples <named tuple>` + ``FrameInfo(frame, filename, lineno, function, code_context, index)`` + is returned. + .. function:: trace(context=1) @@ -948,6 +1089,11 @@ line. entry in the list represents the caller; the last entry represents where the exception was raised. + .. versionchanged:: 3.5 + A list of :term:`named tuples <named tuple>` + ``FrameInfo(frame, filename, lineno, function, code_context, index)`` + is returned. + Fetching attributes statically ------------------------------ @@ -1007,8 +1153,8 @@ code execution:: pass -Current State of a Generator ----------------------------- +Current State of Generators and Coroutines +------------------------------------------ When implementing coroutine schedulers and for other advanced uses of generators, it is useful to determine whether a generator is currently @@ -1028,6 +1174,21 @@ generator to be determined easily. .. versionadded:: 3.2 +.. function:: getcoroutinestate(coroutine) + + Get current state of a coroutine object. The function is intended to be + used with coroutine objects created by :keyword:`async def` functions, but + will accept any coroutine-like object that has ``cr_running`` and + ``cr_frame`` attributes. + + Possible states are: + * CORO_CREATED: Waiting to start execution. + * CORO_RUNNING: Currently being executed by the interpreter. + * CORO_SUSPENDED: Currently suspended at an await expression. + * CORO_CLOSED: Execution has completed. + + .. versionadded:: 3.5 + The current internal state of the generator can also be queried. This is mostly useful for testing purposes, to ensure that internal state is being updated as expected: @@ -1052,6 +1213,13 @@ updated as expected: .. versionadded:: 3.3 +.. function:: getcoroutinelocals(coroutine) + + This function is analogous to :func:`~inspect.getgeneratorlocals`, but + works for coroutine objects created by :keyword:`async def` functions. + + .. versionadded:: 3.5 + .. _inspect-module-cli: diff --git a/Doc/library/io.rst b/Doc/library/io.rst index 9b12045ee7..48fd226130 100644 --- a/Doc/library/io.rst +++ b/Doc/library/io.rst @@ -339,8 +339,11 @@ I/O Base Classes if *size* is not specified). The current stream position isn't changed. This resizing can extend or reduce the current file size. In case of extension, the contents of the new file area depend on the platform - (on most systems, additional bytes are zero-filled, on Windows they're - undetermined). The new file size is returned. + (on most systems, additional bytes are zero-filled). The new file size + is returned. + + .. versionchanged:: 3.5 + Windows will now zero-fill files when extending. .. method:: writable() @@ -391,8 +394,8 @@ I/O Base Classes .. method:: readinto(b) Read up to ``len(b)`` bytes into :class:`bytearray` *b* and return the - number of bytes read. If the object is in non-blocking mode and no - bytes are available, ``None`` is returned. + number of bytes read. If the object is in non-blocking mode and no bytes + are available, ``None`` is returned. .. method:: write(b) @@ -465,9 +468,10 @@ I/O Base Classes .. method:: read1(size=-1) - Read and return up to *size* bytes, with at most one call to the underlying - raw stream's :meth:`~RawIOBase.read` method. This can be useful if you - are implementing your own buffering on top of a :class:`BufferedIOBase` + Read and return up to *size* bytes, with at most one call to the + underlying raw stream's :meth:`~RawIOBase.read` (or + :meth:`~RawIOBase.readinto`) method. This can be useful if you are + implementing your own buffering on top of a :class:`BufferedIOBase` object. .. method:: readinto(b) @@ -478,8 +482,19 @@ I/O Base Classes Like :meth:`read`, multiple reads may be issued to the underlying raw stream, unless the latter is interactive. - A :exc:`BlockingIOError` is raised if the underlying raw stream is in - non blocking-mode, and has no data available at the moment. + A :exc:`BlockingIOError` is raised if the underlying raw stream is in non + blocking-mode, and has no data available at the moment. + + .. method:: readinto1(b) + + Read up to ``len(b)`` bytes into bytearray *b*, using at most one call to + the underlying raw stream's :meth:`~RawIOBase.read` (or + :meth:`~RawIOBase.readinto`) method. Return the number of bytes read. + + A :exc:`BlockingIOError` is raised if the underlying raw stream is in non + blocking-mode, and has no data available at the moment. + + .. versionadded:: 3.5 .. method:: write(b) @@ -507,9 +522,12 @@ Raw File I/O The *name* can be one of two things: * a character string or :class:`bytes` object representing the path to the - file which will be opened; + file which will be opened. In this case closefd must be True (the default) + otherwise an error will be raised. * an integer representing the number of an existing OS-level file descriptor - to which the resulting :class:`FileIO` object will give access. + to which the resulting :class:`FileIO` object will give access. When the + FileIO object is closed this fd will be closed as well, unless *closefd* + is set to ``False``. The *mode* can be ``'r'``, ``'w'``, ``'x'`` or ``'a'`` for reading (default), writing, exclusive creation or appending. The file will be @@ -598,6 +616,11 @@ than raw I/O does. In :class:`BytesIO`, this is the same as :meth:`read`. + .. method:: readinto1() + + In :class:`BytesIO`, this is the same as :meth:`readinto`. + + .. versionadded:: 3.5 .. class:: BufferedReader(raw, buffer_size=DEFAULT_BUFFER_SIZE) @@ -808,11 +831,13 @@ Text I/O exception if there is an encoding error (the default of ``None`` has the same effect), or pass ``'ignore'`` to ignore errors. (Note that ignoring encoding errors can lead to data loss.) ``'replace'`` causes a replacement marker - (such as ``'?'``) to be inserted where there is malformed data. When - writing, ``'xmlcharrefreplace'`` (replace with the appropriate XML character - reference) or ``'backslashreplace'`` (replace with backslashed escape - sequences) can be used. Any other error handling name that has been - registered with :func:`codecs.register_error` is also valid. + (such as ``'?'``) to be inserted where there is malformed data. + ``'backslashreplace'`` causes malformed data to be replaced by a + backslashed escape sequence. When writing, ``'xmlcharrefreplace'`` + (replace with the appropriate XML character reference) or ``'namereplace'`` + (replace with ``\N{...}`` escape sequences) can be used. Any other error + handling name that has been registered with + :func:`codecs.register_error` is also valid. .. index:: single: universal newlines; io.TextIOWrapper class diff --git a/Doc/library/ipaddress.rst b/Doc/library/ipaddress.rst index 301048ef6a..90fcc748e7 100644 --- a/Doc/library/ipaddress.rst +++ b/Doc/library/ipaddress.rst @@ -146,6 +146,20 @@ write code that handles both IP versions correctly. the appropriate length (most significant octet first). This is 4 bytes for IPv4 and 16 bytes for IPv6. + .. attribute:: reverse_pointer + + The name of the reverse DNS PTR record for the IP address, e.g.:: + + >>> ipaddress.ip_address("127.0.0.1").reverse_pointer + '1.0.0.127.in-addr.arpa' + >>> ipaddress.ip_address("2001:db8::1").reverse_pointer + '1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.b.d.0.1.0.0.2.ip6.arpa' + + This is the name that could be used for performing a PTR lookup, not the + resolved hostname itself. + + .. versionadded:: 3.5 + .. attribute:: is_multicast ``True`` if the address is reserved for multicast use. See @@ -226,6 +240,7 @@ write code that handles both IP versions correctly. :class:`IPv4Address` class: .. attribute:: packed + .. attribute:: reverse_pointer .. attribute:: version .. attribute:: max_prefixlen .. attribute:: is_multicast @@ -377,6 +392,12 @@ so to avoid duplication they are only documented for :class:`IPv4Network`. 3. An integer packed into a :class:`bytes` object of length 4, big-endian. The interpretation is similar to an integer *address*. + 4. A two-tuple of an address description and a netmask, where the address + description is either a string, a 32-bits integer, a 4-bytes packed + integer, or an existing IPv4Address object; and the netmask is either + an integer representing the prefix length (e.g. ``24``) or a string + representing the prefix mask (e.g. ``255.255.255.0``). + An :exc:`AddressValueError` is raised if *address* is not a valid IPv4 address. A :exc:`NetmaskValueError` is raised if the mask is not valid for an IPv4 address. @@ -389,6 +410,10 @@ so to avoid duplication they are only documented for :class:`IPv4Network`. objects will raise :exc:`TypeError` if the argument's IP version is incompatible to ``self`` + .. versionchanged:: 3.5 + + Added the two-tuple form for the *address* constructor parameter. + .. attribute:: version .. attribute:: max_prefixlen @@ -553,6 +578,11 @@ so to avoid duplication they are only documented for :class:`IPv4Network`. 3. An integer packed into a :class:`bytes` object of length 16, big-endian. The interpretation is similar to an integer *address*. + 4. A two-tuple of an address description and a netmask, where the address + description is either a string, a 128-bits integer, a 16-bytes packed + integer, or an existing IPv4Address object; and the netmask is an + integer representing the prefix length. + An :exc:`AddressValueError` is raised if *address* is not a valid IPv6 address. A :exc:`NetmaskValueError` is raised if the mask is not valid for an IPv6 address. @@ -561,6 +591,10 @@ so to avoid duplication they are only documented for :class:`IPv4Network`. then :exc:`ValueError` is raised. Otherwise, the host bits are masked out to determine the appropriate network address. + .. versionchanged:: 3.5 + + Added the two-tuple form for the *address* constructor parameter. + .. attribute:: version .. attribute:: max_prefixlen .. attribute:: is_multicast diff --git a/Doc/library/itertools.rst b/Doc/library/itertools.rst index f489535c53..8c7592d17e 100644 --- a/Doc/library/itertools.rst +++ b/Doc/library/itertools.rst @@ -87,10 +87,15 @@ loops that truncate the stream. .. function:: accumulate(iterable[, func]) - Make an iterator that returns accumulated sums. Elements may be any addable - type including :class:`~decimal.Decimal` or :class:`~fractions.Fraction`. - If the optional *func* argument is supplied, it should be a function of two - arguments and it will be used instead of addition. + Make an iterator that returns accumulated sums, or accumulated + results of other binary functions (specified via the optional + *func* argument). If *func* is supplied, it should be a function + of two arguments. Elements of the input *iterable* may be any type + that can be accepted as arguments to *func*. (For example, with + the default operation of addition, elements may be any addable + type including :class:`~decimal.Decimal` or + :class:`~fractions.Fraction`.) If the input iterable is empty, the + output iterable will also be empty. Equivalent to:: @@ -99,7 +104,10 @@ loops that truncate the stream. # accumulate([1,2,3,4,5]) --> 1 3 6 10 15 # accumulate([1,2,3,4,5], operator.mul) --> 1 2 6 24 120 it = iter(iterable) - total = next(it) + try: + total = next(it) + except StopIteration: + return yield total for element in it: total = func(total, element) @@ -400,7 +408,10 @@ loops that truncate the stream. def _grouper(self, tgtkey): while self.currkey == tgtkey: yield self.currvalue - self.currvalue = next(self.it) # Exit on StopIteration + try: + self.currvalue = next(self.it) + except StopIteration: + return self.currkey = self.keyfunc(self.currvalue) @@ -424,7 +435,10 @@ loops that truncate the stream. # islice('ABCDEFG', 0, None, 2) --> A C E G s = slice(*args) it = iter(range(s.start or 0, s.stop or sys.maxsize, s.step or 1)) - nexti = next(it) + try: + nexti = next(it) + except StopIteration: + return for i, element in enumerate(iterable): if i == nexti: yield element @@ -582,7 +596,10 @@ loops that truncate the stream. def gen(mydeque): while True: if not mydeque: # when the local deque is empty - newval = next(it) # fetch a new value and + try: + newval = next(it) # fetch a new value and + except StopIteration: + return for d in deques: # load it to all the deques d.append(newval) yield mydeque.popleft() @@ -657,6 +674,11 @@ which incur interpreter overhead. "Return function(0), function(1), ..." return map(function, count(start)) + def tail(n, iterable): + "Return an iterator over the last n items" + # tail(3, 'ABCDEFG') --> E F G + return iter(collections.deque(iterable, maxlen=n)) + def consume(iterator, n): "Advance the iterator n-steps ahead. If n is none, consume entirely." # Use functions that consume iterators at C speed. diff --git a/Doc/library/json.rst b/Doc/library/json.rst index 6f5f8b15a8..49bb0905e0 100644 --- a/Doc/library/json.rst +++ b/Doc/library/json.rst @@ -106,6 +106,8 @@ Using json.tool from the shell to validate and pretty-print:: $ echo '{1.2:3.4}' | python -m json.tool Expecting property name enclosed in double quotes: line 1 column 2 (char 1) +See :ref:`json-commandline` for detailed documentation. + .. highlight:: python3 .. note:: @@ -248,7 +250,7 @@ Basic Usage will be passed to the constructor of the class. If the data being deserialized is not a valid JSON document, a - :exc:`ValueError` will be raised. + :exc:`JSONDecodeError` will be raised. .. function:: loads(s, encoding=None, cls=None, object_hook=None, parse_float=None, parse_int=None, parse_constant=None, object_pairs_hook=None, **kw) @@ -259,7 +261,7 @@ Basic Usage *encoding* which is ignored and deprecated. If the data being deserialized is not a valid JSON document, a - :exc:`ValueError` will be raised. + :exc:`JSONDecodeError` will be raised. Encoders and Decoders --------------------- @@ -332,13 +334,16 @@ Encoders and Decoders ``'\n'``, ``'\r'`` and ``'\0'``. If the data being deserialized is not a valid JSON document, a - :exc:`ValueError` will be raised. + :exc:`JSONDecodeError` will be raised. .. method:: decode(s) Return the Python representation of *s* (a :class:`str` instance containing a JSON document) + :exc:`JSONDecodeError` will be raised if the given JSON document is not + valid. + .. method:: raw_decode(s) Decode a JSON document from *s* (a :class:`str` beginning with a @@ -467,6 +472,36 @@ Encoders and Decoders mysocket.write(chunk) +Exceptions +---------- + +.. exception:: JSONDecodeError(msg, doc, pos, end=None) + + Subclass of :exc:`ValueError` with the following additional attributes: + + .. attribute:: msg + + The unformatted error message. + + .. attribute:: doc + + The JSON document being parsed. + + .. attribute:: pos + + The start index of *doc* where parsing failed. + + .. attribute:: lineno + + The line corresponding to *pos*. + + .. attribute:: colno + + The column corresponding to *pos*. + + .. versionadded:: 3.5 + + Standard Compliance and Interoperability ---------------------------------------- @@ -588,6 +623,68 @@ when serializing Python :class:`int` values of extremely large magnitude, or when serializing instances of "exotic" numerical types such as :class:`decimal.Decimal`. +.. highlight:: bash +.. module:: json.tool + +.. _json-commandline: + +Command Line Interface +---------------------- + +The :mod:`json.tool` module provides a simple command line interface to validate +and pretty-print JSON objects. + +If the optional ``infile`` and ``outfile`` arguments are not +specified, :attr:`sys.stdin` and :attr:`sys.stdout` will be used respectively:: + + $ echo '{"json": "obj"}' | python -m json.tool + { + "json": "obj" + } + $ echo '{1.2:3.4}' | python -m json.tool + Expecting property name enclosed in double quotes: line 1 column 2 (char 1) + +.. versionchanged:: 3.5 + The output is now in the same order as the input. Use the + :option:`--sort-keys` option to sort the output of dictionaries + alphabetically by key. + +Command line options +^^^^^^^^^^^^^^^^^^^^ + +.. cmdoption:: infile + + The JSON file to be validated or pretty-printed:: + + $ python -m json.tool mp_films.json + [ + { + "title": "And Now for Something Completely Different", + "year": 1971 + }, + { + "title": "Monty Python and the Holy Grail", + "year": 1975 + } + ] + + If *infile* is not specified, read from :attr:`sys.stdin`. + +.. cmdoption:: outfile + + Write the output of the *infile* to the given *outfile*. Otherwise, write it + to :attr:`sys.stdout`. + +.. cmdoption:: --sort-keys + + Sort the output of dictionaries alphabetically by key. + + .. versionadded:: 3.5 + +.. cmdoption:: -h, --help + + Show the help message. + .. rubric:: Footnotes diff --git a/Doc/library/linecache.rst b/Doc/library/linecache.rst index f18b1cdac0..6c92cc58d2 100644 --- a/Doc/library/linecache.rst +++ b/Doc/library/linecache.rst @@ -47,6 +47,14 @@ The :mod:`linecache` module defines the following functions: changed on disk, and you require the updated version. If *filename* is omitted, it will check all the entries in the cache. +.. function:: lazycache(filename, module_globals) + + Capture enough detail about a non-file based module to permit getting its + lines later via :func:`getline` even if *module_globals* is None in the later + call. This avoids doing I/O until a line is actually needed, without having + to carry the module globals around indefinitely. + + .. versionadded:: 3.5 Example:: diff --git a/Doc/library/locale.rst b/Doc/library/locale.rst index 9600193547..bc7f5f932b 100644 --- a/Doc/library/locale.rst +++ b/Doc/library/locale.rst @@ -387,6 +387,14 @@ The :mod:`locale` module defines the following exception and functions: ``str(float)``, but takes the decimal point into account. +.. function:: delocalize(string) + + Converts a string into a normalized number string, following the + :const:`LC_NUMERIC` settings. + + .. versionadded:: 3.5 + + .. function:: atof(string) Converts a string to a floating point number, following the :const:`LC_NUMERIC` diff --git a/Doc/library/logging.handlers.rst b/Doc/library/logging.handlers.rst index d946892ecc..67403a9bc6 100644 --- a/Doc/library/logging.handlers.rst +++ b/Doc/library/logging.handlers.rst @@ -853,7 +853,7 @@ supports sending logging messages to a Web server, using either ``GET`` or credentials, you should also specify secure=True so that your userid and password are not passed in cleartext across the wire. - .. versionchanged:: 3.4.3 + .. versionchanged:: 3.5 The *context* parameter was added. .. method:: mapLogRecord(record) @@ -953,13 +953,20 @@ applications where threads servicing clients need to respond as quickly as possible, while any potentially slow operations (such as sending an email via :class:`SMTPHandler`) are done on a separate thread. -.. class:: QueueListener(queue, *handlers) +.. class:: QueueListener(queue, *handlers, respect_handler_level=False) Returns a new instance of the :class:`QueueListener` class. The instance is initialized with the queue to send messages to and a list of handlers which will handle entries placed on the queue. The queue can be any queue- like object; it's passed as-is to the :meth:`dequeue` method, which needs - to know how to get messages from it. + to know how to get messages from it. If ``respect_handler_level`` is ``True``, + a handler's level is respected (compared with the level for the message) when + deciding whether to pass messages to that handler; otherwise, the behaviour + is as in previous Python versions - to always pass each message to each + handler. + + .. versionchanged:: 3.5 + The ``respect_handler_levels`` argument was added. .. method:: dequeue(block) diff --git a/Doc/library/logging.rst b/Doc/library/logging.rst index 90fb9b06d1..bf821abac1 100644 --- a/Doc/library/logging.rst +++ b/Doc/library/logging.rst @@ -159,11 +159,13 @@ is the module's name in the Python package namespace. *msg* using the string formatting operator. (Note that this means that you can use keywords in the format string, together with a single dictionary argument.) - There are three keyword arguments in *kwargs* which are inspected: *exc_info* - which, if it does not evaluate as false, causes exception information to be + There are three keyword arguments in *kwargs* which are inspected: + *exc_info*, *stack_info*, and *extra*. + + If *exc_info* does not evaluate as false, it causes exception information to be added to the logging message. If an exception tuple (in the format returned by - :func:`sys.exc_info`) is provided, it is used; otherwise, :func:`sys.exc_info` - is called to get the exception information. + :func:`sys.exc_info`) or an exception instance is provided, it is used; + otherwise, :func:`sys.exc_info` is called to get the exception information. The second optional keyword argument is *stack_info*, which defaults to ``False``. If true, stack information is added to the logging @@ -220,6 +222,9 @@ is the module's name in the Python package namespace. .. versionadded:: 3.2 The *stack_info* parameter was added. + .. versionchanged:: 3.5 + The *exc_info* parameter can now accept exception instances. + .. method:: Logger.info(msg, *args, **kwargs) diff --git a/Doc/library/lzma.rst b/Doc/library/lzma.rst index b71051da85..054600530b 100644 --- a/Doc/library/lzma.rst +++ b/Doc/library/lzma.rst @@ -110,6 +110,10 @@ Reading and writing compressed files .. versionchanged:: 3.4 Added support for the ``"x"`` and ``"xb"`` modes. + .. versionchanged:: 3.5 + The :meth:`~io.BufferedIOBase.read` method now accepts an argument of + ``None``. + Compressing and decompressing data in memory -------------------------------------------- @@ -221,13 +225,32 @@ Compressing and decompressing data in memory decompress a multi-stream input with :class:`LZMADecompressor`, you must create a new decompressor for each stream. - .. method:: decompress(data) + .. method:: decompress(data, max_length=-1) + + Decompress *data* (a :term:`bytes-like object`), returning + uncompressed data as bytes. Some of *data* may be buffered + internally, for use in later calls to :meth:`decompress`. The + returned data should be concatenated with the output of any + previous calls to :meth:`decompress`. + + If *max_length* is nonnegative, returns at most *max_length* + bytes of decompressed data. If this limit is reached and further + output can be produced, the :attr:`~.needs_input` attribute will + be set to ``False``. In this case, the next call to + :meth:`~.decompress` may provide *data* as ``b''`` to obtain + more of the output. + + If all of the input data was decompressed and returned (either + because this was less than *max_length* bytes, or because + *max_length* was negative), the :attr:`~.needs_input` attribute + will be set to ``True``. - Decompress *data* (a :class:`bytes` object), returning a :class:`bytes` - object containing the decompressed data for at least part of the input. - Some of *data* may be buffered internally, for use in later calls to - :meth:`decompress`. The returned data should be concatenated with the - output of any previous calls to :meth:`decompress`. + Attempting to decompress data after the end of stream is reached + raises an `EOFError`. Any data found after the end of the + stream is ignored and saved in the :attr:`~.unused_data` attribute. + + .. versionchanged:: 3.5 + Added the *max_length* parameter. .. attribute:: check @@ -245,6 +268,12 @@ Compressing and decompressing data in memory Before the end of the stream is reached, this will be ``b""``. + .. attribute:: needs_input + + ``False`` if the :meth:`.decompress` method can provide more + decompressed data before requiring new uncompressed input. + + .. versionadded:: 3.5 .. function:: compress(data, format=FORMAT_XZ, check=-1, preset=None, filters=None) diff --git a/Doc/library/math.rst b/Doc/library/math.rst index 0fc7c7c9b9..244663eda9 100644 --- a/Doc/library/math.rst +++ b/Doc/library/math.rst @@ -100,6 +100,48 @@ Number-theoretic and representation functions <http://code.activestate.com/recipes/393090/>`_\. +.. function:: gcd(a, b) + + Return the greatest common divisor of the integers *a* and *b*. If either + *a* or *b* is nonzero, then the value of ``gcd(a, b)`` is the largest + positive integer that divides both *a* and *b*. ``gcd(0, 0)`` returns + ``0``. + + .. versionadded:: 3.5 + + +.. function:: isclose(a, b, *, rel_tol=1e-09, abs_tol=0.0) + + Return ``True`` if the values *a* and *b* are close to each other and + ``False`` otherwise. + + Whether or not two values are considered close is determined according to + given absolute and relative tolerances. + + *rel_tol* is the relative tolerance -- it is the maximum allowed difference + between *a* and *b*, relative to the larger absolute value of *a* or *b*. + For example, to set a tolerance of 5%, pass ``rel_tol=0.05``. The default + tolerance is ``1e-09``, which assures that the two values are the same + within about 9 decimal digits. *rel_tol* must be greater than zero. + + *abs_tol* is the minimum absolute tolerance -- useful for comparisons near + zero. *abs_tol* must be at least zero. + + If no errors occur, the result will be: + ``abs(a-b) <= max(rel_tol * max(abs(a), abs(b)), abs_tol)``. + + The IEEE 754 special values of ``NaN``, ``inf``, and ``-inf`` will be + handled according to IEEE rules. Specifically, ``NaN`` is not considered + close to any other value, including ``NaN``. ``inf`` and ``-inf`` are only + considered close to themselves. + + .. versionadded:: 3.5 + + .. seealso:: + + :pep:`485` -- A function for testing approximate equality + + .. function:: isfinite(x) Return ``True`` if *x* is neither an infinity nor a NaN, and @@ -383,6 +425,22 @@ Constants The mathematical constant e = 2.718281..., to available precision. +.. data:: inf + + A floating-point positive infinity. (For negative infinity, use + ``-math.inf``.) Equivalent to the output of ``float('inf')``. + + .. versionadded:: 3.5 + + +.. data:: nan + + A floating-point "not a number" (NaN) value. Equivalent to the output of + ``float('nan')``. + + .. versionadded:: 3.5 + + .. impl-detail:: The :mod:`math` module consists mostly of thin wrappers around the platform C diff --git a/Doc/library/mmap.rst b/Doc/library/mmap.rst index 18e05e31f7..b74a8231ea 100644 --- a/Doc/library/mmap.rst +++ b/Doc/library/mmap.rst @@ -174,6 +174,9 @@ To map anonymous memory, -1 should be passed as the fileno along with the length Optional arguments *start* and *end* are interpreted as in slice notation. Returns ``-1`` on failure. + .. versionchanged: 3.5 + Writable :term:`bytes-like object` is now accepted. + .. method:: flush([offset[, size]]) @@ -234,6 +237,9 @@ To map anonymous memory, -1 should be passed as the fileno along with the length Optional arguments *start* and *end* are interpreted as in slice notation. Returns ``-1`` on failure. + .. versionchanged: 3.5 + Writable :term:`bytes-like object` is now accepted. + .. method:: seek(pos[, whence]) @@ -261,6 +267,9 @@ To map anonymous memory, -1 should be passed as the fileno along with the length were written. If the mmap was created with :const:`ACCESS_READ`, then writing to it will raise a :exc:`TypeError` exception. + .. versionchanged: 3.5 + Writable :term:`bytes-like object` is now accepted. + .. method:: write_byte(byte) diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst index 82de19b5c8..a4d42abb34 100644 --- a/Doc/library/multiprocessing.rst +++ b/Doc/library/multiprocessing.rst @@ -1458,6 +1458,9 @@ processes. Note that accessing the ctypes object through the wrapper can be a lot slower than accessing the raw ctypes object. + .. versionchanged:: 3.5 + Synchronized objects support the :term:`context manager` protocol. + The table below compares the syntax for creating shared ctypes objects from shared memory with the normal ctypes syntax. (In the table ``MyStruct`` is some @@ -1945,9 +1948,9 @@ itself. This means, for example, that one shared object can contain a second: >>> l = manager.list(range(10)) >>> l._callmethod('__len__') 10 - >>> l._callmethod('__getitem__', (slice(2, 7),)) # equiv to `l[2:7]` + >>> l._callmethod('__getitem__', (slice(2, 7),)) # equivalent to l[2:7] [2, 3, 4, 5, 6] - >>> l._callmethod('__getitem__', (20,)) # equiv to `l[20]` + >>> l._callmethod('__getitem__', (20,)) # equivalent to l[20] Traceback (most recent call last): ... IndexError: list index out of range diff --git a/Doc/library/operator.rst b/Doc/library/operator.rst index f9e2a3d003..c01e63b77a 100644 --- a/Doc/library/operator.rst +++ b/Doc/library/operator.rst @@ -138,6 +138,14 @@ The mathematical and bitwise operations are the most numerous: Return ``a * b``, for *a* and *b* numbers. +.. function:: matmul(a, b) + __matmul__(a, b) + + Return ``a @ b``. + + .. versionadded:: 3.5 + + .. function:: neg(obj) __neg__(obj) @@ -391,6 +399,8 @@ Python syntax and the functions in the :mod:`operator` module. +-----------------------+-------------------------+---------------------------------------+ | Multiplication | ``a * b`` | ``mul(a, b)`` | +-----------------------+-------------------------+---------------------------------------+ +| Matrix Multiplication | ``a @ b`` | ``matmul(a, b)`` | ++-----------------------+-------------------------+---------------------------------------+ | Negation (Arithmetic) | ``- a`` | ``neg(a)`` | +-----------------------+-------------------------+---------------------------------------+ | Negation (Logical) | ``not a`` | ``not_(a)`` | @@ -499,6 +509,14 @@ will perform the update, so no subsequent assignment is necessary: ``a = imul(a, b)`` is equivalent to ``a *= b``. +.. function:: imatmul(a, b) + __imatmul__(a, b) + + ``a = imatmul(a, b)`` is equivalent to ``a @= b``. + + .. versionadded:: 3.5 + + .. function:: ior(a, b) __ior__(a, b) diff --git a/Doc/library/os.path.rst b/Doc/library/os.path.rst index 92631b2e41..a3fe73c152 100644 --- a/Doc/library/os.path.rst +++ b/Doc/library/os.path.rst @@ -66,11 +66,37 @@ the :mod:`glob` module.) empty string (``''``). +.. function:: commonpath(paths) + + Return the longest common sub-path of each pathname in the sequence + *paths*. Raise ValueError if *paths* contains both absolute and relative + pathnames, or if *paths* is empty. Unlike :func:`commonprefix`, this + returns a valid path. + + Availability: Unix, Windows + + .. versionadded:: 3.5 + + .. function:: commonprefix(list) - Return the longest path prefix (taken character-by-character) that is a prefix - of all paths in *list*. If *list* is empty, return the empty string (``''``). - Note that this may return invalid paths because it works a character at a time. + Return the longest path prefix (taken character-by-character) that is a + prefix of all paths in *list*. If *list* is empty, return the empty string + (``''``). + + .. note:: + + This function may return invalid paths because it works a + character at a time. To obtain a valid path, see + :func:`commonpath`. + + :: + + >>> os.path.commonprefix(['/usr/lib', '/usr/local/lib']) + '/usr/l' + + >>> os.path.commonpath(['/usr/lib', '/usr/local/lib']) + '/usr' .. function:: dirname(path) diff --git a/Doc/library/os.rst b/Doc/library/os.rst index 916f5e56cf..30f1605c10 100644 --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -78,9 +78,10 @@ uses the file system encoding to perform this conversion (see .. versionchanged:: 3.1 On some systems, conversion using the file system encoding may fail. In this - case, Python uses the ``surrogateescape`` encoding error handler, which means - that undecodable bytes are replaced by a Unicode character U+DCxx on - decoding, and these are again translated to the original byte on encoding. + case, Python uses the :ref:`surrogateescape encoding error handler + <surrogateescape>`, which means that undecodable bytes are replaced by a + Unicode character U+DCxx on decoding, and these are again translated to the + original byte on encoding. The file system encoding must guarantee to successfully decode all bytes @@ -311,8 +312,6 @@ process and user. Return the current process id. - Availability: Unix, Windows. - .. function:: getppid() @@ -549,8 +548,6 @@ process and user. On platforms where :c:func:`strerror` returns ``NULL`` when given an unknown error number, :exc:`ValueError` is raised. - Availability: Unix, Windows. - .. data:: supports_bytes_environ @@ -564,8 +561,6 @@ process and user. Set the current numeric umask and return the previous umask. - Availability: Unix, Windows. - .. function:: uname() @@ -656,8 +651,6 @@ as internal buffering of data. Close file descriptor *fd*. - Availability: Unix, Windows. - .. note:: This function is intended for low-level I/O and must be applied to a file @@ -677,8 +670,6 @@ as internal buffering of data. except OSError: pass - Availability: Unix, Windows. - .. function:: device_encoding(fd) @@ -695,8 +686,6 @@ as internal buffering of data. 2: stderr), the new file descriptor is :ref:`inheritable <fd_inheritance>`. - Availability: Unix, Windows. - .. versionchanged:: 3.4 The new file descriptor is now non-inheritable. @@ -707,8 +696,6 @@ as internal buffering of data. The file descriptor *fd2* is :ref:`inheritable <fd_inheritance>` by default, or non-inheritable if *inheritable* is ``False``. - Availability: Unix, Windows. - .. versionchanged:: 3.4 Add the optional *inheritable* parameter. @@ -774,8 +761,6 @@ as internal buffering of data. The :func:`.stat` function. - Availability: Unix, Windows. - .. function:: fstatvfs(fd) @@ -804,8 +789,21 @@ as internal buffering of data. most *length* bytes in size. As of Python 3.3, this is equivalent to ``os.truncate(fd, length)``. + Availability: Unix, Windows. + + .. versionchanged:: 3.5 + Added support for Windows + +.. function:: get_blocking(fd) + + Get the blocking mode of the file descriptor: ``False`` if the + :data:`O_NONBLOCK` flag is set, ``True`` if the flag is cleared. + + See also :func:`set_blocking` and :meth:`socket.socket.setblocking`. + Availability: Unix. + .. versionadded:: 3.5 .. function:: isatty(fd) @@ -846,8 +844,6 @@ as internal buffering of data. current position; :const:`SEEK_END` or ``2`` to set it relative to the end of the file. Return the new cursor position in bytes, starting from the beginning. - Availability: Unix, Windows. - .. data:: SEEK_SET SEEK_CUR @@ -856,8 +852,6 @@ as internal buffering of data. Parameters to the :func:`lseek` function. Their values are 0, 1, and 2, respectively. - Availability: Unix, Windows. - .. versionadded:: 3.3 Some operating systems could support additional values, like :data:`os.SEEK_HOLE` or :data:`os.SEEK_DATA`. @@ -878,8 +872,6 @@ as internal buffering of data. This function can support :ref:`paths relative to directory descriptors <dir_fd>` with the *dir_fd* parameter. - Availability: Unix, Windows. - .. versionchanged:: 3.4 The new file descriptor is now non-inheritable. @@ -893,6 +885,11 @@ as internal buffering of data. .. versionadded:: 3.3 The *dir_fd* argument. + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise an + exception, the function now retries the system call instead of raising an + :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + The following constants are options for the *flags* parameter to the :func:`~os.open` function. They can be combined using the bitwise OR operator ``|``. Some of them are not available on all platforms. For descriptions of @@ -1060,8 +1057,6 @@ or `the MSDN <http://msdn.microsoft.com/en-us/library/z0kc8e3z.aspx>`_ on Window bytes read. If the end of the file referred to by *fd* has been reached, an empty bytes object is returned. - Availability: Unix, Windows. - .. note:: This function is intended for low-level I/O and must be applied to a file @@ -1070,6 +1065,11 @@ or `the MSDN <http://msdn.microsoft.com/en-us/library/z0kc8e3z.aspx>`_ on Window :func:`popen` or :func:`fdopen`, or :data:`sys.stdin`, use its :meth:`~file.read` or :meth:`~file.readline` methods. + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise an + exception, the function now retries the system call instead of raising an + :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + .. function:: sendfile(out, in, offset, count) sendfile(out, in, offset, count, [headers], [trailers], flags=0) @@ -1099,9 +1099,26 @@ or `the MSDN <http://msdn.microsoft.com/en-us/library/z0kc8e3z.aspx>`_ on Window Availability: Unix. + .. note:: + + For a higher-level wrapper of :func:`sendfile`, see + :meth:`socket.socket.sendfile`. + .. versionadded:: 3.3 +.. function:: set_blocking(fd, blocking) + + Set the blocking mode of the specified file descriptor. Set the + :data:`O_NONBLOCK` flag if blocking is ``False``, clear the flag otherwise. + + See also :func:`get_blocking` and :meth:`socket.socket.setblocking`. + + Availability: Unix. + + .. versionadded:: 3.5 + + .. data:: SF_NODISKIO SF_MNOWAIT SF_SYNC @@ -1158,8 +1175,6 @@ or `the MSDN <http://msdn.microsoft.com/en-us/library/z0kc8e3z.aspx>`_ on Window Write the bytestring in *str* to file descriptor *fd*. Return the number of bytes actually written. - Availability: Unix, Windows. - .. note:: This function is intended for low-level I/O and must be applied to a file @@ -1168,6 +1183,11 @@ or `the MSDN <http://msdn.microsoft.com/en-us/library/z0kc8e3z.aspx>`_ on Window :func:`fdopen`, or :data:`sys.stdout` or :data:`sys.stderr`, use its :meth:`~file.write` method. + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise an + exception, the function now retries the system call instead of raising an + :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + .. function:: writev(fd, buffers) @@ -1330,8 +1350,6 @@ features: or not it is available using :data:`os.supports_effective_ids`. If it is unavailable, using it will raise a :exc:`NotImplementedError`. - Availability: Unix, Windows. - .. note:: Using :func:`access` to check if a user is authorized to e.g. open a file @@ -1384,8 +1402,6 @@ features: This function can support :ref:`specifying a file descriptor <path_fd>`. The descriptor must refer to an opened directory, not an open file. - Availability: Unix, Windows. - .. versionadded:: 3.3 Added support for specifying *path* as a file descriptor on some platforms. @@ -1447,8 +1463,6 @@ features: :ref:`paths relative to directory descriptors <dir_fd>` and :ref:`not following symlinks <follow_symlinks>`. - Availability: Unix, Windows. - .. note:: Although Windows supports :func:`chmod`, you can only set the file's @@ -1499,15 +1513,11 @@ features: Return a string representing the current working directory. - Availability: Unix, Windows. - .. function:: getcwdb() Return a bytestring representing the current working directory. - Availability: Unix, Windows. - .. function:: lchflags(path, flags) @@ -1570,7 +1580,11 @@ features: .. note:: To encode ``str`` filenames to ``bytes``, use :func:`~os.fsencode`. - Availability: Unix, Windows. + .. seealso:: + + The :func:`scandir` function returns directory entries along with + file attribute information, giving better performance for many + common use cases. .. versionchanged:: 3.2 The *path* parameter became optional. @@ -1619,8 +1633,6 @@ features: It is also possible to create temporary directories; see the :mod:`tempfile` module's :func:`tempfile.mkdtemp` function. - Availability: Unix, Windows. - .. versionadded:: 3.3 The *dir_fd* argument. @@ -1779,8 +1791,6 @@ features: This function is identical to :func:`unlink`. - Availability: Unix, Windows. - .. versionadded:: 3.3 The *dir_fd* argument. @@ -1814,8 +1824,6 @@ features: If you want cross-platform overwriting of the destination, use :func:`replace`. - Availability: Unix, Windows. - .. versionadded:: 3.3 The *src_dir_fd* and *dst_dir_fd* arguments. @@ -1844,8 +1852,6 @@ features: This function can support specifying *src_dir_fd* and/or *dst_dir_fd* to supply :ref:`paths relative to directory descriptors <dir_fd>`. - Availability: Unix, Windows. - .. versionadded:: 3.3 @@ -1858,12 +1864,180 @@ features: This function can support :ref:`paths relative to directory descriptors <dir_fd>`. - Availability: Unix, Windows. - .. versionadded:: 3.3 The *dir_fd* parameter. +.. function:: scandir(path='.') + + Return an iterator of :class:`DirEntry` objects corresponding to the entries + in the directory given by *path*. The entries are yielded in arbitrary + order, and the special entries ``'.'`` and ``'..'`` are not included. + + Using :func:`scandir` instead of :func:`listdir` can significantly + increase the performance of code that also needs file type or file + attribute information, because :class:`DirEntry` objects expose this + information if the operating system provides it when scanning a directory. + All :class:`DirEntry` methods may perform a system call, but + :func:`~DirEntry.is_dir` and :func:`~DirEntry.is_file` usually only + require a system call for symbolic links; :func:`DirEntry.stat` + always requires a system call on Unix but only requires one for + symbolic links on Windows. + + On Unix, *path* can be of type :class:`str` or :class:`bytes` (use + :func:`~os.fsencode` and :func:`~os.fsdecode` to encode and decode + :class:`bytes` paths). On Windows, *path* must be of type :class:`str`. + On both sytems, the type of the :attr:`~DirEntry.name` and + :attr:`~DirEntry.path` attributes of each :class:`DirEntry` will be of + the same type as *path*. + + The following example shows a simple use of :func:`scandir` to display all + the files (excluding directories) in the given *path* that don't start with + ``'.'``. The ``entry.is_file()`` call will generally not make an additional + system call:: + + for entry in os.scandir(path): + if not entry.name.startswith('.') and entry.is_file(): + print(entry.name) + + .. note:: + + On Unix-based systems, :func:`scandir` uses the system's + `opendir() <http://pubs.opengroup.org/onlinepubs/009695399/functions/opendir.html>`_ + and + `readdir() <http://pubs.opengroup.org/onlinepubs/009695399/functions/readdir_r.html>`_ + functions. On Windows, it uses the Win32 + `FindFirstFileW <http://msdn.microsoft.com/en-us/library/windows/desktop/aa364418(v=vs.85).aspx>`_ + and + `FindNextFileW <http://msdn.microsoft.com/en-us/library/windows/desktop/aa364428(v=vs.85).aspx>`_ + functions. + + .. versionadded:: 3.5 + + +.. class:: DirEntry + + Object yielded by :func:`scandir` to expose the file path and other file + attributes of a directory entry. + + :func:`scandir` will provide as much of this information as possible without + making additional system calls. When a ``stat()`` or ``lstat()`` system call + is made, the ``DirEntry`` object will cache the result. + + ``DirEntry`` instances are not intended to be stored in long-lived data + structures; if you know the file metadata has changed or if a long time has + elapsed since calling :func:`scandir`, call ``os.stat(entry.path)`` to fetch + up-to-date information. + + Because the ``DirEntry`` methods can make operating system calls, they may + also raise :exc:`OSError`. If you need very fine-grained + control over errors, you can catch :exc:`OSError` when calling one of the + ``DirEntry`` methods and handle as appropriate. + + Attributes and methods on a ``DirEntry`` instance are as follows: + + .. attribute:: name + + The entry's base filename, relative to the :func:`scandir` *path* + argument. + + The :attr:`name` attribute will be of the same type (``str`` or + ``bytes``) as the :func:`scandir` *path* argument. Use + :func:`~os.fsdecode` to decode byte filenames. + + .. attribute:: path + + The entry's full path name: equivalent to ``os.path.join(scandir_path, + entry.name)`` where *scandir_path* is the :func:`scandir` *path* + argument. The path is only absolute if the :func:`scandir` *path* + argument was absolute. + + The :attr:`path` attribute will be of the same type (``str`` or + ``bytes``) as the :func:`scandir` *path* argument. Use + :func:`~os.fsdecode` to decode byte filenames. + + .. method:: inode() + + Return the inode number of the entry. + + The result is cached on the ``DirEntry`` object, use ``os.stat(entry.path, + follow_symlinks=False).st_ino`` to fetch up-to-date information. + + On Unix, no system call is required. + + .. method:: is_dir(\*, follow_symlinks=True) + + If *follow_symlinks* is ``True`` (the default), return ``True`` if the + entry is a directory or a symbolic link pointing to a directory; + return ``False`` if it is or points to any other kind of file, or if it + doesn't exist anymore. + + If *follow_symlinks* is ``False``, return ``True`` only if this entry + is a directory; return ``False`` if it is any other kind of file + or if it doesn't exist anymore. + + The result is cached on the ``DirEntry`` object. Call :func:`os.stat` + along with :func:`stat.S_ISDIR` to fetch up-to-date information. + + This method can raise :exc:`OSError`, such as :exc:`PermissionError`, + but :exc:`FileNotFoundError` is caught and not raised. + + In most cases, no system call is required. + + .. method:: is_file(\*, follow_symlinks=True) + + If *follow_symlinks* is ``True`` (the default), return ``True`` if the + entry is a file or a symbolic link pointing to a file; return ``False`` + if it is or points to a directory or other non-file entry, or if it + doesn't exist anymore. + + If *follow_symlinks* is ``False``, return ``True`` only if this entry + is a file; return ``False`` if it is a directory or other non-file entry, + or if it doesn't exist anymore. + + The result is cached on the ``DirEntry`` object. Call :func:`os.stat` + along with :func:`stat.S_ISREG` to fetch up-to-date information. + + This method can raise :exc:`OSError`, such as :exc:`PermissionError`, + but :exc:`FileNotFoundError` is caught and not raised. + + In most cases, no system call is required. + + .. method:: is_symlink() + + Return ``True`` if this entry is a symbolic link (even if broken); + return ``False`` if it points to a directory or any kind of file, + or if it doesn't exist anymore. + + The result is cached on the ``DirEntry`` object. Call + :func:`os.path.islink` to fetch up-to-date information. + + The method can raise :exc:`OSError`, such as :exc:`PermissionError`, + but :exc:`FileNotFoundError` is caught and not raised. + + In most cases, no system call is required. + + .. method:: stat(\*, follow_symlinks=True) + + Return a :class:`stat_result` object for this entry. This method + follows symbolic links by default; to stat a symbolic link add the + ``follow_symlinks=False`` argument. + + On Unix, this method always requires a system call. On Windows, + ``DirEntry.stat()`` requires a system call only if the + entry is a symbolic link, and ``DirEntry.stat(follow_symlinks=False)`` + never requires a system call. + + On Windows, the ``st_ino``, ``st_dev`` and ``st_nlink`` attributes of the + :class:`stat_result` are always set to zero. Call :func:`os.stat` to + get these attributes. + + The result is cached on the ``DirEntry`` object. Call :func:`os.stat` + to fetch up-to-date information. + + .. versionadded:: 3.5 + + .. function:: stat(path, \*, dir_fd=None, follow_symlinks=True) Get the status of a file or a file descriptor. Perform the equivalent of a @@ -1890,8 +2064,6 @@ features: >>> statinfo.st_size 264 - Availability: Unix, Windows. - .. seealso:: :func:`fstat` and :func:`lstat` functions. @@ -2039,6 +2211,15 @@ features: File type. + On Windows systems, the following attribute is also available: + + .. attribute:: st_file_attributes + + Windows file attributes: ``dwFileAttributes`` member of the + ``BY_HANDLE_FILE_INFORMATION`` structure returned by + :c:func:`GetFileInformationByHandle`. See the ``FILE_ATTRIBUTE_*`` + constants in the :mod:`stat` module. + The standard module :mod:`stat` defines functions and constants that are useful for extracting information from a :c:type:`stat` structure. (On Windows, some items are filled with dummy values.) @@ -2056,6 +2237,9 @@ features: Added the :attr:`st_atime_ns`, :attr:`st_mtime_ns`, and :attr:`st_ctime_ns` members. + .. versionadded:: 3.5 + Added the :attr:`st_file_attributes` member on Windows. + .. function:: stat_float_times([newvalue]) @@ -2259,10 +2443,12 @@ features: This function can support :ref:`specifying a file descriptor <path_fd>`. - Availability: Unix. + Availability: Unix, Windows. .. versionadded:: 3.3 + .. versionchanged:: 3.5 + Added support for Windows .. function:: unlink(path, *, dir_fd=None) @@ -2271,8 +2457,6 @@ features: name. Please see the documentation for :func:`remove` for further information. - Availability: Unix, Windows. - .. versionadded:: 3.3 The *dir_fd* parameter. @@ -2309,8 +2493,6 @@ features: :ref:`paths relative to directory descriptors <dir_fd>` and :ref:`not following symlinks <follow_symlinks>`. - Availability: Unix, Windows. - .. versionadded:: 3.3 Added support for specifying an open file descriptor for *path*, and the *dir_fd*, *follow_symlinks*, and *ns* parameters. @@ -2401,6 +2583,10 @@ features: for name in dirs: os.rmdir(os.path.join(root, name)) + .. versionchanged:: 3.5 + This function now calls :func:`os.scandir` instead of :func:`os.listdir`, + making it faster by reducing the number of calls to :func:`os.stat`. + .. function:: fwalk(top='.', topdown=True, onerror=None, *, follow_symlinks=False, dir_fd=None) @@ -2557,8 +2743,6 @@ to be ignored. Python signal handler registered for :const:`SIGABRT` with :func:`signal.signal`. - Availability: Unix, Windows. - .. function:: execl(path, arg0, arg1, ...) execle(path, arg0, arg1, ..., env) @@ -2622,8 +2806,6 @@ to be ignored. Exit the process with status *n*, without calling cleanup handlers, flushing stdio buffers, etc. - Availability: Unix, Windows. - .. note:: The standard way to exit is ``sys.exit(n)``. :func:`_exit` should @@ -2986,6 +3168,10 @@ written in Python, such as a mail server's external command delivery program. doesn't work if it is. Use the :func:`os.path.normpath` function to ensure that the path is properly encoded for Win32. + To reduce interpreter startup overhead, the Win32 :c:func:`ShellExecute` + function is not resolved until this function is first called. If the function + cannot be resolved, :exc:`NotImplementedError` will be raised. + Availability: Windows. @@ -3133,6 +3319,11 @@ written in Python, such as a mail server's external command delivery program. id is known, not necessarily a child process. The :func:`spawn\* <spawnl>` functions called with :const:`P_NOWAIT` return suitable process handles. + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise an + exception, the function now retries the system call instead of raising an + :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + .. function:: wait3(options) @@ -3540,3 +3731,9 @@ Miscellaneous Functions For an easy-to-use interface to the random number generator provided by your platform, please see :class:`random.SystemRandom`. + + .. versionchanged:: 3.5 + On Linux 3.17 and newer, the ``getrandom()`` syscall is now used + when available. On OpenBSD 5.6 and newer, the C ``getentropy()`` + function is now used. These functions avoid the usage of an internal file + descriptor. diff --git a/Doc/library/ossaudiodev.rst b/Doc/library/ossaudiodev.rst index bb5081afde..c60d596506 100644 --- a/Doc/library/ossaudiodev.rst +++ b/Doc/library/ossaudiodev.rst @@ -148,21 +148,30 @@ and (read-only) attributes: .. method:: oss_audio_device.write(data) - Write the Python string *data* to the audio device and return the number of - bytes written. If the audio device is in blocking mode (the default), the - entire string is always written (again, this is different from usual Unix device - semantics). If the device is in non-blocking mode, some data may not be written + Write a :term:`bytes-like object` *data* to the audio device and return the + number of bytes written. If the audio device is in blocking mode (the + default), the entire data is always written (again, this is different from + usual Unix device semantics). If the device is in non-blocking mode, some + data may not be written ---see :meth:`writeall`. + .. versionchanged: 3.5 + Writable :term:`bytes-like object` is now accepted. + .. method:: oss_audio_device.writeall(data) - Write the entire Python string *data* to the audio device: waits until the audio - device is able to accept data, writes as much data as it will accept, and - repeats until *data* has been completely written. If the device is in blocking - mode (the default), this has the same effect as :meth:`write`; :meth:`writeall` - is only useful in non-blocking mode. Has no return value, since the amount of - data written is always equal to the amount of data supplied. + Write a :term:`bytes-like object` *data* to the audio device: waits until + the audio device is able to accept data, writes as much data as it will + accept, and repeats until *data* has been completely written. If the device + is in blocking mode (the default), this has the same effect as + :meth:`write`; :meth:`writeall` is only useful in non-blocking mode. Has + no return value, since the amount of data written is always equal to the + amount of data supplied. + + .. versionchanged: 3.5 + Writable :term:`bytes-like object` is now accepted. + .. versionchanged:: 3.2 Audio device objects also support the context management protocol, i.e. they can diff --git a/Doc/library/pathlib.rst b/Doc/library/pathlib.rst index 24e2a308df..2f0654440d 100644 --- a/Doc/library/pathlib.rst +++ b/Doc/library/pathlib.rst @@ -628,6 +628,17 @@ call fails (for example because the path doesn't exist): PosixPath('/home/antoine/pathlib') +.. classmethod:: Path.home() + + Return a new path object representing the user's home directory (as + returned by :func:`os.path.expanduser` with ``~`` construct):: + + >>> Path.home() + PosixPath('/home/antoine') + + .. versionadded:: 3.5 + + .. method:: Path.stat() Return information about this path (similarly to :func:`os.stat`). @@ -670,6 +681,18 @@ call fails (for example because the path doesn't exist): symlink *points to* an existing file or directory. +.. method:: Path.expanduser() + + Return a new path with expanded ``~`` and ``~user`` constructs, + as returned by :meth:`os.path.expanduser`:: + + >>> p = PosixPath('~/films/Monty Python') + >>> p.expanduser() + PosixPath('/home/eric/films/Monty Python') + + .. versionadded:: 3.5 + + .. method:: Path.glob(pattern) Glob the given *pattern* in the directory represented by this path, @@ -791,7 +814,7 @@ call fails (for example because the path doesn't exist): the symbolic link's information rather than its target's. -.. method:: Path.mkdir(mode=0o777, parents=False) +.. method:: Path.mkdir(mode=0o777, parents=False, exist_ok=False) Create a new directory at this given path. If *mode* is given, it is combined with the process' ``umask`` value to determine the file mode @@ -805,6 +828,16 @@ call fails (for example because the path doesn't exist): If *parents* is false (the default), a missing parent raises :exc:`FileNotFoundError`. + If *exist_ok* is false (the default), an :exc:`FileExistsError` is + raised if the target directory already exists. + + If *exist_ok* is true, :exc:`FileExistsError` exceptions will be + ignored (same behavior as the POSIX ``mkdir -p`` command), but only if the + last path component is not an existing non-directory file. + + .. versionchanged:: 3.5 + The *exist_ok* parameter was added. + .. method:: Path.open(mode='r', buffering=-1, encoding=None, errors=None, newline=None) @@ -824,6 +857,34 @@ call fails (for example because the path doesn't exist): if the file's uid isn't found in the system database. +.. method:: Path.read_bytes() + + Return the binary contents of the pointed-to file as a bytes object:: + + >>> p = Path('my_binary_file') + >>> p.write_bytes(b'Binary file contents') + 20 + >>> p.read_bytes() + b'Binary file contents' + + .. versionadded:: 3.5 + + +.. method:: Path.read_text(encoding=None, errors=None) + + Return the decoded contents of the pointed-to file as a string:: + + >>> p = Path('my_text_file') + >>> p.write_text('Text file contents') + 18 + >>> p.read_text() + 'Text file contents' + + The optional parameters have the same meaning as in :func:`open`. + + .. versionadded:: 3.5 + + .. method:: Path.rename(target) Rename this file or directory to the given *target*. *target* can be @@ -884,6 +945,25 @@ call fails (for example because the path doesn't exist): Remove this directory. The directory must be empty. +.. method:: Path.samefile(other_path) + + Return whether this path points to the same file as *other_path*, which + can be either a Path object, or a string. The semantics are similar + to :func:`os.path.samefile` and :func:`os.path.samestat`. + + An :exc:`OSError` can be raised if either file cannot be accessed for some + reason. + + >>> p = Path('spam') + >>> q = Path('eggs') + >>> p.samefile(q) + False + >>> p.samefile('spam') + True + + .. versionadded:: 3.5 + + .. method:: Path.symlink_to(target, target_is_directory=False) Make this path a symbolic link to *target*. Under Windows, @@ -917,3 +997,33 @@ call fails (for example because the path doesn't exist): Remove this file or symbolic link. If the path points to a directory, use :func:`Path.rmdir` instead. + + +.. method:: Path.write_bytes(data) + + Open the file pointed to in bytes mode, write *data* to it, and close the + file:: + + >>> p = Path('my_binary_file') + >>> p.write_bytes(b'Binary file contents') + 20 + >>> p.read_bytes() + b'Binary file contents' + + An existing file of the same name is overwritten. + + .. versionadded:: 3.5 + + +.. method:: Path.write_text(data, encoding=None, errors=None) + + Open the file pointed to in text mode, write *data* to it, and close the + file:: + + >>> p = Path('my_text_file') + >>> p.write_text('Text file contents') + 18 + >>> p.read_text() + 'Text file contents' + + .. versionadded:: 3.5 diff --git a/Doc/library/pickle.rst b/Doc/library/pickle.rst index 0f48cb188f..a92526f5a3 100644 --- a/Doc/library/pickle.rst +++ b/Doc/library/pickle.rst @@ -425,7 +425,7 @@ The following types can be pickled: Attempts to pickle unpicklable objects will raise the :exc:`PicklingError` exception; when this happens, an unspecified number of bytes may have already been written to the underlying file. Trying to pickle a highly recursive data -structure may exceed the maximum recursion depth, a :exc:`RuntimeError` will be +structure may exceed the maximum recursion depth, a :exc:`RecursionError` will be raised in this case. You can carefully raise this limit with :func:`sys.setrecursionlimit`. @@ -859,7 +859,7 @@ For the simplest code, use the :func:`dump` and :func:`load` functions. :: data = { 'a': [1, 2.0, 3, 4+6j], 'b': ("character string", b"byte string"), - 'c': set([None, True, False]) + 'c': {None, True, False} } with open('data.pickle', 'wb') as f: diff --git a/Doc/library/pkgutil.rst b/Doc/library/pkgutil.rst index 13ea7b9b33..5d3295db7e 100644 --- a/Doc/library/pkgutil.rst +++ b/Doc/library/pkgutil.rst @@ -58,7 +58,7 @@ support. .. deprecated:: 3.3 This emulation is no longer needed, as the standard import mechanism - is now fully PEP 302 compliant and available in :mod:`importlib` + is now fully PEP 302 compliant and available in :mod:`importlib`. .. class:: ImpLoader(fullname, file, filename, etc) @@ -67,7 +67,7 @@ support. .. deprecated:: 3.3 This emulation is no longer needed, as the standard import mechanism - is now fully PEP 302 compliant and available in :mod:`importlib` + is now fully PEP 302 compliant and available in :mod:`importlib`. .. function:: find_loader(fullname) diff --git a/Doc/library/platform.rst b/Doc/library/platform.rst index e27f2ad8b0..679cc6b23c 100644 --- a/Doc/library/platform.rst +++ b/Doc/library/platform.rst @@ -247,6 +247,8 @@ Unix Platforms This is another name for :func:`linux_distribution`. + .. deprecated-removed:: 3.5 3.7 + .. function:: linux_distribution(distname='', version='', id='', supported_dists=('SuSE','debian','redhat','mandrake',...), full_distribution_name=1) Tries to determine the name of the Linux OS distribution name. @@ -263,6 +265,8 @@ Unix Platforms parameters. ``id`` is the item in parentheses after the version number. It is usually the version codename. + .. deprecated-removed:: 3.5 3.7 + .. function:: libc_ver(executable=sys.executable, lib='', version='', chunksize=2048) Tries to determine the libc version against which the file executable (defaults diff --git a/Doc/library/poplib.rst b/Doc/library/poplib.rst index 45baad96af..8468f4c498 100644 --- a/Doc/library/poplib.rst +++ b/Doc/library/poplib.rst @@ -194,6 +194,15 @@ An :class:`POP3` instance has the following methods: the unique id for that message in the form ``'response mesgnum uid``, otherwise result is list ``(response, ['mesgnum uid', ...], octets)``. + +.. method:: POP3.utf8() + + Try to switch to UTF-8 mode. Returns the server response if successful, + raises :class:`error_proto` if not. Specified in :RFC:`6856`. + + .. versionadded:: 3.5 + + .. method:: POP3.stls(context=None) Start a TLS session on the active connection as specified in :rfc:`2595`. diff --git a/Doc/library/pprint.rst b/Doc/library/pprint.rst index c0589a31a9..0b44dc865e 100644 --- a/Doc/library/pprint.rst +++ b/Doc/library/pprint.rst @@ -235,10 +235,10 @@ In its basic form, :func:`pprint` shows the whole object:: 'classifiers': ['Programming Language :: Python :: 2.6', 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 2 :: Only'], - 'description': 'An extensible framework for Python programming, ' - 'with special focus\r\n' - 'on event-based network programming and ' - 'multiprotocol integration.', + 'description': 'An extensible framework for Python programming, with ' + 'special focus\r\n' + 'on event-based network programming and multiprotocol ' + 'integration.', 'docs_url': '', 'download_url': 'UNKNOWN', 'home_page': 'http://twistedmatrix.com/', @@ -288,10 +288,10 @@ contents):: 'cheesecake_documentation_id': None, 'cheesecake_installability_id': None, 'classifiers': [...], - 'description': 'An extensible framework for Python programming, ' - 'with special focus\r\n' - 'on event-based network programming and ' - 'multiprotocol integration.', + 'description': 'An extensible framework for Python programming, with ' + 'special focus\r\n' + 'on event-based network programming and multiprotocol ' + 'integration.', 'docs_url': '', 'download_url': 'UNKNOWN', 'home_page': 'http://twistedmatrix.com/', @@ -323,13 +323,12 @@ cannot be split, the specified width will be exceeded:: 'cheesecake_installability_id': None, 'classifiers': [...], 'description': 'An extensible ' - 'framework for ' - 'Python programming, ' - 'with special ' - 'focus\r\n' - 'on event-based ' - 'network programming ' - 'and multiprotocol ' + 'framework for Python ' + 'programming, with ' + 'special focus\r\n' + 'on event-based network ' + 'programming and ' + 'multiprotocol ' 'integration.', 'docs_url': '', 'download_url': 'UNKNOWN', @@ -344,8 +343,8 @@ cannot be split, the specified width will be exceeded:: 'release_url': 'http://pypi.python.org/pypi/Twisted/12.3.0', 'requires_python': None, 'stable_version': None, - 'summary': 'An asynchronous ' - 'networking framework ' - 'written in Python', + 'summary': 'An asynchronous networking ' + 'framework written in ' + 'Python', 'version': '12.3.0'}, 'urls': [{...}, {...}]} diff --git a/Doc/library/profile.rst b/Doc/library/profile.rst index 2928821d55..959d9b98a8 100644 --- a/Doc/library/profile.rst +++ b/Doc/library/profile.rst @@ -33,7 +33,7 @@ profiling interface: 2. :mod:`profile`, a pure Python module whose interface is imitated by :mod:`cProfile`, but which adds significant overhead to profiled programs. If you're trying to extend the profiler in some way, the task might be easier - with this module. + with this module. Originally designed and written by Jim Roskind. .. note:: diff --git a/Doc/library/py_compile.rst b/Doc/library/py_compile.rst index bae8450b7c..97f2b20d2a 100644 --- a/Doc/library/py_compile.rst +++ b/Doc/library/py_compile.rst @@ -29,9 +29,9 @@ byte-code cache files in the directory containing the source code. .. function:: compile(file, cfile=None, dfile=None, doraise=False, optimize=-1) Compile a source file to byte-code and write out the byte-code cache file. - The source code is loaded from the file name *file*. The byte-code is - written to *cfile*, which defaults to the :PEP:`3147` path, ending in - ``.pyc`` (``.pyo`` if optimization is enabled in the current interpreter). + The source code is loaded from the file name *file*. The byte-code is + written to *cfile*, which defaults to the :pep:`3147`/:pep:`488` path, ending + in ``.pyc``. For example, if *file* is ``/foo/bar/baz.py`` *cfile* will default to ``/foo/bar/__pycache__/baz.cpython-32.pyc`` for Python 3.2. If *dfile* is specified, it is used as the name of the source file in error messages when @@ -68,7 +68,7 @@ byte-code cache files in the directory containing the source code. .. function:: main(args=None) Compile several source files. The files named in *args* (or on the command - line, if *args* is ``None``) are compiled and the resulting bytecode is + line, if *args* is ``None``) are compiled and the resulting byte-code is cached in the normal manner. This function does not search a directory structure to locate source files; it only compiles files named explicitly. If ``'-'`` is the only parameter in args, the list of files is taken from @@ -86,4 +86,3 @@ could not be compiled. Module :mod:`compileall` Utilities to compile all Python source files in a directory tree. - diff --git a/Doc/library/queue.rst b/Doc/library/queue.rst index 680d690052..1cb0935377 100644 --- a/Doc/library/queue.rst +++ b/Doc/library/queue.rst @@ -158,22 +158,32 @@ fully processed by daemon consumer threads. Example of how to wait for enqueued tasks to be completed:: - def worker(): - while True: - item = q.get() - do_work(item) - q.task_done() - - q = Queue() - for i in range(num_worker_threads): - t = Thread(target=worker) - t.daemon = True + def worker(): + while True: + item = q.get() + if item is None: + break + do_work(item) + q.task_done() + + q = queue.Queue() + threads = [] + for i in range(num_worker_threads): + t = threading.Thread(target=worker) t.start() + threads.append(t) - for item in source(): - q.put(item) + for item in source(): + q.put(item) - q.join() # block until all tasks are done + # block until all tasks are done + q.join() + + # stop workers + for i in range(num_worker_threads): + q.put(None) + for t in threads: + t.join() .. seealso:: diff --git a/Doc/library/random.rst b/Doc/library/random.rst index 11dd367f8a..f8b772749f 100644 --- a/Doc/library/random.rst +++ b/Doc/library/random.rst @@ -46,8 +46,7 @@ from sources provided by the operating system. .. warning:: The pseudo-random generators of this module should not be used for - security purposes. Use :func:`os.urandom` or :class:`SystemRandom` if - you require a cryptographically secure pseudo-random number generator. + security purposes. Bookkeeping functions: diff --git a/Doc/library/re.rst b/Doc/library/re.rst index c3c8b65d8d..888458449a 100644 --- a/Doc/library/re.rst +++ b/Doc/library/re.rst @@ -281,9 +281,7 @@ The special characters are: assertion`. ``(?<=abc)def`` will find a match in ``abcdef``, since the lookbehind will back up 3 characters and check if the contained pattern matches. The contained pattern must only match strings of some fixed length, meaning that - ``abc`` or ``a|b`` are allowed, but ``a*`` and ``a{3,4}`` are not. Group - references are not supported even if they match strings of some fixed length. - Note that + ``abc`` or ``a|b`` are allowed, but ``a*`` and ``a{3,4}`` are not. Note that patterns which start with positive lookbehind assertions will not match at the beginning of the string being searched; you will most likely want to use the :func:`search` function rather than the :func:`match` function: @@ -299,12 +297,14 @@ The special characters are: >>> m.group(0) 'egg' + .. versionchanged: 3.5 + Added support for group references of fixed length. + ``(?<!...)`` Matches if the current position in the string is not preceded by a match for ``...``. This is called a :dfn:`negative lookbehind assertion`. Similar to positive lookbehind assertions, the contained pattern must only match strings of - some fixed length and shouldn't contain group references. - Patterns which start with negative lookbehind assertions may + some fixed length. Patterns which start with negative lookbehind assertions may match at the beginning of the string being searched. ``(?(id/name)yes-pattern|no-pattern)`` @@ -438,6 +438,10 @@ three digits in length. .. versionchanged:: 3.3 The ``'\u'`` and ``'\U'`` escape sequences have been added. +.. deprecated-removed:: 3.5 3.6 + Unknown escapes consist of ``'\'`` and ASCII letter now raise a + deprecation warning and will be forbidden in Python 3.6. + .. seealso:: @@ -524,7 +528,11 @@ form. current locale. The use of this flag is discouraged as the locale mechanism is very unreliable, and it only handles one "culture" at a time anyway; you should use Unicode matching instead, which is the default in Python 3 - for Unicode (str) patterns. + for Unicode (str) patterns. This flag makes sense only with bytes patterns. + + .. deprecated-removed:: 3.5 3.6 + Deprecated the use of :const:`re.LOCALE` with string patterns or + :const:`re.ASCII`. .. data:: M @@ -625,17 +633,37 @@ form. That way, separator components are always found at the same relative indices within the result list. - Note that *split* will never split a string on an empty pattern match. - For example: + .. note:: + + :func:`split` doesn't currently split a string on an empty pattern match. + For example: - >>> re.split('x*', 'foo') - ['foo'] - >>> re.split("(?m)^$", "foo\n\nbar\n") - ['foo\n\nbar\n'] + >>> re.split('x*', 'axbc') + ['a', 'bc'] + + Even though ``'x*'`` also matches 0 'x' before 'a', between 'b' and 'c', + and after 'c', currently these matches are ignored. The correct behavior + (i.e. splitting on empty matches too and returning ``['', 'a', 'b', 'c', + '']``) will be implemented in future versions of Python, but since this + is a backward incompatible change, a :exc:`FutureWarning` will be raised + in the meanwhile. + + Patterns that can only match empty strings currently never split the + string. Since this doesn't match the expected behavior, a + :exc:`ValueError` will be raised starting from Python 3.5:: + + >>> re.split("^$", "foo\n\nbar\n", flags=re.M) + Traceback (most recent call last): + File "<stdin>", line 1, in <module> + ... + ValueError: split() requires a non-empty pattern match. .. versionchanged:: 3.1 Added the optional flags argument. + .. versionchanged:: 3.5 + Splitting on a pattern that could match an empty string now raises + a warning. Patterns that can only match empty strings are now rejected. .. function:: findall(pattern, string, flags=0) @@ -663,7 +691,7 @@ form. *string* is returned unchanged. *repl* can be a string or a function; if it is a string, any backslash escapes in it are processed. That is, ``\n`` is converted to a single newline character, ``\r`` is converted to a carriage return, and - so forth. Unknown escapes such as ``\j`` are left alone. Backreferences, such + so forth. Unknown escapes such as ``\&`` are left alone. Backreferences, such as ``\6``, are replaced with the substring matched by group 6 in the pattern. For example: @@ -705,6 +733,13 @@ form. .. versionchanged:: 3.1 Added the optional flags argument. + .. versionchanged:: 3.5 + Unmatched groups are replaced with an empty string. + + .. deprecated-removed:: 3.5 3.6 + Unknown escapes consist of ``'\'`` and ASCII letter now raise a + deprecation warning and will be forbidden in Python 3.6. + .. function:: subn(pattern, repl, string, count=0, flags=0) @@ -714,6 +749,9 @@ form. .. versionchanged:: 3.1 Added the optional flags argument. + .. versionchanged:: 3.5 + Unmatched groups are replaced with an empty string. + .. function:: escape(string) @@ -730,13 +768,36 @@ form. Clear the regular expression cache. -.. exception:: error +.. exception:: error(msg, pattern=None, pos=None) Exception raised when a string passed to one of the functions here is not a valid regular expression (for example, it might contain unmatched parentheses) or when some other error occurs during compilation or matching. It is never an - error if a string contains no match for a pattern. + error if a string contains no match for a pattern. The error instance has + the following additional attributes: + + .. attribute:: msg + + The unformatted error message. + + .. attribute:: pattern + + The regular expression pattern. + + .. attribute:: pos + + The index of *pattern* where compilation failed. + + .. attribute:: lineno + + The line corresponding to *pos*. + + .. attribute:: colno + + The column corresponding to *pos*. + .. versionchanged:: 3.5 + Added additional attributes. .. _re-objects: @@ -889,6 +950,8 @@ Match objects support the following methods and attributes: (``\g<1>``, ``\g<name>``) are replaced by the contents of the corresponding group. + .. versionchanged:: 3.5 + Unmatched groups are replaced with an empty string. .. method:: match.group([group1, ...]) diff --git a/Doc/library/readline.rst b/Doc/library/readline.rst index 692310b941..3864f0d225 100644 --- a/Doc/library/readline.rst +++ b/Doc/library/readline.rst @@ -59,6 +59,14 @@ The :mod:`readline` module defines the following functions: Save a readline history file. The default filename is :file:`~/.history`. +.. function:: append_history_file(nelements[, filename]) + + Append the last *nelements* of history to a file. The default filename is + :file:`~/.history`. The file must already exist. + + .. versionadded:: 3.5 + + .. function:: clear_history() Clear the current history. (Note: this function is not available if the @@ -209,6 +217,26 @@ from the user's :envvar:`PYTHONSTARTUP` file. :: This code is actually automatically run when Python is run in :ref:`interactive mode <tut-interactive>` (see :ref:`rlcompleter-config`). +The following example achieves the same goal but supports concurrent interactive +sessions, by only appending the new history. :: + + import atexit + import os + import realine + histfile = os.path.join(os.path.expanduser("~"), ".python_history") + + try: + readline.read_history_file(histfile) + h_len = readline.get_history_length() + except FileNotFoundError: + open(histfile, 'wb').close() + h_len = 0 + + def save(prev_h_len, histfile): + new_h_len = readline.get_history_length() + readline.append_history_file(new_h_len - prev_h_len, histfile) + atexit.register(save, h_len, histfile) + The following example extends the :class:`code.InteractiveConsole` class to support history save/restore. :: @@ -234,4 +262,3 @@ support history save/restore. :: def save_history(self, histfile): readline.write_history_file(histfile) - diff --git a/Doc/library/select.rst b/Doc/library/select.rst index 5334af8ea4..a62dc844e4 100644 --- a/Doc/library/select.rst +++ b/Doc/library/select.rst @@ -145,6 +145,13 @@ The module defines the following: library, and does not handle file descriptors that don't originate from WinSock. + .. versionchanged:: 3.5 + The function is now retried with a recomputed timeout when interrupted by + a signal, except if the signal handler raises an exception (see + :pep:`475` for the rationale), instead of raising + :exc:`InterruptedError`. + + .. attribute:: PIPE_BUF The minimum number of bytes which can be written without blocking to a pipe @@ -242,6 +249,12 @@ object. returning. If *timeout* is omitted, -1, or :const:`None`, the call will block until there is an event for this poll object. + .. versionchanged:: 3.5 + The function is now retried with a recomputed timeout when interrupted by + a signal, except if the signal handler raises an exception (see + :pep:`475` for the rationale), instead of raising + :exc:`InterruptedError`. + .. _epoll-objects: @@ -322,6 +335,12 @@ Edge and Level Trigger Polling (epoll) Objects Wait for events. timeout in seconds (float) + .. versionchanged:: 3.5 + The function is now retried with a recomputed timeout when interrupted by + a signal, except if the signal handler raises an exception (see + :pep:`475` for the rationale), instead of raising + :exc:`InterruptedError`. + .. _poll-objects: @@ -401,6 +420,12 @@ linearly scanned again. :c:func:`select` is O(highest file descriptor), while returning. If *timeout* is omitted, negative, or :const:`None`, the call will block until there is an event for this poll object. + .. versionchanged:: 3.5 + The function is now retried with a recomputed timeout when interrupted by + a signal, except if the signal handler raises an exception (see + :pep:`475` for the rationale), instead of raising + :exc:`InterruptedError`. + .. _kqueue-objects: @@ -435,6 +460,12 @@ Kqueue Objects - max_events must be 0 or a positive integer - timeout in seconds (floats possible) + .. versionchanged:: 3.5 + The function is now retried with a recomputed timeout when interrupted by + a signal, except if the signal handler raises an exception (see + :pep:`475` for the rationale), instead of raising + :exc:`InterruptedError`. + .. _kevent-objects: diff --git a/Doc/library/selectors.rst b/Doc/library/selectors.rst index 98377c890f..56cfc6bdb7 100644 --- a/Doc/library/selectors.rst +++ b/Doc/library/selectors.rst @@ -45,12 +45,13 @@ Classes hierarchy:: +-- SelectSelector +-- PollSelector +-- EpollSelector + +-- DevpollSelector +-- KqueueSelector In the following, *events* is a bitwise mask indicating which I/O events should -be waited for on a given file object. It can be a combination of the constants -below: +be waited for on a given file object. It can be a combination of the modules +constants below: +-----------------------+-----------------------------------------------+ | Constant | Meaning | @@ -158,6 +159,12 @@ below: timeout has elapsed if the current process receives a signal: in this case, an empty list will be returned. + .. versionchanged:: 3.5 + The selector is now retried with a recomputed timeout when interrupted + by a signal if the signal handler did not raise an exception (see + :pep:`475` for the rationale), instead of returning an empty list + of events before the timeout. + .. method:: close() Close the selector. @@ -207,6 +214,16 @@ below: This returns the file descriptor used by the underlying :func:`select.epoll` object. +.. class:: DevpollSelector() + + :func:`select.devpoll`-based selector. + + .. method:: fileno() + + This returns the file descriptor used by the underlying + :func:`select.devpoll` object. + + .. versionadded:: 3.5 .. class:: KqueueSelector() diff --git a/Doc/library/shutil.rst b/Doc/library/shutil.rst index 04afe928c8..3b467e01c2 100644 --- a/Doc/library/shutil.rst +++ b/Doc/library/shutil.rst @@ -191,7 +191,8 @@ Directory and files operations match one of the glob-style *patterns* provided. See the example below. -.. function:: copytree(src, dst, symlinks=False, ignore=None, copy_function=copy2, ignore_dangling_symlinks=False) +.. function:: copytree(src, dst, symlinks=False, ignore=None, \ + copy_function=copy2, ignore_dangling_symlinks=False) Recursively copy an entire directory tree rooted at *src*, returning the destination directory. The destination @@ -282,7 +283,7 @@ Directory and files operations .. versionadded:: 3.3 -.. function:: move(src, dst) +.. function:: move(src, dst, copy_function=copy2) Recursively move a file or directory (*src*) to another location (*dst*) and return the destination. @@ -292,15 +293,26 @@ Directory and files operations be overwritten depending on :func:`os.rename` semantics. If the destination is on the current filesystem, then :func:`os.rename` is - used. Otherwise, *src* is copied (using :func:`shutil.copy2`) to *dst* and - then removed. In case of symlinks, a new symlink pointing to the target of - *src* will be created in or as *dst* and *src* will be removed. + used. Otherwise, *src* is copied to *dst* using *copy_function* and then + removed. In case of symlinks, a new symlink pointing to the target of *src* + will be created in or as *dst* and *src* will be removed. + + If *copy_function* is given, it must be a callable that takes two arguments + *src* and *dst*, and will be used to copy *src* to *dest* if + :func:`os.rename` cannot be used. If the source is a directory, + :func:`copytree` is called, passing it the :func:`copy_function`. The + default *copy_function* is :func:`copy2`. Using :func:`copy` as the + *copy_function* allows the move to succeed when it is not possible to also + copy the metadata, at the expense of not copying any of the metadata. .. versionchanged:: 3.3 Added explicit symlink handling for foreign filesystems, thus adapting it to the behavior of GNU's :program:`mv`. Now returns *dst*. + .. versionchanged:: 3.5 + Added the *copy_function* keyword argument. + .. function:: disk_usage(path) Return disk usage statistics about the given path as a :term:`named tuple` @@ -418,6 +430,26 @@ Another example that uses the *ignore* argument to add a logging call:: copytree(source, destination, ignore=_logpath) +.. _shutil-rmtree-example: + +rmtree example +~~~~~~~~~~~~~~ + +This example shows how to remove a directory tree on Windows where some +of the files have their read-only bit set. It uses the onerror callback +to clear the readonly bit and reattempt the remove. Any subsequent failure +will propagate. :: + + import os, stat + import shutil + + def remove_readonly(func, path, _): + "Clear the readonly bit and reattempt the removal" + os.chmod(path, stat.S_IWRITE) + func(path) + + shutil.rmtree(directory, onerror=remove_readonly) + .. _archiving-operations: Archiving operations @@ -434,7 +466,8 @@ provided. They rely on the :mod:`zipfile` and :mod:`tarfile` modules. *base_name* is the name of the file to create, including the path, minus any format-specific extension. *format* is the archive format: one of - "zip", "tar", "bztar" (if the :mod:`bz2` module is available) or "gztar". + "zip", "tar", "bztar" (if the :mod:`bz2` module is available), "xztar" + (if the :mod:`lzma` module is available) or "gztar". *root_dir* is a directory that will be the root directory of the archive; for example, we typically chdir into *root_dir* before creating the @@ -457,6 +490,9 @@ provided. They rely on the :mod:`zipfile` and :mod:`tarfile` modules. The *verbose* argument is unused and deprecated. + .. versionchanged:: 3.5 + Added support for the *xztar* format. + .. function:: get_archive_formats() @@ -467,6 +503,7 @@ provided. They rely on the :mod:`zipfile` and :mod:`tarfile` modules. - *gztar*: gzip'ed tar-file - *bztar*: bzip2'ed tar-file (if the :mod:`bz2` module is available.) + - *xztar*: xz'ed tar-file (if the :mod:`lzma` module is available.) - *tar*: uncompressed tar file - *zip*: ZIP file @@ -542,6 +579,7 @@ provided. They rely on the :mod:`zipfile` and :mod:`tarfile` modules. - *gztar*: gzip'ed tar-file - *bztar*: bzip2'ed tar-file (if the :mod:`bz2` module is available.) + - *xztar*: xz'ed tar-file (if the :mod:`lzma` module is available.) - *tar*: uncompressed tar file - *zip*: ZIP file diff --git a/Doc/library/signal.rst b/Doc/library/signal.rst index 8b90994846..8f814df8bc 100644 --- a/Doc/library/signal.rst +++ b/Doc/library/signal.rst @@ -65,6 +65,16 @@ Besides, only the main thread is allowed to set a new signal handler. Module contents --------------- +.. versionchanged:: 3.5 + signal (SIG*), handler (:const:`SIG_DFL`, :const:`SIG_IGN`) and sigmask + (:const:`SIG_BLOCK`, :const:`SIG_UNBLOCK`, :const:`SIG_SETMASK`) + related constants listed below were turned into + :class:`enums <enum.IntEnum>`. + :func:`getsignal`, :func:`pthread_sigmask`, :func:`sigpending` and + :func:`sigwait` functions return human-readable + :class:`enums <enum.IntEnum>`. + + The variables defined in the :mod:`signal` module are: @@ -209,21 +219,21 @@ The :mod:`signal` module defines the following functions: :func:`sigpending`. -.. function:: pthread_kill(thread_id, signum) +.. function:: pthread_kill(thread_id, signalnum) - Send the signal *signum* to the thread *thread_id*, another thread in the + Send the signal *signalnum* to the thread *thread_id*, another thread in the same process as the caller. The target thread can be executing any code (Python or not). However, if the target thread is executing the Python interpreter, the Python signal handlers will be :ref:`executed by the main - thread <signals-and-threads>`. Therefore, the only point of sending a signal to a particular - Python thread would be to force a running system call to fail with - :exc:`InterruptedError`. + thread <signals-and-threads>`. Therefore, the only point of sending a + signal to a particular Python thread would be to force a running system call + to fail with :exc:`InterruptedError`. Use :func:`threading.get_ident()` or the :attr:`~threading.Thread.ident` attribute of :class:`threading.Thread` objects to get a suitable value for *thread_id*. - If *signum* is 0, then no signal is sent, but error checking is still + If *signalnum* is 0, then no signal is sent, but error checking is still performed; this can be used to check if the target thread is still running. Availability: Unix (see the man page :manpage:`pthread_kill(3)` for further @@ -308,6 +318,9 @@ The :mod:`signal` module defines the following functions: attempting to call it from other threads will cause a :exc:`ValueError` exception to be raised. + .. versionchanged:: 3.5 + On Windows, the function now also supports socket handles. + .. function:: siginterrupt(signalnum, flag) @@ -395,6 +408,11 @@ The :mod:`signal` module defines the following functions: .. versionadded:: 3.3 + .. versionchanged:: 3.5 + The function is now retried if interrupted by a signal not in *sigset* + and the signal handler does not raise an exception (see :pep:`475` for + the rationale). + .. function:: sigtimedwait(sigset, timeout) @@ -409,6 +427,11 @@ The :mod:`signal` module defines the following functions: .. versionadded:: 3.3 + .. versionchanged:: 3.5 + The function is now retried with the recomputed *timeout* if interrupted + by a signal not in *sigset* and the signal handler does not raise an + exception (see :pep:`475` for the rationale). + .. _signal-example: diff --git a/Doc/library/site.rst b/Doc/library/site.rst index 51e5da81d1..43daf790b7 100644 --- a/Doc/library/site.rst +++ b/Doc/library/site.rst @@ -26,24 +26,23 @@ additions, call the :func:`site.main` function. :option:`-S`. .. index:: - pair: site-python; directory pair: site-packages; directory It starts by constructing up to four directories from a head and a tail part. For the head part, it uses ``sys.prefix`` and ``sys.exec_prefix``; empty heads are skipped. For the tail part, it uses the empty string and then :file:`lib/site-packages` (on Windows) or -:file:`lib/python{X.Y}/site-packages` and then :file:`lib/site-python` (on -Unix and Macintosh). For each of the distinct head-tail combinations, it sees -if it refers to an existing directory, and if so, adds it to ``sys.path`` and -also inspects the newly added path for configuration files. +:file:`lib/python{X.Y}/site-packages` (on Unix and Macintosh). For each +of the distinct head-tail combinations, it sees if it refers to an existing +directory, and if so, adds it to ``sys.path`` and also inspects the newly +added path for configuration files. -.. deprecated:: 3.4 - Support for the "site-python" directory will be removed in 3.5. +.. versionchanged:: 3.5 + Support for the "site-python" directory has been removed. If a file named "pyvenv.cfg" exists one directory above sys.executable, sys.prefix and sys.exec_prefix are set to that directory and -it is also checked for site-packages and site-python (sys.base_prefix and +it is also checked for site-packages (sys.base_prefix and sys.base_exec_prefix will always be the "real" prefixes of the Python installation). If "pyvenv.cfg" (a bootstrap configuration file) contains the key "include-system-site-packages" set to anything other than "false" @@ -195,8 +194,7 @@ Module contents .. function:: getsitepackages() - Return a list containing all global site-packages directories (and possibly - site-python). + Return a list containing all global site-packages directories. .. versionadded:: 3.2 diff --git a/Doc/library/smtpd.rst b/Doc/library/smtpd.rst index 3ebed06260..977f9a8748 100644 --- a/Doc/library/smtpd.rst +++ b/Doc/library/smtpd.rst @@ -20,7 +20,8 @@ specific mail-sending strategies. Additionally the SMTPChannel may be extended to implement very specific interaction behaviour with SMTP clients. -The code supports :RFC:`5321`, plus the :rfc:`1870` SIZE extension. +The code supports :RFC:`5321`, plus the :rfc:`1870` SIZE and :rfc:`6531` +SMTPUTF8 extensions. SMTPServer Objects @@ -28,7 +29,7 @@ SMTPServer Objects .. class:: SMTPServer(localaddr, remoteaddr, data_size_limit=33554432,\ - map=None) + map=None, enable_SMTPUTF8=False, decode_data=True) Create a new :class:`SMTPServer` object, which binds to local address *localaddr*. It will treat *remoteaddr* as an upstream SMTP relayer. It @@ -39,25 +40,77 @@ SMTPServer Objects accepted in a ``DATA`` command. A value of ``None`` or ``0`` means no limit. - A dictionary can be specified in *map* to avoid using a global socket map. - - .. method:: process_message(peer, mailfrom, rcpttos, data) - - Raise :exc:`NotImplementedError` exception. Override this in subclasses to + *map* is the socket map to use for connections (an initially empty + dictionary is a suitable value). If not specified the :mod:`asyncore` + global socket map is used. + + *enable_SMTPUTF8* determins whether the ``SMTPUTF8`` extension (as defined + in :RFC:`6531`) should be enabled. The default is ``False``. If set to + ``True``, *decode_data* must be ``False`` (otherwise an error is raised). + When ``True``, ``SMTPUTF8`` is accepted as a parameter to the ``MAIL`` + command and when present is passed to :meth:`process_message` in the + ``kwargs['mail_options']`` list. + + *decode_data* specifies whether the data portion of the SMTP transaction + should be decoded using UTF-8. The default is ``True`` for backward + compatibility reasons, but will change to ``False`` in Python 3.6; specify + the keyword value explicitly to avoid the :exc:`DeprecationWarning`. When + *decode_data* is set to ``False`` the server advertises the ``8BITMIME`` + extension (:rfc:`6152`), accepts the ``BODY=8BITMIME`` parameter to + the ``MAIL`` command, and when present passes it to :meth:`process_message` + in the ``kwargs['mail_options']`` list. + + .. method:: process_message(peer, mailfrom, rcpttos, data, **kwargs) + + Raise a :exc:`NotImplementedError` exception. Override this in subclasses to do something useful with this message. Whatever was passed in the constructor as *remoteaddr* will be available as the :attr:`_remoteaddr` attribute. *peer* is the remote host's address, *mailfrom* is the envelope originator, *rcpttos* are the envelope recipients and *data* is a string - containing the contents of the e-mail (which should be in :rfc:`2822` + containing the contents of the e-mail (which should be in :rfc:`5321` format). + If the *decode_data* constructor keyword is set to ``True``, the *data* + argument will be a unicode string. If it is set to ``False``, it + will be a bytes object. + + *kwargs* is a dictionary containing additional information. It is empty + unless at least one of ``decode_data=False`` or ``enable_SMTPUTF8=True`` + was given as an init parameter, in which case it contains the following + keys: + + *mail_options*: + a list of all received parameters to the ``MAIL`` + command (the elements are uppercase strings; example: + ``['BODY=8BITMIME', 'SMTPUTF8']``). + + *rcpt_options*: + same as *mail_options* but for the ``RCPT`` command. + Currently no ``RCPT TO`` options are supported, so for now + this will always be an empty list. + + Implementations of ``process_message`` should use the ``**kwargs`` + signature to accept arbitrary keyword arguments, since future feature + enhancements may add keys to the kwargs dictionary. + + Return ``None`` to request a normal ``250 Ok`` response; otherwise + return the desired response string in :RFC:`5321` format. + .. attribute:: channel_class Override this in subclasses to use a custom :class:`SMTPChannel` for managing SMTP clients. - .. versionchanged:: 3.4 - The *map* argument was added. + .. versionadded:: 3.4 + The *map* constructor argument. + + .. versionchanged:: 3.5 + *localaddr* and *remoteaddr* may now contain IPv6 addresses. + + .. versionadded:: 3.5 + the *decode_data* and *enable_SMTPUTF8* constructor arguments, and the + *kwargs* argument to :meth:`process_message` when one or more of these is + specified. DebuggingServer Objects @@ -97,7 +150,7 @@ SMTPChannel Objects ------------------- .. class:: SMTPChannel(server, conn, addr, data_size_limit=33554432,\ - map=None)) + map=None, enable_SMTPUTF8=False, decode_data=True) Create a new :class:`SMTPChannel` object which manages the communication between the server and a single SMTP client. @@ -108,11 +161,24 @@ SMTPChannel Objects accepted in a ``DATA`` command. A value of ``None`` or ``0`` means no limit. + *enable_SMTPUTF8* determins whether the ``SMTPUTF8`` extension (as defined + in :RFC:`6531`) should be enabled. The default is ``False``. A + :exc:`ValueError` is raised if both *enable_SMTPUTF8* and *decode_data* are + set to ``True`` at the same time. + A dictionary can be specified in *map* to avoid using a global socket map. + *decode_data* specifies whether the data portion of the SMTP transaction + should be decoded using UTF-8. The default is ``True`` for backward + compatibility reasons, but will change to ``False`` in Python 3.6. Specify + the keyword value explicitly to avoid the :exc:`DeprecationWarning`. + To use a custom SMTPChannel implementation you need to override the :attr:`SMTPServer.channel_class` of your :class:`SMTPServer`. + .. versionchanged:: 3.5 + the *decode_data* and *enable_SMTPUTF8* arguments were added. + The :class:`SMTPChannel` has the following instance variables: .. attribute:: smtp_server diff --git a/Doc/library/smtplib.rst b/Doc/library/smtplib.rst index 8e1bfb57f9..a71ee585a3 100644 --- a/Doc/library/smtplib.rst +++ b/Doc/library/smtplib.rst @@ -61,6 +61,10 @@ Protocol) and :rfc:`1869` (SMTP Service Extensions). .. versionchanged:: 3.3 source_address argument was added. + .. versionadded:: 3.5 + The SMTPUTF8 extension (:rfc:`6531`) is now supported. + + .. class:: SMTP_SSL(host='', port=0, local_hostname=None, keyfile=None, \ certfile=None [, timeout], context=None, \ source_address=None) @@ -161,6 +165,13 @@ A nice selection of exceptions is defined as well: The server refused our ``HELO`` message. +.. exception:: SMTPNotSupportedError + + The command or option attempted is not supported by the server. + + .. versionadded:: 3.5 + + .. exception:: SMTPAuthenticationError SMTP authentication went wrong. Most probably the server didn't accept the @@ -189,8 +200,12 @@ An :class:`SMTP` instance has the following methods: .. method:: SMTP.set_debuglevel(level) - Set the debug output level. A true value for *level* results in debug messages - for connection and for all messages sent to and received from the server. + Set the debug output level. A value of 1 or ``True`` for *level* results in + debug messages for connection and for all messages sent to and received from + the server. A value of 2 for *level* results in these messages being + timestamped. + + .. versionchanged:: 3.5 Added debuglevel 2. .. method:: SMTP.docmd(cmd, args='') @@ -240,8 +255,7 @@ An :class:`SMTP` instance has the following methods: the server is stored as the :attr:`ehlo_resp` attribute, :attr:`does_esmtp` is set to true or false depending on whether the server supports ESMTP, and :attr:`esmtp_features` will be a dictionary containing the names of the - SMTP service extensions this server supports, and their - parameters (if any). + SMTP service extensions this server supports, and their parameters (if any). Unless you wish to use :meth:`has_extn` before sending mail, it should not be necessary to call this method explicitly. It will be implicitly called by @@ -274,7 +288,7 @@ An :class:`SMTP` instance has the following methods: Many sites disable SMTP ``VRFY`` in order to foil spammers. -.. method:: SMTP.login(user, password) +.. method:: SMTP.login(user, password, *, initial_response_ok=True) Log in on an SMTP server that requires authentication. The arguments are the username and the password to authenticate with. If there has been no previous @@ -288,9 +302,68 @@ An :class:`SMTP` instance has the following methods: :exc:`SMTPAuthenticationError` The server didn't accept the username/password combination. + :exc:`SMTPNotSupportedError` + The ``AUTH`` command is not supported by the server. + :exc:`SMTPException` No suitable authentication method was found. + Each of the authentication methods supported by :mod:`smtplib` are tried in + turn if they are advertised as supported by the server. See :meth:`auth` + for a list of supported authentication methods. *initial_response_ok* is + passed through to :meth:`auth`. + + Optional keyword argument *initial_response_ok* specifies whether, for + authentication methods that support it, an "initial response" as specified + in :rfc:`4954` can be sent along with the ``AUTH`` command, rather than + requiring a challenge/response. + + .. versionchanged:: 3.5 + :exc:`SMTPNotSupportedError` may be raised, and the + *initial_response_ok* parameter was added. + + +.. method:: SMTP.auth(mechanism, authobject, *, initial_response_ok=True) + + Issue an ``SMTP`` ``AUTH`` command for the specified authentication + *mechanism*, and handle the challenge response via *authobject*. + + *mechanism* specifies which authentication mechanism is to + be used as argument to the ``AUTH`` command; the valid values are + those listed in the ``auth`` element of :attr:`esmtp_features`. + + *authobject* must be a callable object taking an optional single argument: + + data = authobject(challenge=None) + + If optional keyword argument *initial_response_ok* is true, + ``authobject()`` will be called first with no argument. It can return the + :rfc:`4954` "initial response" bytes which will be encoded and sent with + the ``AUTH`` command as below. If the ``authobject()`` does not support an + initial response (e.g. because it requires a challenge), it should return + None when called with ``challenge=None``. If *initial_response_ok* is + false, then ``authobject()`` will not be called first with None. + + If the initial response check returns None, or if *initial_response_ok* is + false, ``authobject()`` will be called to process the server's challenge + response; the *challenge* argument it is passed will be a ``bytes``. It + should return ``bytes`` *data* that will be base64 encoded and sent to the + server. + + The ``SMTP`` class provides ``authobjects`` for the ``CRAM-MD5``, ``PLAIN``, + and ``LOGIN`` mechanisms; they are named ``SMTP.auth_cram_md5``, + ``SMTP.auth_plain``, and ``SMTP.auth_login`` respectively. They all require + that the ``user`` and ``password`` properties of the ``SMTP`` instance are + set to appropriate values. + + User code does not normally need to call ``auth`` directly, but can instead + call the :meth:`login` method, which will try each of the above mechanisms + in turn, in the order listed. ``auth`` is exposed to facilitate the + implementation of authentication methods not (or not yet) supported + directly by :mod:`smtplib`. + + .. versionadded:: 3.5 + .. method:: SMTP.starttls(keyfile=None, certfile=None, context=None) @@ -310,7 +383,7 @@ An :class:`SMTP` instance has the following methods: :exc:`SMTPHeloError` The server didn't reply properly to the ``HELO`` greeting. - :exc:`SMTPException` + :exc:`SMTPNotSupportedError` The server does not support the STARTTLS extension. :exc:`RuntimeError` @@ -324,6 +397,11 @@ An :class:`SMTP` instance has the following methods: :attr:`SSLContext.check_hostname` and *Server Name Indicator* (see :data:`~ssl.HAS_SNI`). + .. versionchanged:: 3.5 + The error raised for lack of STARTTLS support is now the + :exc:`SMTPNotSupportedError` subclass instead of the base + :exc:`SMTPException`. + .. method:: SMTP.sendmail(from_addr, to_addrs, msg, mail_options=[], rcpt_options=[]) @@ -360,6 +438,9 @@ An :class:`SMTP` instance has the following methods: recipient that was refused. Each entry contains a tuple of the SMTP error code and the accompanying error message sent by the server. + If ``SMTPUTF8`` is included in *mail_options*, and the server supports it, + *from_addr* and *to_addr* may contain non-ASCII characters. + This method may raise the following exceptions: :exc:`SMTPRecipientsRefused` @@ -378,12 +459,20 @@ An :class:`SMTP` instance has the following methods: The server replied with an unexpected error code (other than a refusal of a recipient). + :exc:`SMTPNotSupportedError` + ``SMTPUTF8`` was given in the *mail_options* but is not supported by the + server. + Unless otherwise noted, the connection will be open even after an exception is raised. .. versionchanged:: 3.2 *msg* may be a byte string. + .. versionchanged:: 3.5 + ``SMTPUTF8`` support added, and :exc:`SMTPNotSupportedError` may be + raised if ``SMTPUTF8`` is specified but the server does not support it. + .. method:: SMTP.send_message(msg, from_addr=None, to_addrs=None, \ mail_options=[], rcpt_options=[]) @@ -395,7 +484,7 @@ An :class:`SMTP` instance has the following methods: If *from_addr* is ``None`` or *to_addrs* is ``None``, ``send_message`` fills those arguments with addresses extracted from the headers of *msg* as - specified in :rfc:`2822`\: *from_addr* is set to the :mailheader:`Sender` + specified in :rfc:`5322`\: *from_addr* is set to the :mailheader:`Sender` field if it is present, and otherwise to the :mailheader:`From` field. *to_adresses* combines the values (if any) of the :mailheader:`To`, :mailheader:`Cc`, and :mailheader:`Bcc` fields from *msg*. If exactly one @@ -410,10 +499,18 @@ An :class:`SMTP` instance has the following methods: calls :meth:`sendmail` to transmit the resulting message. Regardless of the values of *from_addr* and *to_addrs*, ``send_message`` does not transmit any :mailheader:`Bcc` or :mailheader:`Resent-Bcc` headers that may appear - in *msg*. + in *msg*. If any of the addresses in *from_addr* and *to_addrs* contain + non-ASCII characters and the server does not advertise ``SMTPUTF8`` support, + an :exc:`SMTPNotSupported` error is raised. Otherwise the ``Message`` is + serialized with a clone of its :mod:`~email.policy` with the + :attr:`~email.policy.EmailPolicy.utf8` attribute set to ``True``, and + ``SMTPUTF8`` and ``BODY=8BITMIME`` are added to *mail_options*. .. versionadded:: 3.2 + .. versionadded:: 3.5 + Support for internationalized addresses (``SMTPUTF8``). + .. method:: SMTP.quit() diff --git a/Doc/library/sndhdr.rst b/Doc/library/sndhdr.rst index f36df68703..f8b5d8b246 100644 --- a/Doc/library/sndhdr.rst +++ b/Doc/library/sndhdr.rst @@ -16,8 +16,9 @@ The :mod:`sndhdr` provides utility functions which attempt to determine the type of sound data which is in a file. When these functions are able to determine -what type of sound data is stored in a file, they return a tuple ``(type, -sampling_rate, channels, frames, bits_per_sample)``. The value for *type* +what type of sound data is stored in a file, they return a +:func:`~collections.namedtuple`, containing five attributes: (``filetype``, +``framerate``, ``nchannels``, ``nframes``, ``sampwidth``). The value for *type* indicates the data type and will be one of the strings ``'aifc'``, ``'aiff'``, ``'au'``, ``'hcom'``, ``'sndr'``, ``'sndt'``, ``'voc'``, ``'wav'``, ``'8svx'``, ``'sb'``, ``'ub'``, or ``'ul'``. The *sampling_rate* will be either the actual @@ -31,13 +32,19 @@ be the sample size in bits or ``'A'`` for A-LAW or ``'U'`` for u-LAW. .. function:: what(filename) Determines the type of sound data stored in the file *filename* using - :func:`whathdr`. If it succeeds, returns a tuple as described above, otherwise + :func:`whathdr`. If it succeeds, returns a namedtuple as described above, otherwise ``None`` is returned. + .. versionchanged:: 3.5 + Result changed from a tuple to a namedtuple. + .. function:: whathdr(filename) Determines the type of sound data stored in a file based on the file header. - The name of the file is given by *filename*. This function returns a tuple as + The name of the file is given by *filename*. This function returns a namedtuple as described above on success, or ``None``. + .. versionchanged:: 3.5 + Result changed from a tuple to a namedtuple. + diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst index 271dda8212..e583ebd83d 100644 --- a/Doc/library/socket.rst +++ b/Doc/library/socket.rst @@ -46,17 +46,20 @@ created. Socket addresses are represented as follows: - The address of an :const:`AF_UNIX` socket bound to a file system node is represented as a string, using the file system encoding and the ``'surrogateescape'`` error handler (see :pep:`383`). An address in - Linux's abstract namespace is returned as a :class:`bytes` object with + Linux's abstract namespace is returned as a :term:`bytes-like object` with an initial null byte; note that sockets in this namespace can communicate with normal file system sockets, so programs intended to run on Linux may need to deal with both types of address. A string or - :class:`bytes` object can be used for either type of address when + bytes-like object can be used for either type of address when passing it as an argument. .. versionchanged:: 3.3 Previously, :const:`AF_UNIX` socket paths were assumed to use UTF-8 encoding. + .. versionchanged: 3.5 + Writable :term:`bytes-like object` is now accepted. + - A pair ``(host, port)`` is used for the :const:`AF_INET` address family, where *host* is a string representing either a hostname in Internet domain notation like ``'daring.cwi.nl'`` or an IPv4 address like ``'100.50.200.5'``, @@ -298,6 +301,18 @@ Constants .. versionadded:: 3.4 +.. data:: CAN_RAW_FD_FRAMES + + Enables CAN FD support in a CAN_RAW socket. This is disabled by default. + This allows your application to send both CAN and CAN FD frames; however, + you one must accept both CAN and CAN FD frames when reading from the socket. + + This constant is documented in the Linux documentation. + + Availability: Linux >= 3.6. + + .. versionadded:: 3.5 + .. data:: AF_RDS PF_RDS SOL_RDS @@ -390,7 +405,6 @@ The following functions all create :ref:`socket objects <socket-objects>`. type, and protocol number. Address family, socket type, and protocol number are as for the :func:`.socket` function above. The default family is :const:`AF_UNIX` if defined on the platform; otherwise, the default is :const:`AF_INET`. - Availability: Unix. The newly created sockets are :ref:`non-inheritable <fd_inheritance>`. @@ -401,6 +415,9 @@ The following functions all create :ref:`socket objects <socket-objects>`. .. versionchanged:: 3.4 The returned sockets are now non-inheritable. + .. versionchanged:: 3.5 + Windows support added. + .. function:: create_connection(address[, timeout[, source_address]]) @@ -647,8 +664,8 @@ The :mod:`socket` module also offers various network-related services: .. function:: inet_ntoa(packed_ip) - Convert a 32-bit packed IPv4 address (a bytes object four characters in - length) to its standard dotted-quad string representation (for example, + Convert a 32-bit packed IPv4 address (a :term:`bytes-like object` four + bytes in length) to its standard dotted-quad string representation (for example, '123.45.67.89'). This is useful when conversing with a program that uses the standard C library and needs objects of type :c:type:`struct in_addr`, which is the C type for the 32-bit packed binary data this function takes as an @@ -659,6 +676,9 @@ The :mod:`socket` module also offers various network-related services: support IPv6, and :func:`inet_ntop` should be used instead for IPv4/v6 dual stack support. + .. versionchanged: 3.5 + Writable :term:`bytes-like object` is now accepted. + .. function:: inet_pton(address_family, ip_string) @@ -681,22 +701,26 @@ The :mod:`socket` module also offers various network-related services: .. function:: inet_ntop(address_family, packed_ip) - Convert a packed IP address (a bytes object of some number of characters) to its - standard, family-specific string representation (for example, ``'7.10.0.5'`` or - ``'5aef:2b::8'``). :func:`inet_ntop` is useful when a library or network protocol - returns an object of type :c:type:`struct in_addr` (similar to :func:`inet_ntoa`) - or :c:type:`struct in6_addr`. + Convert a packed IP address (a :term:`bytes-like object` of some number of + bytes) to its standard, family-specific string representation (for + example, ``'7.10.0.5'`` or ``'5aef:2b::8'``). + :func:`inet_ntop` is useful when a library or network protocol returns an + object of type :c:type:`struct in_addr` (similar to :func:`inet_ntoa`) or + :c:type:`struct in6_addr`. Supported values for *address_family* are currently :const:`AF_INET` and - :const:`AF_INET6`. If the string *packed_ip* is not the correct length for the - specified address family, :exc:`ValueError` will be raised. A - :exc:`OSError` is raised for errors from the call to :func:`inet_ntop`. + :const:`AF_INET6`. If the bytes object *packed_ip* is not the correct + length for the specified address family, :exc:`ValueError` will be raised. + A :exc:`OSError` is raised for errors from the call to :func:`inet_ntop`. Availability: Unix (maybe not all platforms), Windows. .. versionchanged:: 3.4 Windows support added + .. versionchanged: 3.5 + Writable :term:`bytes-like object` is now accepted. + .. XXX: Are sendmsg(), recvmsg() and CMSG_*() available on any @@ -821,6 +845,11 @@ to sockets. .. versionchanged:: 3.4 The socket is now non-inheritable. + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise + an exception, the method now retries the system call instead of raising + an :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + .. method:: socket.bind(address) @@ -853,6 +882,19 @@ to sockets. Connect to a remote socket at *address*. (The format of *address* depends on the address family --- see above.) + If the connection is interrupted by a signal, the method waits until the + connection completes, or raise a :exc:`socket.timeout` on timeout, if the + signal handler doesn't raise an exception and the socket is blocking or has + a timeout. For non-blocking sockets, the method raises an + :exc:`InterruptedError` exception if the connection is interrupted by a + signal (or the exception raised by the signal handler). + + .. versionchanged:: 3.5 + The method now waits until the connection completes instead of raising an + :exc:`InterruptedError` exception if the connection is interrupted by a + signal, the signal handler doesn't raise an exception and the socket is + blocking or has a timeout (see the :pep:`475` for the rationale). + .. method:: socket.connect_ex(address) @@ -948,12 +990,15 @@ to sockets. On other platforms, the generic :func:`fcntl.fcntl` and :func:`fcntl.ioctl` functions may be used; they accept a socket object as their first argument. -.. method:: socket.listen(backlog) +.. method:: socket.listen([backlog]) - Listen for connections made to the socket. The *backlog* argument specifies the - maximum number of queued connections and should be at least 0; the maximum value - is system-dependent (usually 5), the minimum value is forced to 0. + Enable a server to accept connections. If *backlog* is specified, it must + be at least 0 (if it is lower, it is set to 0); it specifies the number of + unaccepted connections that the system will allow before refusing new + connections. If not specified, a default reasonable value is chosen. + .. versionchanged:: 3.5 + The *backlog* parameter is now optional. .. method:: socket.makefile(mode='r', buffering=None, *, encoding=None, \ errors=None, newline=None) @@ -991,6 +1036,11 @@ to sockets. For best match with hardware and network realities, the value of *bufsize* should be a relatively small power of 2, for example, 4096. + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise + an exception, the method now retries the system call instead of raising + an :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + .. method:: socket.recvfrom(bufsize[, flags]) @@ -1000,6 +1050,11 @@ to sockets. :manpage:`recv(2)` for the meaning of the optional argument *flags*; it defaults to zero. (The format of *address* depends on the address family --- see above.) + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise + an exception, the method now retries the system call instead of raising + an :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + .. method:: socket.recvmsg(bufsize[, ancbufsize[, flags]]) @@ -1066,6 +1121,11 @@ to sockets. .. versionadded:: 3.3 + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise + an exception, the method now retries the system call instead of raising + an :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + .. method:: socket.recvmsg_into(buffers[, ancbufsize[, flags]]) @@ -1132,6 +1192,11 @@ to sockets. application needs to attempt delivery of the remaining data. For further information on this topic, consult the :ref:`socket-howto`. + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise + an exception, the method now retries the system call instead of raising + an :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + .. method:: socket.sendall(bytes[, flags]) @@ -1142,6 +1207,15 @@ to sockets. success. On error, an exception is raised, and there is no way to determine how much data, if any, was successfully sent. + .. versionchanged:: 3.5 + The socket timeout is no more reset each time data is sent successfuly. + The socket timeout is now the maximum total duration to send all data. + + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise + an exception, the method now retries the system call instead of raising + an :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + .. method:: socket.sendto(bytes, address) socket.sendto(bytes, flags, address) @@ -1152,6 +1226,11 @@ to sockets. bytes sent. (The format of *address* depends on the address family --- see above.) + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise + an exception, the method now retries the system call instead of raising + an :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + .. method:: socket.sendmsg(buffers[, ancdata[, flags[, address]]]) @@ -1188,6 +1267,26 @@ to sockets. .. versionadded:: 3.3 + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise + an exception, the method now retries the system call instead of raising + an :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + +.. method:: socket.sendfile(file, offset=0, count=None) + + Send a file until EOF is reached by using high-performance + :mod:`os.sendfile` and return the total number of bytes which were sent. + *file* must be a regular file object opened in binary mode. If + :mod:`os.sendfile` is not available (e.g. Windows) or *file* is not a + regular file :meth:`send` will be used instead. *offset* tells from where to + start reading the file. If specified, *count* is the total number of bytes + to transmit as opposed to sending the file until EOF is reached. File + position is updated on return or also in case of error in which case + :meth:`file.tell() <io.IOBase.tell>` can be used to figure out the number of + bytes which were sent. The socket must be of :const:`SOCK_STREAM` type. Non- + blocking sockets are not supported. + + .. versionadded:: 3.5 .. method:: socket.set_inheritable(inheritable) @@ -1227,11 +1326,15 @@ to sockets. Set the value of the given socket option (see the Unix manual page :manpage:`setsockopt(2)`). The needed symbolic constants are defined in the - :mod:`socket` module (:const:`SO_\*` etc.). The value can be an integer or a - bytes object representing a buffer. In the latter case it is up to the caller to + :mod:`socket` module (:const:`SO_\*` etc.). The value can be an integer or + a :term:`bytes-like object` representing a buffer. In the latter case it is + up to the caller to ensure that the bytestring contains the proper bits (see the optional built-in module :mod:`struct` for a way to encode C structures as bytestrings). + .. versionchanged: 3.5 + Writable :term:`bytes-like object` is now accepted. + .. method:: socket.shutdown(how) diff --git a/Doc/library/socketserver.rst b/Doc/library/socketserver.rst index 3e49af6bba..18be936b08 100644 --- a/Doc/library/socketserver.rst +++ b/Doc/library/socketserver.rst @@ -114,7 +114,7 @@ the request handler class :meth:`handle` method. Another approach to handling multiple simultaneous requests in an environment that supports neither threads nor :func:`~os.fork` (or where these are too expensive or inappropriate for the service) is to maintain an explicit table of -partially finished requests and to use :func:`~select.select` to decide which +partially finished requests and to use :mod:`selectors` to decide which request to work on next (or whether to handle a new incoming request). This is particularly important for stream services where each client can potentially be connected for a long time (if threads or subprocesses cannot be used). See @@ -137,7 +137,7 @@ Server Objects .. method:: BaseServer.fileno() Return an integer file descriptor for the socket on which the server is - listening. This function is most commonly passed to :func:`select.select`, to + listening. This function is most commonly passed to :mod:`selectors`, to allow monitoring multiple servers in the same process. diff --git a/Doc/library/sqlite3.rst b/Doc/library/sqlite3.rst index 6097e7a30d..fc69a804d5 100644 --- a/Doc/library/sqlite3.rst +++ b/Doc/library/sqlite3.rst @@ -649,6 +649,9 @@ Row Objects This method returns a list of column names. Immediately after a query, it is the first member of each tuple in :attr:`Cursor.description`. + .. versionchanged:: 3.5 + Added support of slicing. + Let's assume we initialize a table as in the example given above:: conn = sqlite3.connect(":memory:") diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst index 233de8dd91..40751347ae 100644 --- a/Doc/library/ssl.rst +++ b/Doc/library/ssl.rst @@ -315,6 +315,8 @@ Random generation For almost all applications :func:`os.urandom` is preferable. + For almost all applications :func:`os.urandom` is preferable. + .. versionadded:: 3.3 .. function:: RAND_status() @@ -335,6 +337,8 @@ Random generation See http://egd.sourceforge.net/ or http://prngd.sourceforge.net/ for sources of entropy-gathering daemons. + Availability: not available with LibreSSL. + .. function:: RAND_add(bytes, entropy) Mix the given *bytes* into the SSL pseudo-random number generator. The @@ -342,6 +346,9 @@ Random generation string (so you can always use :const:`0.0`). See :rfc:`1750` for more information on sources of entropy. + .. versionchanged: 3.5 + Writable :term:`bytes-like object` is now accepted. + Certificate handling ^^^^^^^^^^^^^^^^^^^^ @@ -350,10 +357,9 @@ Certificate handling Verify that *cert* (in decoded format as returned by :meth:`SSLSocket.getpeercert`) matches the given *hostname*. The rules applied are those for checking the identity of HTTPS servers as outlined - in :rfc:`2818` and :rfc:`6125`, except that IP addresses are not currently - supported. In addition to HTTPS, this function should be suitable for - checking the identity of servers in various SSL-based protocols such as - FTPS, IMAPS, POPS and others. + in :rfc:`2818` and :rfc:`6125`. In addition to HTTPS, this function + should be suitable for checking the identity of servers in various + SSL-based protocols such as FTPS, IMAPS, POPS and others. :exc:`CertificateError` is raised on failure. On success, the function returns nothing:: @@ -375,22 +381,38 @@ Certificate handling IDN A-labels such as ``www*.xn--pthon-kva.org`` are still supported, but ``x*.python.org`` no longer matches ``xn--tda.python.org``. -.. function:: cert_time_to_seconds(timestring) + .. versionchanged:: 3.5 + Matching of IP addresses, when present in the subjectAltName field + of the certificate, is now supported. + +.. function:: cert_time_to_seconds(cert_time) - Returns a floating-point value containing a normal seconds-after-the-epoch - time value, given the time-string representing the "notBefore" or "notAfter" - date from a certificate. + Return the time in seconds since the Epoch, given the ``cert_time`` + string representing the "notBefore" or "notAfter" date from a + certificate in ``"%b %d %H:%M:%S %Y %Z"`` strptime format (C + locale). - Here's an example:: + Here's an example: - >>> import ssl - >>> ssl.cert_time_to_seconds("May 9 00:00:00 2007 GMT") - 1178694000.0 - >>> import time - >>> time.ctime(ssl.cert_time_to_seconds("May 9 00:00:00 2007 GMT")) - 'Wed May 9 00:00:00 2007' + .. doctest:: newcontext -.. function:: get_server_certificate(addr, ssl_version=PROTOCOL_SSLv3, ca_certs=None) + >>> import ssl + >>> timestamp = ssl.cert_time_to_seconds("Jan 5 09:34:43 2018 GMT") + >>> timestamp + 1515144883 + >>> from datetime import datetime + >>> print(datetime.utcfromtimestamp(timestamp)) + 2018-01-05 09:34:43 + + "notBefore" or "notAfter" dates must use GMT (:rfc:`5280`). + + .. versionchanged:: 3.5 + Interpret the input time as a time in UTC as specified by 'GMT' + timezone in the input string. Local timezone was used + previously. Return an integer (no fractions of a second in the + input format) + +.. function:: get_server_certificate(addr, ssl_version=PROTOCOL_SSLv23, ca_certs=None) Given the address ``addr`` of an SSL-protected server, as a (*hostname*, *port-number*) pair, fetches the server's certificate, and returns it as a @@ -404,6 +426,10 @@ Certificate handling .. versionchanged:: 3.3 This function is now IPv6-compatible. + .. versionchanged:: 3.5 + The default *ssl_version* is changed from :data:`PROTOCOL_SSLv3` to + :data:`PROTOCOL_SSLv23` for maximum compatibility with modern servers. + .. function:: DER_cert_to_PEM_cert(DER_cert_bytes) Given a certificate as a DER-encoded blob of bytes, returns a PEM-encoded @@ -671,6 +697,13 @@ Constants .. versionadded:: 3.3 +.. data:: HAS_ALPN + + Whether the OpenSSL library has built-in support for the *Application-Layer + Protocol Negotiation* TLS extension as described in :rfc:`7301`. + + .. versionadded:: 3.5 + .. data:: HAS_ECDH Whether the OpenSSL library has built-in support for Elliptic Curve-based @@ -788,6 +821,8 @@ SSL Sockets (but passing a non-zero ``flags`` argument is not allowed) - :meth:`~socket.socket.send()`, :meth:`~socket.socket.sendall()` (with the same limitation) + - :meth:`~socket.socket.sendfile()` (but :mod:`os.sendfile` will be used + for plain-text sockets only, else :meth:`~socket.socket.send()` will be used) - :meth:`~socket.socket.shutdown()` However, since the SSL (and TLS) protocol has its own framing atop @@ -798,6 +833,15 @@ SSL Sockets Usually, :class:`SSLSocket` are not created directly, but using the :func:`wrap_socket` function or the :meth:`SSLContext.wrap_socket` method. + .. versionchanged:: 3.5 + The :meth:`sendfile` method was added. + + .. versionchanged:: 3.5 + The :meth:`shutdown` does not reset the socket timeout each time bytes + are received or sent. The socket timeout is now to maximum total duration + of the shutdown. + + SSL sockets also have the following additional methods and attributes: .. method:: SSLSocket.read(len=0, buffer=None) @@ -812,6 +856,11 @@ SSL sockets also have the following additional methods and attributes: As at any time a re-negotiation is possible, a call to :meth:`read` can also cause write operations. + .. versionchanged:: 3.5 + The socket timeout is no more reset each time bytes are received or sent. + The socket timeout is now to maximum total duration to read up to *len* + bytes. + .. method:: SSLSocket.write(buf) Write *buf* to the SSL socket and return the number of bytes written. The @@ -823,6 +872,10 @@ SSL sockets also have the following additional methods and attributes: As at any time a re-negotiation is possible, a call to :meth:`write` can also cause read operations. + .. versionchanged:: 3.5 + The socket timeout is no more reset each time bytes are received or sent. + The socket timeout is now to maximum total duration to write *buf*. + .. note:: The :meth:`~SSLSocket.read` and :meth:`~SSLSocket.write` methods are the @@ -844,6 +897,10 @@ SSL sockets also have the following additional methods and attributes: :attr:`~SSLContext.check_hostname` attribute of the socket's :attr:`~SSLSocket.context` is true. + .. versionchanged:: 3.5 + The socket timeout is no more reset each time bytes are received or sent. + The socket timeout is now to maximum total duration of the handshake. + .. method:: SSLSocket.getpeercert(binary_form=False) If there is no certificate for the peer on the other end of the connection, @@ -917,6 +974,17 @@ SSL sockets also have the following additional methods and attributes: version of the SSL protocol that defines its use, and the number of secret bits being used. If no connection has been established, returns ``None``. +.. method:: SSLSocket.shared_ciphers() + + Return the list of ciphers shared by the client during the handshake. Each + entry of the returned list is a three-value tuple containing the name of the + cipher, the version of the SSL protocol that defines its use, and the number + of secret bits the cipher uses. :meth:`~SSLSocket.shared_ciphers` returns + ``None`` if no connection has been established or the socket is a client + socket. + + .. versionadded:: 3.5 + .. method:: SSLSocket.compression() Return the compression algorithm being used as a string, or ``None`` @@ -940,12 +1008,22 @@ SSL sockets also have the following additional methods and attributes: .. versionadded:: 3.3 +.. method:: SSLSocket.selected_alpn_protocol() + + Return the protocol that was selected during the TLS handshake. If + :meth:`SSLContext.set_alpn_protocols` was not called, if the other party does + not support ALPN, if this socket does not support any of the client's + proposed protocols, or if the handshake has not happened yet, ``None`` is + returned. + + .. versionadded:: 3.5 + .. method:: SSLSocket.selected_npn_protocol() - Returns the protocol that was selected during the TLS/SSL handshake. If - :meth:`SSLContext.set_npn_protocols` was not called, or if the other party - does not support NPN, or if the handshake has not yet happened, this will - return ``None``. + Return the higher-level protocol that was selected during the TLS/SSL + handshake. If :meth:`SSLContext.set_npn_protocols` was not called, or + if the other party does not support NPN, or if the handshake has not yet + happened, this will return ``None``. .. versionadded:: 3.3 @@ -957,6 +1035,16 @@ SSL sockets also have the following additional methods and attributes: returned socket should always be used for further communication with the other side of the connection, rather than the original socket. +.. method:: SSLSocket.version() + + Return the actual SSL protocol version negotiated by the connection + as a string, or ``None`` is no secure connection is established. + As of this writing, possible return values include ``"SSLv2"``, + ``"SSLv3"``, ``"TLSv1"``, ``"TLSv1.1"`` and ``"TLSv1.2"``. + Recent OpenSSL versions may define more return values. + + .. versionadded:: 3.5 + .. method:: SSLSocket.pending() Returns the number of already decrypted bytes available for read, pending on @@ -1135,6 +1223,20 @@ to speed up repeated connections from the same clients. when connected, the :meth:`SSLSocket.cipher` method of SSL sockets will give the currently selected cipher. +.. method:: SSLContext.set_alpn_protocols(protocols) + + Specify which protocols the socket should advertise during the SSL/TLS + handshake. It should be a list of ASCII strings, like ``['http/1.1', + 'spdy/2']``, ordered by preference. The selection of a protocol will happen + during the handshake, and will play out according to :rfc:`7301`. After a + successful handshake, the :meth:`SSLSocket.selected_alpn_protocol` method will + return the agreed-upon protocol. + + This method will raise :exc:`NotImplementedError` if :data:`HAS_ALPN` is + False. + + .. versionadded:: 3.5 + .. method:: SSLContext.set_npn_protocols(protocols) Specify which protocols the socket should advertise during the SSL/TLS @@ -1175,7 +1277,7 @@ to speed up repeated connections from the same clients. Due to the early negotiation phase of the TLS connection, only limited methods and attributes are usable like - :meth:`SSLSocket.selected_npn_protocol` and :attr:`SSLSocket.context`. + :meth:`SSLSocket.selected_alpn_protocol` and :attr:`SSLSocket.context`. :meth:`SSLSocket.getpeercert`, :meth:`SSLSocket.getpeercert`, :meth:`SSLSocket.cipher` and :meth:`SSLSocket.compress` methods require that the TLS connection has progressed beyond the TLS Client Hello and therefore @@ -1251,10 +1353,20 @@ to speed up repeated connections from the same clients. quite similarly to HTTP virtual hosts. Specifying *server_hostname* will raise a :exc:`ValueError` if *server_side* is true. - .. versionchanged:: 3.4.3 + .. versionchanged:: 3.5 Always allow a server_hostname to be passed, even if OpenSSL does not have SNI. +.. method:: SSLContext.wrap_bio(incoming, outgoing, server_side=False, \ + server_hostname=None) + + Create a new :class:`SSLObject` instance by wrapping the BIO objects + *incoming* and *outgoing*. The SSL routines will read input data from the + incoming BIO and write data to the outgoing BIO. + + The *server_side* and *server_hostname* parameters have the same meaning as + in :meth:`SSLContext.wrap_socket`. + .. method:: SSLContext.session_stats() Get statistics about the SSL sessions created or managed by this context. @@ -1627,7 +1739,7 @@ are finished with the client (or the client is finished with you):: And go back to listening for new client connections (of course, a real server would probably handle each client connection in a separate thread, or put -the sockets in non-blocking mode and use an event loop). +the sockets in :ref:`non-blocking mode <ssl-nonblocking>` and use an event loop). .. _ssl-nonblocking: @@ -1649,6 +1761,12 @@ thus several things you need to be aware of: socket first, and attempts to *read* from the SSL socket may require a prior *write* to the underlying socket. + .. versionchanged:: 3.5 + + In earlier Python versions, the :meth:`!SSLSocket.send` method + returned zero instead of raising :exc:`SSLWantWriteError` or + :exc:`SSLWantReadError`. + - Calling :func:`~select.select` tells you that the OS-level socket can be read from (or written to), but it does not imply that there is sufficient data at the upper SSL layer. For example, only part of an SSL frame might @@ -1681,13 +1799,143 @@ thus several things you need to be aware of: .. seealso:: - The :mod:`asyncio` module supports non-blocking SSL sockets and provides a + The :mod:`asyncio` module supports :ref:`non-blocking SSL sockets + <ssl-nonblocking>` and provides a higher level API. It polls for events using the :mod:`selectors` module and handles :exc:`SSLWantWriteError`, :exc:`SSLWantReadError` and :exc:`BlockingIOError` exceptions. It runs the SSL handshake asynchronously as well. +Memory BIO Support +------------------ + +.. versionadded:: 3.5 + +Ever since the SSL module was introduced in Python 2.6, the :class:`SSLSocket` +class has provided two related but distinct areas of functionality: + +- SSL protocol handling +- Network IO + +The network IO API is identical to that provided by :class:`socket.socket`, +from which :class:`SSLSocket` also inherits. This allows an SSL socket to be +used as a drop-in replacement for a regular socket, making it very easy to add +SSL support to an existing application. + +Combining SSL protocol handling and network IO usually works well, but there +are some cases where it doesn't. An example is async IO frameworks that want to +use a different IO multiplexing model than the "select/poll on a file +descriptor" (readiness based) model that is assumed by :class:`socket.socket` +and by the internal OpenSSL socket IO routines. This is mostly relevant for +platforms like Windows where this model is not efficient. For this purpose, a +reduced scope variant of :class:`SSLSocket` called :class:`SSLObject` is +provided. + +.. class:: SSLObject + + A reduced-scope variant of :class:`SSLSocket` representing an SSL protocol + instance that does not contain any network IO methods. This class is + typically used by framework authors that want to implement asynchronous IO + for SSL through memory buffers. + + This class implements an interface on top of a low-level SSL object as + implemented by OpenSSL. This object captures the state of an SSL connection + but does not provide any network IO itself. IO needs to be performed through + separate "BIO" objects which are OpenSSL's IO abstraction layer. + + An :class:`SSLObject` instance can be created using the + :meth:`~SSLContext.wrap_bio` method. This method will create the + :class:`SSLObject` instance and bind it to a pair of BIOs. The *incoming* + BIO is used to pass data from Python to the SSL protocol instance, while the + *outgoing* BIO is used to pass data the other way around. + + The following methods are available: + + - :attr:`~SSLSocket.context` + - :attr:`~SSLSocket.server_side` + - :attr:`~SSLSocket.server_hostname` + - :meth:`~SSLSocket.read` + - :meth:`~SSLSocket.write` + - :meth:`~SSLSocket.getpeercert` + - :meth:`~SSLSocket.selected_npn_protocol` + - :meth:`~SSLSocket.cipher` + - :meth:`~SSLSocket.shared_ciphers` + - :meth:`~SSLSocket.compression` + - :meth:`~SSLSocket.pending` + - :meth:`~SSLSocket.do_handshake` + - :meth:`~SSLSocket.unwrap` + - :meth:`~SSLSocket.get_channel_binding` + + When compared to :class:`SSLSocket`, this object lacks the following + features: + + - Any form of network IO incluging methods such as ``recv()`` and + ``send()``. + + - There is no *do_handshake_on_connect* machinery. You must always manually + call :meth:`~SSLSocket.do_handshake` to start the handshake. + + - There is no handling of *suppress_ragged_eofs*. All end-of-file conditions + that are in violation of the protocol are reported via the + :exc:`SSLEOFError` exception. + + - The method :meth:`~SSLSocket.unwrap` call does not return anything, + unlike for an SSL socket where it returns the underlying socket. + + - The *server_name_callback* callback passed to + :meth:`SSLContext.set_servername_callback` will get an :class:`SSLObject` + instance instead of a :class:`SSLSocket` instance as its first parameter. + + Some notes related to the use of :class:`SSLObject`: + + - All IO on an :class:`SSLObject` is :ref:`non-blocking <ssl-nonblocking>`. + This means that for example :meth:`~SSLSocket.read` will raise an + :exc:`SSLWantReadError` if it needs more data than the incoming BIO has + available. + + - There is no module-level ``wrap_bio()`` call like there is for + :meth:`~SSLContext.wrap_socket`. An :class:`SSLObject` is always created + via an :class:`SSLContext`. + +An SSLObject communicates with the outside world using memory buffers. The +class :class:`MemoryBIO` provides a memory buffer that can be used for this +purpose. It wraps an OpenSSL memory BIO (Basic IO) object: + +.. class:: MemoryBIO + + A memory buffer that can be used to pass data between Python and an SSL + protocol instance. + + .. attribute:: MemoryBIO.pending + + Return the number of bytes currently in the memory buffer. + + .. attribute:: MemoryBIO.eof + + A boolean indicating whether the memory BIO is current at the end-of-file + position. + + .. method:: MemoryBIO.read(n=-1) + + Read up to *n* bytes from the memory buffer. If *n* is not specified or + negative, all bytes are returned. + + .. method:: MemoryBIO.write(buf) + + Write the bytes from *buf* to the memory BIO. The *buf* argument must be an + object supporting the buffer protocol. + + The return value is the number of bytes written, which is always equal to + the length of *buf*. + + .. method:: MemoryBIO.write_eof() + + Write an EOF marker to the memory BIO. After this method has been called, it + is illegal to call :meth:`~MemoryBIO.write`. The attribute :attr:`eof` will + become true after all data currently in the buffer has been read. + + .. _ssl-security: Security considerations diff --git a/Doc/library/stat.rst b/Doc/library/stat.rst index 24769f6893..845b2ef7da 100644 --- a/Doc/library/stat.rst +++ b/Doc/library/stat.rst @@ -126,7 +126,7 @@ Example:: if __name__ == '__main__': walktree(sys.argv[1], visitfile) -An additional utility function is provided to covert a file's mode in a human +An additional utility function is provided to convert a file's mode in a human readable string: .. function:: filemode(mode) @@ -399,3 +399,29 @@ The following flags can be used in the *flags* argument of :func:`os.chflags`: The file is a snapshot file. See the \*BSD or Mac OS systems man page :manpage:`chflags(2)` for more information. + +On Windows, the following file attribute constants are available for use when +testing bits in the ``st_file_attributes`` member returned by :func:`os.stat`. +See the `Windows API documentation +<http://msdn.microsoft.com/en-us/library/windows/desktop/gg258117.aspx>`_ +for more detail on the meaning of these constants. + +.. data:: FILE_ATTRIBUTE_ARCHIVE + FILE_ATTRIBUTE_COMPRESSED + FILE_ATTRIBUTE_DEVICE + FILE_ATTRIBUTE_DIRECTORY + FILE_ATTRIBUTE_ENCRYPTED + FILE_ATTRIBUTE_HIDDEN + FILE_ATTRIBUTE_INTEGRITY_STREAM + FILE_ATTRIBUTE_NORMAL + FILE_ATTRIBUTE_NOT_CONTENT_INDEXED + FILE_ATTRIBUTE_NO_SCRUB_DATA + FILE_ATTRIBUTE_OFFLINE + FILE_ATTRIBUTE_READONLY + FILE_ATTRIBUTE_REPARSE_POINT + FILE_ATTRIBUTE_SPARSE_FILE + FILE_ATTRIBUTE_SYSTEM + FILE_ATTRIBUTE_TEMPORARY + FILE_ATTRIBUTE_VIRTUAL + + .. versionadded:: 3.5 diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst index c3dc63de9f..b3c9e23512 100644 --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -354,7 +354,7 @@ Notes: The numeric literals accepted include the digits ``0`` to ``9`` or any Unicode equivalent (code points with the ``Nd`` property). - See http://www.unicode.org/Public/6.3.0/ucd/extracted/DerivedNumericType.txt + See http://www.unicode.org/Public/8.0.0/ucd/extracted/DerivedNumericType.txt for a complete list of code points with the ``Nd`` property. @@ -1940,6 +1940,16 @@ expression support in the :mod:`re` module). >>> 'www.example.com'.strip('cmowz.') 'example' + The outermost leading and trailing *chars* argument values are stripped + from the string. Characters are removed from the leading end until + reaching a string character that is not contained in the set of + characters in *chars*. A similar action takes place on the trailing end. + For example:: + + >>> comment_string = '#....... Section 3.2.1 Issue #32 .......' + >>> comment_string.strip('.#! ') + 'Section 3.2.1 Issue #32' + .. method:: str.swapcase() @@ -2293,6 +2303,19 @@ the bytes type has an additional class method to read data in that format: >>> bytes.fromhex('2Ef0 F1f2 ') b'.\xf0\xf1\xf2' +A reverse conversion function exists to transform a bytes object into its +hexadecimal representation. + +.. method:: bytes.hex() + + Return a string object containing two hexadecimal digits for each + byte in the instance. + + >>> b'\xf0\xf1\xf2'.hex() + 'f0f1f2' + + .. versionadded:: 3.5 + Since bytes objects are sequences of integers (akin to a tuple), for a bytes object *b*, ``b[0]`` will be an integer, while ``b[0:1]`` will be a bytes object of length 1. (This contrasts with text strings, where both indexing @@ -2348,6 +2371,19 @@ the bytearray type has an additional class method to read data in that format: >>> bytearray.fromhex('2Ef0 F1f2 ') bytearray(b'.\xf0\xf1\xf2') +A reverse conversion function exists to transform a bytearray object into its +hexadecimal representation. + +.. method:: bytearray.hex() + + Return a string object containing two hexadecimal digits for each + byte in the instance. + + >>> bytearray(b'\xf0\xf1\xf2').hex() + 'f0f1f2' + + .. versionadded:: 3.5 + Since bytearray objects are sequences of integers (akin to a list), for a bytearray object *b*, ``b[0]`` will be an integer, while ``b[0:1]`` will be a bytearray object of length 1. (This contrasts with text strings, where @@ -3093,6 +3129,203 @@ place, and instead produce new objects. always produces a new object, even if no changes were made. +.. _bytes-formatting: + +``printf``-style Bytes Formatting +---------------------------------- + +.. index:: + single: formatting, bytes (%) + single: formatting, bytearray (%) + single: interpolation, bytes (%) + single: interpolation, bytearray (%) + single: bytes; formatting + single: bytearray; formatting + single: bytes; interpolation + single: bytearray; interpolation + single: printf-style formatting + single: sprintf-style formatting + single: % formatting + single: % interpolation + +.. note:: + + The formatting operations described here exhibit a variety of quirks that + lead to a number of common errors (such as failing to display tuples and + dictionaries correctly). If the value being printed may be a tuple or + dictionary, wrap it in a tuple. + +Bytes objects (``bytes``/``bytearray``) have one unique built-in operation: +the ``%`` operator (modulo). +This is also known as the bytes *formatting* or *interpolation* operator. +Given ``format % values`` (where *format* is a bytes object), ``%`` conversion +specifications in *format* are replaced with zero or more elements of *values*. +The effect is similar to using the :c:func:`sprintf` in the C language. + +If *format* requires a single argument, *values* may be a single non-tuple +object. [5]_ Otherwise, *values* must be a tuple with exactly the number of +items specified by the format bytes object, or a single mapping object (for +example, a dictionary). + +A conversion specifier contains two or more characters and has the following +components, which must occur in this order: + +#. The ``'%'`` character, which marks the start of the specifier. + +#. Mapping key (optional), consisting of a parenthesised sequence of characters + (for example, ``(somename)``). + +#. Conversion flags (optional), which affect the result of some conversion + types. + +#. Minimum field width (optional). If specified as an ``'*'`` (asterisk), the + actual width is read from the next element of the tuple in *values*, and the + object to convert comes after the minimum field width and optional precision. + +#. Precision (optional), given as a ``'.'`` (dot) followed by the precision. If + specified as ``'*'`` (an asterisk), the actual precision is read from the next + element of the tuple in *values*, and the value to convert comes after the + precision. + +#. Length modifier (optional). + +#. Conversion type. + +When the right argument is a dictionary (or other mapping type), then the +formats in the bytes object *must* include a parenthesised mapping key into that +dictionary inserted immediately after the ``'%'`` character. The mapping key +selects the value to be formatted from the mapping. For example: + + >>> print(b'%(language)s has %(number)03d quote types.' % + ... {b'language': b"Python", b"number": 2}) + b'Python has 002 quote types.' + +In this case no ``*`` specifiers may occur in a format (since they require a +sequential parameter list). + +The conversion flag characters are: + ++---------+---------------------------------------------------------------------+ +| Flag | Meaning | ++=========+=====================================================================+ +| ``'#'`` | The value conversion will use the "alternate form" (where defined | +| | below). | ++---------+---------------------------------------------------------------------+ +| ``'0'`` | The conversion will be zero padded for numeric values. | ++---------+---------------------------------------------------------------------+ +| ``'-'`` | The converted value is left adjusted (overrides the ``'0'`` | +| | conversion if both are given). | ++---------+---------------------------------------------------------------------+ +| ``' '`` | (a space) A blank should be left before a positive number (or empty | +| | string) produced by a signed conversion. | ++---------+---------------------------------------------------------------------+ +| ``'+'`` | A sign character (``'+'`` or ``'-'``) will precede the conversion | +| | (overrides a "space" flag). | ++---------+---------------------------------------------------------------------+ + +A length modifier (``h``, ``l``, or ``L``) may be present, but is ignored as it +is not necessary for Python -- so e.g. ``%ld`` is identical to ``%d``. + +The conversion types are: + ++------------+-----------------------------------------------------+-------+ +| Conversion | Meaning | Notes | ++============+=====================================================+=======+ +| ``'d'`` | Signed integer decimal. | | ++------------+-----------------------------------------------------+-------+ +| ``'i'`` | Signed integer decimal. | | ++------------+-----------------------------------------------------+-------+ +| ``'o'`` | Signed octal value. | \(1) | ++------------+-----------------------------------------------------+-------+ +| ``'u'`` | Obsolete type -- it is identical to ``'d'``. | \(8) | ++------------+-----------------------------------------------------+-------+ +| ``'x'`` | Signed hexadecimal (lowercase). | \(2) | ++------------+-----------------------------------------------------+-------+ +| ``'X'`` | Signed hexadecimal (uppercase). | \(2) | ++------------+-----------------------------------------------------+-------+ +| ``'e'`` | Floating point exponential format (lowercase). | \(3) | ++------------+-----------------------------------------------------+-------+ +| ``'E'`` | Floating point exponential format (uppercase). | \(3) | ++------------+-----------------------------------------------------+-------+ +| ``'f'`` | Floating point decimal format. | \(3) | ++------------+-----------------------------------------------------+-------+ +| ``'F'`` | Floating point decimal format. | \(3) | ++------------+-----------------------------------------------------+-------+ +| ``'g'`` | Floating point format. Uses lowercase exponential | \(4) | +| | format if exponent is less than -4 or not less than | | +| | precision, decimal format otherwise. | | ++------------+-----------------------------------------------------+-------+ +| ``'G'`` | Floating point format. Uses uppercase exponential | \(4) | +| | format if exponent is less than -4 or not less than | | +| | precision, decimal format otherwise. | | ++------------+-----------------------------------------------------+-------+ +| ``'c'`` | Single byte (accepts integer or single | | +| | byte objects). | | ++------------+-----------------------------------------------------+-------+ +| ``'b'`` | Bytes (any object that follows the | \(5) | +| | :ref:`buffer protocol <bufferobjects>` or has | | +| | :meth:`__bytes__`). | | ++------------+-----------------------------------------------------+-------+ +| ``'s'`` | ``'s'`` is an alias for ``'b'`` and should only | \(6) | +| | be used for Python2/3 code bases. | | ++------------+-----------------------------------------------------+-------+ +| ``'a'`` | Bytes (converts any Python object using | \(5) | +| | ``repr(obj).encode('ascii','backslashreplace)``). | | ++------------+-----------------------------------------------------+-------+ +| ``'r'`` | ``'r'`` is an alias for ``'a'`` and should only | \(7) | +| | be used for Python2/3 code bases. | | ++------------+-----------------------------------------------------+-------+ +| ``'%'`` | No argument is converted, results in a ``'%'`` | | +| | character in the result. | | ++------------+-----------------------------------------------------+-------+ + +Notes: + +(1) + The alternate form causes a leading zero (``'0'``) to be inserted between + left-hand padding and the formatting of the number if the leading character + of the result is not already a zero. + +(2) + The alternate form causes a leading ``'0x'`` or ``'0X'`` (depending on whether + the ``'x'`` or ``'X'`` format was used) to be inserted between left-hand padding + and the formatting of the number if the leading character of the result is not + already a zero. + +(3) + The alternate form causes the result to always contain a decimal point, even if + no digits follow it. + + The precision determines the number of digits after the decimal point and + defaults to 6. + +(4) + The alternate form causes the result to always contain a decimal point, and + trailing zeroes are not removed as they would otherwise be. + + The precision determines the number of significant digits before and after the + decimal point and defaults to 6. + +(5) + If precision is ``N``, the output is truncated to ``N`` characters. + +(6) + ``b'%s'`` is deprecated, but will not be removed during the 3.x series. + +(7) + ``b'%r'`` is deprecated, but will not be removed during the 3.x series. + +(8) + See :pep:`237`. + +.. note:: + + The bytearray version of this method does *not* operate in place - it + always produces a new object, even if no changes were made. + +.. seealso:: :pep:`461`. +.. versionadded:: 3.5 + .. _typememoryview: Memory Views @@ -3121,10 +3354,8 @@ copying. the view. The :class:`~memoryview.itemsize` attribute will give you the number of bytes in a single element. - A :class:`memoryview` supports slicing to expose its data. If - :class:`~memoryview.format` is one of the native format specifiers - from the :mod:`struct` module, indexing will return a single element - with the correct type. Full slicing will result in a subview:: + A :class:`memoryview` supports slicing and indexing to expose its data. + One-dimensional slicing will result in a subview:: >>> v = memoryview(b'abcefg') >>> v[1] @@ -3136,25 +3367,29 @@ copying. >>> bytes(v[1:4]) b'bce' - Other native formats:: + If :class:`~memoryview.format` is one of the native format specifiers + from the :mod:`struct` module, indexing with an integer or a tuple of + integers is also supported and returns a single *element* with + the correct type. One-dimensional memoryviews can be indexed + with an integer or a one-integer tuple. Multi-dimensional memoryviews + can be indexed with tuples of exactly *ndim* integers where *ndim* is + the number of dimensions. Zero-dimensional memoryviews can be indexed + with the empty tuple. + + Here is an example with a non-byte format:: >>> import array >>> a = array.array('l', [-11111111, 22222222, -33333333, 44444444]) - >>> a[0] + >>> m = memoryview(a) + >>> m[0] -11111111 - >>> a[-1] + >>> m[-1] 44444444 - >>> a[2:3].tolist() - [-33333333] - >>> a[::2].tolist() + >>> m[::2].tolist() [-11111111, -33333333] - >>> a[::-1].tolist() - [44444444, -33333333, 22222222, -11111111] - - .. versionadded:: 3.3 - If the underlying object is writable, the memoryview supports slice - assignment. Resizing is not allowed:: + If the underlying object is writable, the memoryview supports + one-dimensional slice assignment. Resizing is not allowed:: >>> data = bytearray(b'abcefg') >>> v = memoryview(data) @@ -3187,12 +3422,16 @@ copying. True .. versionchanged:: 3.3 + One-dimensional memoryviews can now be sliced. One-dimensional memoryviews with formats 'B', 'b' or 'c' are now hashable. .. versionchanged:: 3.4 memoryview is now registered automatically with :class:`collections.abc.Sequence` + .. versionchanged:: 3.5 + memoryviews can now be indexed with tuple of integers. + :class:`memoryview` has several methods: .. method:: __eq__(exporter) @@ -3259,6 +3498,17 @@ copying. supports all format strings, including those that are not in :mod:`struct` module syntax. + .. method:: hex() + + Return a string object containing two hexadecimal digits for each + byte in the buffer. :: + + >>> m = memoryview(b"abc") + >>> m.hex() + '616263' + + .. versionadded:: 3.5 + .. method:: tolist() Return the data in the buffer as a list of elements. :: @@ -3314,10 +3564,10 @@ copying. Cast a memoryview to a new format or shape. *shape* defaults to ``[byte_length//new_itemsize]``, which means that the result view will be one-dimensional. The return value is a new memoryview, but - the buffer itself is not copied. Supported casts are 1D -> C-contiguous + the buffer itself is not copied. Supported casts are 1D -> C-:term:`contiguous` and C-contiguous -> 1D. - Both formats are restricted to single element native formats in + The destination format is restricted to a single element native format in :mod:`struct` syntax. One of the formats must be a byte format ('B', 'b' or 'c'). The byte length of the result must be the same as the original length. @@ -3398,6 +3648,9 @@ copying. .. versionadded:: 3.3 + .. versionchanged:: 3.5 + The source format is no longer restricted when casting to a byte view. + There are also several readonly attributes available: .. attribute:: obj @@ -3502,19 +3755,19 @@ copying. .. attribute:: c_contiguous - A bool indicating whether the memory is C-contiguous. + A bool indicating whether the memory is C-:term:`contiguous`. .. versionadded:: 3.3 .. attribute:: f_contiguous - A bool indicating whether the memory is Fortran contiguous. + A bool indicating whether the memory is Fortran :term:`contiguous`. .. versionadded:: 3.3 .. attribute:: contiguous - A bool indicating whether the memory is contiguous. + A bool indicating whether the memory is :term:`contiguous`. .. versionadded:: 3.3 diff --git a/Doc/library/string.rst b/Doc/library/string.rst index 19bdb21b3e..2bd8dfdc96 100644 --- a/Doc/library/string.rst +++ b/Doc/library/string.rst @@ -95,6 +95,10 @@ implementation as the built-in :meth:`format` method. an arbitrary set of positional and keyword arguments. :meth:`format` is just a wrapper that calls :meth:`vformat`. + .. deprecated:: 3.5 + Passing a format string as keyword argument *format_string* has been + deprecated. + .. method:: vformat(format_string, args, kwargs) This function does the actual work of formatting. It is exposed as a diff --git a/Doc/library/subprocess.rst b/Doc/library/subprocess.rst index 9ac8882d32..adf99ec362 100644 --- a/Doc/library/subprocess.rst +++ b/Doc/library/subprocess.rst @@ -25,160 +25,99 @@ modules and functions can be found in the following sections. Using the :mod:`subprocess` Module ---------------------------------- -The recommended approach to invoking subprocesses is to use the following -convenience functions for all use cases they can handle. For more advanced -use cases, the underlying :class:`Popen` interface can be used directly. +The recommended approach to invoking subprocesses is to use the :func:`run` +function for all use cases it can handle. For more advanced use cases, the +underlying :class:`Popen` interface can be used directly. +The :func:`run` function was added in Python 3.5; if you need to retain +compatibility with older versions, see the :ref:`call-function-trio` section. -.. function:: call(args, *, stdin=None, stdout=None, stderr=None, shell=False, timeout=None) + +.. function:: run(args, *, stdin=None, input=None, stdout=None, stderr=None,\ + shell=False, timeout=None, check=False) Run the command described by *args*. Wait for command to complete, then - return the :attr:`returncode` attribute. + return a :class:`CompletedProcess` instance. The arguments shown above are merely the most common ones, described below in :ref:`frequently-used-arguments` (hence the use of keyword-only notation in the abbreviated signature). The full function signature is largely the - same as that of the :class:`Popen` constructor - this function passes all - supplied arguments other than *timeout* directly through to that interface. + same as that of the :class:`Popen` constructor - apart from *timeout*, + *input* and *check*, all the arguments to this function are passed through to + that interface. + + This does not capture stdout or stderr by default. To do so, pass + :data:`PIPE` for the *stdout* and/or *stderr* arguments. - The *timeout* argument is passed to :meth:`Popen.wait`. If the timeout - expires, the child process will be killed and then waited for again. The + The *timeout* argument is passed to :meth:`Popen.communicate`. If the timeout + expires, the child process will be killed and waited for. The :exc:`TimeoutExpired` exception will be re-raised after the child process has terminated. - Examples:: - - >>> subprocess.call(["ls", "-l"]) - 0 - - >>> subprocess.call("exit 1", shell=True) - 1 - - .. note:: - - Do not use ``stdout=PIPE`` or ``stderr=PIPE`` with this - function. The child process will block if it generates enough - output to a pipe to fill up the OS pipe buffer as the pipes are - not being read from. - - .. versionchanged:: 3.3 - *timeout* was added. - - -.. function:: check_call(args, *, stdin=None, stdout=None, stderr=None, shell=False, timeout=None) - - Run command with arguments. Wait for command to complete. If the return - code was zero then return, otherwise raise :exc:`CalledProcessError`. The - :exc:`CalledProcessError` object will have the return code in the - :attr:`~CalledProcessError.returncode` attribute. - - The arguments shown above are merely the most common ones, described below - in :ref:`frequently-used-arguments` (hence the use of keyword-only notation - in the abbreviated signature). The full function signature is largely the - same as that of the :class:`Popen` constructor - this function passes all - supplied arguments other than *timeout* directly through to that interface. + The *input* argument is passed to :meth:`Popen.communicate` and thus to the + subprocess's stdin. If used it must be a byte sequence, or a string if + ``universal_newlines=True``. When used, the internal :class:`Popen` object + is automatically created with ``stdin=PIPE``, and the *stdin* argument may + not be used as well. - The *timeout* argument is passed to :meth:`Popen.wait`. If the timeout - expires, the child process will be killed and then waited for again. The - :exc:`TimeoutExpired` exception will be re-raised after the child process - has terminated. + If *check* is True, and the process exits with a non-zero exit code, a + :exc:`CalledProcessError` exception will be raised. Attributes of that + exception hold the arguments, the exit code, and stdout and stderr if they + were captured. Examples:: - >>> subprocess.check_call(["ls", "-l"]) - 0 + >>> subprocess.run(["ls", "-l"]) # doesn't capture output + CompletedProcess(args=['ls', '-l'], returncode=0) - >>> subprocess.check_call("exit 1", shell=True) + >>> subprocess.run("exit 1", shell=True, check=True) Traceback (most recent call last): - ... + ... subprocess.CalledProcessError: Command 'exit 1' returned non-zero exit status 1 - .. note:: + >>> subprocess.run(["ls", "-l", "/dev/null"], stdout=subprocess.PIPE) + CompletedProcess(args=['ls', '-l', '/dev/null'], returncode=0, + stdout=b'crw-rw-rw- 1 root root 1, 3 Jan 23 16:23 /dev/null\n') - Do not use ``stdout=PIPE`` or ``stderr=PIPE`` with this - function. The child process will block if it generates enough - output to a pipe to fill up the OS pipe buffer as the pipes are - not being read from. + .. versionadded:: 3.5 - .. versionchanged:: 3.3 - *timeout* was added. - - -.. function:: check_output(args, *, input=None, stdin=None, stderr=None, shell=False, universal_newlines=False, timeout=None) - - Run command with arguments and return its output. - - If the return code was non-zero it raises a :exc:`CalledProcessError`. The - :exc:`CalledProcessError` object will have the return code in the - :attr:`~CalledProcessError.returncode` attribute and any output in the - :attr:`~CalledProcessError.output` attribute. - - The arguments shown above are merely the most common ones, described below - in :ref:`frequently-used-arguments` (hence the use of keyword-only notation - in the abbreviated signature). The full function signature is largely the - same as that of the :class:`Popen` constructor - this functions passes all - supplied arguments other than *input* and *timeout* directly through to - that interface. In addition, *stdout* is not permitted as an argument, as - it is used internally to collect the output from the subprocess. - - The *timeout* argument is passed to :meth:`Popen.wait`. If the timeout - expires, the child process will be killed and then waited for again. The - :exc:`TimeoutExpired` exception will be re-raised after the child process - has terminated. - - The *input* argument is passed to :meth:`Popen.communicate` and thus to the - subprocess's stdin. If used it must be a byte sequence, or a string if - ``universal_newlines=True``. When used, the internal :class:`Popen` object - is automatically created with ``stdin=PIPE``, and the *stdin* argument may - not be used as well. +.. class:: CompletedProcess - Examples:: + The return value from :func:`run`, representing a process that has finished. - >>> subprocess.check_output(["echo", "Hello World!"]) - b'Hello World!\n' + .. attribute:: args - >>> subprocess.check_output(["echo", "Hello World!"], universal_newlines=True) - 'Hello World!\n' + The arguments used to launch the process. This may be a list or a string. - >>> subprocess.check_output(["sed", "-e", "s/foo/bar/"], - ... input=b"when in the course of fooman events\n") - b'when in the course of barman events\n' + .. attribute:: returncode - >>> subprocess.check_output("exit 1", shell=True) - Traceback (most recent call last): - ... - subprocess.CalledProcessError: Command 'exit 1' returned non-zero exit status 1 + Exit status of the child process. Typically, an exit status of 0 indicates + that it ran successfully. - By default, this function will return the data as encoded bytes. The actual - encoding of the output data may depend on the command being invoked, so the - decoding to text will often need to be handled at the application level. + A negative value ``-N`` indicates that the child was terminated by signal + ``N`` (POSIX only). - This behaviour may be overridden by setting *universal_newlines* to - ``True`` as described below in :ref:`frequently-used-arguments`. + .. attribute:: stdout - To also capture standard error in the result, use - ``stderr=subprocess.STDOUT``:: + Captured stdout from the child process. A bytes sequence, or a string if + :func:`run` was called with ``universal_newlines=True``. None if stdout + was not captured. - >>> subprocess.check_output( - ... "ls non_existent_file; exit 0", - ... stderr=subprocess.STDOUT, - ... shell=True) - 'ls: non_existent_file: No such file or directory\n' + If you ran the process with ``stderr=subprocess.STDOUT``, stdout and + stderr will be combined in this attribute, and :attr:`stderr` will be + None. - .. note:: + .. attribute:: stderr - Do not use ``stdout=PIPE`` or ``stderr=PIPE`` with this - function. The child process will block if it generates enough - output to a pipe to fill up the OS pipe buffer as the pipes are - not being read from. + Captured stderr from the child process. A bytes sequence, or a string if + :func:`run` was called with ``universal_newlines=True``. None if stderr + was not captured. - .. versionadded:: 3.1 + .. method:: check_returncode() - .. versionchanged:: 3.3 - *timeout* was added. + If :attr:`returncode` is non-zero, raise a :exc:`CalledProcessError`. - .. versionchanged:: 3.4 - *input* was added. + .. versionadded:: 3.5 .. data:: DEVNULL @@ -225,11 +164,22 @@ use cases, the underlying :class:`Popen` interface can be used directly. .. attribute:: output - Output of the child process if this exception is raised by + Output of the child process if it was captured by :func:`run` or :func:`check_output`. Otherwise, ``None``. + .. attribute:: stdout + + Alias for output, for symmetry with :attr:`stderr`. + + .. attribute:: stderr + + Stderr output of the child process if it was captured by :func:`run`. + Otherwise, ``None``. + .. versionadded:: 3.3 + .. versionchanged:: 3.5 + *stdout* and *stderr* attributes added .. exception:: CalledProcessError @@ -246,9 +196,20 @@ use cases, the underlying :class:`Popen` interface can be used directly. .. attribute:: output - Output of the child process if this exception is raised by + Output of the child process if it was captured by :func:`run` or :func:`check_output`. Otherwise, ``None``. + .. attribute:: stdout + + Alias for output, for symmetry with :attr:`stderr`. + + .. attribute:: stderr + + Stderr output of the child process if it was captured by :func:`run`. + Otherwise, ``None``. + + .. versionchanged:: 3.5 + *stdout* and *stderr* attributes added .. _frequently-used-arguments: @@ -635,6 +596,7 @@ Instances of the :class:`Popen` class have the following methods: must be bytes or, if *universal_newlines* was ``True``, a string. :meth:`communicate` returns a tuple ``(stdout_data, stderr_data)``. + The data will be bytes or, if *universal_newlines* was ``True``, strings. Note that if you want to send data to the process's stdin, you need to create the Popen object with ``stdin=PIPE``. Similarly, to get anything other than @@ -851,6 +813,112 @@ The :mod:`subprocess` module exposes the following constants. This flag is ignored if :data:`CREATE_NEW_CONSOLE` is specified. +.. _call-function-trio: + +Older high-level API +-------------------- + +Prior to Python 3.5, these three functions comprised the high level API to +subprocess. You can now use :func:`run` in many cases, but lots of existing code +calls these functions. + +.. function:: call(args, *, stdin=None, stdout=None, stderr=None, shell=False, timeout=None) + + Run the command described by *args*. Wait for command to complete, then + return the :attr:`~Popen.returncode` attribute. + + This is equivalent to:: + + run(...).returncode + + (except that the *input* and *check* parameters are not supported) + + The arguments shown above are merely the most + common ones. The full function signature is largely the + same as that of the :class:`Popen` constructor - this function passes all + supplied arguments other than *timeout* directly through to that interface. + + .. note:: + + Do not use ``stdout=PIPE`` or ``stderr=PIPE`` with this + function. The child process will block if it generates enough + output to a pipe to fill up the OS pipe buffer as the pipes are + not being read from. + + .. versionchanged:: 3.3 + *timeout* was added. + +.. function:: check_call(args, *, stdin=None, stdout=None, stderr=None, shell=False, timeout=None) + + Run command with arguments. Wait for command to complete. If the return + code was zero then return, otherwise raise :exc:`CalledProcessError`. The + :exc:`CalledProcessError` object will have the return code in the + :attr:`~CalledProcessError.returncode` attribute. + + This is equivalent to:: + + run(..., check=True) + + (except that the *input* parameter is not supported) + + The arguments shown above are merely the most + common ones. The full function signature is largely the + same as that of the :class:`Popen` constructor - this function passes all + supplied arguments other than *timeout* directly through to that interface. + + .. note:: + + Do not use ``stdout=PIPE`` or ``stderr=PIPE`` with this + function. The child process will block if it generates enough + output to a pipe to fill up the OS pipe buffer as the pipes are + not being read from. + + .. versionchanged:: 3.3 + *timeout* was added. + + +.. function:: check_output(args, *, stdin=None, stderr=None, shell=False, universal_newlines=False, timeout=None) + + Run command with arguments and return its output. + + If the return code was non-zero it raises a :exc:`CalledProcessError`. The + :exc:`CalledProcessError` object will have the return code in the + :attr:`~CalledProcessError.returncode` attribute and any output in the + :attr:`~CalledProcessError.output` attribute. + + This is equivalent to:: + + run(..., check=True, stdout=PIPE).stdout + + The arguments shown above are merely the most common ones. + The full function signature is largely the same as that of :func:`run` - + most arguments are passed directly through to that interface. + However, explicitly passing ``input=None`` to inherit the parent's + standard input file handle is not supported. + + By default, this function will return the data as encoded bytes. The actual + encoding of the output data may depend on the command being invoked, so the + decoding to text will often need to be handled at the application level. + + This behaviour may be overridden by setting *universal_newlines* to + ``True`` as described above in :ref:`frequently-used-arguments`. + + To also capture standard error in the result, use + ``stderr=subprocess.STDOUT``:: + + >>> subprocess.check_output( + ... "ls non_existent_file; exit 0", + ... stderr=subprocess.STDOUT, + ... shell=True) + 'ls: non_existent_file: No such file or directory\n' + + .. versionadded:: 3.1 + + .. versionchanged:: 3.3 + *timeout* was added. + + .. versionchanged:: 3.4 + Support for the *input* keyword argument was added. .. _subprocess-replacements: diff --git a/Doc/library/symtable.rst b/Doc/library/symtable.rst index 2503d335e8..ba2caff589 100644 --- a/Doc/library/symtable.rst +++ b/Doc/library/symtable.rst @@ -71,10 +71,6 @@ Examining Symbol Tables Return ``True`` if the block uses ``exec``. - .. method:: has_import_star() - - Return ``True`` if the block uses a starred from-import. - .. method:: get_identifiers() Return a list of names of symbols in this table. diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst index 3024086a22..bb9bdc86a1 100644 --- a/Doc/library/sys.rst +++ b/Doc/library/sys.rst @@ -167,7 +167,7 @@ always available. .. data:: dont_write_bytecode - If this is true, Python won't try to write ``.pyc`` or ``.pyo`` files on the + If this is true, Python won't try to write ``.pyc`` files on the import of source modules. This value is initially set to ``True`` or ``False`` depending on the :option:`-B` command line option and the :envvar:`PYTHONDONTWRITEBYTECODE` environment variable, but you can set it @@ -576,6 +576,18 @@ always available. *service_pack_major*, *suite_mask*, and *product_type*. +.. function:: get_coroutine_wrapper() + + Returns ``None``, or a wrapper set by :func:`set_coroutine_wrapper`. + + .. versionadded:: 3.5 + See :pep:`492` for more details. + + .. note:: + This function has been added on a provisional basis (see :pep:`411` + for details.) Use it only for debugging purposes. + + .. data:: hash_info A :term:`struct sequence` giving parameters of the numeric hash @@ -718,6 +730,14 @@ always available. value of :func:`intern` around to benefit from it. +.. function:: is_finalizing() + + Return :const:`True` if the Python interpreter is + :term:`shutting down <interpreter shutdown>`, :const:`False` otherwise. + + .. versionadded:: 3.5 + + .. data:: last_type last_value last_traceback @@ -1053,6 +1073,46 @@ always available. thus not likely to be implemented elsewhere. +.. function:: set_coroutine_wrapper(wrapper) + + Allows intercepting creation of :term:`coroutine` objects (only ones that + are created by an :keyword:`async def` function; generators decorated with + :func:`types.coroutine` or :func:`asyncio.coroutine` will not be + intercepted). + + The *wrapper* argument must be either: + + * a callable that accepts one argument (a coroutine object); + * ``None``, to reset the wrapper. + + If called twice, the new wrapper replaces the previous one. The function + is thread-specific. + + The *wrapper* callable cannot define new coroutines directly or indirectly:: + + def wrapper(coro): + async def wrap(coro): + return await coro + return wrap(coro) + sys.set_coroutine_wrapper(wrapper) + + async def foo(): + pass + + # The following line will fail with a RuntimeError, because + # ``wrapper`` creates a ``wrap(coro)`` coroutine: + foo() + + See also :func:`get_coroutine_wrapper`. + + .. versionadded:: 3.5 + See :pep:`492` for more details. + + .. note:: + This function has been added on a provisional basis (see :pep:`411` + for details.) Use it only for debugging purposes. + + .. data:: stdin stdout stderr @@ -1223,4 +1283,3 @@ always available. .. rubric:: Citations .. [C99] ISO/IEC 9899:1999. "Programming languages -- C." A public draft of this standard is available at http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1256.pdf\ . - diff --git a/Doc/library/tarfile.rst b/Doc/library/tarfile.rst index 05f29adb0b..adacb0ab3c 100644 --- a/Doc/library/tarfile.rst +++ b/Doc/library/tarfile.rst @@ -62,6 +62,23 @@ Some facts and figures: +------------------+---------------------------------------------+ | ``'r:xz'`` | Open for reading with lzma compression. | +------------------+---------------------------------------------+ + | ``'x'`` or | Create a tarfile exclusively without | + | ``'x:'`` | compression. | + | | Raise an :exc:`FileExistsError` exception | + | | if it is already exists. | + +------------------+---------------------------------------------+ + | ``'x:gz'`` | Create a tarfile with gzip compression. | + | | Raise an :exc:`FileExistsError` exception | + | | if it is already exists. | + +------------------+---------------------------------------------+ + | ``'x:bz2'`` | Create a tarfile with bzip2 compression. | + | | Raise an :exc:`FileExistsError` exception | + | | if it is already exists. | + +------------------+---------------------------------------------+ + | ``'x:xz'`` | Create a tarfile with lzma compression. | + | | Raise an :exc:`FileExistsError` exception | + | | if it is already exists. | + +------------------+---------------------------------------------+ | ``'a' or 'a:'`` | Open for appending with no compression. The | | | file is created if it does not exist. | +------------------+---------------------------------------------+ @@ -82,9 +99,9 @@ Some facts and figures: If *fileobj* is specified, it is used as an alternative to a :term:`file object` opened in binary mode for *name*. It is supposed to be at position 0. - For modes ``'w:gz'``, ``'r:gz'``, ``'w:bz2'``, ``'r:bz2'``, :func:`tarfile.open` - accepts the keyword argument *compresslevel* to specify the compression level of - the file. + For modes ``'w:gz'``, ``'r:gz'``, ``'w:bz2'``, ``'r:bz2'``, ``'x:gz'``, + ``'x:bz2'``, :func:`tarfile.open` accepts the keyword argument + *compresslevel* to specify the compression level of the file. For special purposes, there is a second format for *mode*: ``'filemode|[compression]'``. :func:`tarfile.open` will return a :class:`TarFile` @@ -127,6 +144,8 @@ Some facts and figures: | | writing. | +-------------+--------------------------------------------+ + .. versionchanged:: 3.5 + The ``'x'`` (exclusive creation) mode was added. .. class:: TarFile @@ -252,8 +271,8 @@ be finalized; only the internally used file object will be closed. See the In this case, the file object's :attr:`name` attribute is used if it exists. *mode* is either ``'r'`` to read from an existing archive, ``'a'`` to append - data to an existing file or ``'w'`` to create a new file overwriting an existing - one. + data to an existing file, ``'w'`` to create a new file overwriting an existing + one or ``'x'`` to create a new file only if it's not exists. If *fileobj* is given, it is used for reading or writing data. If it can be determined, *mode* is overridden by *fileobj*'s mode. *fileobj* will be used @@ -292,12 +311,14 @@ be finalized; only the internally used file object will be closed. See the to be handled. The default settings will work for most users. See section :ref:`tar-unicode` for in-depth information. - .. versionchanged:: 3.2 - Use ``'surrogateescape'`` as the default for the *errors* argument. - The *pax_headers* argument is an optional dictionary of strings which will be added as a pax global header if *format* is :const:`PAX_FORMAT`. + .. versionchanged:: 3.2 + Use ``'surrogateescape'`` as the default for the *errors* argument. + + .. versionchanged:: 3.5 + The ``'x'`` (exclusive creation) mode was added. .. classmethod:: TarFile.open(...) @@ -328,11 +349,15 @@ be finalized; only the internally used file object will be closed. See the returned by :meth:`getmembers`. -.. method:: TarFile.list(verbose=True) +.. method:: TarFile.list(verbose=True, *, members=None) Print a table of contents to ``sys.stdout``. If *verbose* is :const:`False`, only the names of the members are printed. If it is :const:`True`, output - similar to that of :program:`ls -l` is produced. + similar to that of :program:`ls -l` is produced. If optional *members* is + given, it must be a subset of the list returned by :meth:`getmembers`. + + .. versionchanged:: 3.5 + Added the *members* parameter. .. method:: TarFile.next() @@ -342,7 +367,7 @@ be finalized; only the internally used file object will be closed. See the available. -.. method:: TarFile.extractall(path=".", members=None) +.. method:: TarFile.extractall(path=".", members=None, *, numeric_owner=False) Extract all members from the archive to the current working directory or directory *path*. If optional *members* is given, it must be a subset of the @@ -352,6 +377,10 @@ be finalized; only the internally used file object will be closed. See the reset each time a file is created in it. And, if a directory's permissions do not allow writing, extracting files to it will fail. + If *numeric_owner* is :const:`True`, the uid and gid numbers from the tarfile + are used to set the owner/group for the extracted files. Otherwise, the named + values from the tarfile are used. + .. warning:: Never extract archives from untrusted sources without prior inspection. @@ -359,8 +388,11 @@ be finalized; only the internally used file object will be closed. See the that have absolute filenames starting with ``"/"`` or filenames with two dots ``".."``. + .. versionchanged:: 3.5 + Added the *numeric_only* parameter. -.. method:: TarFile.extract(member, path="", set_attrs=True) + +.. method:: TarFile.extract(member, path="", set_attrs=True, *, numeric_owner=False) Extract a member from the archive to the current working directory, using its full name. Its file information is extracted as accurately as possible. *member* @@ -368,6 +400,10 @@ be finalized; only the internally used file object will be closed. See the directory using *path*. File attributes (owner, mtime, mode) are set unless *set_attrs* is false. + If *numeric_owner* is :const:`True`, the uid and gid numbers from the tarfile + are used to set the owner/group for the extracted files. Otherwise, the named + values from the tarfile are used. + .. note:: The :meth:`extract` method does not take care of several extraction issues. @@ -380,6 +416,9 @@ be finalized; only the internally used file object will be closed. See the .. versionchanged:: 3.2 Added the *set_attrs* parameter. + .. versionchanged:: 3.5 + Added the *numeric_only* parameter. + .. method:: TarFile.extractfile(member) Extract a member from the archive as a file object. *member* may be a filename @@ -802,4 +841,3 @@ In case of :const:`PAX_FORMAT` archives, *encoding* is generally not needed because all the metadata is stored using *UTF-8*. *encoding* is only used in the rare cases when binary pax headers are decoded or when strings with surrogate characters are stored. - diff --git a/Doc/library/tempfile.rst b/Doc/library/tempfile.rst index 44d025defe..83f994170a 100644 --- a/Doc/library/tempfile.rst +++ b/Doc/library/tempfile.rst @@ -16,16 +16,18 @@ -------------- -This module generates temporary files and directories. It works on all -supported platforms. It provides three new functions, -:func:`NamedTemporaryFile`, :func:`mkstemp`, and :func:`mkdtemp`, which should -eliminate all remaining need to use the insecure :func:`mktemp` function. -Temporary file names created by this module no longer contain the process ID; -instead a string of six random characters is used. - -Also, all the user-callable functions now take additional arguments which -allow direct control over the location and name of temporary files. It is -no longer necessary to use the global *tempdir* variable. +This module creates temporary files and directories. It works on all +supported platforms. :class:`TemporaryFile`, :class:`NamedTemporaryFile`, +:class:`TemporaryDirectory`, and :class:`SpooledTemporaryFile` are high-level +interfaces which provide automatic cleanup and can be used as +context managers. :func:`mkstemp` and +:func:`mkdtemp` are lower-level functions which require manual cleanup. + +All the user-callable functions and constructors take additional arguments which +allow direct control over the location and name of temporary files and +directories. Files names used by this module include a string of +random characters which allows those files to be securely created in +shared temporary directories. To maintain backward compatibility, the argument order is somewhat odd; it is recommended to use keyword arguments for clarity. @@ -34,25 +36,37 @@ The module defines the following user-callable items: .. function:: TemporaryFile(mode='w+b', buffering=None, encoding=None, newline=None, suffix='', prefix='tmp', dir=None) Return a :term:`file-like object` that can be used as a temporary storage area. - The file is created using :func:`mkstemp`. It will be destroyed as soon + The file is created securely, using the same rules as :func:`mkstemp`. It will be destroyed as soon as it is closed (including an implicit close when the object is garbage - collected). Under Unix, the directory entry for the file is removed + collected). Under Unix, the directory entry for the file is either not created at all or is removed immediately after the file is created. Other platforms do not support this; your code should not rely on a temporary file created using this function having or not having a visible name in the file system. + The resulting object can be used as a context manager (see + :ref:`tempfile-examples`). On completion of the context or + destruction of the file object the temporary file will be removed + from the filesystem. + The *mode* parameter defaults to ``'w+b'`` so that the file created can be read and written without being closed. Binary mode is used so that it behaves consistently on all platforms without regard for the data that is stored. *buffering*, *encoding* and *newline* are interpreted as for :func:`open`. - The *dir*, *prefix* and *suffix* parameters are passed to :func:`mkstemp`. + The *dir*, *prefix* and *suffix* parameters have the same meaning + as with :func:`mkstemp`. The returned object is a true file object on POSIX platforms. On other platforms, it is a file-like object whose :attr:`!file` attribute is the - underlying true file object. This file-like object can be used in a - :keyword:`with` statement, just like a normal file. + underlying true file object. + + The :py:data:`os.O_TMPFILE` flag is used if it is available and works + (Linux-specific, requires Linux kernel 3.11 or later). + + .. versionchanged:: 3.5 + + The :py:data:`os.O_TMPFILE` flag is now used if available. .. function:: NamedTemporaryFile(mode='w+b', buffering=None, encoding=None, newline=None, suffix='', prefix='tmp', dir=None, delete=True) @@ -94,10 +108,9 @@ The module defines the following user-callable items: .. function:: TemporaryDirectory(suffix='', prefix='tmp', dir=None) - This function creates a temporary directory using :func:`mkdtemp` - (the supplied arguments are passed directly to the underlying function). + This function securely creates a temporary directory using the same rules as :func:`mkdtemp`. The resulting object can be used as a context manager (see - :ref:`context-managers`). On completion of the context or destruction + :ref:`tempfile-examples`). On completion of the context or destruction of the temporary directory object the newly created temporary directory and all its contents are removed from the filesystem. @@ -112,7 +125,7 @@ The module defines the following user-callable items: .. versionadded:: 3.2 -.. function:: mkstemp(suffix='', prefix='tmp', dir=None, text=False) +.. function:: mkstemp(suffix=None, prefix=None, dir=None, text=False) Creates a temporary file in the most secure manner possible. There are no race conditions in the file's creation, assuming that the platform @@ -141,6 +154,16 @@ The module defines the following user-callable items: filename will have any nice properties, such as not requiring quoting when passed to external commands via ``os.popen()``. + *suffix*, *prefix*, and *dir* must all contain the same type, if specified. + If they are bytes, the returned name will be bytes instead of str. + If you want to force a bytes return value with otherwise default behavior, + pass ``suffix=b''``. + + A *prefix* value of ``None`` means use the return value of + :func:`gettempprefix` or :func:`gettempprefixb` as appropriate. + + A *suffix* value of ``None`` means use an appropriate empty value. + If *text* is specified, it indicates whether to open the file in binary mode (the default) or text mode. On some platforms, this makes no difference. @@ -149,8 +172,14 @@ The module defines the following user-callable items: file (as would be returned by :func:`os.open`) and the absolute pathname of that file, in that order. + .. versionchanged:: 3.5 + *suffix*, *prefix*, and *dir* may now be supplied in bytes in order to + obtain a bytes return value. Prior to this, only str was allowed. + *suffix* and *prefix* now accept and default to ``None`` to cause + an appropriate default value to be used. + -.. function:: mkdtemp(suffix='', prefix='tmp', dir=None) +.. function:: mkdtemp(suffix=None, prefix=None, dir=None) Creates a temporary directory in the most secure manner possible. There are no race conditions in the directory's creation. The directory is @@ -164,50 +193,21 @@ The module defines the following user-callable items: :func:`mkdtemp` returns the absolute pathname of the new directory. + .. versionchanged:: 3.5 + *suffix*, *prefix*, and *dir* may now be supplied in bytes in order to + obtain a bytes return value. Prior to this, only str was allowed. + *suffix* and *prefix* now accept and default to ``None`` to cause + an appropriate default value to be used. -.. function:: mktemp(suffix='', prefix='tmp', dir=None) - - .. deprecated:: 2.3 - Use :func:`mkstemp` instead. - - Return an absolute pathname of a file that did not exist at the time the - call is made. The *prefix*, *suffix*, and *dir* arguments are the same - as for :func:`mkstemp`. - .. warning:: - - Use of this function may introduce a security hole in your program. By - the time you get around to doing anything with the file name it returns, - someone else may have beaten you to the punch. :func:`mktemp` usage can - be replaced easily with :func:`NamedTemporaryFile`, passing it the - ``delete=False`` parameter:: - - >>> f = NamedTemporaryFile(delete=False) - >>> f.name - '/tmp/tmptjujjt' - >>> f.write(b"Hello World!\n") - 13 - >>> f.close() - >>> os.unlink(f.name) - >>> os.path.exists(f.name) - False - -The module uses a global variable that tell it how to construct a -temporary name. They are initialized at the first call to any of the -functions above. The caller may change them, but this is discouraged; use -the appropriate function arguments, instead. - - -.. data:: tempdir +.. function:: gettempdir() - When set to a value other than ``None``, this variable defines the - default value for the *dir* argument to all the functions defined in this - module. + Return the name of the directory used for temporary files. This + defines the default value for the *dir* argument to all functions + in this module. - If ``tempdir`` is unset or ``None`` at any call to any of the above - functions, Python searches a standard list of directories and sets - *tempdir* to the first one which the calling user can create files in. - The list is: + Python searches a standard list of directories to find one which + the calling user can create files in. The list is: #. The directory named by the :envvar:`TMPDIR` environment variable. @@ -225,19 +225,43 @@ the appropriate function arguments, instead. #. As a last resort, the current working directory. + The result of this search is cached, see the description of + :data:`tempdir` below. -.. function:: gettempdir() +.. function:: gettempdirb() - Return the directory currently selected to create temporary files in. If - :data:`tempdir` is not ``None``, this simply returns its contents; otherwise, - the search described above is performed, and the result returned. + Same as :func:`gettempdir` but the return value is in bytes. + .. versionadded:: 3.5 .. function:: gettempprefix() Return the filename prefix used to create temporary files. This does not contain the directory component. +.. function:: gettempprefixb() + + Same as :func:`gettempprefixb` but the return value is in bytes. + + .. versionadded:: 3.5 + +The module uses a global variable to store the name of the directory +used for temporary files returned by :func:`gettempdir`. It can be +set directly to override the selection process, but this is discouraged. +All functions in this module take a *dir* argument which can be used +to specify the directory and this is the recommend approach. + +.. data:: tempdir + + When set to a value other than ``None``, this variable defines the + default value for the *dir* argument to all the functions defined in this + module. + + If ``tempdir`` is unset or ``None`` at any call to any of the above + functions except :func:`gettempprefix` it is initalized following the + algorithm described in :func:`gettempdir`. + +.. _tempfile-examples: Examples -------- @@ -271,3 +295,42 @@ Here are some examples of typical usage of the :mod:`tempfile` module:: >>> # directory and contents have been removed + +Deprecated functions and variables +---------------------------------- + +A historical way to create temporary files was to first generate a +file name with the :func:`mktemp` function and then create a file +using this name. Unfortunately this is not secure, because a different +process may create a file with this name in the time between the call +to :func:`mktemp` and the subsequent attempt to create the file by the +first process. The solution is to combine the two steps and create the +file immediately. This approach is used by :func:`mkstemp` and the +other functions described above. + +.. function:: mktemp(suffix='', prefix='tmp', dir=None) + + .. deprecated:: 2.3 + Use :func:`mkstemp` instead. + + Return an absolute pathname of a file that did not exist at the time the + call is made. The *prefix*, *suffix*, and *dir* arguments are the same + as for :func:`mkstemp`. + + .. warning:: + + Use of this function may introduce a security hole in your program. By + the time you get around to doing anything with the file name it returns, + someone else may have beaten you to the punch. :func:`mktemp` usage can + be replaced easily with :func:`NamedTemporaryFile`, passing it the + ``delete=False`` parameter:: + + >>> f = NamedTemporaryFile(delete=False) + >>> f.name + '/tmp/tmptjujjt' + >>> f.write(b"Hello World!\n") + 13 + >>> f.close() + >>> os.unlink(f.name) + >>> os.path.exists(f.name) + False diff --git a/Doc/library/test.rst b/Doc/library/test.rst index 974909e182..f7ad475402 100644 --- a/Doc/library/test.rst +++ b/Doc/library/test.rst @@ -568,6 +568,17 @@ The :mod:`test.support` module defines the following functions: def load_tests(*args): return load_package_tests(os.path.dirname(__file__), *args) +.. function:: detect_api_mismatch(ref_api, other_api, *, ignore=()): + + Returns the set of attributes, functions or methods of *ref_api* not + found on *other_api*, except for a defined list of items to be + ignored in this check specified in *ignore*. + + By default this skips private attributes beginning with '_' but + includes all magic methods, i.e. those starting and ending in '__'. + + .. versionadded:: 3.5 + The :mod:`test.support` module defines the following classes: diff --git a/Doc/library/time.rst b/Doc/library/time.rst index 8d8b7d4c55..3d335c8867 100644 --- a/Doc/library/time.rst +++ b/Doc/library/time.rst @@ -314,9 +314,9 @@ The module defines the following functions and data items: processes running for more than 49 days. On more recent versions of Windows and on other operating systems, :func:`monotonic` is system-wide. - Availability: Windows, Mac OS X, Linux, FreeBSD, OpenBSD, Solaris. - .. versionadded:: 3.3 + .. versionchanged:: 3.5 + The function is now always available. .. function:: perf_counter() @@ -350,6 +350,11 @@ The module defines the following functions and data items: requested by an arbitrary amount because of the scheduling of other activity in the system. + .. versionchanged:: 3.5 + The function now sleeps at least *secs* even if the sleep is interrupted + by a signal, except if the signal handler raises an exception (see + :pep:`475` for the rationale). + .. function:: strftime(format[, t]) diff --git a/Doc/library/timeit.rst b/Doc/library/timeit.rst index 70df409fac..d1051f688c 100644 --- a/Doc/library/timeit.rst +++ b/Doc/library/timeit.rst @@ -59,18 +59,26 @@ Python Interface The module defines three convenience functions and a public class: -.. function:: timeit(stmt='pass', setup='pass', timer=<default timer>, number=1000000) +.. function:: timeit(stmt='pass', setup='pass', timer=<default timer>, number=1000000, globals=None) Create a :class:`Timer` instance with the given statement, *setup* code and *timer* function and run its :meth:`.timeit` method with *number* executions. + The optional *globals* argument specifies a namespace in which to execute the + code. + .. versionchanged:: 3.5 + The optional *globals* parameter was added. -.. function:: repeat(stmt='pass', setup='pass', timer=<default timer>, repeat=3, number=1000000) + +.. function:: repeat(stmt='pass', setup='pass', timer=<default timer>, repeat=3, number=1000000, globals=None) Create a :class:`Timer` instance with the given statement, *setup* code and *timer* function and run its :meth:`.repeat` method with the given *repeat* - count and *number* executions. + count and *number* executions. The optional *globals* argument specifies a + namespace in which to execute the code. + .. versionchanged:: 3.5 + The optional *globals* parameter was added. .. function:: default_timer() @@ -80,7 +88,7 @@ The module defines three convenience functions and a public class: :func:`time.perf_counter` is now the default timer. -.. class:: Timer(stmt='pass', setup='pass', timer=<timer function>) +.. class:: Timer(stmt='pass', setup='pass', timer=<timer function>, globals=None) Class for timing execution speed of small code snippets. @@ -88,7 +96,9 @@ The module defines three convenience functions and a public class: for setup, and a timer function. Both statements default to ``'pass'``; the timer function is platform-dependent (see the module doc string). *stmt* and *setup* may also contain multiple statements separated by ``;`` - or newlines, as long as they don't contain multi-line string literals. + or newlines, as long as they don't contain multi-line string literals. The + statement will by default be executed within timeit's namespace; this behavior + can be controlled by passing a namespace to *globals*. To measure the execution time of the first statement, use the :meth:`.timeit` method. The :meth:`.repeat` method is a convenience to call :meth:`.timeit` @@ -101,6 +111,8 @@ The module defines three convenience functions and a public class: will then be executed by :meth:`.timeit`. Note that the timing overhead is a little larger in this case because of the extra function calls. + .. versionchanged:: 3.5 + The optional *globals* parameter was added. .. method:: Timer.timeit(number=1000000) @@ -169,7 +181,7 @@ Command-Line Interface When called as a program from the command line, the following form is used:: - python -m timeit [-n N] [-r N] [-s S] [-t] [-c] [-h] [statement ...] + python -m timeit [-n N] [-r N] [-u U] [-s S] [-t] [-c] [-h] [statement ...] Where the following options are understood: @@ -198,6 +210,12 @@ Where the following options are understood: use :func:`time.time` (deprecated) +.. cmdoption:: -u, --unit=U + + specify a time unit for timer output; can select usec, msec, or sec + + .. versionadded:: 3.5 + .. cmdoption:: -c, --clock use :func:`time.clock` (deprecated) @@ -320,3 +338,17 @@ To give the :mod:`timeit` module access to functions you define, you can pass a if __name__ == '__main__': import timeit print(timeit.timeit("test()", setup="from __main__ import test")) + +Another option is to pass :func:`globals` to the *globals* parameter, which will cause the code +to be executed within your current global namespace. This can be more convenient +than individually specifying imports:: + + def f(x): + return x**2 + def g(x): + return x**4 + def h(x): + return x**8 + + import timeit + print(timeit.timeit('[func(42) for func in (f,g,h)]', globals=globals())) diff --git a/Doc/library/token.rst b/Doc/library/token.rst index 4cd709814c..88fb38bc1d 100644 --- a/Doc/library/token.rst +++ b/Doc/library/token.rst @@ -93,6 +93,7 @@ The token constants are: DOUBLESLASH DOUBLESLASHEQUAL AT + ATEQUAL RARROW ELLIPSIS OP diff --git a/Doc/library/traceback.rst b/Doc/library/traceback.rst index 15fbedcf33..8d216d07e3 100644 --- a/Doc/library/traceback.rst +++ b/Doc/library/traceback.rst @@ -22,15 +22,20 @@ The module defines the following functions: .. function:: print_tb(traceback, limit=None, file=None) - Print up to *limit* stack trace entries from *traceback*. If *limit* is omitted - or ``None``, all entries are printed. If *file* is omitted or ``None``, the - output goes to ``sys.stderr``; otherwise it should be an open file or file-like - object to receive the output. + Print up to *limit* stack trace entries from *traceback* (starting from + the caller's frame) if *limit* is positive. Otherwise, print the last + ``abs(limit)`` entries. If *limit* is omitted or ``None``, all entries + are printed. If *file* is omitted or ``None``, the output goes to + ``sys.stderr``; otherwise it should be an open file or file-like object + to receive the output. + + .. versionchanged:: 3.5 + Added negative *limit* support. .. function:: print_exception(type, value, traceback, limit=None, file=None, chain=True) - Print exception information and up to *limit* stack trace entries from + Print exception information and stack trace entries from *traceback* to *file*. This differs from :func:`print_tb` in the following ways: @@ -41,6 +46,7 @@ The module defines the following functions: prints the line where the syntax error occurred with a caret indicating the approximate position of the error. + The optional *limit* argument has the same meaning as for :func:`print_tb`. If *chain* is true (the default), then chained exceptions (the :attr:`__cause__` or :attr:`__context__` attributes of the exception) will be printed as well, like the interpreter itself does when printing an unhandled @@ -49,33 +55,41 @@ The module defines the following functions: .. function:: print_exc(limit=None, file=None, chain=True) - This is a shorthand for ``print_exception(*sys.exc_info())``. + This is a shorthand for ``print_exception(*sys.exc_info(), limit, file, + chain)``. .. function:: print_last(limit=None, file=None, chain=True) This is a shorthand for ``print_exception(sys.last_type, sys.last_value, - sys.last_traceback, limit, file)``. In general it will work only after - an exception has reached an interactive prompt (see :data:`sys.last_type`). + sys.last_traceback, limit, file, chain)``. In general it will work only + after an exception has reached an interactive prompt (see + :data:`sys.last_type`). .. function:: print_stack(f=None, limit=None, file=None) - This function prints a stack trace from its invocation point. The optional *f* - argument can be used to specify an alternate stack frame to start. The optional - *limit* and *file* arguments have the same meaning as for - :func:`print_exception`. + Print up to *limit* stack trace entries (starting from the invocation + point) if *limit* is positive. Otherwise, print the last ``abs(limit)`` + entries. If *limit* is omitted or ``None``, all entries are printed. + The optional *f* argument can be used to specify an alternate stack frame + to start. The optional *file* argument has the same meaning as for + :func:`print_tb`. + + .. versionchanged:: 3.5 + Added negative *limit* support. .. function:: extract_tb(traceback, limit=None) - Return a list of up to *limit* "pre-processed" stack trace entries extracted - from the traceback object *traceback*. It is useful for alternate formatting of - stack traces. If *limit* is omitted or ``None``, all entries are extracted. A - "pre-processed" stack trace entry is a 4-tuple (*filename*, *line number*, - *function name*, *text*) representing the information that is usually printed - for a stack trace. The *text* is a string with leading and trailing whitespace - stripped; if the source is not available it is ``None``. + Return a list of "pre-processed" stack trace entries extracted from the + traceback object *traceback*. It is useful for alternate formatting of + stack traces. The optional *limit* argument has the same meaning as for + :func:`print_tb`. A "pre-processed" stack trace entry is a 4-tuple + (*filename*, *line number*, *function name*, *text*) representing the + information that is usually printed for a stack trace. The *text* is a + string with leading and trailing whitespace stripped; if the source is + not available it is ``None``. .. function:: extract_stack(f=None, limit=None) @@ -136,6 +150,162 @@ The module defines the following functions: .. versionadded:: 3.4 +.. function:: walk_stack(f) + + Walk a stack following ``f.f_back`` from the given frame, yielding the frame + and line number for each frame. If *f* is ``None``, the current stack is + used. This helper is used with :meth:`StackSummary.extract`. + + .. versionadded:: 3.5 + +.. function:: walk_tb(tb) + + Walk a traceback following ``tb_next`` yielding the frame and line number + for each frame. This helper is used with :meth:`StackSummary.extract`. + + .. versionadded:: 3.5 + +The module also defines the following classes: + +:class:`TracebackException` Objects +----------------------------------- + +.. versionadded:: 3.5 + +:class:`TracebackException` objects are created from actual exceptions to +capture data for later printing in a lightweight fashion. + +.. class:: TracebackException(exc_type, exc_value, exc_traceback, *, limit=None, lookup_lines=True, capture_locals=False) + + Capture an exception for later rendering. *limit*, *lookup_lines* and + *capture_locals* are as for the :class:`StackSummary` class. + + Note that when locals are captured, they are also shown in the traceback. + + .. attribute:: __cause__ + + A :class:`TracebackException` of the original ``__cause__``. + + .. attribute:: __context__ + + A :class:`TracebackException` of the original ``__context__``. + + .. attribute:: __suppress_context__ + + The ``__suppress_context__`` value from the original exception. + + .. attribute:: stack + + A :class:`StackSummary` representing the traceback. + + .. attribute:: exc_type + + The class of the original traceback. + + .. attribute:: filename + + For syntax errors - the file name where the error occurred. + + .. attribute:: lineno + + For syntax errors - the line number where the error occurred. + + .. attribute:: text + + For syntax errors - the text where the error occurred. + + .. attribute:: offset + + For syntax errors - the offset into the text where the error occurred. + + .. attribute:: msg + + For syntax errors - the compiler error message. + + .. classmethod:: from_exception(exc, *, limit=None, lookup_lines=True, capture_locals=False) + + Capture an exception for later rendering. *limit*, *lookup_lines* and + *capture_locals* are as for the :class:`StackSummary` class. + + Note that when locals are captured, they are also shown in the traceback. + + .. method:: format(*, chain=True) + + Format the exception. + + If *chain* is not ``True``, ``__cause__`` and ``__context__`` will not + be formatted. + + The return value is a generator of strings, each ending in a newline and + some containing internal newlines. :func:`~traceback.print_exception` + is a wrapper around this method which just prints the lines to a file. + + The message indicating which exception occurred is always the last + string in the output. + + .. method:: format_exception_only() + + Format the exception part of the traceback. + + The return value is a generator of strings, each ending in a newline. + + Normally, the generator emits a single string; however, for + :exc:`SyntaxError` exceptions, it emits several lines that (when + printed) display detailed information about where the syntax + error occurred. + + The message indicating which exception occurred is always the last + string in the output. + + +:class:`StackSummary` Objects +----------------------------- + +.. versionadded:: 3.5 + +:class:`StackSummary` objects represent a call stack ready for formatting. + +.. class:: StackSummary + + .. classmethod:: extract(frame_gen, *, limit=None, lookup_lines=True, capture_locals=False) + + Construct a :class:`StackSummary` object from a frame generator (such as + is returned by :func:`~traceback.walk_stack` or + :func:`~traceback.walk_tb`). + + If *limit* is supplied, only this many frames are taken from *frame_gen*. + If *lookup_lines* is ``False``, the returned :class:`FrameSummary` + objects will not have read their lines in yet, making the cost of + creating the :class:`StackSummary` cheaper (which may be valuable if it + may not actually get formatted). If *capture_locals* is ``True`` the + local variables in each :class:`FrameSummary` are captured as object + representations. + + .. classmethod:: from_list(a_list) + + Construct a :class:`StackSummary` object from a supplied old-style list + of tuples. Each tuple should be a 4-tuple with filename, lineno, name, + line as the elements. + + +:class:`FrameSummary` Objects +----------------------------- + +.. versionadded:: 3.5 + +:class:`FrameSummary` objects represent a single frame in a traceback. + +.. class:: FrameSummary(filename, lineno, name, lookup_line=True, locals=None, line=None) + + Represent a single frame in the traceback or stack that is being formatted + or printed. It may optionally have a stringified version of the frames + locals included in it. If *lookup_line* is ``False``, the source code is not + looked up until the :class:`FrameSummary` has the :attr:`~FrameSummary.line` + attribute accessed (which also happens when casting it to a tuple). + :attr:`~FrameSummary.line` may be directly provided, and will prevent line + lookups happening at all. *locals* is an optional local variable + dictionary, and if supplied the variable representations are stored in the + summary for later display. .. _traceback-example: diff --git a/Doc/library/tracemalloc.rst b/Doc/library/tracemalloc.rst index a04a432716..13c81a7fed 100644 --- a/Doc/library/tracemalloc.rst +++ b/Doc/library/tracemalloc.rst @@ -363,7 +363,7 @@ Filter Filter on traces of memory blocks. See the :func:`fnmatch.fnmatch` function for the syntax of - *filename_pattern*. The ``'.pyc'`` and ``'.pyo'`` file extensions are + *filename_pattern*. The ``'.pyc'`` file extension is replaced with ``'.py'``. Examples: @@ -374,6 +374,10 @@ Filter :mod:`tracemalloc` module * ``Filter(False, "<unknown>")`` excludes empty tracebacks + + .. versionchanged:: 3.5 + The ``'.pyo'`` file extension is no longer replaced with ``'.py'``. + .. attribute:: inclusive If *inclusive* is ``True`` (include), only trace memory blocks allocated @@ -631,4 +635,3 @@ Traceback obj = Object() File "test.py", line 12 tb = tracemalloc.get_object_traceback(f()) - diff --git a/Doc/library/tulip_coro.dia b/Doc/library/tulip_coro.dia Binary files differindex eccf4fa7f5..70a33e3c00 100644 --- a/Doc/library/tulip_coro.dia +++ b/Doc/library/tulip_coro.dia diff --git a/Doc/library/tulip_coro.png b/Doc/library/tulip_coro.png Binary files differindex 65b6951550..36ced8ddbf 100644 --- a/Doc/library/tulip_coro.png +++ b/Doc/library/tulip_coro.png diff --git a/Doc/library/turtle.rst b/Doc/library/turtle.rst index efe5c54edf..30dd6eff2d 100644 --- a/Doc/library/turtle.rst +++ b/Doc/library/turtle.rst @@ -1879,7 +1879,7 @@ Settings and special methods >>> cv = screen.getcanvas() >>> cv - <turtle.ScrolledCanvas object at ...> + <turtle.ScrolledCanvas object ...> .. function:: getshapes() @@ -2351,6 +2351,9 @@ The demo scripts are: | | pairwise in opposite | shapesize, tilt, | | | direction | get_shapepoly, update | +----------------+------------------------------+-----------------------+ +| sorting_animate| visual demonstration of | simple alignment, | +| | different sorting methods | randomization | ++----------------+------------------------------+-----------------------+ | tree | a (graphical) breadth | :func:`clone` | | | first tree (using generators)| | +----------------+------------------------------+-----------------------+ diff --git a/Doc/library/types.rst b/Doc/library/types.rst index abdb939d1a..eb27846aab 100644 --- a/Doc/library/types.rst +++ b/Doc/library/types.rst @@ -86,8 +86,16 @@ Standard names are defined for the following types: .. data:: GeneratorType - The type of :term:`generator`-iterator objects, produced by calling a - generator function. + The type of :term:`generator`-iterator objects, created by + generator functions. + + +.. data:: CoroutineType + + The type of :term:`coroutine` objects, created by + :keyword:`async def` functions. + + .. versionadded:: 3.5 .. data:: CodeType @@ -115,6 +123,10 @@ Standard names are defined for the following types: The type of :term:`modules <module>`. Constructor takes the name of the module to be created and optionally its :term:`docstring`. + .. note:: + Use :func:`importlib.util.module_from_spec` to create a new module if you + wish to set the various import-controlled attributes. + .. attribute:: __doc__ The :term:`docstring` of the module. Defaults to ``None``. @@ -267,3 +279,25 @@ Additional Utility Classes and Functions attributes on the class with the same name (see Enum for an example). .. versionadded:: 3.4 + + +Coroutine Utility Functions +--------------------------- + +.. function:: coroutine(gen_func) + + This function transforms a :term:`generator` function into a + :term:`coroutine function` which returns a generator-based coroutine. + The generator-based coroutine is still a :term:`generator iterator`, + but is also considered to be a :term:`coroutine` object and is + :term:`awaitable`. However, it may not necessarily implement + the :meth:`__await__` method. + + If *gen_func* is a generator function, it will be modified in-place. + + If *gen_func* is not a generator function, it will be wrapped. If it + returns an instance of :class:`collections.abc.Generator`, the instance + will be wrapped in an *awaitable* proxy object. All other types + of objects will be returned as is. + + .. versionadded:: 3.5 diff --git a/Doc/library/typing.rst b/Doc/library/typing.rst new file mode 100644 index 0000000000..668523b52d --- /dev/null +++ b/Doc/library/typing.rst @@ -0,0 +1,524 @@ +:mod:`typing` --- Support for type hints +======================================== + +.. module:: typing + :synopsis: Support for type hints (see PEP 484). + +**Source code:** :source:`Lib/typing.py` + +-------------- + +This module supports type hints as specified by :pep:`484`. The most +fundamental support consists of the type :class:`Any`, :class:`Union`, +:class:`Tuple`, :class:`Callable`, :class:`TypeVar`, and +:class:`Generic`. For full specification please see :pep:`484`. For +a simplified introduction to type hints see :pep:`483`. + + +The function below takes and returns a string and is annotated as follows:: + + def greeting(name: str) -> str: + return 'Hello ' + name + +In the function ``greeting``, the argument ``name`` is expected to by of type +:class:`str` and the return type :class:`str`. Subtypes are accepted as +arguments. + +Type aliases +------------ + +A type alias is defined by assigning the type to the alias:: + + Vector = List[float] + +Callable +-------- + +Frameworks expecting callback functions of specific signatures might be +type hinted using ``Callable[[Arg1Type, Arg2Type], ReturnType]``. + +For example:: + + from typing import Callable + + def feeder(get_next_item: Callable[[], str]) -> None: + # Body + + def async_query(on_success: Callable[[int], None], + on_error: Callable[[int, Exception], None]) -> None: + # Body + +It is possible to declare the return type of a callable without specifying +the call signature by substituting a literal ellipsis +for the list of arguments in the type hint: ``Callable[..., ReturnType]``. +``None`` as a type hint is a special case and is replaced by ``type(None)``. + +Generics +-------- + +Since type information about objects kept in containers cannot be statically +inferred in a generic way, abstract base classes have been extended to support +subscription to denote expected types for container elements. + +:: + + from typing import Mapping, Sequence + + def notify_by_email(employees: Sequence[Employee], + overrides: Mapping[str, str]) -> None: ... + +Generics can be parametrized by using a new factory available in typing +called :class:`TypeVar`. + +:: + + from typing import Sequence, TypeVar + + T = TypeVar('T') # Declare type variable + + def first(l: Sequence[T]) -> T: # Generic function + return l[0] + + +User-defined generic types +-------------------------- + +A user-defined class can be defined as a generic class. + +:: + + from typing import TypeVar, Generic + from logging import Logger + + T = TypeVar('T') + + class LoggedVar(Generic[T]): + def __init__(self, value: T, name: str, logger: Logger) -> None: + self.name = name + self.logger = logger + self.value = value + + def set(self, new: T) -> None: + self.log('Set ' + repr(self.value)) + self.value = new + + def get(self) -> T: + self.log('Get ' + repr(self.value)) + return self.value + + def log(self, message: str) -> None: + self.logger.info('{}: {}'.format(self.name, message)) + +``Generic[T]`` as a base class defines that the class ``LoggedVar`` takes a +single type parameter ``T`` . This also makes ``T`` valid as a type within the +class body. + +The :class:`Generic` base class uses a metaclass that defines +:meth:`__getitem__` so that ``LoggedVar[t]`` is valid as a type:: + + from typing import Iterable + + def zero_all_vars(vars: Iterable[LoggedVar[int]]) -> None: + for var in vars: + var.set(0) + +A generic type can have any number of type variables, and type variables may +be constrained:: + + from typing import TypeVar, Generic + ... + + T = TypeVar('T') + S = TypeVar('S', int, str) + + class StrangePair(Generic[T, S]): + ... + +Each type variable argument to :class:`Generic` must be distinct. +This is thus invalid:: + + from typing import TypeVar, Generic + ... + + T = TypeVar('T') + + class Pair(Generic[T, T]): # INVALID + ... + +You can use multiple inheritance with :class:`Generic`:: + + from typing import TypeVar, Generic, Sized + + T = TypeVar('T') + + class LinkedList(Sized, Generic[T]): + ... + +When inheriting from generic classes, some type variables could fixed:: + + from typing import TypeVar, Mapping + + T = TypeVar('T') + + class MyDict(Mapping[str, T]): + ... + +In this case ``MyDict`` has a single parameter, ``T``. + +Subclassing a generic class without specifying type parameters assumes +:class:`Any` for each position. In the following example, ``MyIterable`` is +not generic but implicitly inherits from ``Iterable[Any]``:: + + from typing import Iterable + + class MyIterable(Iterable): # Same as Iterable[Any] + +The metaclass used by :class:`Generic` is a subclass of :class:`abc.ABCMeta`. +A generic class can be an ABC by including abstract methods or properties, +and generic classes can also have ABCs as base classes without a metaclass +conflict. Generic metaclasses are not supported. + + +The :class:`Any` type +--------------------- + +A special kind of type is :class:`Any`. Every type is a subtype of +:class:`Any`. This is also true for the builtin type object. However, to the +static type checker these are completely different. + +When the type of a value is :class:`object`, the type checker will reject +almost all operations on it, and assigning it to a variable (or using it as a +return value) of a more specialized type is a type error. On the other hand, +when a value has type :class:`Any`, the type checker will allow all operations +on it, and a value of type :class:`Any` can be assigned to a variable (or used +as a return value) of a more constrained type. + + +Classes, functions, and decorators +---------------------------------- + +The module defines the following classes, functions and decorators: + +.. class:: Any + + Special type indicating an unconstrained type. + + * Any object is an instance of :class:`Any`. + * Any class is a subclass of :class:`Any`. + * As a special case, :class:`Any` and :class:`object` are subclasses of + each other. + +.. class:: TypeVar + + Type variable. + + Usage:: + + T = TypeVar('T') # Can be anything + A = TypeVar('A', str, bytes) # Must be str or bytes + + Type variables exist primarily for the benefit of static type + checkers. They serve as the parameters for generic types as well + as for generic function definitions. See class Generic for more + information on generic types. Generic functions work as follows:: + + def repeat(x: T, n: int) -> Sequence[T]: + """Return a list containing n references to x.""" + return [x]*n + + def longest(x: A, y: A) -> A: + """Return the longest of two strings.""" + return x if len(x) >= len(y) else y + + The latter example's signature is essentially the overloading + of ``(str, str) -> str`` and ``(bytes, bytes) -> bytes``. Also note + that if the arguments are instances of some subclass of :class:`str`, + the return type is still plain :class:`str`. + + At runtime, ``isinstance(x, T)`` will raise :exc:`TypeError`. In general, + :func:`isinstance` and :func:`issubclass` should not be used with types. + + Type variables may be marked covariant or contravariant by passing + ``covariant=True`` or ``contravariant=True``. See :pep:`484` for more + details. By default type variables are invariant. Alternatively, + a type variable may specify an upper bound using ``bound=<type>``. + This means that an actual type substituted (explicitly or implicitly) + for the type variable must be a subclass of the boundary type, + see :pep:`484`. + +.. class:: Union + + Union type; ``Union[X, Y]`` means either X or Y. + + To define a union, use e.g. ``Union[int, str]``. Details: + + * The arguments must be types and there must be at least one. + + * Unions of unions are flattened, e.g.:: + + Union[Union[int, str], float] == Union[int, str, float] + + * Unions of a single argument vanish, e.g.:: + + Union[int] == int # The constructor actually returns int + + * Redundant arguments are skipped, e.g.:: + + Union[int, str, int] == Union[int, str] + + * When comparing unions, the argument order is ignored, e.g.:: + + Union[int, str] == Union[str, int] + + * If :class:`Any` is present it is the sole survivor, e.g.:: + + Union[int, Any] == Any + + * You cannot subclass or instantiate a union. + + * You cannot write ``Union[X][Y]``. + + * You can use ``Optional[X]`` as a shorthand for ``Union[X, None]``. + +.. class:: Optional + + Optional type. + + ``Optional[X]`` is equivalent to ``Union[X, type(None)]``. + +.. class:: Tuple + + Tuple type; ``Tuple[X, Y]`` is the is the type of a tuple of two items + with the first item of type X and the second of type Y. + + Example: ``Tuple[T1, T2]`` is a tuple of two elements corresponding + to type variables T1 and T2. ``Tuple[int, float, str]`` is a tuple + of an int, a float and a string. + + To specify a variable-length tuple of homogeneous type, + use literal ellipsis, e.g. ``Tuple[int, ...]``. + +.. class:: Callable + + Callable type; ``Callable[[int], str]`` is a function of (int) -> str. + + The subscription syntax must always be used with exactly two + values: the argument list and the return type. The argument list + must be a list of types; the return type must be a single type. + + There is no syntax to indicate optional or keyword arguments, + such function types are rarely used as callback types. + ``Callable[..., ReturnType]`` could be used to type hint a callable + taking any number of arguments and returning ``ReturnType``. + A plain :class:`Callable` is equivalent to ``Callable[..., Any]``. + +.. class:: Generic + + Abstract base class for generic types. + + A generic type is typically declared by inheriting from an + instantiation of this class with one or more type variables. + For example, a generic mapping type might be defined as:: + + class Mapping(Generic[KT, VT]): + def __getitem__(self, key: KT) -> VT: + ... + # Etc. + + This class can then be used as follows:: + + X = TypeVar('X') + Y = TypeVar('Y') + + def lookup_name(mapping: Mapping[X, Y], key: X, default: Y) -> Y: + try: + return mapping[key] + except KeyError: + return default + +.. class:: Iterable(Generic[T_co]) + + A generic version of the :class:`collections.abc.Iterable`. + +.. class:: Iterator(Iterable[T_co]) + + A generic version of the :class:`collections.abc.Iterator`. + +.. class:: SupportsInt + + An ABC with one abstract method ``__int__``. + +.. class:: SupportsFloat + + An ABC with one abstract method ``__float__``. + +.. class:: SupportsAbs + + An ABC with one abstract method ``__abs__`` that is covariant + in its return type. + +.. class:: SupportsRound + + An ABC with one abstract method ``__round__`` + that is covariant in its return type. + +.. class:: Reversible + + An ABC with one abstract method ``__reversed__`` returning + an ``Iterator[T_co]``. + +.. class:: Container(Generic[T_co]) + + A generic version of :class:`collections.abc.Container`. + +.. class:: AbstractSet(Sized, Iterable[T_co], Container[T_co]) + + A generic version of :class:`collections.abc.Set`. + +.. class:: MutableSet(AbstractSet[T]) + + A generic version of :class:`collections.abc.MutableSet`. + +.. class:: Mapping(Sized, Iterable[KT], Container[KT], Generic[VT_co]) + + A generic version of :class:`collections.abc.Mapping`. + +.. class:: MutableMapping(Mapping[KT, VT]) + + A generic version of :class:`collections.abc.MutableMapping`. + +.. class:: Sequence(Sized, Iterable[T_co], Container[T_co]) + + A generic version of :class:`collections.abc.Sequence`. + +.. class:: MutableSequence(Sequence[T]) + + A generic version of :class:`collections.abc.MutableSequence`. + +.. class:: ByteString(Sequence[int]) + + A generic version of :class:`collections.abc.ByteString`. + + This type represents the types :class:`bytes`, :class:`bytearray`, + and :class:`memoryview`. + + As a shorthand for this type, :class:`bytes` can be used to + annotate arguments of any of the types mentioned above. + +.. class:: List(list, MutableSequence[T]) + + Generic version of :class:`list`. + Useful for annotating return types. To annotate arguments it is preferred + to use abstract collection types such as :class:`Mapping`, :class:`Sequence`, + or :class:`AbstractSet`. + + This type may be used as follows:: + + T = TypeVar('T', int, float) + + def vec2(x: T, y: T) -> List[T]: + return [x, y] + + def slice__to_4(vector: Sequence[T]) -> List[T]: + return vector[0:4] + +.. class:: AbstractSet(set, MutableSet[T]) + + A generic version of :class:`collections.abc.Set`. + +.. class:: MappingView(Sized, Iterable[T_co]) + + A generic version of :class:`collections.abc.MappingView`. + +.. class:: KeysView(MappingView[KT_co], AbstractSet[KT_co]) + + A generic version of :class:`collections.abc.KeysView`. + +.. class:: ItemsView(MappingView, Generic[KT_co, VT_co]) + + A generic version of :class:`collections.abc.ItemsView`. + +.. class:: ValuesView(MappingView[VT_co]) + + A generic version of :class:`collections.abc.ValuesView`. + +.. class:: Dict(dict, MutableMapping[KT, VT]) + + A generic version of :class:`dict`. + The usage of this type is as follows:: + + def get_position_in_index(word_list: Dict[str, int], word: str) -> int: + return word_list[word] + +.. class:: Generator(Iterator[T_co], Generic[T_co, T_contra, V_co]) + +.. class:: io + + Wrapper namespace for I/O stream types. + + This defines the generic type ``IO[AnyStr]`` and aliases ``TextIO`` + and ``BinaryIO`` for respectively ``IO[str]`` and ``IO[bytes]``. + These representing the types of I/O streams such as returned by + :func:`open`. + +.. class:: re + + Wrapper namespace for regular expression matching types. + + This defines the type aliases ``Pattern`` and ``Match`` which + correspond to the return types from :func:`re.compile` and + :func:`re.match`. These types (and the corresponding functions) + are generic in ``AnyStr`` and can be made specific by writing + ``Pattern[str]``, ``Pattern[bytes]``, ``Match[str]``, or + ``Match[bytes]``. + +.. function:: NamedTuple(typename, fields) + + Typed version of namedtuple. + + Usage:: + + Employee = typing.NamedTuple('Employee', [('name', str), 'id', int)]) + + This is equivalent to:: + + Employee = collections.namedtuple('Employee', ['name', 'id']) + + The resulting class has one extra attribute: _field_types, + giving a dict mapping field names to types. (The field names + are in the _fields attribute, which is part of the namedtuple + API.) + +.. function:: cast(typ, val) + + Cast a value to a type. + + This returns the value unchanged. To the type checker this + signals that the return value has the designated type, but at + runtime we intentionally don't check anything (we want this + to be as fast as possible). + +.. function:: get_type_hints(obj) + + Return type hints for a function or method object. + + This is often the same as ``obj.__annotations__``, but it handles + forward references encoded as string literals, and if necessary + adds ``Optional[t]`` if a default value equal to None is set. + +.. decorator:: no_type_check(arg) + + Decorator to indicate that annotations are not type hints. + + The argument must be a class or function; if it is a class, it + applies recursively to all methods defined in that class (but not + to methods defined in its superclasses or subclasses). + + This mutates the function(s) in place. + +.. decorator:: no_type_check_decorator(decorator) + + Decorator to give another decorator the :func:`no_type_check` effect. + + This wraps the decorator with something that wraps the decorated + function in :func:`no_type_check`. diff --git a/Doc/library/unicodedata.rst b/Doc/library/unicodedata.rst index 3b3d3a0a8f..1430d9b403 100644 --- a/Doc/library/unicodedata.rst +++ b/Doc/library/unicodedata.rst @@ -15,8 +15,8 @@ This module provides access to the Unicode Character Database (UCD) which defines character properties for all Unicode characters. The data contained in -this database is compiled from the `UCD version 6.3.0 -<http://www.unicode.org/Public/6.3.0/ucd>`_. +this database is compiled from the `UCD version 8.0.0 +<http://www.unicode.org/Public/8.0.0/ucd>`_. The module uses the same names and symbols as defined by Unicode Standard Annex #44, `"Unicode Character Database" @@ -166,6 +166,6 @@ Examples: .. rubric:: Footnotes -.. [#] http://www.unicode.org/Public/6.3.0/ucd/NameAliases.txt +.. [#] http://www.unicode.org/Public/8.0.0/ucd/NameAliases.txt -.. [#] http://www.unicode.org/Public/6.3.0/ucd/NamedSequences.txt +.. [#] http://www.unicode.org/Public/8.0.0/ucd/NamedSequences.txt diff --git a/Doc/library/unittest.mock.rst b/Doc/library/unittest.mock.rst index 48eefe900f..1b271c83bd 100644 --- a/Doc/library/unittest.mock.rst +++ b/Doc/library/unittest.mock.rst @@ -198,7 +198,7 @@ a :class:`MagicMock` for you. You can specify an alternative class of :class:`Mo the *new_callable* argument to :func:`patch`. -.. class:: Mock(spec=None, side_effect=None, return_value=DEFAULT, wraps=None, name=None, spec_set=None, **kwargs) +.. class:: Mock(spec=None, side_effect=None, return_value=DEFAULT, wraps=None, name=None, spec_set=None, unsafe=False, **kwargs) Create a new :class:`Mock` object. :class:`Mock` takes several optional arguments that specify the behaviour of the Mock object: @@ -235,6 +235,12 @@ the *new_callable* argument to :func:`patch`. this is a new Mock (created on first access). See the :attr:`return_value` attribute. + * *unsafe*: By default if any attribute starts with *assert* or + *assret* will raise an :exc:`AttributeError`. Passing ``unsafe=True`` + will allow access to these attributes. + + .. versionadded:: 3.5 + * *wraps*: Item for the mock object to wrap. If *wraps* is not None then calling the Mock will pass the call through to the wrapped object (returning the real result). Attribute access on the mock will return a @@ -315,6 +321,20 @@ the *new_callable* argument to :func:`patch`. >>> calls = [call(4), call(2), call(3)] >>> mock.assert_has_calls(calls, any_order=True) + .. method:: assert_not_called(*args, **kwargs) + + Assert the mock was never called. + + >>> m = Mock() + >>> m.hello.assert_not_called() + >>> obj = m.hello() + >>> m.hello.assert_not_called() + Traceback (most recent call last): + ... + AssertionError: Expected 'hello' to not have been called. Called 1 times. + + .. versionadded:: 3.5 + .. method:: reset_mock() @@ -1032,6 +1052,12 @@ patch default because it can be dangerous. With it switched on you can write passing tests against APIs that don't actually exist! + .. note:: + + .. versionchanged:: 3.5 + If you are patching builtins in a module then you don't + need to pass ``create=True``, it will be added by default. + Patch can be used as a :class:`TestCase` class decorator. It works by decorating each test method in the class. This reduces the boilerplate code when your test methods share a common patchings set. :func:`patch` finds @@ -1403,6 +1429,22 @@ It is also possible to stop all patches which have been started by using Stop all active patches. Only stops patches started with ``start``. +.. _patch-builtins: + +patch builtins +~~~~~~~~~~~~~~ +You can patch any builtins within a module. The following example patches +builtin :func:`ord`: + + >>> @patch('__main__.ord') + ... def test(mock_ord): + ... mock_ord.return_value = 101 + ... print(ord('c')) + ... + >>> test() + 101 + + TEST_PREFIX ~~~~~~~~~~~ @@ -1587,7 +1629,7 @@ The full list of supported magic methods is: * Context manager: ``__enter__`` and ``__exit__`` * Unary numeric methods: ``__neg__``, ``__pos__`` and ``__invert__`` * The numeric methods (including right hand and in-place variants): - ``__add__``, ``__sub__``, ``__mul__``, ``__div__``, ``__truediv__``, + ``__add__``, ``__sub__``, ``__mul__``, ``__matmul__``, ``__div__``, ``__truediv__``, ``__floordiv__``, ``__mod__``, ``__divmod__``, ``__lshift__``, ``__rshift__``, ``__and__``, ``__xor__``, ``__or__``, and ``__pow__`` * Numeric conversion methods: ``__complex__``, ``__int__``, ``__float__`` @@ -2006,7 +2048,7 @@ mock_open The mock of :meth:`~io.IOBase.read` changed to consume *read_data* rather than returning it on each call. - .. versionchanged:: 3.4.4 + .. versionchanged:: 3.5 *read_data* is now reset on each call to the *mock*. Using :func:`open` as a context manager is a great way to ensure your file handles @@ -2023,7 +2065,7 @@ Mocking context managers with a :class:`MagicMock` is common enough and fiddly enough that a helper function is useful. >>> m = mock_open() - >>> with patch('__main__.open', m, create=True): + >>> with patch('__main__.open', m): ... with open('foo', 'w') as h: ... h.write('some stuff') ... @@ -2038,7 +2080,7 @@ enough that a helper function is useful. And for reading files: - >>> with patch('__main__.open', mock_open(read_data='bibble'), create=True) as m: + >>> with patch('__main__.open', mock_open(read_data='bibble')) as m: ... with open('foo') as h: ... result = h.read() ... diff --git a/Doc/library/unittest.rst b/Doc/library/unittest.rst index 1930f785a7..d3a3649f84 100644 --- a/Doc/library/unittest.rst +++ b/Doc/library/unittest.rst @@ -214,9 +214,16 @@ Command-line options Stop the test run on the first error or failure. +.. cmdoption:: --locals + + Show local variables in tracebacks. + .. versionadded:: 3.2 The command-line options ``-b``, ``-c`` and ``-f`` were added. +.. versionadded:: 3.5 + The command-line option ``--locals``. + The command line can also be used for test discovery, for running all of the tests in a project or just a subset. @@ -1541,6 +1548,20 @@ Loading and running tests :data:`unittest.defaultTestLoader`. Using a subclass or instance, however, allows customization of some configurable properties. + :class:`TestLoader` objects have the following attributes: + + + .. attribute:: errors + + A list of the non-fatal errors encountered while loading tests. Not reset + by the loader at any point. Fatal errors are signalled by the relevant + a method raising an exception to the caller. Non-fatal errors are also + indicated by a synthetic test that will raise the original error when + run. + + .. versionadded:: 3.5 + + :class:`TestLoader` objects have the following methods: @@ -1556,7 +1577,7 @@ Loading and running tests case is created for that method instead. - .. method:: loadTestsFromModule(module) + .. method:: loadTestsFromModule(module, pattern=None) Return a suite of all tests cases contained in the given module. This method searches *module* for classes derived from :class:`TestCase` and @@ -1573,11 +1594,18 @@ Loading and running tests If a module provides a ``load_tests`` function it will be called to load the tests. This allows modules to customize test loading. - This is the `load_tests protocol`_. + This is the `load_tests protocol`_. The *pattern* argument is passed as + the third argument to ``load_tests``. .. versionchanged:: 3.2 Support for ``load_tests`` added. + .. versionchanged:: 3.5 + The undocumented and unofficial *use_load_tests* default argument is + deprecated and ignored, although it is still accepted for backward + compatibility. The method also now accepts a keyword-only argument + *pattern* which is passed to ``load_tests`` as the third argument. + .. method:: loadTestsFromName(name, module=None) @@ -1603,6 +1631,12 @@ Loading and running tests The method optionally resolves *name* relative to the given *module*. + .. versionchanged:: 3.5 + If an :exc:`ImportError` or :exc:`AttributeError` occurs while traversing + *name* then a synthetic test that raises that error when run will be + returned. These errors are included in the errors accumulated by + self.errors. + .. method:: loadTestsFromNames(names, module=None) @@ -1629,18 +1663,22 @@ Loading and running tests the start directory is not the top level directory then the top level directory must be specified separately. - If importing a module fails, for example due to a syntax error, then this - will be recorded as a single error and discovery will continue. If the - import failure is due to :exc:`SkipTest` being raised, it will be recorded - as a skip instead of an error. + If importing a module fails, for example due to a syntax error, then + this will be recorded as a single error and discovery will continue. If + the import failure is due to :exc:`SkipTest` being raised, it will be + recorded as a skip instead of an error. - If a test package name (directory with :file:`__init__.py`) matches the - pattern then the package will be checked for a ``load_tests`` - function. If this exists then it will be called with *loader*, *tests*, - *pattern*. + If a package (a directory containing a file named :file:`__init__.py`) is + found, the package will be checked for a ``load_tests`` function. If this + exists then it will be called + ``package.load_tests(loader, tests, pattern)``. Test discovery takes care + to ensure that a package is only checked for tests once during an + invocation, even if the load_tests function itself calls + ``loader.discover``. - If ``load_tests`` exists then discovery does *not* recurse into the package, - ``load_tests`` is responsible for loading all tests in the package. + If ``load_tests`` exists then discovery does *not* recurse into the + package, ``load_tests`` is responsible for loading all tests in the + package. The pattern is deliberately not stored as a loader attribute so that packages can continue discovery themselves. *top_level_dir* is stored so @@ -1659,6 +1697,11 @@ Loading and running tests the same even if the underlying file system's ordering is not dependent on file name. + .. versionchanged:: 3.5 + Found packages are now checked for ``load_tests`` regardless of + whether their path matches *pattern*, because it is impossible for + a package name to match the default pattern. + The following attributes of a :class:`TestLoader` can be configured either by subclassing or assignment on an instance: @@ -1741,12 +1784,10 @@ Loading and running tests Set to ``True`` when the execution of tests should stop by :meth:`stop`. - .. attribute:: testsRun The total number of tests run so far. - .. attribute:: buffer If set to true, ``sys.stdout`` and ``sys.stderr`` will be buffered in between @@ -1756,7 +1797,6 @@ Loading and running tests .. versionadded:: 3.2 - .. attribute:: failfast If set to true :meth:`stop` will be called on the first failure or error, @@ -1764,6 +1804,11 @@ Loading and running tests .. versionadded:: 3.2 + .. attribute:: tb_locals + + If set to true then local variables will be shown in tracebacks. + + .. versionadded:: 3.5 .. method:: wasSuccessful() @@ -1774,7 +1819,6 @@ Loading and running tests Returns ``False`` if there were any :attr:`unexpectedSuccesses` from tests marked with the :func:`expectedFailure` decorator. - .. method:: stop() This method can be called to signal that the set of tests being run should @@ -1906,12 +1950,14 @@ Loading and running tests .. class:: TextTestRunner(stream=None, descriptions=True, verbosity=1, failfast=False, \ - buffer=False, resultclass=None, warnings=None) + buffer=False, resultclass=None, warnings=None, *, tb_locals=False) A basic test runner implementation that outputs results to a stream. If *stream* is ``None``, the default, :data:`sys.stderr` is used as the output stream. This class has a few configurable parameters, but is essentially very simple. Graphical - applications which run test suites should provide alternate implementations. + applications which run test suites should provide alternate implementations. Such + implementations should accept ``**kwargs`` as the interface to construct runners + changes when features are added to unittest. By default this runner shows :exc:`DeprecationWarning`, :exc:`PendingDeprecationWarning`, :exc:`ResourceWarning` and @@ -1930,6 +1976,9 @@ Loading and running tests The default stream is set to :data:`sys.stderr` at instantiation time rather than import time. + .. versionchanged:: 3.5 + Added the tb_locals parameter. + .. method:: _makeResult() This method returns the instance of ``TestResult`` used by :meth:`run`. @@ -2027,7 +2076,10 @@ test runs or test discovery by implementing a function called ``load_tests``. If a test module defines ``load_tests`` it will be called by :meth:`TestLoader.loadTestsFromModule` with the following arguments:: - load_tests(loader, standard_tests, None) + load_tests(loader, standard_tests, pattern) + +where *pattern* is passed straight through from ``loadTestsFromModule``. It +defaults to ``None``. It should return a :class:`TestSuite`. @@ -2049,21 +2101,12 @@ A typical ``load_tests`` function that loads tests from a specific set of suite.addTests(tests) return suite -If discovery is started, either from the command line or by calling -:meth:`TestLoader.discover`, with a pattern that matches a package -name then the package :file:`__init__.py` will be checked for ``load_tests``. - -.. note:: - - The default pattern is ``'test*.py'``. This matches all Python files - that start with ``'test'`` but *won't* match any test directories. - - A pattern like ``'test*'`` will match test packages as well as - modules. - -If the package :file:`__init__.py` defines ``load_tests`` then it will be -called and discovery not continued into the package. ``load_tests`` -is called with the following arguments:: +If discovery is started in a directory containing a package, either from the +command line or by calling :meth:`TestLoader.discover`, then the package +:file:`__init__.py` will be checked for ``load_tests``. If that function does +not exist, discovery will recurse into the package as though it were just +another directory. Otherwise, discovery of the package's tests will be left up +to ``load_tests`` which is called with the following arguments:: load_tests(loader, standard_tests, pattern) @@ -2082,6 +2125,11 @@ continue (and potentially modify) test discovery. A 'do nothing' standard_tests.addTests(package_tests) return standard_tests +.. versionchanged:: 3.5 + Discovery no longer checks package names for matching *pattern* due to the + impossibility of package names matching the default pattern. + + Class and Module Fixtures ------------------------- diff --git a/Doc/library/urllib.parse.rst b/Doc/library/urllib.parse.rst index fbbabcadf9..40098d0496 100644 --- a/Doc/library/urllib.parse.rst +++ b/Doc/library/urllib.parse.rst @@ -270,6 +270,11 @@ or on combining URL components into a URL string. :func:`urlunsplit`, removing possible *scheme* and *netloc* parts. + .. versionchanged:: 3.5 + + Behaviour updated to match the semantics defined in :rfc:`3986`. + + .. function:: urldefrag(url) If *url* contains a fragment identifier, return a modified version of *url* @@ -516,7 +521,8 @@ task isn't already covered by the URL parsing functions above. Example: ``unquote_to_bytes('a%26%EF')`` yields ``b'a&\xef'``. -.. function:: urlencode(query, doseq=False, safe='', encoding=None, errors=None) +.. function:: urlencode(query, doseq=False, safe='', encoding=None, \ + errors=None, quote_via=quote_plus) Convert a mapping object or a sequence of two-element tuples, which may contain :class:`str` or :class:`bytes` objects, to a "percent-encoded" @@ -525,8 +531,16 @@ task isn't already covered by the URL parsing functions above. properly encoded to bytes, otherwise it would result in a :exc:`TypeError`. The resulting string is a series of ``key=value`` pairs separated by ``'&'`` - characters, where both *key* and *value* are quoted using :func:`quote_plus` - above. When a sequence of two-element tuples is used as the *query* + characters, where both *key* and *value* are quoted using the *quote_via* + function. By default, :func:`quote_plus` is used to quote the values, which + means spaces are quoted as a ``'+'`` character and '/' characters are + encoded as ``%2F``, which follows the standard for GET requests + (``application/x-www-form-urlencoded``). An alternate function that can be + passed as *quote_via* is :func:`quote`, which will encode spaces as ``%20`` + and not encode '/' characters. For maximum control of what is quoted, use + ``quote`` and specify a value for *safe*. + + When a sequence of two-element tuples is used as the *query* argument, the first element of each tuple is a key and the second is a value. The value element in itself can be a sequence and in that case, if the optional parameter *doseq* is evaluates to *True*, individual @@ -535,7 +549,7 @@ task isn't already covered by the URL parsing functions above. string will match the order of parameter tuples in the sequence. The *safe*, *encoding*, and *errors* parameters are passed down to - :func:`quote_plus` (the *encoding* and *errors* parameters are only passed + *quote_via* (the *encoding* and *errors* parameters are only passed when a query element is a :class:`str`). To reverse this encoding process, :func:`parse_qs` and :func:`parse_qsl` are @@ -547,6 +561,9 @@ task isn't already covered by the URL parsing functions above. .. versionchanged:: 3.2 Query parameter supports bytes and string objects. + .. versionadded:: 3.5 + *quote_via* parameter. + .. seealso:: diff --git a/Doc/library/urllib.request.rst b/Doc/library/urllib.request.rst index 75b95d9b4a..349ba70d36 100644 --- a/Doc/library/urllib.request.rst +++ b/Doc/library/urllib.request.rst @@ -115,6 +115,7 @@ The :mod:`urllib.request` module defines the following functions: .. versionchanged:: 3.4.3 *context* was added. + .. function:: install_opener(opener) Install an :class:`OpenerDirector` instance as the default global opener. @@ -287,13 +288,37 @@ The following classes are provided: fits. +.. class:: HTTPPasswordMgrWithPriorAuth() + + A variant of :class:`HTTPPasswordMgrWithDefaultRealm` that also has a + database of ``uri -> is_authenticated`` mappings. Can be used by a + BasicAuth handler to determine when to send authentication credentials + immediately instead of waiting for a ``401`` response first. + + .. versionadded:: 3.5 + + .. class:: AbstractBasicAuthHandler(password_mgr=None) This is a mixin class that helps with HTTP authentication, both to the remote host and to a proxy. *password_mgr*, if given, should be something that is compatible with :class:`HTTPPasswordMgr`; refer to section :ref:`http-password-mgr` for information on the interface that must be - supported. + supported. If *passwd_mgr* also provides ``is_authenticated`` and + ``update_authenticated`` methods (see + :ref:`http-password-mgr-with-prior-auth`), then the handler will use the + ``is_authenticated`` result for a given URI to determine whether or not to + send authentication credentials with the request. If ``is_authenticated`` + returns ``True`` for the URI, credentials are sent. If ``is_authenticated`` + is ``False``, credentials are not sent, and then if a ``401`` response is + received the request is re-sent with the authentication credentials. If + authentication succeeds, ``update_authenticated`` is called to set + ``is_authenticated`` ``True`` for the URI, so that subsequent requests to + the URI or any of its super-URIs will automatically include the + authentication credentials. + + .. versionadded:: 3.5 + Added ``is_authenticated`` support. .. class:: HTTPBasicAuthHandler(password_mgr=None) @@ -845,6 +870,42 @@ These methods are available on :class:`HTTPPasswordMgr` and searched if the given *realm* has no matching user/password. +.. _http-password-mgr-with-prior-auth: + +HTTPPasswordMgrWithPriorAuth Objects +------------------------------------ + +This password manager extends :class:`HTTPPasswordMgrWithDefaultRealm` to support +tracking URIs for which authentication credentials should always be sent. + + +.. method:: HTTPPasswordMgrWithPriorAuth.add_password(realm, uri, user, \ + passwd, is_authenticated=False) + + *realm*, *uri*, *user*, *passwd* are as for + :meth:`HTTPPasswordMgr.add_password`. *is_authenticated* sets the initial + value of the ``is_authenticated`` flag for the given URI or list of URIs. + If *is_authenticated* is specified as ``True``, *realm* is ignored. + + +.. method:: HTTPPasswordMgr.find_user_password(realm, authuri) + + Same as for :class:`HTTPPasswordMgrWithDefaultRealm` objects + + +.. method:: HTTPPasswordMgrWithPriorAuth.update_authenticated(self, uri, \ + is_authenticated=False) + + Update the ``is_authenticated`` flag for the given *uri* or list + of URIs. + + +.. method:: HTTPPasswordMgrWithPriorAuth.is_authenticated(self, authuri) + + Returns the current state of the ``is_authenticated`` flag for + the given URI. + + .. _abstract-basic-auth-handler: AbstractBasicAuthHandler Objects diff --git a/Doc/library/weakref.rst b/Doc/library/weakref.rst index cc883b1b51..2e16077b8a 100644 --- a/Doc/library/weakref.rst +++ b/Doc/library/weakref.rst @@ -258,7 +258,7 @@ These method have the same issues as the and :meth:`keyrefs` method of are called in reverse order of creation. A finalizer will never invoke its callback during the later part of - the interpreter shutdown when module globals are liable to have + the :term:`interpreter shutdown` when module globals are liable to have been replaced by :const:`None`. .. method:: __call__() @@ -527,8 +527,8 @@ follows:: Starting with Python 3.4, :meth:`__del__` methods no longer prevent reference cycles from being garbage collected, and module globals are -no longer forced to :const:`None` during interpreter shutdown. So this -code should work without any issues on CPython. +no longer forced to :const:`None` during :term:`interpreter shutdown`. +So this code should work without any issues on CPython. However, handling of :meth:`__del__` methods is notoriously implementation specific, since it depends on internal details of the interpreter's garbage diff --git a/Doc/library/wsgiref.rst b/Doc/library/wsgiref.rst index a9e19dad98..de74c30250 100644 --- a/Doc/library/wsgiref.rst +++ b/Doc/library/wsgiref.rst @@ -184,10 +184,11 @@ This module provides a single class, :class:`Headers`, for convenient manipulation of WSGI response headers using a mapping-like interface. -.. class:: Headers(headers) +.. class:: Headers([headers]) Create a mapping-like object wrapping *headers*, which must be a list of header - name/value tuples as described in :pep:`3333`. + name/value tuples as described in :pep:`3333`. The default value of *headers* is + an empty list. :class:`Headers` objects support typical mapping operations including :meth:`__getitem__`, :meth:`get`, :meth:`__setitem__`, :meth:`setdefault`, @@ -251,6 +252,10 @@ manipulation of WSGI response headers using a mapping-like interface. Content-Disposition: attachment; filename="bud.gif" + .. versionchanged:: 3.5 + *headers* parameter is optional. + + :mod:`wsgiref.simple_server` -- a simple WSGI HTTP server --------------------------------------------------------- diff --git a/Doc/library/xml.sax.reader.rst b/Doc/library/xml.sax.reader.rst index 3ab60636c5..31ca260f7c 100644 --- a/Doc/library/xml.sax.reader.rst +++ b/Doc/library/xml.sax.reader.rst @@ -100,8 +100,10 @@ The :class:`XMLReader` interface supports the following methods: system identifier (a string identifying the input source -- typically a file name or an URL), a file-like object, or an :class:`InputSource` object. When :meth:`parse` returns, the input is completely processed, and the parser object - can be discarded or reset. As a limitation, the current implementation only - accepts byte streams; processing of character streams is for further study. + can be discarded or reset. + + .. versionchanged:: 3.5 + Added support of character streams. .. method:: XMLReader.getContentHandler() @@ -288,8 +290,7 @@ InputSource Objects .. method:: InputSource.setByteStream(bytefile) - Set the byte stream (a Python file-like object which does not perform - byte-to-character conversion) for this input source. + Set the byte stream (a :term:`binary file`) for this input source. The SAX parser will ignore this if there is also a character stream specified, but it will use a byte stream in preference to opening a URI connection itself. @@ -308,8 +309,7 @@ InputSource Objects .. method:: InputSource.setCharacterStream(charfile) - Set the character stream for this input source. (The stream must be a Python 1.6 - Unicode-wrapped file-like that performs conversion to strings.) + Set the character stream (a :term:`text file`) for this input source. If there is a character stream specified, the SAX parser will ignore any byte stream and will not attempt to open a URI connection to the system identifier. diff --git a/Doc/library/xml.sax.rst b/Doc/library/xml.sax.rst index e95d6b0f85..55f97999de 100644 --- a/Doc/library/xml.sax.rst +++ b/Doc/library/xml.sax.rst @@ -47,7 +47,11 @@ The convenience functions are: .. function:: parseString(string, handler, error_handler=handler.ErrorHandler()) Similar to :func:`parse`, but parses from a buffer *string* received as a - parameter. + parameter. *string* must be a :class:`str` instance or a + :term:`bytes-like object`. + + .. versionchanged:: 3.5 + Added support of :class:`str` instances. A typical SAX application uses three kinds of objects: readers, handlers and input sources. "Reader" in this context is another term for parser, i.e. some diff --git a/Doc/library/xmlrpc.client.rst b/Doc/library/xmlrpc.client.rst index cc5e83a706..e199931b55 100644 --- a/Doc/library/xmlrpc.client.rst +++ b/Doc/library/xmlrpc.client.rst @@ -27,7 +27,7 @@ between conformable Python objects and XML on the wire. constructed data. If you need to parse untrusted or unauthenticated data see :ref:`xml-vulnerabilities`. -.. versionchanged:: 3.4.3 +.. versionchanged:: 3.5 For https URIs, :mod:`xmlrpc.client` now performs all the necessary certificate and hostname checks by default @@ -129,7 +129,7 @@ between conformable Python objects and XML on the wire. :class:`Server` is retained as an alias for :class:`ServerProxy` for backwards compatibility. New code should use :class:`ServerProxy`. - .. versionchanged:: 3.4.3 + .. versionchanged:: 3.5 Added the *context* argument. @@ -200,6 +200,11 @@ grouped under the reserved :attr:`system` attribute: no such string is available, an empty string is returned. The documentation string may contain HTML markup. +.. versionchanged:: 3.5 + + Instances of :class:`ServerProxy` support the :term:`context manager` protocol + for closing the underlying transport. + A working example follows. The server code:: @@ -217,9 +222,9 @@ The client code for the preceding server:: import xmlrpc.client - proxy = xmlrpc.client.ServerProxy("http://localhost:8000/") - print("3 is even: %s" % str(proxy.is_even(3))) - print("100 is even: %s" % str(proxy.is_even(100))) + with xmlrpc.client.ServerProxy("http://localhost:8000/") as proxy: + print("3 is even: %s" % str(proxy.is_even(3))) + print("100 is even: %s" % str(proxy.is_even(100))) .. _datetime-objects: @@ -527,14 +532,14 @@ Example of Client Usage from xmlrpc.client import ServerProxy, Error # server = ServerProxy("http://localhost:8000") # local server - server = ServerProxy("http://betty.userland.com") + with ServerProxy("http://betty.userland.com") as proxy: - print(server) + print(proxy) - try: - print(server.examples.getStateName(41)) - except Error as v: - print("ERROR", v) + try: + print(proxy.examples.getStateName(41)) + except Error as v: + print("ERROR", v) To access an XML-RPC server through a proxy, you need to define a custom transport. The following example shows how: diff --git a/Doc/library/zipapp.rst b/Doc/library/zipapp.rst new file mode 100644 index 0000000000..b4484c87d0 --- /dev/null +++ b/Doc/library/zipapp.rst @@ -0,0 +1,259 @@ +:mod:`zipapp` --- Manage executable python zip archives +======================================================= + +.. module:: zipapp + :synopsis: Manage executable python zip archives + + +.. index:: + single: Executable Zip Files + +.. versionadded:: 3.5 + +**Source code:** :source:`Lib/zipapp.py` + +-------------- + +This module provides tools to manage the creation of zip files containing +Python code, which can be :ref:`executed directly by the Python interpreter +<using-on-interface-options>`. The module provides both a +:ref:`zipapp-command-line-interface` and a :ref:`zipapp-python-api`. + + +Basic Example +------------- + +The following example shows how the :ref:`command-line-interface` +can be used to create an executable archive from a directory containing +Python code. When run, the archive will execute the ``main`` function from +the module ``myapp`` in the archive. + +.. code-block:: sh + + $ python -m zipapp myapp -m "myapp:main" + $ python myapp.pyz + <output from myapp> + + +.. _zipapp-command-line-interface: + +Command-Line Interface +---------------------- + +When called as a program from the command line, the following form is used: + +.. code-block:: sh + + $ python -m zipapp source [options] + +If *source* is a directory, this will create an archive from the contents of +*source*. If *source* is a file, it should be an archive, and it will be +copied to the target archive (or the contents of its shebang line will be +displayed if the --info option is specified). + +The following options are understood: + +.. program:: zipapp + +.. cmdoption:: -o <output>, --output=<output> + + Write the output to a file named *output*. If this option is not specified, + the output filename will be the same as the input *source*, with the + extension ``.pyz`` added. If an explicit filename is given, it is used as + is (so a ``.pyz`` extension should be included if required). + + An output filename must be specified if the *source* is an archive (and in + that case, *output* must not be the same as *source*). + +.. cmdoption:: -p <interpreter>, --python=<interpreter> + + Add a ``#!`` line to the archive specifying *interpreter* as the command + to run. Also, on POSIX, make the archive executable. The default is to + write no ``#!`` line, and not make the file executable. + +.. cmdoption:: -m <mainfn>, --main=<mainfn> + + Write a ``__main__.py`` file to the archive that executes *mainfn*. The + *mainfn* argument should have the form "pkg.mod:fn", where "pkg.mod" is a + package/module in the archive, and "fn" is a callable in the given module. + The ``__main__.py`` file will execute that callable. + + :option:`--main` cannot be specified when copying an archive. + +.. cmdoption:: --info + + Display the interpreter embedded in the archive, for diagnostic purposes. In + this case, any other options are ignored and SOURCE must be an archive, not a + directory. + +.. cmdoption:: -h, --help + + Print a short usage message and exit. + + +.. _zipapp-python-api: + +Python API +---------- + +The module defines two convenience functions: + + +.. function:: create_archive(source, target=None, interpreter=None, main=None) + + Create an application archive from *source*. The source can be any + of the following: + + * The name of a directory, or a :class:`pathlib.Path` object referring + to a directory, in which case a new application archive will be + created from the content of that directory. + * The name of an existing application archive file, or a :class:`pathlib.Path` + object referring to such a file, in which case the file is copied to + the target (modifying it to reflect the value given for the *interpreter* + argument). The file name should include the ``.pyz`` extension, if required. + * A file object open for reading in bytes mode. The content of the + file should be an application archive, and the file object is + assumed to be positioned at the start of the archive. + + The *target* argument determines where the resulting archive will be + written: + + * If it is the name of a file, or a :class:`pathlb.Path` object, + the archive will be written to that file. + * If it is an open file object, the archive will be written to that + file object, which must be open for writing in bytes mode. + * If the target is omitted (or None), the source must be a directory + and the target will be a file with the same name as the source, with + a ``.pyz`` extension added. + + The *interpreter* argument specifies the name of the Python + interpreter with which the archive will be executed. It is written as + a "shebang" line at the start of the archive. On POSIX, this will be + interpreted by the OS, and on Windows it will be handled by the Python + launcher. Omitting the *interpreter* results in no shebang line being + written. If an interpreter is specified, and the target is a + filename, the executable bit of the target file will be set. + + The *main* argument specifies the name of a callable which will be + used as the main program for the archive. It can only be specified if + the source is a directory, and the source does not already contain a + ``__main__.py`` file. The *main* argument should take the form + "pkg.module:callable" and the archive will be run by importing + "pkg.module" and executing the given callable with no arguments. It + is an error to omit *main* if the source is a directory and does not + contain a ``__main__.py`` file, as otherwise the resulting archive + would not be executable. + + If a file object is specified for *source* or *target*, it is the + caller's responsibility to close it after calling create_archive. + + When copying an existing archive, file objects supplied only need + ``read`` and ``readline``, or ``write`` methods. When creating an + archive from a directory, if the target is a file object it will be + passed to the ``zipfile.ZipFile`` class, and must supply the methods + needed by that class. + +.. function:: get_interpreter(archive) + + Return the interpreter specified in the ``#!`` line at the start of the + archive. If there is no ``#!`` line, return :const:`None`. + The *archive* argument can be a filename or a file-like object open + for reading in bytes mode. It is assumed to be at the start of the archive. + + +.. _zipapp-examples: + +Examples +-------- + +Pack up a directory into an archive, and run it. + +.. code-block:: sh + + $ python -m zipapp myapp + $ python myapp.pyz + <output from myapp> + +The same can be done using the :func:`create_archive` functon:: + + >>> import zipapp + >>> zipapp.create_archive('myapp.pyz', 'myapp') + +To make the application directly executable on POSIX, specify an interpreter +to use. + +.. code-block:: sh + + $ python -m zipapp myapp -p "/usr/bin/env python" + $ ./myapp.pyz + <output from myapp> + +To replace the shebang line on an existing archive, create a modified archive +using the :func:`create_archive` function:: + + >>> import zipapp + >>> zipapp.create_archive('old_archive.pyz', 'new_archive.pyz', '/usr/bin/python3') + +To update the file in place, do the replacement in memory using a :class:`BytesIO` +object, and then overwrite the source afterwards. Note that there is a risk +when overwriting a file in place that an error will result in the loss of +the original file. This code does not protect against such errors, but +production code should do so. Also, this method will only work if the archive +fits in memory:: + + >>> import zipapp + >>> import io + >>> temp = io.BytesIO() + >>> zipapp.create_archive('myapp.pyz', temp, '/usr/bin/python2') + >>> with open('myapp.pyz', 'wb') as f: + >>> f.write(temp.getvalue()) + +Note that if you specify an interpreter and then distribute your application +archive, you need to ensure that the interpreter used is portable. The Python +launcher for Windows supports most common forms of POSIX ``#!`` line, but there +are other issues to consider: + +* If you use "/usr/bin/env python" (or other forms of the "python" command, + such as "/usr/bin/python"), you need to consider that your users may have + either Python 2 or Python 3 as their default, and write your code to work + under both versions. +* If you use an explicit version, for example "/usr/bin/env python3" your + application will not work for users who do not have that version. (This + may be what you want if you have not made your code Python 2 compatible). +* There is no way to say "python X.Y or later", so be careful of using an + exact version like "/usr/bin/env python3.4" as you will need to change your + shebang line for users of Python 3.5, for example. + +The Python Zip Application Archive Format +----------------------------------------- + +Python has been able to execute zip files which contain a ``__main__.py`` file +since version 2.6. In order to be executed by Python, an application archive +simply has to be a standard zip file containing a ``__main__.py`` file which +will be run as the entry point for the application. As usual for any Python +script, the parent of the script (in this case the zip file) will be placed on +:data:`sys.path` and thus further modules can be imported from the zip file. + +The zip file format allows arbitrary data to be prepended to a zip file. The +zip application format uses this ability to prepend a standard POSIX "shebang" +line to the file (``#!/path/to/interpreter``). + +Formally, the Python zip application format is therefore: + +1. An optional shebang line, containing the characters ``b'#!'`` followed by an + interpreter name, and then a newline (``b'\n'``) character. The interpreter + name can be anything acceptable to the OS "shebang" processing, or the Python + launcher on Windows. The interpreter should be encoded in UTF-8 on Windows, + and in :func:`sys.getfilesystemencoding()` on POSIX. +2. Standard zipfile data, as generated by the :mod:`zipfile` module. The + zipfile content *must* include a file called ``__main__.py`` (which must be + in the "root" of the zipfile - i.e., it cannot be in a subdirectory). The + zipfile data can be compressed or uncompressed. + +If an application archive has a shebang line, it may have the executable bit set +on POSIX systems, to allow it to be executed directly. + +There is no requirement that the tools in this module are used to create +application archives - the module is a convenience, but archives in the above +format created by any means are acceptable to Python. + diff --git a/Doc/library/zipfile.rst b/Doc/library/zipfile.rst index 10a094f8b1..d40315eaf8 100644 --- a/Doc/library/zipfile.rst +++ b/Doc/library/zipfile.rst @@ -134,12 +134,16 @@ ZipFile Objects Open a ZIP file, where *file* can be either a path to a file (a string) or a file-like object. The *mode* parameter should be ``'r'`` to read an existing - file, ``'w'`` to truncate and write a new file, or ``'a'`` to append to an - existing file. If *mode* is ``'a'`` and *file* refers to an existing ZIP + file, ``'w'`` to truncate and write a new file, ``'x'`` to exclusive create + and write a new file, or ``'a'`` to append to an existing file. + If *mode* is ``'x'`` and *file* refers to an existing file, + a :exc:`FileExistsError` will be raised. + If *mode* is ``'a'`` and *file* refers to an existing ZIP file, then additional files are added to it. If *file* does not refer to a ZIP file, then a new ZIP archive is appended to the file. This is meant for adding a ZIP archive to another file (such as :file:`python.exe`). If *mode* is ``a`` and the file does not exist at all, it is created. + If *mode* is ``r`` or ``a``, the file should be seekable. *compression* is the ZIP compression method to use when writing the archive, and should be :const:`ZIP_STORED`, :const:`ZIP_DEFLATED`, :const:`ZIP_BZIP2` or :const:`ZIP_LZMA`; unrecognized @@ -151,7 +155,7 @@ ZipFile Objects extensions when the zipfile is larger than 2 GiB. If it is false :mod:`zipfile` will raise an exception when the ZIP file would require ZIP64 extensions. - If the file is created with mode ``'a'`` or ``'w'`` and then + If the file is created with mode ``'w'``, ``'x'`` or ``'a'`` and then :meth:`closed <close>` without adding any files to the archive, the appropriate ZIP structures for an empty archive will be written to the file. @@ -171,6 +175,10 @@ ZipFile Objects .. versionchanged:: 3.4 ZIP64 extensions are enabled by default. + .. versionchanged:: 3.5 + Added support for writing to unseekable streams. + Added support for the ``'x'`` mode. + .. method:: ZipFile.close() @@ -226,14 +234,8 @@ ZipFile Objects .. note:: - If the ZipFile was created by passing in a file-like object as the first - argument to the constructor, then the object returned by :meth:`.open` shares the - ZipFile's file pointer. Under these circumstances, the object returned by - :meth:`.open` should not be used after any additional operations are performed - on the ZipFile object. If the ZipFile was created by passing in a string (the - filename) as the first argument to the constructor, then :meth:`.open` will - create a new file object that will be held by the ZipExtFile, allowing it to - operate independently of the ZipFile. + Objects returned by :meth:`.open` can operate independently of the + ZipFile. .. note:: @@ -318,7 +320,8 @@ ZipFile Objects *arcname* (by default, this will be the same as *filename*, but without a drive letter and with leading path separators removed). If given, *compress_type* overrides the value given for the *compression* parameter to the constructor for - the new entry. The archive must be open with mode ``'w'`` or ``'a'`` -- calling + the new entry. + The archive must be open with mode ``'w'``, ``'x'`` or ``'a'`` -- calling :meth:`write` on a ZipFile created with mode ``'r'`` will raise a :exc:`RuntimeError`. Calling :meth:`write` on a closed ZipFile will raise a :exc:`RuntimeError`. @@ -340,16 +343,16 @@ ZipFile Objects If ``arcname`` (or ``filename``, if ``arcname`` is not given) contains a null byte, the name of the file in the archive will be truncated at the null byte. - .. method:: ZipFile.writestr(zinfo_or_arcname, bytes[, compress_type]) Write the string *bytes* to the archive; *zinfo_or_arcname* is either the file name it will be given in the archive, or a :class:`ZipInfo` instance. If it's an instance, at least the filename, date, and time must be given. If it's a - name, the date and time is set to the current date and time. The archive must be - opened with mode ``'w'`` or ``'a'`` -- calling :meth:`writestr` on a ZipFile - created with mode ``'r'`` will raise a :exc:`RuntimeError`. Calling - :meth:`writestr` on a closed ZipFile will raise a :exc:`RuntimeError`. + name, the date and time is set to the current date and time. + The archive must be opened with mode ``'w'``, ``'x'`` or ``'a'`` -- calling + :meth:`writestr` on a ZipFile created with mode ``'r'`` will raise a + :exc:`RuntimeError`. Calling :meth:`writestr` on a closed ZipFile will + raise a :exc:`RuntimeError`. If given, *compress_type* overrides the value given for the *compression* parameter to the constructor for the new entry, or in the *zinfo_or_arcname* @@ -377,7 +380,8 @@ The following data attributes are also available: .. attribute:: ZipFile.comment The comment text associated with the ZIP file. If assigning a comment to a - :class:`ZipFile` instance created with mode 'a' or 'w', this should be a + :class:`ZipFile` instance created with mode ``'w'``, ``'x'`` or ``'a'``, + this should be a string no longer than 65535 bytes. Comments longer than this will be truncated in the written archive when :meth:`close` is called. @@ -407,8 +411,7 @@ The :class:`PyZipFile` constructor takes the same parameters as the archive. If the *optimize* parameter to :class:`PyZipFile` was not given or ``-1``, - the corresponding file is a :file:`\*.pyo` file if available, else a - :file:`\*.pyc` file, compiling if necessary. + the corresponding file is a :file:`\*.pyc` file, compiling if necessary. If the *optimize* parameter to :class:`PyZipFile` was ``0``, ``1`` or ``2``, only files with that optimization level (see :func:`compile`) are @@ -571,4 +574,3 @@ Instances have the following attributes: .. attribute:: ZipInfo.file_size Size of the uncompressed file. - diff --git a/Doc/library/zipimport.rst b/Doc/library/zipimport.rst index 2cf508b85c..8a5d5d19df 100644 --- a/Doc/library/zipimport.rst +++ b/Doc/library/zipimport.rst @@ -20,10 +20,10 @@ subdirectory. For example, the path :file:`example.zip/lib/` would only import from the :file:`lib/` subdirectory within the archive. Any files may be present in the ZIP archive, but only files :file:`.py` and -:file:`.py[co]` are available for import. ZIP import of dynamic modules +:file:`.pyc` are available for import. ZIP import of dynamic modules (:file:`.pyd`, :file:`.so`) is disallowed. Note that if an archive only contains :file:`.py` files, Python will not attempt to modify the archive by adding the -corresponding :file:`.pyc` or :file:`.pyo` file, meaning that if a ZIP archive +corresponding :file:`.pyc` file, meaning that if a ZIP archive doesn't contain :file:`.pyc` files, importing may be rather slow. ZIP archives with an archive comment are currently not supported. @@ -161,4 +161,3 @@ Here is an example that imports a module from a ZIP archive - note that the >>> import jwzthreading >>> jwzthreading.__file__ 'example.zip/jwzthreading.py' - |
