diff options
Diffstat (limited to 'Doc')
392 files changed, 14469 insertions, 5688 deletions
diff --git a/Doc/Makefile b/Doc/Makefile index 2220d92d91..878685dd7a 100644 --- a/Doc/Makefile +++ b/Doc/Makefile @@ -15,11 +15,12 @@ ALLSPHINXOPTS = -b $(BUILDER) -d build/doctrees -D latex_paper_size=$(PAPER) \ .PHONY: help build html htmlhelp latex text changes linkcheck \ suspicious coverage doctest pydoc-topics htmlview clean dist check serve \ - autobuild-dev autobuild-stable + autobuild-dev autobuild-stable venv help: @echo "Please use \`make <target>' where <target> is one of" @echo " clean to remove build files" + @echo " venv to create a venv with necessary tools" @echo " html to make standalone HTML files" @echo " htmlview to open the index page built by the html target in your browser" @echo " htmlhelp to make HTML files and a HTML help project" @@ -95,14 +96,18 @@ doctest: pydoc-topics: BUILDER = pydoc-topics pydoc-topics: build - @echo "Building finished; now copy build/pydoc-topics/topics.py" \ - "to ../Lib/pydoc_data/topics.py" + @echo "Building finished; now run this:" \ + "cp build/pydoc-topics/topics.py ../Lib/pydoc_data/topics.py" htmlview: html $(PYTHON) -c "import webbrowser; webbrowser.open('build/html/index.html')" clean: - -rm -rf build/* + -rm -rf build/* venv/* + +venv: + $(PYTHON) -m venv venv + ./venv/bin/python3 -m pip install -U Sphinx dist: rm -rf dist diff --git a/Doc/README.txt b/Doc/README.txt index f985e6e3a8..4f8e9f8f14 100644 --- a/Doc/README.txt +++ b/Doc/README.txt @@ -3,7 +3,7 @@ Python Documentation README This directory contains the reStructuredText (reST) sources to the Python documentation. You don't need to build them yourself, prebuilt versions are -available at <https://docs.python.org/3.4/download.html>. +available at <https://docs.python.org/dev/download.html>. Documentation on authoring Python documentation, including information about both style and markup, is available in the "Documenting Python" chapter of the diff --git a/Doc/bugs.rst b/Doc/bugs.rst index f01ae0e390..1b0a5a9a93 100644 --- a/Doc/bugs.rst +++ b/Doc/bugs.rst @@ -1,13 +1,16 @@ .. _reporting-bugs: -************** -Reporting Bugs -************** +***************** +Dealing with Bugs +***************** Python is a mature programming language which has established a reputation for stability. In order to maintain this reputation, the developers would like to know of any deficiencies you find in Python. +It can be sometimes faster to fix bugs yourself and contribute patches to +Python as it streamlines the process and involves less people. Learn how to +:ref:`contribute <contributing-to-python>`. Documentation bugs ================== @@ -16,7 +19,8 @@ If you find a bug in this documentation or would like to propose an improvement, please submit a bug report on the :ref:`tracker <using-the-tracker>`. If you have a suggestion how to fix it, include that as well. -If you're short on time, you can also email your bug report to docs@python.org. +If you're short on time, you can also email documentation bug reports to +docs@python.org (behavioral bugs can be sent to python-list@python.org). 'docs@' is a mailing list run by volunteers; your request will be noticed, though it may take a while to be processed. @@ -72,6 +76,7 @@ taken on the bug. Information about writing a good bug report. Some of this is specific to the Mozilla project, but describes general good practices. +.. _contributing-to-python: Getting started contributing to Python yourself =============================================== diff --git a/Doc/c-api/arg.rst b/Doc/c-api/arg.rst index 8427bc4d13..6d493aaf3b 100644 --- a/Doc/c-api/arg.rst +++ b/Doc/c-api/arg.rst @@ -30,7 +30,7 @@ variable(s) whose address should be passed. Strings and buffers ------------------- -These formats allow to access an object as a contiguous chunk of memory. +These formats allow accessing an object as a contiguous chunk of memory. You don't have to provide raw storage for the returned unicode or bytes area. Also, you won't have to release any memory yourself, except with the ``es``, ``es#``, ``et`` and ``et#`` formats. @@ -59,8 +59,8 @@ Unless otherwise stated, buffers are not NUL-terminated. Convert a Unicode object to a C pointer to a character string. A pointer to an existing string is stored in the character pointer variable whose address you pass. The C string is NUL-terminated. - The Python string must not contain embedded NUL bytes; if it does, - a :exc:`TypeError` exception is raised. Unicode objects are converted + The Python string must not contain embedded null characters; if it does, + a :exc:`ValueError` exception is raised. Unicode objects are converted to C strings using ``'utf-8'`` encoding. If this conversion fails, a :exc:`UnicodeError` is raised. @@ -71,6 +71,10 @@ Unless otherwise stated, buffers are not NUL-terminated. preferable to use the ``O&`` format with :c:func:`PyUnicode_FSConverter` as *converter*. + .. versionchanged:: 3.5 + Previously, :exc:`TypeError` was raised when embedded null characters + were encountered in the Python string. + ``s*`` (:class:`str` or :term:`bytes-like object`) [Py_buffer] This format accepts Unicode objects as well as bytes-like objects. It fills a :c:type:`Py_buffer` structure provided by the caller. @@ -99,9 +103,13 @@ Unless otherwise stated, buffers are not NUL-terminated. ``y`` (read-only :term:`bytes-like object`) [const char \*] This format converts a bytes-like object to a C pointer to a character string; it does not accept Unicode objects. The bytes buffer must not - contain embedded NUL bytes; if it does, a :exc:`TypeError` + contain embedded null bytes; if it does, a :exc:`ValueError` exception is raised. + .. versionchanged:: 3.5 + Previously, :exc:`TypeError` was raised when embedded null bytes were + encountered in the bytes buffer. + ``y*`` (:term:`bytes-like object`) [Py_buffer] This variant on ``s*`` doesn't accept Unicode objects, only bytes-like objects. **This is the recommended way to accept @@ -127,14 +135,18 @@ Unless otherwise stated, buffers are not NUL-terminated. pointer variable, which will be filled with the pointer to an existing Unicode buffer. Please note that the width of a :c:type:`Py_UNICODE` character depends on compilation options (it is either 16 or 32 bits). - The Python string must not contain embedded NUL characters; if it does, - a :exc:`TypeError` exception is raised. + The Python string must not contain embedded null characters; if it does, + a :exc:`ValueError` exception is raised. .. note:: Since ``u`` doesn't give you back the length of the string, and it may contain embedded NUL characters, it is recommended to use ``u#`` or ``U`` instead. + .. versionchanged:: 3.5 + Previously, :exc:`TypeError` was raised when embedded null characters + were encountered in the Python string. + ``u#`` (:class:`str`) [Py_UNICODE \*, int] This variant on ``u`` stores into two C variables, the first one a pointer to a Unicode data buffer, the second one its length. @@ -152,7 +164,7 @@ Unless otherwise stated, buffers are not NUL-terminated. any conversion. Raises :exc:`TypeError` if the object is not a Unicode object. The C variable may also be declared as :c:type:`PyObject\*`. -``w*`` (:class:`bytearray` or read-write byte-oriented buffer) [Py_buffer] +``w*`` (read-write :term:`bytes-like object`) [Py_buffer] This format accepts any object which implements the read-write buffer interface. It fills a :c:type:`Py_buffer` structure provided by the caller. The buffer may contain embedded null bytes. The caller have to call @@ -206,7 +218,8 @@ Unless otherwise stated, buffers are not NUL-terminated. :c:func:`PyArg_ParseTuple` will use this location as the buffer and interpret the initial value of *\*buffer_length* as the buffer size. It will then copy the encoded data into the buffer and NUL-terminate it. If the buffer is not large - enough, a :exc:`ValueError` will be set. + enough, a :exc:`TypeError` will be set. + Note: starting from Python 3.6 a :exc:`ValueError` will be set. In both cases, *\*buffer_length* is set to the length of the encoded data without the trailing NUL byte. diff --git a/Doc/c-api/buffer.rst b/Doc/c-api/buffer.rst index 7361099123..45c94887ed 100644 --- a/Doc/c-api/buffer.rst +++ b/Doc/c-api/buffer.rst @@ -96,8 +96,8 @@ a buffer, see :c:func:`PyObject_GetBuffer`. block of the exporter. For example, with negative :c:member:`~Py_buffer.strides` the value may point to the end of the memory block. - For contiguous arrays, the value points to the beginning of the memory - block. + For :term:`contiguous` arrays, the value points to the beginning of + the memory block. .. c:member:: void \*obj @@ -281,11 +281,14 @@ of the flags below it. +-----------------------------+-------+---------+------------+ +.. index:: contiguous, C-contiguous, Fortran contiguous + contiguity requests ~~~~~~~~~~~~~~~~~~~ -C or Fortran contiguity can be explicitly requested, with and without stride -information. Without stride information, the buffer must be C-contiguous. +C or Fortran :term:`contiguity <contiguous>` can be explicitly requested, +with and without stride information. Without stride information, the buffer +must be C-contiguous. .. tabularcolumns:: |p{0.35\linewidth}|l|l|l|l| @@ -466,13 +469,13 @@ Buffer-related functions .. c:function:: int PyBuffer_IsContiguous(Py_buffer *view, char order) Return 1 if the memory defined by the *view* is C-style (*order* is - ``'C'``) or Fortran-style (*order* is ``'F'``) contiguous or either one + ``'C'``) or Fortran-style (*order* is ``'F'``) :term:`contiguous` or either one (*order* is ``'A'``). Return 0 otherwise. .. c:function:: void PyBuffer_FillContiguousStrides(int ndim, Py_ssize_t *shape, Py_ssize_t *strides, Py_ssize_t itemsize, char order) - Fill the *strides* array with byte-strides of a contiguous (C-style if + Fill the *strides* array with byte-strides of a :term:`contiguous` (C-style if *order* is ``'C'`` or Fortran-style if *order* is ``'F'``) array of the given shape with the given number of bytes per element. diff --git a/Doc/c-api/bytes.rst b/Doc/c-api/bytes.rst index 23b712881a..dcd7088629 100644 --- a/Doc/c-api/bytes.rst +++ b/Doc/c-api/bytes.rst @@ -158,7 +158,7 @@ called with a non-bytes parameter. If *length* is *NULL*, the bytes object may not contain embedded null bytes; - if it does, the function returns ``-1`` and a :exc:`TypeError` is raised. + if it does, the function returns ``-1`` and a :exc:`ValueError` is raised. The buffer refers to an internal buffer of *obj*, which includes an additional null byte at the end (not counted in *length*). The data @@ -167,6 +167,10 @@ called with a non-bytes parameter. *obj* is not a bytes object at all, :c:func:`PyBytes_AsStringAndSize` returns ``-1`` and raises :exc:`TypeError`. + .. versionchanged:: 3.5 + Previously, :exc:`TypeError` was raised when embedded null bytes were + encountered in the bytes object. + .. c:function:: void PyBytes_Concat(PyObject **bytes, PyObject *newpart) diff --git a/Doc/c-api/code.rst b/Doc/c-api/code.rst index 9c93563382..10d89f297c 100644 --- a/Doc/c-api/code.rst +++ b/Doc/c-api/code.rst @@ -2,15 +2,13 @@ .. _codeobjects: +.. index:: object; code, code object + Code Objects ------------ .. sectionauthor:: Jeffrey Yasskin <jyasskin@gmail.com> - -.. index:: - object: code - Code objects are a low-level detail of the CPython implementation. Each one represents a chunk of executable code that hasn't yet been bound into a function. diff --git a/Doc/c-api/codec.rst b/Doc/c-api/codec.rst index 83252afbb7..dfe3d436e5 100644 --- a/Doc/c-api/codec.rst +++ b/Doc/c-api/codec.rst @@ -116,3 +116,8 @@ Registry API for Unicode encoding error handlers Replace the unicode encode error with backslash escapes (``\x``, ``\u`` and ``\U``). +.. c:function:: PyObject* PyCodec_NameReplaceErrors(PyObject *exc) + + Replace the unicode encode error with ``\N{...}`` escapes. + + .. versionadded:: 3.5 diff --git a/Doc/c-api/concrete.rst b/Doc/c-api/concrete.rst index 2d5638670e..47dab81891 100644 --- a/Doc/c-api/concrete.rst +++ b/Doc/c-api/concrete.rst @@ -112,5 +112,6 @@ Other Objects weakref.rst capsule.rst gen.rst + coro.rst datetime.rst diff --git a/Doc/c-api/coro.rst b/Doc/c-api/coro.rst new file mode 100644 index 0000000000..2fe50b5d8c --- /dev/null +++ b/Doc/c-api/coro.rst @@ -0,0 +1,34 @@ +.. highlightlang:: c + +.. _coro-objects: + +Coroutine Objects +----------------- + +.. versionadded:: 3.5 + +Coroutine objects are what functions declared with an ``async`` keyword +return. + + +.. c:type:: PyCoroObject + + The C structure used for coroutine objects. + + +.. c:var:: PyTypeObject PyCoro_Type + + The type object corresponding to coroutine objects. + + +.. c:function:: int PyCoro_CheckExact(PyObject *ob) + + Return true if *ob*'s type is *PyCoro_Type*; *ob* must not be *NULL*. + + +.. c:function:: PyObject* PyCoro_New(PyFrameObject *frame, PyObject *name, PyObject *qualname) + + Create and return a new coroutine object based on the *frame* object, + with ``__name__`` and ``__qualname__`` set to *name* and *qualname*. + A reference to *frame* is stolen by this function. The *frame* argument + must not be *NULL*. diff --git a/Doc/c-api/dict.rst b/Doc/c-api/dict.rst index aeff640564..cfa5e13b9f 100644 --- a/Doc/c-api/dict.rst +++ b/Doc/c-api/dict.rst @@ -118,6 +118,7 @@ Dictionary Objects is returned. This function evaluates the hash function of *key* only once, instead of evaluating it independently for the lookup and the insertion. + .. versionadded:: 3.4 .. c:function:: PyObject* PyDict_Items(PyObject *p) diff --git a/Doc/c-api/exceptions.rst b/Doc/c-api/exceptions.rst index c2df767998..19cbb3bcb8 100644 --- a/Doc/c-api/exceptions.rst +++ b/Doc/c-api/exceptions.rst @@ -9,13 +9,19 @@ Exception Handling The functions described in this chapter will let you handle and raise Python exceptions. It is important to understand some of the basics of Python -exception handling. It works somewhat like the Unix :c:data:`errno` variable: +exception handling. It works somewhat like the POSIX :c:data:`errno` variable: there is a global indicator (per thread) of the last error that occurred. Most -functions don't clear this on success, but will set it to indicate the cause of -the error on failure. Most functions also return an error indicator, usually -*NULL* if they are supposed to return a pointer, or ``-1`` if they return an -integer (exception: the :c:func:`PyArg_\*` functions return ``1`` for success and -``0`` for failure). +C API functions don't clear this on success, but will set it to indicate the +cause of the error on failure. Most C API functions also return an error +indicator, usually *NULL* if they are supposed to return a pointer, or ``-1`` +if they return an integer (exception: the :c:func:`PyArg_\*` functions +return ``1`` for success and ``0`` for failure). + +Concretely, the error indicator consists of three object pointers: the +exception's type, the exception's value, and the traceback object. Any +of those pointers can be NULL if non-set (although some combinations are +forbidden, for example you can't have a non-NULL traceback if the exception +type is NULL). When a function must fail because some function it called failed, it generally doesn't set the error indicator; the function it called already set it. It is @@ -27,12 +33,21 @@ the caller that an error has been set. If the error is not handled or carefully propagated, additional calls into the Python/C API may not behave as intended and may fail in mysterious ways. -The error indicator consists of three Python objects corresponding to the result -of ``sys.exc_info()``. API functions exist to interact with the error indicator -in various ways. There is a separate error indicator for each thread. +.. note:: + The error indicator is **not** the result of :func:`sys.exc_info()`. + The former corresponds to an exception that is not yet caught (and is + therefore still propagating), while the latter returns an exception after + it is caught (and has therefore stopped propagating). -.. XXX Order of these should be more thoughtful. - Either alphabetical or some kind of structure. + +Printing and clearing +===================== + + +.. c:function:: void PyErr_Clear() + + Clear the error indicator. If the error indicator is not set, there is no + effect. .. c:function:: void PyErr_PrintEx(int set_sys_last_vars) @@ -51,127 +66,24 @@ in various ways. There is a separate error indicator for each thread. Alias for ``PyErr_PrintEx(1)``. -.. c:function:: PyObject* PyErr_Occurred() - - Test whether the error indicator is set. If set, return the exception *type* - (the first argument to the last call to one of the :c:func:`PyErr_Set\*` - functions or to :c:func:`PyErr_Restore`). If not set, return *NULL*. You do not - own a reference to the return value, so you do not need to :c:func:`Py_DECREF` - it. - - .. note:: - - Do not compare the return value to a specific exception; use - :c:func:`PyErr_ExceptionMatches` instead, shown below. (The comparison could - easily fail since the exception may be an instance instead of a class, in the - case of a class exception, or it may be a subclass of the expected exception.) - - -.. c:function:: int PyErr_ExceptionMatches(PyObject *exc) - - Equivalent to ``PyErr_GivenExceptionMatches(PyErr_Occurred(), exc)``. This - should only be called when an exception is actually set; a memory access - violation will occur if no exception has been raised. - - -.. c:function:: int PyErr_GivenExceptionMatches(PyObject *given, PyObject *exc) - - Return true if the *given* exception matches the exception in *exc*. If - *exc* is a class object, this also returns true when *given* is an instance - of a subclass. If *exc* is a tuple, all exceptions in the tuple (and - recursively in subtuples) are searched for a match. - - -.. c:function:: void PyErr_NormalizeException(PyObject**exc, PyObject**val, PyObject**tb) - - Under certain circumstances, the values returned by :c:func:`PyErr_Fetch` below - can be "unnormalized", meaning that ``*exc`` is a class object but ``*val`` is - not an instance of the same class. This function can be used to instantiate - the class in that case. If the values are already normalized, nothing happens. - The delayed normalization is implemented to improve performance. - - .. note:: - - This function *does not* implicitly set the ``__traceback__`` - attribute on the exception value. If setting the traceback - appropriately is desired, the following additional snippet is needed:: - - if (tb != NULL) { - PyException_SetTraceback(val, tb); - } - - -.. c:function:: void PyErr_Clear() - - Clear the error indicator. If the error indicator is not set, there is no - effect. - - -.. c:function:: void PyErr_Fetch(PyObject **ptype, PyObject **pvalue, PyObject **ptraceback) - - Retrieve the error indicator into three variables whose addresses are passed. - If the error indicator is not set, set all three variables to *NULL*. If it is - set, it will be cleared and you own a reference to each object retrieved. The - value and traceback object may be *NULL* even when the type object is not. - - .. note:: - - This function is normally only used by code that needs to handle exceptions or - by code that needs to save and restore the error indicator temporarily. - - -.. c:function:: void PyErr_Restore(PyObject *type, PyObject *value, PyObject *traceback) - - Set the error indicator from the three objects. If the error indicator is - already set, it is cleared first. If the objects are *NULL*, the error - indicator is cleared. Do not pass a *NULL* type and non-*NULL* value or - traceback. The exception type should be a class. Do not pass an invalid - exception type or value. (Violating these rules will cause subtle problems - later.) This call takes away a reference to each object: you must own a - reference to each object before the call and after the call you no longer own - these references. (If you don't understand this, don't use this function. I - warned you.) - - .. note:: - - This function is normally only used by code that needs to save and restore the - error indicator temporarily; use :c:func:`PyErr_Fetch` to save the current - exception state. - - -.. c:function:: void PyErr_GetExcInfo(PyObject **ptype, PyObject **pvalue, PyObject **ptraceback) - - Retrieve the exception info, as known from ``sys.exc_info()``. This refers - to an exception that was already caught, not to an exception that was - freshly raised. Returns new references for the three objects, any of which - may be *NULL*. Does not modify the exception info state. - - .. note:: - - This function is not normally used by code that wants to handle exceptions. - Rather, it can be used when code needs to save and restore the exception - state temporarily. Use :c:func:`PyErr_SetExcInfo` to restore or clear the - exception state. - - .. versionadded:: 3.3 - +.. c:function:: void PyErr_WriteUnraisable(PyObject *obj) -.. c:function:: void PyErr_SetExcInfo(PyObject *type, PyObject *value, PyObject *traceback) + This utility function prints a warning message to ``sys.stderr`` when an + exception has been set but it is impossible for the interpreter to actually + raise the exception. It is used, for example, when an exception occurs in an + :meth:`__del__` method. - Set the exception info, as known from ``sys.exc_info()``. This refers - to an exception that was already caught, not to an exception that was - freshly raised. This function steals the references of the arguments. - To clear the exception state, pass *NULL* for all three arguments. - For general rules about the three arguments, see :c:func:`PyErr_Restore`. + The function is called with a single argument *obj* that identifies the context + in which the unraisable exception occurred. If possible, + the repr of *obj* will be printed in the warning message. - .. note:: - This function is not normally used by code that wants to handle exceptions. - Rather, it can be used when code needs to save and restore the exception - state temporarily. Use :c:func:`PyErr_GetExcInfo` to read the exception - state. +Raising exceptions +================== - .. versionadded:: 3.3 +These functions help you set the current thread's error indicator. +For convenience, some of these functions will always return a +NULL pointer for use in a ``return`` statement. .. c:function:: void PyErr_SetString(PyObject *type, const char *message) @@ -197,6 +109,14 @@ in various ways. There is a separate error indicator for each thread. string. +.. c:function:: PyObject* PyErr_FormatV(PyObject *exception, const char *format, va_list vargs) + + Same as :c:func:`PyErr_Format`, but taking a :c:type:`va_list` argument rather + than a variable number of arguments. + + .. versionadded:: 3.5 + + .. c:function:: void PyErr_SetNone(PyObject *type) This is a shorthand for ``PyErr_SetObject(type, Py_None)``. @@ -346,27 +266,31 @@ in various ways. There is a separate error indicator for each thread. use. +Issuing warnings +================ + +Use these functions to issue warnings from C code. They mirror similar +functions exported by the Python :mod:`warnings` module. They normally +print a warning message to *sys.stderr*; however, it is +also possible that the user has specified that warnings are to be turned into +errors, and in that case they will raise an exception. It is also possible that +the functions raise an exception because of a problem with the warning machinery. +The return value is ``0`` if no exception is raised, or ``-1`` if an exception +is raised. (It is not possible to determine whether a warning message is +actually printed, nor what the reason is for the exception; this is +intentional.) If an exception is raised, the caller should do its normal +exception handling (for example, :c:func:`Py_DECREF` owned references and return +an error value). + .. c:function:: int PyErr_WarnEx(PyObject *category, const char *message, Py_ssize_t stack_level) Issue a warning message. The *category* argument is a warning category (see - below) or *NULL*; the *message* argument is an UTF-8 encoded string. *stack_level* is a + below) or *NULL*; the *message* argument is a UTF-8 encoded string. *stack_level* is a positive number giving a number of stack frames; the warning will be issued from the currently executing line of code in that stack frame. A *stack_level* of 1 is the function calling :c:func:`PyErr_WarnEx`, 2 is the function above that, and so forth. - This function normally prints a warning message to *sys.stderr*; however, it is - also possible that the user has specified that warnings are to be turned into - errors, and in that case this will raise an exception. It is also possible that - the function raises an exception because of a problem with the warning machinery - (the implementation imports the :mod:`warnings` module to do the heavy lifting). - The return value is ``0`` if no exception is raised, or ``-1`` if an exception - is raised. (It is not possible to determine whether a warning message is - actually printed, nor what the reason is for the exception; this is - intentional.) If an exception is raised, the caller should do its normal - exception handling (for example, :c:func:`Py_DECREF` owned references and return - an error value). - Warning categories must be subclasses of :c:data:`Warning`; the default warning category is :c:data:`RuntimeWarning`. The standard Python warning categories are available as global variables whose names are ``PyExc_`` followed by the Python @@ -410,6 +334,139 @@ in various ways. There is a separate error indicator for each thread. .. versionadded:: 3.2 +Querying the error indicator +============================ + +.. c:function:: PyObject* PyErr_Occurred() + + Test whether the error indicator is set. If set, return the exception *type* + (the first argument to the last call to one of the :c:func:`PyErr_Set\*` + functions or to :c:func:`PyErr_Restore`). If not set, return *NULL*. You do not + own a reference to the return value, so you do not need to :c:func:`Py_DECREF` + it. + + .. note:: + + Do not compare the return value to a specific exception; use + :c:func:`PyErr_ExceptionMatches` instead, shown below. (The comparison could + easily fail since the exception may be an instance instead of a class, in the + case of a class exception, or it may be a subclass of the expected exception.) + + +.. c:function:: int PyErr_ExceptionMatches(PyObject *exc) + + Equivalent to ``PyErr_GivenExceptionMatches(PyErr_Occurred(), exc)``. This + should only be called when an exception is actually set; a memory access + violation will occur if no exception has been raised. + + +.. c:function:: int PyErr_GivenExceptionMatches(PyObject *given, PyObject *exc) + + Return true if the *given* exception matches the exception type in *exc*. If + *exc* is a class object, this also returns true when *given* is an instance + of a subclass. If *exc* is a tuple, all exception types in the tuple (and + recursively in subtuples) are searched for a match. + + +.. c:function:: void PyErr_Fetch(PyObject **ptype, PyObject **pvalue, PyObject **ptraceback) + + Retrieve the error indicator into three variables whose addresses are passed. + If the error indicator is not set, set all three variables to *NULL*. If it is + set, it will be cleared and you own a reference to each object retrieved. The + value and traceback object may be *NULL* even when the type object is not. + + .. note:: + + This function is normally only used by code that needs to catch exceptions or + by code that needs to save and restore the error indicator temporarily, e.g.:: + + { + PyObject **type, **value, **traceback; + PyErr_Fetch(&type, &value, &traceback); + + /* ... code that might produce other errors ... */ + + PyErr_Restore(type, value, traceback); + } + + +.. c:function:: void PyErr_Restore(PyObject *type, PyObject *value, PyObject *traceback) + + Set the error indicator from the three objects. If the error indicator is + already set, it is cleared first. If the objects are *NULL*, the error + indicator is cleared. Do not pass a *NULL* type and non-*NULL* value or + traceback. The exception type should be a class. Do not pass an invalid + exception type or value. (Violating these rules will cause subtle problems + later.) This call takes away a reference to each object: you must own a + reference to each object before the call and after the call you no longer own + these references. (If you don't understand this, don't use this function. I + warned you.) + + .. note:: + + This function is normally only used by code that needs to save and restore the + error indicator temporarily. Use :c:func:`PyErr_Fetch` to save the current + error indicator. + + +.. c:function:: void PyErr_NormalizeException(PyObject**exc, PyObject**val, PyObject**tb) + + Under certain circumstances, the values returned by :c:func:`PyErr_Fetch` below + can be "unnormalized", meaning that ``*exc`` is a class object but ``*val`` is + not an instance of the same class. This function can be used to instantiate + the class in that case. If the values are already normalized, nothing happens. + The delayed normalization is implemented to improve performance. + + .. note:: + + This function *does not* implicitly set the ``__traceback__`` + attribute on the exception value. If setting the traceback + appropriately is desired, the following additional snippet is needed:: + + if (tb != NULL) { + PyException_SetTraceback(val, tb); + } + + +.. c:function:: void PyErr_GetExcInfo(PyObject **ptype, PyObject **pvalue, PyObject **ptraceback) + + Retrieve the exception info, as known from ``sys.exc_info()``. This refers + to an exception that was *already caught*, not to an exception that was + freshly raised. Returns new references for the three objects, any of which + may be *NULL*. Does not modify the exception info state. + + .. note:: + + This function is not normally used by code that wants to handle exceptions. + Rather, it can be used when code needs to save and restore the exception + state temporarily. Use :c:func:`PyErr_SetExcInfo` to restore or clear the + exception state. + + .. versionadded:: 3.3 + + +.. c:function:: void PyErr_SetExcInfo(PyObject *type, PyObject *value, PyObject *traceback) + + Set the exception info, as known from ``sys.exc_info()``. This refers + to an exception that was *already caught*, not to an exception that was + freshly raised. This function steals the references of the arguments. + To clear the exception state, pass *NULL* for all three arguments. + For general rules about the three arguments, see :c:func:`PyErr_Restore`. + + .. note:: + + This function is not normally used by code that wants to handle exceptions. + Rather, it can be used when code needs to save and restore the exception + state temporarily. Use :c:func:`PyErr_GetExcInfo` to read the exception + state. + + .. versionadded:: 3.3 + + +Signal Handling +=============== + + .. c:function:: int PyErr_CheckSignals() .. index:: @@ -443,13 +500,21 @@ in various ways. There is a separate error indicator for each thread. .. c:function:: int PySignal_SetWakeupFd(int fd) - This utility function specifies a file descriptor to which a ``'\0'`` byte will - be written whenever a signal is received. It returns the previous such file - descriptor. The value ``-1`` disables the feature; this is the initial state. + This utility function specifies a file descriptor to which the signal number + is written as a single byte whenever a signal is received. *fd* must be + non-blocking. It returns the previous such file descriptor. + + The value ``-1`` disables the feature; this is the initial state. This is equivalent to :func:`signal.set_wakeup_fd` in Python, but without any error checking. *fd* should be a valid file descriptor. The function should only be called from the main thread. + .. versionchanged:: 3.5 + On Windows, the function now also supports socket handles. + + +Exception Classes +================= .. c:function:: PyObject* PyErr_NewException(const char *name, PyObject *base, PyObject *dict) @@ -475,18 +540,6 @@ in various ways. There is a separate error indicator for each thread. .. versionadded:: 3.2 -.. c:function:: void PyErr_WriteUnraisable(PyObject *obj) - - This utility function prints a warning message to ``sys.stderr`` when an - exception has been set but it is impossible for the interpreter to actually - raise the exception. It is used, for example, when an exception occurs in an - :meth:`__del__` method. - - The function is called with a single argument *obj* that identifies the context - in which the unraisable exception occurred. The repr of *obj* will be printed in - the warning message. - - Exception Objects ================= @@ -556,7 +609,7 @@ The following functions are used to create and modify Unicode exceptions from C. .. c:function:: PyObject* PyUnicodeTranslateError_Create(const Py_UNICODE *object, Py_ssize_t length, Py_ssize_t start, Py_ssize_t end, const char *reason) Create a :class:`UnicodeTranslateError` object with the attributes *object*, - *length*, *start*, *end* and *reason*. *reason* is an UTF-8 encoded string. + *length*, *start*, *end* and *reason*. *reason* is a UTF-8 encoded string. .. c:function:: PyObject* PyUnicodeDecodeError_GetEncoding(PyObject *exc) PyObject* PyUnicodeEncodeError_GetEncoding(PyObject *exc) @@ -630,12 +683,12 @@ recursion depth automatically). sets a :exc:`MemoryError` and returns a nonzero value. The function then checks if the recursion limit is reached. If this is the - case, a :exc:`RuntimeError` is set and a nonzero value is returned. + case, a :exc:`RecursionError` is set and a nonzero value is returned. Otherwise, zero is returned. *where* should be a string such as ``" in instance check"`` to be - concatenated to the :exc:`RuntimeError` message caused by the recursion depth - limit. + concatenated to the :exc:`RecursionError` message caused by the recursion + depth limit. .. c:function:: void Py_LeaveRecursiveCall() @@ -747,6 +800,8 @@ the variables: +-----------------------------------------+---------------------------------+----------+ | :c:data:`PyExc_ProcessLookupError` | :exc:`ProcessLookupError` | | +-----------------------------------------+---------------------------------+----------+ +| :c:data:`PyExc_RecursionError` | :exc:`RecursionError` | | ++-----------------------------------------+---------------------------------+----------+ | :c:data:`PyExc_ReferenceError` | :exc:`ReferenceError` | \(2) | +-----------------------------------------+---------------------------------+----------+ | :c:data:`PyExc_RuntimeError` | :exc:`RuntimeError` | | @@ -776,6 +831,9 @@ the variables: :c:data:`PyExc_PermissionError`, :c:data:`PyExc_ProcessLookupError` and :c:data:`PyExc_TimeoutError` were introduced following :pep:`3151`. +.. versionadded:: 3.5 + :c:data:`PyExc_RecursionError`. + These are compatibility aliases to :c:data:`PyExc_OSError`: @@ -824,6 +882,7 @@ These are compatibility aliases to :c:data:`PyExc_OSError`: single: PyExc_OverflowError single: PyExc_PermissionError single: PyExc_ProcessLookupError + single: PyExc_RecursionError single: PyExc_ReferenceError single: PyExc_RuntimeError single: PyExc_SyntaxError diff --git a/Doc/c-api/function.rst b/Doc/c-api/function.rst index ad9832233f..17279c732a 100644 --- a/Doc/c-api/function.rst +++ b/Doc/c-api/function.rst @@ -34,13 +34,14 @@ There are a few functions specific to Python functions. Return a new function object associated with the code object *code*. *globals* must be a dictionary with the global variables accessible to the function. - The function's docstring, name and *__module__* are retrieved from the code - object, the argument defaults and closure are set to *NULL*. + The function's docstring and name are retrieved from the code object. *__module__* + is retrieved from *globals*. The argument defaults, annotations and closure are + set to *NULL*. *__qualname__* is set to the same value as the function's name. .. c:function:: PyObject* PyFunction_NewWithQualName(PyObject *code, PyObject *globals, PyObject *qualname) - As :c:func:`PyFunction_New`, but also allows to set the function object's + As :c:func:`PyFunction_New`, but also allows setting the function object's ``__qualname__`` attribute. *qualname* should be a unicode object or NULL; if NULL, the ``__qualname__`` attribute is set to the same value as its ``__name__`` attribute. diff --git a/Doc/c-api/gcsupport.rst b/Doc/c-api/gcsupport.rst index 9f6ad85226..f5e0d7ec9c 100644 --- a/Doc/c-api/gcsupport.rst +++ b/Doc/c-api/gcsupport.rst @@ -126,9 +126,10 @@ must name its arguments exactly *visit* and *arg*: .. c:function:: void Py_VISIT(PyObject *o) - Call the *visit* callback, with arguments *o* and *arg*. If *visit* returns - a non-zero value, then return it. Using this macro, :c:member:`~PyTypeObject.tp_traverse` - handlers look like:: + If *o* is not *NULL*, call the *visit* callback, with arguments *o* + and *arg*. If *visit* returns a non-zero value, then return it. + Using this macro, :c:member:`~PyTypeObject.tp_traverse` handlers + look like:: static int my_traverse(Noddy *self, visitproc visit, void *arg) diff --git a/Doc/c-api/gen.rst b/Doc/c-api/gen.rst index 0c851a75ef..1efbae4fcb 100644 --- a/Doc/c-api/gen.rst +++ b/Doc/c-api/gen.rst @@ -7,7 +7,7 @@ Generator Objects Generator objects are what Python uses to implement generator iterators. They are normally created by iterating over a function that yields values, rather -than explicitly calling :c:func:`PyGen_New`. +than explicitly calling :c:func:`PyGen_New` or :c:func:`PyGen_NewWithQualName`. .. c:type:: PyGenObject @@ -20,19 +20,25 @@ than explicitly calling :c:func:`PyGen_New`. The type object corresponding to generator objects. -.. c:function:: int PyGen_Check(ob) +.. c:function:: int PyGen_Check(PyObject *ob) Return true if *ob* is a generator object; *ob* must not be *NULL*. -.. c:function:: int PyGen_CheckExact(ob) +.. c:function:: int PyGen_CheckExact(PyObject *ob) - Return true if *ob*'s type is *PyGen_Type* is a generator object; *ob* must not - be *NULL*. + Return true if *ob*'s type is *PyGen_Type*; *ob* must not be *NULL*. .. c:function:: PyObject* PyGen_New(PyFrameObject *frame) - Create and return a new generator object based on the *frame* object. A - reference to *frame* is stolen by this function. The parameter must not be + Create and return a new generator object based on the *frame* object. + A reference to *frame* is stolen by this function. The argument must not be *NULL*. + +.. c:function:: PyObject* PyGen_NewWithQualName(PyFrameObject *frame, PyObject *name, PyObject *qualname) + + Create and return a new generator object based on the *frame* object, + with ``__name__`` and ``__qualname__`` set to *name* and *qualname*. + A reference to *frame* is stolen by this function. The *frame* argument + must not be *NULL*. diff --git a/Doc/c-api/import.rst b/Doc/c-api/import.rst index 60865f456f..2936f4ff33 100644 --- a/Doc/c-api/import.rst +++ b/Doc/c-api/import.rst @@ -72,7 +72,7 @@ Importing Modules .. c:function:: PyObject* PyImport_ImportModuleLevel(const char *name, PyObject *globals, PyObject *locals, PyObject *fromlist, int level) - Similar to :c:func:`PyImport_ImportModuleLevelObject`, but the name is an + Similar to :c:func:`PyImport_ImportModuleLevelObject`, but the name is a UTF-8 encoded string instead of a Unicode object. .. versionchanged:: 3.3 @@ -183,9 +183,9 @@ Importing Modules .. c:function:: long PyImport_GetMagicNumber() - Return the magic number for Python bytecode files (a.k.a. :file:`.pyc` and - :file:`.pyo` files). The magic number should be present in the first four bytes - of the bytecode file, in little-endian byte order. Returns -1 on error. + Return the magic number for Python bytecode files (a.k.a. :file:`.pyc` file). + The magic number should be present in the first four bytes of the bytecode + file, in little-endian byte order. Returns -1 on error. .. versionchanged:: 3.3 Return value of -1 upon failure. @@ -236,11 +236,6 @@ Importing Modules For internal use only. -.. c:function:: PyObject* _PyImport_FixupExtension(char *, char *) - - For internal use only. - - .. c:function:: int PyImport_ImportFrozenModuleObject(PyObject *name) Load a frozen module named *name*. Return ``1`` for success, ``0`` if the @@ -277,7 +272,7 @@ Importing Modules }; -.. c:var:: struct _frozen* PyImport_FrozenModules +.. c:var:: const struct _frozen* PyImport_FrozenModules This pointer is initialized to point to an array of :c:type:`struct _frozen` records, terminated by one whose members are all *NULL* or zero. When a frozen diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst index 4bb50649ef..639819ce29 100644 --- a/Doc/c-api/init.rst +++ b/Doc/c-api/init.rst @@ -134,6 +134,9 @@ Process-wide parameters change for the duration of the program's execution. No code in the Python interpreter will change the contents of this storage. + Use :c:func:`Py_DecodeLocale` to decode a bytes string to get a + :c:type:`wchar_*` string. + .. c:function:: wchar* Py_GetProgramName() @@ -245,6 +248,9 @@ Process-wide parameters :data:`sys.exec_prefix` to be empty. It is up to the caller to modify these if required after calling :c:func:`Py_Initialize`. + Use :c:func:`Py_DecodeLocale` to decode a bytes string to get a + :c:type:`wchar_*` string. + The path argument is copied internally, so the caller may free it after the call completes. @@ -344,11 +350,14 @@ Process-wide parameters :data:`sys.path`, which is the same as prepending the current working directory (``"."``). + Use :c:func:`Py_DecodeLocale` to decode a bytes string to get a + :c:type:`wchar_*` string. + .. note:: It is recommended that applications embedding the Python interpreter for purposes other than executing a single script pass 0 as *updatepath*, and update :data:`sys.path` themselves if desired. - See `CVE-2008-5983 <http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2008-5983>`_. + See `CVE-2008-5983 <https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2008-5983>`_. On versions before 3.1.3, you can achieve the same effect by manually popping the first :data:`sys.path` element after having called @@ -368,6 +377,9 @@ Process-wide parameters to 1 unless the :program:`python` interpreter was started with the :option:`-I`. + Use :c:func:`Py_DecodeLocale` to decode a bytes string to get a + :c:type:`wchar_*` string. + .. versionchanged:: 3.4 The *updatepath* value depends on :option:`-I`. @@ -382,6 +394,9 @@ Process-wide parameters execution. No code in the Python interpreter will change the contents of this storage. + Use :c:func:`Py_DecodeLocale` to decode a bytes string to get a + :c:type:`wchar_*` string. + .. c:function:: w_char* Py_GetPythonHome() @@ -858,6 +873,8 @@ been created. instead. +.. _sub-interpreter-support: + Sub-interpreter support ======================= diff --git a/Doc/c-api/mapping.rst b/Doc/c-api/mapping.rst index e34104708c..fe601b605b 100644 --- a/Doc/c-api/mapping.rst +++ b/Doc/c-api/mapping.rst @@ -50,21 +50,21 @@ Mapping Protocol .. c:function:: PyObject* PyMapping_Keys(PyObject *o) - On success, return a list of the keys in object *o*. On failure, return *NULL*. - This is equivalent to the Python expression ``list(o.keys())``. + On success, return a list, a tuple or a dictionary view in case of a dict, + of the keys in object *o*. On failure, return *NULL*. .. c:function:: PyObject* PyMapping_Values(PyObject *o) - On success, return a list of the values in object *o*. On failure, return - *NULL*. This is equivalent to the Python expression ``list(o.values())``. + On success, return a list, a tuple or a dictionary view in case of a dict, of + the values in object *o*. On failure, return *NULL*. .. c:function:: PyObject* PyMapping_Items(PyObject *o) - On success, return a list of the items in object *o*, where each item is a tuple - containing a key-value pair. On failure, return *NULL*. This is equivalent to - the Python expression ``list(o.items())``. + On success, return a list, a tuple or a dictionary view in case of a dict, of + the items in object *o*, where each item is a tuple containing a key-value + pair. On failure, return *NULL*. .. c:function:: PyObject* PyMapping_GetItemString(PyObject *o, const char *key) diff --git a/Doc/c-api/memory.rst b/Doc/c-api/memory.rst index 7908622d2b..290ef09dce 100644 --- a/Doc/c-api/memory.rst +++ b/Doc/c-api/memory.rst @@ -83,6 +83,12 @@ collection, memory compaction or other preventive procedures. Note that by using the C library allocator as shown in the previous example, the allocated memory for the I/O buffer escapes completely the Python memory manager. +.. seealso:: + + The :envvar:`PYTHONMALLOCSTATS` environment variable can be used to print + memory allocation statistics every time a new object arena is created, and + on shutdown. + Raw Memory Interface ==================== @@ -92,38 +98,59 @@ functions are thread-safe, the :term:`GIL <global interpreter lock>` does not need to be held. The default raw memory block allocator uses the following functions: -:c:func:`malloc`, :c:func:`realloc` and :c:func:`free`; call ``malloc(1)`` when -requesting zero bytes. +:c:func:`malloc`, :c:func:`calloc`, :c:func:`realloc` and :c:func:`free`; call +``malloc(1)`` (or ``calloc(1, 1)``) when requesting zero bytes. .. versionadded:: 3.4 .. c:function:: void* PyMem_RawMalloc(size_t n) Allocates *n* bytes and returns a pointer of type :c:type:`void\*` to the - allocated memory, or *NULL* if the request fails. Requesting zero bytes - returns a distinct non-*NULL* pointer if possible, as if - ``PyMem_RawMalloc(1)`` had been called instead. The memory will not have + allocated memory, or *NULL* if the request fails. + + Requesting zero bytes returns a distinct non-*NULL* pointer if possible, as + if ``PyMem_RawMalloc(1)`` had been called instead. The memory will not have been initialized in any way. +.. c:function:: void* PyMem_RawCalloc(size_t nelem, size_t elsize) + + Allocates *nelem* elements each whose size in bytes is *elsize* and returns + a pointer of type :c:type:`void\*` to the allocated memory, or *NULL* if the + request fails. The memory is initialized to zeros. + + Requesting zero elements or elements of size zero bytes returns a distinct + non-*NULL* pointer if possible, as if ``PyMem_RawCalloc(1, 1)`` had been + called instead. + + .. versionadded:: 3.5 + + .. c:function:: void* PyMem_RawRealloc(void *p, size_t n) Resizes the memory block pointed to by *p* to *n* bytes. The contents will - be unchanged to the minimum of the old and the new sizes. If *p* is *NULL*, - the call is equivalent to ``PyMem_RawMalloc(n)``; else if *n* is equal to - zero, the memory block is resized but is not freed, and the returned pointer - is non-*NULL*. Unless *p* is *NULL*, it must have been returned by a - previous call to :c:func:`PyMem_RawMalloc` or :c:func:`PyMem_RawRealloc`. If - the request fails, :c:func:`PyMem_RawRealloc` returns *NULL* and *p* remains - a valid pointer to the previous memory area. + be unchanged to the minimum of the old and the new sizes. + + If *p* is *NULL*, the call is equivalent to ``PyMem_RawMalloc(n)``; else if + *n* is equal to zero, the memory block is resized but is not freed, and the + returned pointer is non-*NULL*. + + Unless *p* is *NULL*, it must have been returned by a previous call to + :c:func:`PyMem_RawMalloc`, :c:func:`PyMem_RawRealloc` or + :c:func:`PyMem_RawCalloc`. + + If the request fails, :c:func:`PyMem_RawRealloc` returns *NULL* and *p* + remains a valid pointer to the previous memory area. .. c:function:: void PyMem_RawFree(void *p) Frees the memory block pointed to by *p*, which must have been returned by a - previous call to :c:func:`PyMem_RawMalloc` or :c:func:`PyMem_RawRealloc`. - Otherwise, or if ``PyMem_Free(p)`` has been called before, undefined - behavior occurs. If *p* is *NULL*, no operation is performed. + previous call to :c:func:`PyMem_RawMalloc`, :c:func:`PyMem_RawRealloc` or + :c:func:`PyMem_RawCalloc`. Otherwise, or if ``PyMem_Free(p)`` has been + called before, undefined behavior occurs. + + If *p* is *NULL*, no operation is performed. .. _memoryinterface: @@ -136,8 +163,8 @@ behavior when requesting zero bytes, are available for allocating and releasing memory from the Python heap. The default memory block allocator uses the following functions: -:c:func:`malloc`, :c:func:`realloc` and :c:func:`free`; call ``malloc(1)`` when -requesting zero bytes. +:c:func:`malloc`, :c:func:`calloc`, :c:func:`realloc` and :c:func:`free`; call +``malloc(1)`` (or ``calloc(1, 1)``) when requesting zero bytes. .. warning:: @@ -147,29 +174,50 @@ requesting zero bytes. .. c:function:: void* PyMem_Malloc(size_t n) Allocates *n* bytes and returns a pointer of type :c:type:`void\*` to the - allocated memory, or *NULL* if the request fails. Requesting zero bytes returns - a distinct non-*NULL* pointer if possible, as if ``PyMem_Malloc(1)`` had - been called instead. The memory will not have been initialized in any way. + allocated memory, or *NULL* if the request fails. + + Requesting zero bytes returns a distinct non-*NULL* pointer if possible, as + if ``PyMem_Malloc(1)`` had been called instead. The memory will not have + been initialized in any way. + + +.. c:function:: void* PyMem_Calloc(size_t nelem, size_t elsize) + + Allocates *nelem* elements each whose size in bytes is *elsize* and returns + a pointer of type :c:type:`void\*` to the allocated memory, or *NULL* if the + request fails. The memory is initialized to zeros. + + Requesting zero elements or elements of size zero bytes returns a distinct + non-*NULL* pointer if possible, as if ``PyMem_Calloc(1, 1)`` had been called + instead. + + .. versionadded:: 3.5 .. c:function:: void* PyMem_Realloc(void *p, size_t n) Resizes the memory block pointed to by *p* to *n* bytes. The contents will be - unchanged to the minimum of the old and the new sizes. If *p* is *NULL*, the - call is equivalent to ``PyMem_Malloc(n)``; else if *n* is equal to zero, - the memory block is resized but is not freed, and the returned pointer is - non-*NULL*. Unless *p* is *NULL*, it must have been returned by a previous call - to :c:func:`PyMem_Malloc` or :c:func:`PyMem_Realloc`. If the request fails, - :c:func:`PyMem_Realloc` returns *NULL* and *p* remains a valid pointer to the - previous memory area. + unchanged to the minimum of the old and the new sizes. + + If *p* is *NULL*, the call is equivalent to ``PyMem_Malloc(n)``; else if *n* + is equal to zero, the memory block is resized but is not freed, and the + returned pointer is non-*NULL*. + + Unless *p* is *NULL*, it must have been returned by a previous call to + :c:func:`PyMem_Malloc`, :c:func:`PyMem_Realloc` or :c:func:`PyMem_Calloc`. + + If the request fails, :c:func:`PyMem_Realloc` returns *NULL* and *p* remains + a valid pointer to the previous memory area. .. c:function:: void PyMem_Free(void *p) Frees the memory block pointed to by *p*, which must have been returned by a - previous call to :c:func:`PyMem_Malloc` or :c:func:`PyMem_Realloc`. Otherwise, or - if ``PyMem_Free(p)`` has been called before, undefined behavior occurs. If - *p* is *NULL*, no operation is performed. + previous call to :c:func:`PyMem_Malloc`, :c:func:`PyMem_Realloc` or + :c:func:`PyMem_Calloc`. Otherwise, or if ``PyMem_Free(p)`` has been called + before, undefined behavior occurs. + + If *p* is *NULL*, no operation is performed. The following type-oriented macros are provided for convenience. Note that *TYPE* refers to any C type. @@ -187,8 +235,10 @@ The following type-oriented macros are provided for convenience. Note that Same as :c:func:`PyMem_Realloc`, but the memory block is resized to ``(n * sizeof(TYPE))`` bytes. Returns a pointer cast to :c:type:`TYPE\*`. On return, *p* will be a pointer to the new memory area, or *NULL* in the event of - failure. This is a C preprocessor macro; p is always reassigned. Save - the original value of p to avoid losing memory when handling errors. + failure. + + This is a C preprocessor macro; *p* is always reassigned. Save the original + value of *p* to avoid losing memory when handling errors. .. c:function:: void PyMem_Del(void *p) @@ -200,9 +250,12 @@ allocator directly, without involving the C API functions listed above. However, note that their use does not preserve binary compatibility across Python versions and is therefore deprecated in extension modules. -:c:func:`PyMem_MALLOC`, :c:func:`PyMem_REALLOC`, :c:func:`PyMem_FREE`. - -:c:func:`PyMem_NEW`, :c:func:`PyMem_RESIZE`, :c:func:`PyMem_DEL`. +* ``PyMem_MALLOC(size)`` +* ``PyMem_NEW(type, size)`` +* ``PyMem_REALLOC(ptr, size)`` +* ``PyMem_RESIZE(ptr, type, size)`` +* ``PyMem_FREE(ptr)`` +* ``PyMem_DEL(ptr)`` Customize Memory Allocators @@ -210,7 +263,7 @@ Customize Memory Allocators .. versionadded:: 3.4 -.. c:type:: PyMemAllocator +.. c:type:: PyMemAllocatorEx Structure used to describe a memory block allocator. The structure has four fields: @@ -222,29 +275,39 @@ Customize Memory Allocators +----------------------------------------------------------+---------------------------------------+ | ``void* malloc(void *ctx, size_t size)`` | allocate a memory block | +----------------------------------------------------------+---------------------------------------+ + | ``void* calloc(void *ctx, size_t nelem, size_t elsize)`` | allocate a memory block initialized | + | | with zeros | + +----------------------------------------------------------+---------------------------------------+ | ``void* realloc(void *ctx, void *ptr, size_t new_size)`` | allocate or resize a memory block | +----------------------------------------------------------+---------------------------------------+ | ``void free(void *ctx, void *ptr)`` | free a memory block | +----------------------------------------------------------+---------------------------------------+ + .. versionchanged:: 3.5 + The :c:type:`PyMemAllocator` structure was renamed to + :c:type:`PyMemAllocatorEx` and a new ``calloc`` field was added. + + .. c:type:: PyMemAllocatorDomain Enum used to identify an allocator domain. Domains: * :c:data:`PYMEM_DOMAIN_RAW`: functions :c:func:`PyMem_RawMalloc`, - :c:func:`PyMem_RawRealloc` and :c:func:`PyMem_RawFree` + :c:func:`PyMem_RawRealloc`, :c:func:`PyMem_RawCalloc` and + :c:func:`PyMem_RawFree` * :c:data:`PYMEM_DOMAIN_MEM`: functions :c:func:`PyMem_Malloc`, - :c:func:`PyMem_Realloc` and :c:func:`PyMem_Free` + :c:func:`PyMem_Realloc`, :c:func:`PyMem_Calloc` and :c:func:`PyMem_Free` * :c:data:`PYMEM_DOMAIN_OBJ`: functions :c:func:`PyObject_Malloc`, - :c:func:`PyObject_Realloc` and :c:func:`PyObject_Free` + :c:func:`PyObject_Realloc`, :c:func:`PyObject_Calloc` and + :c:func:`PyObject_Free` -.. c:function:: void PyMem_GetAllocator(PyMemAllocatorDomain domain, PyMemAllocator *allocator) +.. c:function:: void PyMem_GetAllocator(PyMemAllocatorDomain domain, PyMemAllocatorEx *allocator) Get the memory block allocator of the specified domain. -.. c:function:: void PyMem_SetAllocator(PyMemAllocatorDomain domain, PyMemAllocator *allocator) +.. c:function:: void PyMem_SetAllocator(PyMemAllocatorDomain domain, PyMemAllocatorEx *allocator) Set the memory block allocator of the specified domain. @@ -266,10 +329,11 @@ Customize Memory Allocators functions: - :c:func:`PyMem_RawMalloc`, :c:func:`PyMem_RawRealloc`, - :c:func:`PyMem_RawFree` - - :c:func:`PyMem_Malloc`, :c:func:`PyMem_Realloc`, :c:func:`PyMem_Free` + :c:func:`PyMem_RawCalloc`, :c:func:`PyMem_RawFree` + - :c:func:`PyMem_Malloc`, :c:func:`PyMem_Realloc`, :c:func:`PyMem_Calloc`, + :c:func:`PyMem_Free` - :c:func:`PyObject_Malloc`, :c:func:`PyObject_Realloc`, - :c:func:`PyObject_Free` + :c:func:`PyObject_Calloc`, :c:func:`PyObject_Free` Newly allocated memory is filled with the byte ``0xCB``, freed memory is filled with the byte ``0xDB``. Additional checks: diff --git a/Doc/c-api/memoryview.rst b/Doc/c-api/memoryview.rst index 5e50977cbe..9f6bfd751a 100644 --- a/Doc/c-api/memoryview.rst +++ b/Doc/c-api/memoryview.rst @@ -35,7 +35,7 @@ any other object. .. c:function:: PyObject *PyMemoryView_GetContiguous(PyObject *obj, int buffertype, char order) - Create a memoryview object to a contiguous chunk of memory (in either + Create a memoryview object to a :term:`contiguous` chunk of memory (in either 'C' or 'F'ortran *order*) from an object that defines the buffer interface. If memory is contiguous, the memoryview object points to the original memory. Otherwise, a copy is made and the memoryview points to a diff --git a/Doc/c-api/method.rst b/Doc/c-api/method.rst index acc81e4814..7a2a84fe11 100644 --- a/Doc/c-api/method.rst +++ b/Doc/c-api/method.rst @@ -49,7 +49,7 @@ Method Objects .. index:: object: method Methods are bound function objects. Methods are always bound to an instance of -an user-defined class. Unbound methods (methods bound to a class object) are +a user-defined class. Unbound methods (methods bound to a class object) are no longer available. diff --git a/Doc/c-api/module.rst b/Doc/c-api/module.rst index 985a347d63..34f8443931 100644 --- a/Doc/c-api/module.rst +++ b/Doc/c-api/module.rst @@ -7,8 +7,6 @@ Module Objects .. index:: object: module -There are only a few functions special to module objects. - .. c:var:: PyTypeObject PyModule_Type @@ -52,7 +50,7 @@ There are only a few functions special to module objects. .. c:function:: PyObject* PyModule_New(const char *name) - Similar to :c:func:`PyImport_NewObject`, but the name is an UTF-8 encoded + Similar to :c:func:`PyImport_NewObject`, but the name is a UTF-8 encoded string instead of a Unicode object. @@ -61,10 +59,10 @@ There are only a few functions special to module objects. .. index:: single: __dict__ (module attribute) Return the dictionary object that implements *module*'s namespace; this object - is the same as the :attr:`__dict__` attribute of the module object. This + is the same as the :attr:`~object.__dict__` attribute of the module object. This function never fails. It is recommended extensions use other :c:func:`PyModule_\*` and :c:func:`PyObject_\*` functions rather than directly - manipulate a module's :attr:`__dict__`. + manipulate a module's :attr:`~object.__dict__`. .. c:function:: PyObject* PyModule_GetNameObject(PyObject *module) @@ -84,6 +82,18 @@ There are only a few functions special to module objects. Similar to :c:func:`PyModule_GetNameObject` but return the name encoded to ``'utf-8'``. +.. c:function:: void* PyModule_GetState(PyObject *module) + + Return the "state" of the module, that is, a pointer to the block of memory + allocated at module creation time, or *NULL*. See + :c:member:`PyModuleDef.m_size`. + + +.. c:function:: PyModuleDef* PyModule_GetDef(PyObject *module) + + Return a pointer to the :c:type:`PyModuleDef` struct from which the module was + created, or *NULL* if the module wasn't created from a definition. + .. c:function:: PyObject* PyModule_GetFilenameObject(PyObject *module) @@ -109,55 +119,109 @@ There are only a few functions special to module objects. unencodable filenames, use :c:func:`PyModule_GetFilenameObject` instead. -.. c:function:: void* PyModule_GetState(PyObject *module) +.. _initializing-modules: - Return the "state" of the module, that is, a pointer to the block of memory - allocated at module creation time, or *NULL*. See - :c:member:`PyModuleDef.m_size`. +Initializing C modules +^^^^^^^^^^^^^^^^^^^^^^ +Modules objects are usually created from extension modules (shared libraries +which export an initialization function), or compiled-in modules +(where the initialization function is added using :c:func:`PyImport_AppendInittab`). +See :ref:`building` or :ref:`extending-with-embedding` for details. -.. c:function:: PyModuleDef* PyModule_GetDef(PyObject *module) +The initialization function can either pass a module definition instance +to :c:func:`PyModule_Create`, and return the resulting module object, +or request "multi-phase initialization" by returning the definition struct itself. - Return a pointer to the :c:type:`PyModuleDef` struct from which the module was - created, or *NULL* if the module wasn't created with - :c:func:`PyModule_Create`. +.. c:type:: PyModuleDef -.. c:function:: PyObject* PyState_FindModule(PyModuleDef *def) + The module definition struct, which holds all information needed to create + a module object. There is usually only one statically initialized variable + of this type for each module. - Returns the module object that was created from *def* for the current interpreter. - This method requires that the module object has been attached to the interpreter state with - :c:func:`PyState_AddModule` beforehand. In case the corresponding module object is not - found or has not been attached to the interpreter state yet, it returns NULL. + .. c:member:: PyModuleDef_Base m_base -.. c:function:: int PyState_AddModule(PyObject *module, PyModuleDef *def) + Always initialize this member to :const:`PyModuleDef_HEAD_INIT`. - Attaches the module object passed to the function to the interpreter state. This allows - the module object to be accessible via - :c:func:`PyState_FindModule`. + .. c:member:: char* m_name - .. versionadded:: 3.3 + Name for the new module. -.. c:function:: int PyState_RemoveModule(PyModuleDef *def) + .. c:member:: char* m_doc - Removes the module object created from *def* from the interpreter state. + Docstring for the module; usually a docstring variable created with + :c:func:`PyDoc_STRVAR` is used. - .. versionadded:: 3.3 + .. c:member:: Py_ssize_t m_size -Initializing C modules -^^^^^^^^^^^^^^^^^^^^^^ + Module state may be kept in a per-module memory area that can be + retrieved with :c:func:`PyModule_GetState`, rather than in static globals. + This makes modules safe for use in multiple sub-interpreters. + + This memory area is allocated based on *m_size* on module creation, + and freed when the module object is deallocated, after the + :c:member:`m_free` function has been called, if present. + + Setting ``m_size`` to ``-1`` means that the module does not support + sub-interpreters, because it has global state. -These functions are usually used in the module initialization function. + Setting it to a non-negative value means that the module can be + re-initialized and specifies the additional amount of memory it requires + for its state. Non-negative ``m_size`` is required for multi-phase + initialization. -.. c:function:: PyObject* PyModule_Create(PyModuleDef *module) + See :PEP:`3121` for more details. + + .. c:member:: PyMethodDef* m_methods - Create a new module object, given the definition in *module*. This behaves + A pointer to a table of module-level functions, described by + :c:type:`PyMethodDef` values. Can be *NULL* if no functions are present. + + .. c:member:: PyModuleDef_Slot* m_slots + + An array of slot definitions for multi-phase initialization, terminated by + a ``{0, NULL}`` entry. + When using single-phase initialization, *m_slots* must be *NULL*. + + .. versionchanged:: 3.5 + + Prior to version 3.5, this member was always set to *NULL*, + and was defined as: + + .. c:member:: inquiry m_reload + + .. c:member:: traverseproc m_traverse + + A traversal function to call during GC traversal of the module object, or + *NULL* if not needed. + + .. c:member:: inquiry m_clear + + A clear function to call during GC clearing of the module object, or + *NULL* if not needed. + + .. c:member:: freefunc m_free + + A function to call during deallocation of the module object, or *NULL* if + not needed. + +Single-phase initialization +........................... + +The module initialization function may create and return the module object +directly. This is referred to as "single-phase initialization", and uses one +of the following two module creation functions: + +.. c:function:: PyObject* PyModule_Create(PyModuleDef *def) + + Create a new module object, given the definition in *def*. This behaves like :c:func:`PyModule_Create2` with *module_api_version* set to :const:`PYTHON_API_VERSION`. -.. c:function:: PyObject* PyModule_Create2(PyModuleDef *module, int module_api_version) +.. c:function:: PyObject* PyModule_Create2(PyModuleDef *def, int module_api_version) - Create a new module object, given the definition in *module*, assuming the + Create a new module object, given the definition in *def*, assuming the API version *module_api_version*. If that version does not match the version of the running interpreter, a :exc:`RuntimeWarning` is emitted. @@ -166,69 +230,179 @@ These functions are usually used in the module initialization function. Most uses of this function should be using :c:func:`PyModule_Create` instead; only use this if you are sure you need it. +Before it is returned from in the initialization function, the resulting module +object is typically populated using functions like :c:func:`PyModule_AddObject`. -.. c:type:: PyModuleDef +.. _multi-phase-initialization: - This struct holds all information that is needed to create a module object. - There is usually only one static variable of that type for each module, which - is statically initialized and then passed to :c:func:`PyModule_Create` in the - module initialization function. +Multi-phase initialization +.......................... - .. c:member:: PyModuleDef_Base m_base +An alternate way to specify extensions is to request "multi-phase initialization". +Extension modules created this way behave more like Python modules: the +initialization is split between the *creation phase*, when the module object +is created, and the *execution phase*, when it is populated. +The distinction is similar to the :py:meth:`__new__` and :py:meth:`__init__` methods +of classes. - Always initialize this member to :const:`PyModuleDef_HEAD_INIT`. +Unlike modules created using single-phase initialization, these modules are not +singletons: if the *sys.modules* entry is removed and the module is re-imported, +a new module object is created, and the old module is subject to normal garbage +collection -- as with Python modules. +By default, multiple modules created from the same definition should be +independent: changes to one should not affect the others. +This means that all state should be specific to the module object (using e.g. +using :c:func:`PyModule_GetState`), or its contents (such as the module's +:attr:`__dict__` or individual classes created with :c:func:`PyType_FromSpec`). - .. c:member:: char* m_name +All modules created using multi-phase initialization are expected to support +:ref:`sub-interpreters <sub-interpreter-support>`. Making sure multiple modules +are independent is typically enough to achieve this. - Name for the new module. +To request multi-phase initialization, the initialization function +(PyInit_modulename) returns a :c:type:`PyModuleDef` instance with non-empty +:c:member:`~PyModuleDef.m_slots`. Before it is returned, the ``PyModuleDef`` +instance must be initialized with the following function: - .. c:member:: char* m_doc +.. c:function:: PyObject* PyModuleDef_Init(PyModuleDef *def) - Docstring for the module; usually a docstring variable created with - :c:func:`PyDoc_STRVAR` is used. + Ensures a module definition is a properly initialized Python object that + correctly reports its type and reference count. - .. c:member:: Py_ssize_t m_size + Returns *def* cast to ``PyObject*``, or *NULL* if an error occurred. - Some modules allow re-initialization (calling their ``PyInit_*`` function - more than once). These modules should keep their state in a per-module - memory area that can be retrieved with :c:func:`PyModule_GetState`. + .. versionadded:: 3.5 - This memory should be used, rather than static globals, to hold per-module - state, since it is then safe for use in multiple sub-interpreters. It is - freed when the module object is deallocated, after the :c:member:`m_free` - function has been called, if present. +The *m_slots* member of the module definition must point to an array of +``PyModuleDef_Slot`` structures: - Setting ``m_size`` to ``-1`` means that the module can not be - re-initialized because it has global state. Setting it to a non-negative - value means that the module can be re-initialized and specifies the - additional amount of memory it requires for its state. +.. c:type:: PyModuleDef_Slot - See :PEP:`3121` for more details. + .. c:member:: int slot - .. c:member:: PyMethodDef* m_methods + A slot ID, chosen from the available values explained below. - A pointer to a table of module-level functions, described by - :c:type:`PyMethodDef` values. Can be *NULL* if no functions are present. + .. c:member:: void* value - .. c:member:: inquiry m_reload + Value of the slot, whose meaning depends on the slot ID. - Currently unused, should be *NULL*. + .. versionadded:: 3.5 - .. c:member:: traverseproc m_traverse +The *m_slots* array must be terminated by a slot with id 0. - A traversal function to call during GC traversal of the module object, or - *NULL* if not needed. +The available slot types are: - .. c:member:: inquiry m_clear +.. c:var:: Py_mod_create - A clear function to call during GC clearing of the module object, or - *NULL* if not needed. + Specifies a function that is called to create the module object itself. + The *value* pointer of this slot must point to a function of the signature: - .. c:member:: freefunc m_free + .. c:function:: PyObject* create_module(PyObject *spec, PyModuleDef *def) - A function to call during deallocation of the module object, or *NULL* if - not needed. + The function receives a :py:class:`~importlib.machinery.ModuleSpec` + instance, as defined in :PEP:`451`, and the module definition. + It should return a new module object, or set an error + and return *NULL*. + + This function should be kept minimal. In particular, it should not + call arbitrary Python code, as trying to import the same module again may + result in an infinite loop. + + Multiple ``Py_mod_create`` slots may not be specified in one module + definition. + + If ``Py_mod_create`` is not specified, the import machinery will create + a normal module object using :c:func:`PyModule_New`. The name is taken from + *spec*, not the definition, to allow extension modules to dynamically adjust + to their place in the module hierarchy and be imported under different + names through symlinks, all while sharing a single module definition. + + There is no requirement for the returned object to be an instance of + :c:type:`PyModule_Type`. Any type can be used, as long as it supports + setting and getting import-related attributes. + However, only ``PyModule_Type`` instances may be returned if the + ``PyModuleDef`` has non-*NULL* ``m_methods``, ``m_traverse``, ``m_clear``, + ``m_free``; non-zero ``m_size``; or slots other than ``Py_mod_create``. + +.. c:var:: Py_mod_exec + + Specifies a function that is called to *execute* the module. + This is equivalent to executing the code of a Python module: typically, + this function adds classes and constants to the module. + The signature of the function is: + + .. c:function:: int exec_module(PyObject* module) + + If multiple ``Py_mod_exec`` slots are specified, they are processed in the + order they appear in the *m_slots* array. + +See :PEP:`489` for more details on multi-phase initialization. + +Low-level module creation functions +................................... + +The following functions are called under the hood when using multi-phase +initialization. They can be used directly, for example when creating module +objects dynamically. Note that both ``PyModule_FromDefAndSpec`` and +``PyModule_ExecDef`` must be called to fully initialize a module. + +.. c:function:: PyObject * PyModule_FromDefAndSpec(PyModuleDef *def, PyObject *spec) + + Create a new module object, given the definition in *module* and the + ModuleSpec *spec*. This behaves like :c:func:`PyModule_FromDefAndSpec2` + with *module_api_version* set to :const:`PYTHON_API_VERSION`. + + .. versionadded:: 3.5 + +.. c:function:: PyObject * PyModule_FromDefAndSpec2(PyModuleDef *def, PyObject *spec, int module_api_version) + + Create a new module object, given the definition in *module* and the + ModuleSpec *spec*, assuming the API version *module_api_version*. + If that version does not match the version of the running interpreter, + a :exc:`RuntimeWarning` is emitted. + + .. note:: + Most uses of this function should be using :c:func:`PyModule_FromDefAndSpec` + instead; only use this if you are sure you need it. + + .. versionadded:: 3.5 + +.. c:function:: int PyModule_ExecDef(PyObject *module, PyModuleDef *def) + + Process any execution slots (:c:data:`Py_mod_exec`) given in *def*. + + .. versionadded:: 3.5 + +.. c:function:: int PyModule_SetDocString(PyObject *module, const char *docstring) + + Set the docstring for *module* to *docstring*. + This function is called automatically when creating a module from + ``PyModuleDef``, using either ``PyModule_Create`` or + ``PyModule_FromDefAndSpec``. + + .. versionadded:: 3.5 + +.. c:function:: int PyModule_AddFunctions(PyObject *module, PyMethodDef *functions) + + Add the functions from the *NULL* terminated *functions* array to *module*. + Refer to the :c:type:`PyMethodDef` documentation for details on individual + entries (due to the lack of a shared module namespace, module level + "functions" implemented in C typically receive the module as their first + parameter, making them similar to instance methods on Python classes). + This function is called automatically when creating a module from + ``PyModuleDef``, using either ``PyModule_Create`` or + ``PyModule_FromDefAndSpec``. + + .. versionadded:: 3.5 + +Support functions +................. + +The module initialization function (if using single phase initialization) or +a function called from a module execution slot (if using multi-phase +initialization), can use the following functions to help initialize the module +state: .. c:function:: int PyModule_AddObject(PyObject *module, const char *name, PyObject *value) @@ -236,7 +410,6 @@ These functions are usually used in the module initialization function. be used from the module's initialization function. This steals a reference to *value*. Return ``-1`` on error, ``0`` on success. - .. c:function:: int PyModule_AddIntConstant(PyObject *module, const char *name, long value) Add an integer constant to *module* as *name*. This convenience function can be @@ -248,7 +421,7 @@ These functions are usually used in the module initialization function. Add a string constant to *module* as *name*. This convenience function can be used from the module's initialization function. The string *value* must be - null-terminated. Return ``-1`` on error, ``0`` on success. + *NULL*-terminated. Return ``-1`` on error, ``0`` on success. .. c:function:: int PyModule_AddIntMacro(PyObject *module, macro) @@ -262,3 +435,36 @@ These functions are usually used in the module initialization function. .. c:function:: int PyModule_AddStringMacro(PyObject *module, macro) Add a string constant to *module*. + + +Module lookup +^^^^^^^^^^^^^ + +Single-phase initialization creates singleton modules that can be looked up +in the context of the current interpreter. This allows the module object to be +retrieved later with only a reference to the module definition. + +These functions will not work on modules created using multi-phase initialization, +since multiple such modules can be created from a single definition. + +.. c:function:: PyObject* PyState_FindModule(PyModuleDef *def) + + Returns the module object that was created from *def* for the current interpreter. + This method requires that the module object has been attached to the interpreter state with + :c:func:`PyState_AddModule` beforehand. In case the corresponding module object is not + found or has not been attached to the interpreter state yet, it returns *NULL*. + +.. c:function:: int PyState_AddModule(PyObject *module, PyModuleDef *def) + + Attaches the module object passed to the function to the interpreter state. This allows + the module object to be accessible via :c:func:`PyState_FindModule`. + + Only effective on modules created using single-phase initialization. + + .. versionadded:: 3.3 + +.. c:function:: int PyState_RemoveModule(PyModuleDef *def) + + Removes the module object created from *def* from the interpreter state. + + .. versionadded:: 3.3 diff --git a/Doc/c-api/number.rst b/Doc/c-api/number.rst index 21951c38c0..9bcb649c9d 100644 --- a/Doc/c-api/number.rst +++ b/Doc/c-api/number.rst @@ -30,6 +30,14 @@ Number Protocol the equivalent of the Python expression ``o1 * o2``. +.. c:function:: PyObject* PyNumber_MatrixMultiply(PyObject *o1, PyObject *o2) + + Returns the result of matrix multiplication on *o1* and *o2*, or *NULL* on + failure. This is the equivalent of the Python expression ``o1 @ o2``. + + .. versionadded:: 3.5 + + .. c:function:: PyObject* PyNumber_FloorDivide(PyObject *o1, PyObject *o2) Return the floor of *o1* divided by *o2*, or *NULL* on failure. This is @@ -146,6 +154,15 @@ Number Protocol the Python statement ``o1 *= o2``. +.. c:function:: PyObject* PyNumber_InPlaceMatrixMultiply(PyObject *o1, PyObject *o2) + + Returns the result of matrix multiplication on *o1* and *o2*, or *NULL* on + failure. The operation is done *in-place* when *o1* supports it. This is + the equivalent of the Python statement ``o1 @= o2``. + + .. versionadded:: 3.5 + + .. c:function:: PyObject* PyNumber_InPlaceFloorDivide(PyObject *o1, PyObject *o2) Returns the mathematical floor of dividing *o1* by *o2*, or *NULL* on failure. diff --git a/Doc/c-api/object.rst b/Doc/c-api/object.rst index 97b45b12cb..b761c808fc 100644 --- a/Doc/c-api/object.rst +++ b/Doc/c-api/object.rst @@ -68,25 +68,35 @@ Object Protocol .. c:function:: int PyObject_SetAttr(PyObject *o, PyObject *attr_name, PyObject *v) Set the value of the attribute named *attr_name*, for object *o*, to the value - *v*. Returns ``-1`` on failure. This is the equivalent of the Python statement + *v*. Raise an exception and return ``-1`` on failure; + return ``0`` on success. This is the equivalent of the Python statement ``o.attr_name = v``. + If *v* is *NULL*, the attribute is deleted, however this feature is + deprecated in favour of using :c:func:`PyObject_DelAttr`. + .. c:function:: int PyObject_SetAttrString(PyObject *o, const char *attr_name, PyObject *v) Set the value of the attribute named *attr_name*, for object *o*, to the value - *v*. Returns ``-1`` on failure. This is the equivalent of the Python statement + *v*. Raise an exception and return ``-1`` on failure; + return ``0`` on success. This is the equivalent of the Python statement ``o.attr_name = v``. + If *v* is *NULL*, the attribute is deleted, however this feature is + deprecated in favour of using :c:func:`PyObject_DelAttrString`. + .. c:function:: int PyObject_GenericSetAttr(PyObject *o, PyObject *name, PyObject *value) - Generic attribute setter function that is meant to be put into a type - object's ``tp_setattro`` slot. It looks for a data descriptor in the + Generic attribute setter and deleter function that is meant + to be put into a type object's :c:member:`~PyTypeObject.tp_setattro` + slot. It looks for a data descriptor in the dictionary of classes in the object's MRO, and if found it takes preference - over setting the attribute in the instance dictionary. Otherwise, the - attribute is set in the object's :attr:`~object.__dict__` (if present). - Otherwise, an :exc:`AttributeError` is raised and ``-1`` is returned. + over setting or deleting the attribute in the instance dictionary. Otherwise, the + attribute is set or deleted in the object's :attr:`~object.__dict__` (if present). + On success, ``0`` is returned, otherwise an :exc:`AttributeError` + is raised and ``-1`` is returned. .. c:function:: int PyObject_DelAttr(PyObject *o, PyObject *attr_name) @@ -378,7 +388,8 @@ Object Protocol .. c:function:: int PyObject_SetItem(PyObject *o, PyObject *key, PyObject *v) - Map the object *key* to the value *v*. Returns ``-1`` on failure. This is the + Map the object *key* to the value *v*. Raise an exception and + return ``-1`` on failure; return ``0`` on success. This is the equivalent of the Python statement ``o[key] = v``. diff --git a/Doc/c-api/sequence.rst b/Doc/c-api/sequence.rst index 5960db9d42..f1825f079b 100644 --- a/Doc/c-api/sequence.rst +++ b/Doc/c-api/sequence.rst @@ -62,10 +62,14 @@ Sequence Protocol .. c:function:: int PySequence_SetItem(PyObject *o, Py_ssize_t i, PyObject *v) - Assign object *v* to the *i*\ th element of *o*. Returns ``-1`` on failure. This + Assign object *v* to the *i*\ th element of *o*. Raise an exception + and return ``-1`` on failure; return ``0`` on success. This is the equivalent of the Python statement ``o[i] = v``. This function *does not* steal a reference to *v*. + If *v* is *NULL*, the element is deleted, however this feature is + deprecated in favour of using :c:func:`PySequence_DelItem`. + .. c:function:: int PySequence_DelItem(PyObject *o, Py_ssize_t i) diff --git a/Doc/c-api/set.rst b/Doc/c-api/set.rst index 7f4d534a70..8de0394112 100644 --- a/Doc/c-api/set.rst +++ b/Doc/c-api/set.rst @@ -128,7 +128,7 @@ or :class:`frozenset` or instances of their subtypes. of brand new frozensets before they are exposed to other code). Return 0 on success or -1 on failure. Raise a :exc:`TypeError` if the *key* is unhashable. Raise a :exc:`MemoryError` if there is no room to grow. Raise a - :exc:`SystemError` if *set* is an not an instance of :class:`set` or its + :exc:`SystemError` if *set* is not an instance of :class:`set` or its subtype. @@ -142,7 +142,7 @@ subtypes but not for instances of :class:`frozenset` or its subtypes. error is encountered. Does not raise :exc:`KeyError` for missing keys. Raise a :exc:`TypeError` if the *key* is unhashable. Unlike the Python :meth:`~set.discard` method, this function does not automatically convert unhashable sets into - temporary frozensets. Raise :exc:`PyExc_SystemError` if *set* is an not an + temporary frozensets. Raise :exc:`PyExc_SystemError` if *set* is not an instance of :class:`set` or its subtype. @@ -150,7 +150,7 @@ subtypes but not for instances of :class:`frozenset` or its subtypes. Return a new reference to an arbitrary object in the *set*, and removes the object from the *set*. Return *NULL* on failure. Raise :exc:`KeyError` if the - set is empty. Raise a :exc:`SystemError` if *set* is an not an instance of + set is empty. Raise a :exc:`SystemError` if *set* is not an instance of :class:`set` or its subtype. diff --git a/Doc/c-api/stable.rst b/Doc/c-api/stable.rst index 063f856d22..5b771dd4ad 100644 --- a/Doc/c-api/stable.rst +++ b/Doc/c-api/stable.rst @@ -34,5 +34,5 @@ will work on all subsequent Python releases, but fail to load (because of missing symbols) on the older releases. As of Python 3.2, the set of functions available to the limited API is -documented in PEP 384. In the C API documentation, API elements that are not +documented in :pep:`384`. In the C API documentation, API elements that are not part of the limited API are marked as "Not part of the limited API." diff --git a/Doc/c-api/structures.rst b/Doc/c-api/structures.rst index e9e8add6c1..cc843146bd 100644 --- a/Doc/c-api/structures.rst +++ b/Doc/c-api/structures.rst @@ -150,9 +150,8 @@ specific C type of the *self* object. The :attr:`ml_flags` field is a bitfield which can include the following flags. The individual flags indicate either a calling convention or a binding convention. Of the calling convention flags, only :const:`METH_VARARGS` and -:const:`METH_KEYWORDS` can be combined (but note that :const:`METH_KEYWORDS` -alone is equivalent to ``METH_VARARGS | METH_KEYWORDS``). Any of the calling -convention flags can be combined with a binding flag. +:const:`METH_KEYWORDS` can be combined. Any of the calling convention flags +can be combined with a binding flag. .. data:: METH_VARARGS diff --git a/Doc/c-api/sys.rst b/Doc/c-api/sys.rst index 7cead07081..3d83b279c2 100644 --- a/Doc/c-api/sys.rst +++ b/Doc/c-api/sys.rst @@ -47,6 +47,60 @@ Operating System Utilities not call those functions directly! :c:type:`PyOS_sighandler_t` is a typedef alias for :c:type:`void (\*)(int)`. +.. c:function:: wchar_t* Py_DecodeLocale(const char* arg, size_t *size) + + Decode a byte string from the locale encoding with the :ref:`surrogateescape + error handler <surrogateescape>`: undecodable bytes are decoded as + characters in range U+DC80..U+DCFF. If a byte sequence can be decoded as a + surrogate character, escape the bytes using the surrogateescape error + handler instead of decoding them. + + Return a pointer to a newly allocated wide character string, use + :c:func:`PyMem_RawFree` to free the memory. If size is not ``NULL``, write + the number of wide characters excluding the null character into ``*size`` + + Return ``NULL`` on decoding error or memory allocation error. If *size* is + not ``NULL``, ``*size`` is set to ``(size_t)-1`` on memory error or set to + ``(size_t)-2`` on decoding error. + + Decoding errors should never happen, unless there is a bug in the C + library. + + Use the :c:func:`Py_EncodeLocale` function to encode the character string + back to a byte string. + + .. seealso:: + + The :c:func:`PyUnicode_DecodeFSDefaultAndSize` and + :c:func:`PyUnicode_DecodeLocaleAndSize` functions. + + .. versionadded:: 3.5 + + +.. c:function:: char* Py_EncodeLocale(const wchar_t *text, size_t *error_pos) + + Encode a wide character string to the locale encoding with the + :ref:`surrogateescape error handler <surrogateescape>`: surrogate characters + in the range U+DC80..U+DCFF are converted to bytes 0x80..0xFF. + + Return a pointer to a newly allocated byte string, use :c:func:`PyMem_Free` + to free the memory. Return ``NULL`` on encoding error or memory allocation + error + + If error_pos is not ``NULL``, ``*error_pos`` is set to the index of the + invalid character on encoding error, or set to ``(size_t)-1`` otherwise. + + Use the :c:func:`Py_DecodeLocale` function to decode the bytes string back + to a wide character string. + + .. seealso:: + + The :c:func:`PyUnicode_EncodeFSDefault` and + :c:func:`PyUnicode_EncodeLocale` functions. + + .. versionadded:: 3.5 + + .. _systemfunctions: System Functions diff --git a/Doc/c-api/typeobj.rst b/Doc/c-api/typeobj.rst index 5de8be0680..2c448a0211 100644 --- a/Doc/c-api/typeobj.rst +++ b/Doc/c-api/typeobj.rst @@ -111,10 +111,10 @@ type objects) *must* have the :attr:`ob_size` field. For statically allocated type objects, the tp_name field should contain a dot. Everything before the last dot is made accessible as the :attr:`__module__` attribute, and everything after the last dot is made accessible as the - :attr:`__name__` attribute. + :attr:`~definition.__name__` attribute. If no dot is present, the entire :c:member:`~PyTypeObject.tp_name` field is made accessible as the - :attr:`__name__` attribute, and the :attr:`__module__` attribute is undefined + :attr:`~definition.__name__` attribute, and the :attr:`__module__` attribute is undefined (unless explicitly set in the dictionary, as explained above). This means your type will be impossible to pickle. @@ -208,21 +208,27 @@ type objects) *must* have the :attr:`ob_size` field. .. c:member:: setattrfunc PyTypeObject.tp_setattr - An optional pointer to the set-attribute-string function. + An optional pointer to the function for setting and deleting attributes. This field is deprecated. When it is defined, it should point to a function that acts the same as the :c:member:`~PyTypeObject.tp_setattro` function, but taking a C string instead of a Python string object to give the attribute name. The signature is - the same as for :c:func:`PyObject_SetAttrString`. + the same as for :c:func:`PyObject_SetAttrString`, but setting + *v* to *NULL* to delete an attribute must be supported. This field is inherited by subtypes together with :c:member:`~PyTypeObject.tp_setattro`: a subtype inherits both :c:member:`~PyTypeObject.tp_setattr` and :c:member:`~PyTypeObject.tp_setattro` from its base type when the subtype's :c:member:`~PyTypeObject.tp_setattr` and :c:member:`~PyTypeObject.tp_setattro` are both *NULL*. -.. c:member:: void* PyTypeObject.tp_reserved +.. c:member:: PyAsyncMethods* tp_as_async - Reserved slot, formerly known as tp_compare. + Pointer to an additional structure that contains fields relevant only to + objects which implement :term:`awaitable` and :term:`asynchronous iterator` + protocols at the C-level. See :ref:`async-structs` for details. + + .. versionadded:: 3.5 + Formerly known as ``tp_compare`` and ``tp_reserved``. .. c:member:: reprfunc PyTypeObject.tp_repr @@ -346,9 +352,10 @@ type objects) *must* have the :attr:`ob_size` field. .. c:member:: setattrofunc PyTypeObject.tp_setattro - An optional pointer to the set-attribute function. + An optional pointer to the function for setting and deleting attributes. - The signature is the same as for :c:func:`PyObject_SetAttr`. It is usually + The signature is the same as for :c:func:`PyObject_SetAttr`, but setting + *v* to *NULL* to delete an attribute must be supported. It is usually convenient to set this field to :c:func:`PyObject_GenericSetAttr`, which implements the normal way of setting object attributes. @@ -719,7 +726,7 @@ type objects) *must* have the :attr:`ob_size` field. typedef struct PyGetSetDef { char *name; /* attribute name */ getter get; /* C function to get the attribute */ - setter set; /* C function to set the attribute */ + setter set; /* C function to set or delete the attribute */ char *doc; /* optional doc string */ void *closure; /* optional additional data for getter and setter */ } PyGetSetDef; @@ -770,12 +777,14 @@ type objects) *must* have the :attr:`ob_size` field. .. c:member:: descrsetfunc PyTypeObject.tp_descr_set - An optional pointer to a "descriptor set" function. + An optional pointer to a function for setting and deleting + a descriptor's value. The function signature is :: int tp_descr_set(PyObject *self, PyObject *obj, PyObject *value); + The *value* argument is set to *NULL* to delete the value. This field is inherited by subtypes. .. XXX explain. @@ -1118,6 +1127,9 @@ Number Object Structures binaryfunc nb_inplace_true_divide; unaryfunc nb_index; + + binaryfunc nb_matrix_multiply; + binaryfunc nb_inplace_matrix_multiply; } PyNumberMethods; .. note:: @@ -1163,9 +1175,11 @@ Mapping Object Structures .. c:member:: objobjargproc PyMappingMethods.mp_ass_subscript - This function is used by :c:func:`PyObject_SetItem` and has the same - signature. If this slot is *NULL*, the object does not support item - assignment. + This function is used by :c:func:`PyObject_SetItem` and + :c:func:`PyObject_DelItem`. It has the same signature as + :c:func:`PyObject_SetItem`, but *v* can also be set to *NULL* to delete + an item. If this slot is *NULL*, the object does not support item + assignment and deletion. .. _sequence-structs: @@ -1214,7 +1228,7 @@ Sequence Object Structures This function is used by :c:func:`PySequence_SetItem` and has the same signature. This slot may be left to *NULL* if the object does not support - item assignment. + item assignment and deletion. .. c:member:: objobjproc PySequenceMethods.sq_contains @@ -1329,3 +1343,58 @@ Buffer Object Structures :c:func:`PyBuffer_Release` is the interface for the consumer that wraps this function. + + +.. _async-structs: + + +Async Object Structures +======================= + +.. sectionauthor:: Yury Selivanov <yselivanov@sprymix.com> + +.. versionadded:: 3.5 + +.. c:type:: PyAsyncMethods + + This structure holds pointers to the functions required to implement + :term:`awaitable` and :term:`asynchronous iterator` objects. + + Here is the structure definition:: + + typedef struct { + unaryfunc am_await; + unaryfunc am_aiter; + unaryfunc am_anext; + } PyAsyncMethods; + +.. c:member:: unaryfunc PyAsyncMethods.am_await + + The signature of this function is:: + + PyObject *am_await(PyObject *self) + + The returned object must be an iterator, i.e. :c:func:`PyIter_Check` must + return ``1`` for it. + + This slot may be set to *NULL* if an object is not an :term:`awaitable`. + +.. c:member:: unaryfunc PyAsyncMethods.am_aiter + + The signature of this function is:: + + PyObject *am_aiter(PyObject *self) + + Must return an :term:`awaitable` object. See :meth:`__anext__` for details. + + This slot may be set to *NULL* if an object does not implement + asynchronous iteration protocol. + +.. c:member:: unaryfunc PyAsyncMethods.am_anext + + The signature of this function is:: + + PyObject *am_anext(PyObject *self) + + Must return an :term:`awaitable` object. See :meth:`__anext__` for details. + This slot may be set to *NULL*. diff --git a/Doc/c-api/unicode.rst b/Doc/c-api/unicode.rst index f7e99d6d99..a0672ca257 100644 --- a/Doc/c-api/unicode.rst +++ b/Doc/c-api/unicode.rst @@ -367,7 +367,7 @@ These APIs can be used to work with surrogates: .. c:macro:: Py_UNICODE_IS_HIGH_SURROGATE(ch) - Check if *ch* is an high surrogate (``0xD800 <= ch <= 0xDBFF``). + Check if *ch* is a high surrogate (``0xD800 <= ch <= 0xDBFF``). .. c:macro:: Py_UNICODE_IS_LOW_SURROGATE(ch) @@ -423,7 +423,7 @@ APIs: .. c:function:: PyObject *PyUnicode_FromString(const char *u) - Create a Unicode object from an UTF-8 encoded null-terminated char buffer + Create a Unicode object from a UTF-8 encoded null-terminated char buffer *u*. @@ -450,7 +450,7 @@ APIs: | :attr:`%%` | *n/a* | The literal % character. | +-------------------+---------------------+--------------------------------+ | :attr:`%c` | int | A single character, | - | | | represented as an C int. | + | | | represented as a C int. | +-------------------+---------------------+--------------------------------+ | :attr:`%d` | int | Exactly equivalent to | | | | ``printf("%d")``. | @@ -556,14 +556,13 @@ APIs: .. c:function:: PyObject* PyUnicode_FromEncodedObject(PyObject *obj, \ const char *encoding, const char *errors) - Coerce an encoded object *obj* to an Unicode object and return a reference with - incremented refcount. + Decode an encoded object *obj* to a Unicode object. :class:`bytes`, :class:`bytearray` and other :term:`bytes-like objects <bytes-like object>` are decoded according to the given *encoding* and using the error handling defined by *errors*. Both can be *NULL* to have the interface use the default - values (see the next section for details). + values (see :ref:`builtincodecs` for details). All other objects, including Unicode objects, cause a :exc:`TypeError` to be set. @@ -614,8 +613,7 @@ APIs: This function checks that *unicode* is a Unicode object, that the index is not out of bounds, and that the object can be modified safely (i.e. that it - its reference count is one), in contrast to the macro version - :c:func:`PyUnicode_WRITE_CHAR`. + its reference count is one). .. versionadded:: 3.3 @@ -745,8 +743,11 @@ Extension modules can continue using them, as they will not be removed in Python .. c:function:: PyObject* PyUnicode_FromObject(PyObject *obj) - Shortcut for ``PyUnicode_FromEncodedObject(obj, NULL, "strict")`` which is used - throughout the interpreter whenever coercion to Unicode is needed. + Copy an instance of a Unicode subtype to a new true Unicode object if + necessary. If *obj* is already a true Unicode object (not a subtype), + return the reference with incremented refcount. + + Objects other than Unicode or its subtypes will cause a :exc:`TypeError`. Locale Encoding @@ -765,11 +766,13 @@ system. *errors* is ``NULL``. *str* must end with a null character but cannot contain embedded null characters. + Use :c:func:`PyUnicode_DecodeFSDefaultAndSize` to decode a string from + :c:data:`Py_FileSystemDefaultEncoding` (the locale encoding read at + Python startup). + .. seealso:: - Use :c:func:`PyUnicode_DecodeFSDefaultAndSize` to decode a string from - :c:data:`Py_FileSystemDefaultEncoding` (the locale encoding read at - Python startup). + The :c:func:`Py_DecodeLocale` function. .. versionadded:: 3.3 @@ -790,11 +793,13 @@ system. *errors* is ``NULL``. Return a :class:`bytes` object. *str* cannot contain embedded null characters. + Use :c:func:`PyUnicode_EncodeFSDefault` to encode a string to + :c:data:`Py_FileSystemDefaultEncoding` (the locale encoding read at + Python startup). + .. seealso:: - Use :c:func:`PyUnicode_EncodeFSDefault` to encode a string to - :c:data:`Py_FileSystemDefaultEncoding` (the locale encoding read at - Python startup). + The :c:func:`Py_EncodeLocale` function. .. versionadded:: 3.3 @@ -839,12 +844,14 @@ used, passing :c:func:`PyUnicode_FSDecoder` as the conversion function: If :c:data:`Py_FileSystemDefaultEncoding` is not set, fall back to the locale encoding. + :c:data:`Py_FileSystemDefaultEncoding` is initialized at startup from the + locale encoding and cannot be modified later. If you need to decode a string + from the current locale encoding, use + :c:func:`PyUnicode_DecodeLocaleAndSize`. + .. seealso:: - :c:data:`Py_FileSystemDefaultEncoding` is initialized at startup from the - locale encoding and cannot be modified later. If you need to decode a - string from the current locale encoding, use - :c:func:`PyUnicode_DecodeLocaleAndSize`. + The :c:func:`Py_DecodeLocale` function. .. versionchanged:: 3.2 Use ``"strict"`` error handler on Windows. @@ -874,12 +881,13 @@ used, passing :c:func:`PyUnicode_FSDecoder` as the conversion function: If :c:data:`Py_FileSystemDefaultEncoding` is not set, fall back to the locale encoding. + :c:data:`Py_FileSystemDefaultEncoding` is initialized at startup from the + locale encoding and cannot be modified later. If you need to encode a string + to the current locale encoding, use :c:func:`PyUnicode_EncodeLocale`. + .. seealso:: - :c:data:`Py_FileSystemDefaultEncoding` is initialized at startup from the - locale encoding and cannot be modified later. If you need to encode a - string to the current locale encoding, use - :c:func:`PyUnicode_EncodeLocale`. + The :c:func:`Py_EncodeLocale` function. .. versionadded:: 3.2 @@ -1217,7 +1225,7 @@ These are the UTF-16 codec APIs: If *Py_UNICODE_WIDE* is defined, a single :c:type:`Py_UNICODE` value may get represented as a surrogate pair. If it is not defined, each :c:type:`Py_UNICODE` - values is interpreted as an UCS-2 character. + values is interpreted as a UCS-2 character. Return *NULL* if an exception was raised by the codec. diff --git a/Doc/c-api/veryhigh.rst b/Doc/c-api/veryhigh.rst index f8aaf0f67a..706efdfd55 100644 --- a/Doc/c-api/veryhigh.rst +++ b/Doc/c-api/veryhigh.rst @@ -307,10 +307,16 @@ the same library that the Python runtime is using. cells. +.. c:type:: PyFrameObject + + The C structure of the objects used to describe frame objects. The + fields of this type are subject to change at any time. + + .. c:function:: PyObject* PyEval_EvalFrame(PyFrameObject *f) Evaluate an execution frame. This is a simplified interface to - PyEval_EvalFrameEx, for backward compatibility. + :c:func:`PyEval_EvalFrameEx`, for backward compatibility. .. c:function:: PyObject* PyEval_EvalFrameEx(PyFrameObject *f, int throwflag) diff --git a/Doc/conf.py b/Doc/conf.py index f803de238e..d6f20ba25c 100644 --- a/Doc/conf.py +++ b/Doc/conf.py @@ -17,7 +17,7 @@ extensions = ['sphinx.ext.coverage', 'sphinx.ext.doctest', # General substitutions. project = 'Python' -copyright = '1990-%s, Python Software Foundation' % time.strftime('%Y') +copyright = '2001-%s, Python Software Foundation' % time.strftime('%Y') # We look for the Include/patchlevel.h file in the current Python source tree # and replace the values accordingly. @@ -36,6 +36,9 @@ highlight_language = 'python3' # Require Sphinx 1.2 for build. needs_sphinx = '1.2' +# Ignore any .rst files in the venv/ directory. +exclude_patterns = ['venv/*'] + # Options for HTML output # ----------------------- @@ -135,6 +138,11 @@ latex_appendices = ['glossary', 'about', 'license', 'copyright'] # Get LaTeX to handle Unicode correctly latex_elements = {'inputenc': r'\usepackage[utf8x]{inputenc}', 'utf8extra': ''} +# Options for Epub output +# ----------------------- + +epub_author = 'Python Documentation Authors' +epub_publisher = 'Python Software Foundation' # Options for the coverage checker # -------------------------------- diff --git a/Doc/data/refcounts.dat b/Doc/data/refcounts.dat index 6025617a1b..e388195757 100644 --- a/Doc/data/refcounts.dat +++ b/Doc/data/refcounts.dat @@ -349,6 +349,11 @@ PyErr_Format:PyObject*:exception:+1: PyErr_Format:const char*:format:: PyErr_Format::...:: +PyErr_FormatV:PyObject*::null: +PyErr_FormatV:PyObject*:exception:+1: +PyErr_FormatV:const char*:format:: +PyErr_FormatV:va_list:vargs:: + PyErr_WarnEx:int::: PyErr_WarnEx:PyObject*:category:0: PyErr_WarnEx:const char*:message:: @@ -486,6 +491,12 @@ PyFunction_SetDefaults:PyObject*:defaults:+1: PyGen_New:PyObject*::+1: PyGen_New:PyFrameObject*:frame:0: +PyGen_NewWithQualName:PyObject*::+1: +PyGen_NewWithQualName:PyFrameObject*:frame:0: + +PyCoro_New:PyObject*::+1: +PyCoro_New:PyFrameObject*:frame:0: + Py_InitModule:PyObject*::0: Py_InitModule:const char*:name:: Py_InitModule:PyMethodDef[]:methods:: diff --git a/Doc/distributing/index.rst b/Doc/distributing/index.rst index 1774d23643..82ecd2c1ef 100644 --- a/Doc/distributing/index.rst +++ b/Doc/distributing/index.rst @@ -35,7 +35,7 @@ Key terms repository of open source licensed packages made available for use by other Python users * the `Python Packaging Authority - <https://packaging.python.org/en/latest/future.html>`__ are the group of + <https://www.pypa.io/>`__ are the group of developers and documentation authors responsible for the maintenance and evolution of the standard packaging tools and the associated metadata and file format standards. They maintain a variety of tools, documentation @@ -61,8 +61,8 @@ Key terms extensions, to be installed on a system without needing to be built locally. -.. _setuptools: https://setuptools.pypa.io/en/latest/setuptools.html -.. _wheel: http://wheel.readthedocs.org +.. _setuptools: https://setuptools.readthedocs.io/en/latest/ +.. _wheel: https://wheel.readthedocs.org Open source licensing and collaboration ======================================= @@ -111,7 +111,7 @@ by invoking the ``pip`` module at the command line:: The Python Packaging User Guide includes more details on the `currently recommended tools`_. -.. _currently recommended tools: https://packaging.python.org/en/latest/current.html#packaging-tool-recommendations +.. _currently recommended tools: https://packaging.python.org/en/latest/current/#packaging-tool-recommendations Reading the guide ================= @@ -124,11 +124,11 @@ involved in creating a project: * `Uploading the project to the Python Packaging Index`_ .. _Project structure: \ - https://packaging.python.org/en/latest/distributing.html#creating-your-own-project + https://packaging.python.org/en/latest/distributing/ .. _Building and packaging the project: \ - https://packaging.python.org/en/latest/distributing.html#packaging-your-project + https://packaging.python.org/en/latest/distributing/#packaging-your-project .. _Uploading the project to the Python Packaging Index: \ - https://packaging.python.org/en/latest/distributing.html#uploading-your-project-to-pypi + https://packaging.python.org/en/latest/distributing/#uploading-your-project-to-pypi How do I...? @@ -160,11 +160,11 @@ Python Packaging User Guide for more information and recommendations. .. seealso:: `Python Packaging User Guide: Binary Extensions - <https://packaging.python.org/en/latest/extensions.html>`__ + <https://packaging.python.org/en/latest/extensions>`__ .. other topics: Once the Development & Deployment part of PPUG is fleshed out, some of those sections should be linked from new questions here (most notably, we should have a question about avoiding depending on PyPI that links to - https://packaging.python.org/en/latest/deployment.html#pypi-mirrors-and-caches) + https://packaging.python.org/en/latest/mirrors/) diff --git a/Doc/distutils/apiref.rst b/Doc/distutils/apiref.rst index f67fc5ae25..0672253471 100644 --- a/Doc/distutils/apiref.rst +++ b/Doc/distutils/apiref.rst @@ -319,12 +319,12 @@ This module provides the following functions. .. function:: gen_preprocess_options(macros, include_dirs) - Generate C pre-processor options (:option:`-D`, :option:`-U`, :option:`-I`) as + Generate C pre-processor options (:option:`-D`, :option:`!-U`, :option:`!-I`) as used by at least two types of compilers: the typical Unix compiler and Visual C++. *macros* is the usual thing, a list of 1- or 2-tuples, where ``(name,)`` - means undefine (:option:`-U`) macro *name*, and ``(name, value)`` means define + means undefine (:option:`!-U`) macro *name*, and ``(name, value)`` means define (:option:`-D`) macro *name* to *value*. *include_dirs* is just a list of - directory names to be added to the header file search path (:option:`-I`). + directory names to be added to the header file search path (:option:`!-I`). Returns a list of command-line options suitable for either Unix compilers or Visual C++. @@ -799,7 +799,7 @@ This module provides the :class:`UnixCCompiler` class, a subclass of * library search directories specified with :option:`-Ldir` -* compile handled by :program:`cc` (or similar) executable with :option:`-c` +* compile handled by :program:`cc` (or similar) executable with :option:`!-c` option: compiles :file:`.c` to :file:`.o` * link static library handled by :program:`ar` command (possibly with @@ -837,7 +837,7 @@ selection by :class:`MSVCCompiler`. .. module:: distutils.bcppcompiler -This module provides :class:`BorlandCCompiler`, an subclass of the abstract +This module provides :class:`BorlandCCompiler`, a subclass of the abstract :class:`CCompiler` class for the Borland C++ compiler. @@ -868,23 +868,31 @@ tarballs or zipfiles. Create an archive file (eg. ``zip`` or ``tar``). *base_name* is the name of the file to create, minus any format-specific extension; *format* is the - archive format: one of ``zip``, ``tar``, ``ztar``, or ``gztar``. *root_dir* is - a directory that will be the root directory of the archive; ie. we typically - ``chdir`` into *root_dir* before creating the archive. *base_dir* is the - directory where we start archiving from; ie. *base_dir* will be the common - prefix of all files and directories in the archive. *root_dir* and *base_dir* - both default to the current directory. Returns the name of the archive file. + archive format: one of ``zip``, ``tar``, ``gztar``, ``bztar``, ``xztar``, or + ``ztar``. *root_dir* is a directory that will be the root directory of the + archive; ie. we typically ``chdir`` into *root_dir* before creating the + archive. *base_dir* is the directory where we start archiving from; ie. + *base_dir* will be the common prefix of all files and directories in the + archive. *root_dir* and *base_dir* both default to the current directory. + Returns the name of the archive file. + + .. versionchanged:: 3.5 + Added support for the ``xztar`` format. .. function:: make_tarball(base_name, base_dir[, compress='gzip', verbose=0, dry_run=0]) 'Create an (optional compressed) archive as a tar file from all files in and - under *base_dir*. *compress* must be ``'gzip'`` (the default), ``'compress'``, - ``'bzip2'``, or ``None``. Both :program:`tar` and the compression utility named - by *compress* must be on the default program search path, so this is probably - Unix-specific. The output tar file will be named :file:`base_dir.tar`, - possibly plus the appropriate compression extension (:file:`.gz`, :file:`.bz2` - or :file:`.Z`). Return the output filename. + under *base_dir*. *compress* must be ``'gzip'`` (the default), + ``'bzip2'``, ``'xz'``, ``'compress'``, or ``None``. For the ``'compress'`` + method the compression utility named by :program:`compress` must be on the + default program search path, so this is probably Unix-specific. The output + tar file will be named :file:`base_dir.tar`, possibly plus the appropriate + compression extension (``.gz``, ``.bz2``, ``.xz`` or ``.Z``). Return the + output filename. + + .. versionchanged:: 3.5 + Added support for the ``xz`` compression. .. function:: make_zipfile(base_name, base_dir[, verbose=0, dry_run=0]) @@ -1193,12 +1201,12 @@ other utility module. .. function:: byte_compile(py_files[, optimize=0, force=0, prefix=None, base_dir=None, verbose=1, dry_run=0, direct=None]) - Byte-compile a collection of Python source files to either :file:`.pyc` or - :file:`.pyo` files in a :file:`__pycache__` subdirectory (see :pep:`3147`). + Byte-compile a collection of Python source files to :file:`.pyc` files in a + :file:`__pycache__` subdirectory (see :pep:`3147` and :pep:`488`). *py_files* is a list of files to compile; any files that don't end in :file:`.py` are silently skipped. *optimize* must be one of the following: - * ``0`` - don't optimize (generate :file:`.pyc`) + * ``0`` - don't optimize * ``1`` - normal optimization (like ``python -O``) * ``2`` - extra optimization (like ``python -OO``) @@ -1222,10 +1230,13 @@ other utility module. doing, leave it set to ``None``. .. versionchanged:: 3.2.3 - Create ``.pyc`` or ``.pyo`` files with an :func:`import magic tag + Create ``.pyc`` files with an :func:`import magic tag <imp.get_tag>` in their name, in a :file:`__pycache__` subdirectory instead of files without tag in the current directory. + .. versionchanged:: 3.5 + Create ``.pyc`` files according to :pep:`488`. + .. function:: rfc822_escape(header) @@ -1811,7 +1822,7 @@ Subclasses of :class:`Command` must define the following methods. Builds a `Windows Installer`_ (.msi) binary package. - .. _Windows Installer: http://msdn.microsoft.com/en-us/library/cc185688(VS.85).aspx + .. _Windows Installer: https://msdn.microsoft.com/en-us/library/cc185688(VS.85).aspx In most cases, the ``bdist_msi`` installer is a better choice than the ``bdist_wininst`` installer, because it provides better support for @@ -1896,9 +1907,9 @@ Subclasses of :class:`Command` must define the following methods. that is designed to run with both Python 2.x and 3.x, add:: try: - from distutils.command.build_py import build_py_2to3 as build_py + from distutils.command.build_py import build_py_2to3 as build_py except ImportError: - from distutils.command.build_py import build_py + from distutils.command.build_py import build_py to your setup.py, and later:: diff --git a/Doc/distutils/builtdist.rst b/Doc/distutils/builtdist.rst index c5827b63cf..523d1e0fff 100644 --- a/Doc/distutils/builtdist.rst +++ b/Doc/distutils/builtdist.rst @@ -72,13 +72,19 @@ The available formats for built distributions are: +-------------+------------------------------+---------+ | Format | Description | Notes | +=============+==============================+=========+ -| ``gztar`` | gzipped tar file | (1),(3) | +| ``gztar`` | gzipped tar file | \(1) | | | (:file:`.tar.gz`) | | +-------------+------------------------------+---------+ +| ``bztar`` | bzipped tar file | | +| | (:file:`.tar.bz2`) | | ++-------------+------------------------------+---------+ +| ``xztar`` | xzipped tar file | | +| | (:file:`.tar.xz`) | | ++-------------+------------------------------+---------+ | ``ztar`` | compressed tar file | \(3) | | | (:file:`.tar.Z`) | | +-------------+------------------------------+---------+ -| ``tar`` | tar file (:file:`.tar`) | \(3) | +| ``tar`` | tar file (:file:`.tar`) | | +-------------+------------------------------+---------+ | ``zip`` | zip file (:file:`.zip`) | (2),(4) | +-------------+------------------------------+---------+ @@ -94,6 +100,9 @@ The available formats for built distributions are: | ``msi`` | Microsoft Installer. | | +-------------+------------------------------+---------+ +.. versionchanged:: 3.5 + Added support for the ``xztar`` format. + Notes: @@ -104,8 +113,7 @@ Notes: default on Windows (3) - requires external utilities: :program:`tar` and possibly one of :program:`gzip`, - :program:`bzip2`, or :program:`compress` + requires external :program:`compress` utility. (4) requires either external :program:`zip` utility or :mod:`zipfile` module (part @@ -119,21 +127,22 @@ You don't have to use the :command:`bdist` command with the :option:`--formats` option; you can also use the command that directly implements the format you're interested in. Some of these :command:`bdist` "sub-commands" actually generate several similar formats; for instance, the :command:`bdist_dumb` command -generates all the "dumb" archive formats (``tar``, ``ztar``, ``gztar``, and -``zip``), and :command:`bdist_rpm` generates both binary and source RPMs. The -:command:`bdist` sub-commands, and the formats generated by each, are: - -+--------------------------+-----------------------+ -| Command | Formats | -+==========================+=======================+ -| :command:`bdist_dumb` | tar, ztar, gztar, zip | -+--------------------------+-----------------------+ -| :command:`bdist_rpm` | rpm, srpm | -+--------------------------+-----------------------+ -| :command:`bdist_wininst` | wininst | -+--------------------------+-----------------------+ -| :command:`bdist_msi` | msi | -+--------------------------+-----------------------+ +generates all the "dumb" archive formats (``tar``, ``gztar``, ``bztar``, +``xztar``, ``ztar``, and ``zip``), and :command:`bdist_rpm` generates both +binary and source RPMs. The :command:`bdist` sub-commands, and the formats +generated by each, are: + ++--------------------------+-------------------------------------+ +| Command | Formats | ++==========================+=====================================+ +| :command:`bdist_dumb` | tar, gztar, bztar, xztar, ztar, zip | ++--------------------------+-------------------------------------+ +| :command:`bdist_rpm` | rpm, srpm | ++--------------------------+-------------------------------------+ +| :command:`bdist_wininst` | wininst | ++--------------------------+-------------------------------------+ +| :command:`bdist_msi` | msi | ++--------------------------+-------------------------------------+ The following sections give details on the individual :command:`bdist_\*` commands. diff --git a/Doc/distutils/configfile.rst b/Doc/distutils/configfile.rst index 8faffe6c20..51d88971a4 100644 --- a/Doc/distutils/configfile.rst +++ b/Doc/distutils/configfile.rst @@ -51,7 +51,7 @@ option values can be split across multiple lines simply by indenting the continuation lines. You can find out the list of options supported by a particular command with the -universal :option:`--help` option, e.g. :: +universal :option:`!--help` option, e.g. :: > python setup.py --help build_ext [...] diff --git a/Doc/distutils/examples.rst b/Doc/distutils/examples.rst index af9125a7b3..1f5be9cdb2 100644 --- a/Doc/distutils/examples.rst +++ b/Doc/distutils/examples.rst @@ -245,7 +245,9 @@ Let's take an example with a simple script:: setup(name='foobar') -Running the ``check`` command will display some warnings:: +Running the ``check`` command will display some warnings: + +.. code-block:: shell-session $ python setup.py check running check @@ -274,7 +276,9 @@ For example, if the :file:`setup.py` script is changed like this:: url='http://example.com', long_description=desc) Where the long description is broken, ``check`` will be able to detect it -by using the :mod:`docutils` parser:: +by using the :mod:`docutils` parser: + +.. code-block:: shell-session $ python setup.py check --restructuredtext running check @@ -286,7 +290,9 @@ Reading the metadata The :func:`distutils.core.setup` function provides a command-line interface that allows you to query the metadata fields of a project through the -``setup.py`` script of a given project:: +``setup.py`` script of a given project: + +.. code-block:: shell-session $ python setup.py --name distribute diff --git a/Doc/distutils/index.rst b/Doc/distutils/index.rst index 335f804d42..c565bcc562 100644 --- a/Doc/distutils/index.rst +++ b/Doc/distutils/index.rst @@ -7,6 +7,11 @@ :Authors: Greg Ward, Anthony Baxter :Email: distutils-sig@python.org +.. seealso:: + + :ref:`distributing-index` + The up to date module distribution documentations + This document describes the Python Distribution Utilities ("Distutils") from the module developer's point of view, describing how to use the Distutils to make Python modules and extensions easily available to a wider audience with @@ -20,7 +25,6 @@ very little overhead for build/release/install mechanics. recommendations section <https://packaging.python.org/en/latest/current/>`__ in the Python Packaging User Guide for more information. - .. toctree:: :maxdepth: 2 :numbered: diff --git a/Doc/distutils/introduction.rst b/Doc/distutils/introduction.rst index 0ece646c34..8f46bd74c5 100644 --- a/Doc/distutils/introduction.rst +++ b/Doc/distutils/introduction.rst @@ -156,8 +156,8 @@ module pure Python module a module written in Python and contained in a single :file:`.py` file (and - possibly associated :file:`.pyc` and/or :file:`.pyo` files). Sometimes referred - to as a "pure module." + possibly associated :file:`.pyc` files). Sometimes referred to as a + "pure module." extension module a module written in the low-level language of the Python implementation: C/C++ @@ -210,5 +210,3 @@ distribution root the top-level directory of your source tree (or source distribution); the directory where :file:`setup.py` exists. Generally :file:`setup.py` will be run from this directory. - - diff --git a/Doc/distutils/packageindex.rst b/Doc/distutils/packageindex.rst index 2d7daef784..28ad1289ee 100644 --- a/Doc/distutils/packageindex.rst +++ b/Doc/distutils/packageindex.rst @@ -233,7 +233,9 @@ in the root of the package besides :file:`setup.py`. To prevent registering broken reStructuredText content, you can use the :program:`rst2html` program that is provided by the :mod:`docutils` package and -check the ``long_description`` from the command line:: +check the ``long_description`` from the command line: + +.. code-block:: shell-session $ python setup.py --long-description | rst2html.py > output.html diff --git a/Doc/distutils/sourcedist.rst b/Doc/distutils/sourcedist.rst index b9f0cc863b..35aea1e39a 100644 --- a/Doc/distutils/sourcedist.rst +++ b/Doc/distutils/sourcedist.rst @@ -32,12 +32,18 @@ to create a gzipped tarball and a zip file. The available formats are: | ``bztar`` | bzip2'ed tar file | | | | (:file:`.tar.bz2`) | | +-----------+-------------------------+---------+ +| ``xztar`` | xz'ed tar file | | +| | (:file:`.tar.xz`) | | ++-----------+-------------------------+---------+ | ``ztar`` | compressed tar file | \(4) | | | (:file:`.tar.Z`) | | +-----------+-------------------------+---------+ | ``tar`` | tar file (:file:`.tar`) | | +-----------+-------------------------+---------+ +.. versionchanged:: 3.5 + Added support for the ``xztar`` format. + Notes: (1) @@ -54,7 +60,7 @@ Notes: requires the :program:`compress` program. Notice that this format is now pending for deprecation and will be removed in the future versions of Python. -When using any ``tar`` format (``gztar``, ``bztar``, ``ztar`` or +When using any ``tar`` format (``gztar``, ``bztar``, ``xztar``, ``ztar`` or ``tar``), under Unix you can specify the ``owner`` and ``group`` names that will be set for each member of the archive. @@ -127,7 +133,9 @@ described above does not apply in this case. The manifest template has one command per line, where each command specifies a set of files to include or exclude from the source distribution. For an -example, again we turn to the Distutils' own manifest template:: +example, again we turn to the Distutils' own manifest template: + +.. code-block:: none include *.txt recursive-include examples *.txt *.py diff --git a/Doc/extending/building.rst b/Doc/extending/building.rst index 656af88131..9fe12c2424 100644 --- a/Doc/extending/building.rst +++ b/Doc/extending/building.rst @@ -1,30 +1,63 @@ .. highlightlang:: c - .. _building: -******************************************** -Building C and C++ Extensions with distutils -******************************************** +***************************** +Building C and C++ Extensions +***************************** -.. sectionauthor:: Martin v. Löwis <martin@v.loewis.de> +A C extension for CPython is a shared library (e.g. a ``.so`` file on Linux, +``.pyd`` on Windows), which exports an *initialization function*. + +To be importable, the shared library must be available on :envvar:`PYTHONPATH`, +and must be named after the module name, with an appropriate extension. +When using distutils, the correct filename is generated automatically. + +The initialization function has the signature: +.. c:function:: PyObject* PyInit_modulename(void) -Starting in Python 1.4, Python provides, on Unix, a special make file for -building make files for building dynamically-linked extensions and custom -interpreters. Starting with Python 2.0, this mechanism (known as related to -Makefile.pre.in, and Setup files) is no longer supported. Building custom -interpreters was rarely used, and extension modules can be built using -distutils. +It returns either a fully-initialized module, or a :c:type:`PyModuleDef` +instance. See :ref:`initializing-modules` for details. -Building an extension module using distutils requires that distutils is -installed on the build machine, which is included in Python 2.x and available -separately for Python 1.5. Since distutils also supports creation of binary -packages, users don't necessarily need a compiler and distutils to install the -extension. +.. highlightlang:: python + +For modules with ASCII-only names, the function must be named +``PyInit_<modulename>``, with ``<modulename>`` replaced by the name of the +module. When using :ref:`multi-phase-initialization`, non-ASCII module names +are allowed. In this case, the initialization function name is +``PyInitU_<modulename>``, with ``<modulename>`` encoded using Python's +*punycode* encoding with hyphens replaced by underscores. In Python:: + + def initfunc_name(name): + try: + suffix = b'_' + name.encode('ascii') + except UnicodeEncodeError: + suffix = b'U_' + name.encode('punycode').replace(b'-', b'_') + return b'PyInit' + suffix + +It is possible to export multiple modules from a single shared library by +defining multiple initialization functions. However, importing them requires +using symbolic links or a custom importer, because by default only the +function corresponding to the filename is found. +See the *"Multiple modules in one library"* section in :pep:`489` for details. + + +.. highlightlang:: c + +Building C and C++ Extensions with distutils +============================================ + +.. sectionauthor:: Martin v. Löwis <martin@v.loewis.de> + +Extension modules can be built using distutils, which is included in Python. +Since distutils also supports creation of binary packages, users don't +necessarily need a compiler and distutils to install the extension. A distutils package contains a driver script, :file:`setup.py`. This is a plain -Python file, which, in the most simple case, could look like this:: +Python file, which, in the most simple case, could look like this: + +.. code-block:: python3 from distutils.core import setup, Extension @@ -50,21 +83,24 @@ In the :file:`setup.py`, all execution is performed by calling the ``setup`` function. This takes a variable number of keyword arguments, of which the example above uses only a subset. Specifically, the example specifies meta-information to build packages, and it specifies the contents of the -package. Normally, a package will contain of addition modules, like Python +package. Normally, a package will contain additional modules, like Python source modules, documentation, subpackages, etc. Please refer to the distutils documentation in :ref:`distutils-index` to learn more about the features of distutils; this section explains building extension modules only. It is common to pre-compute arguments to :func:`setup`, to better structure the driver script. In the example above, the ``ext_modules`` argument to -:func:`setup` is a list of extension modules, each of which is an instance of +:func:`~distutils.core.setup` is a list of extension modules, each of which is +an instance of the :class:`~distutils.extension.Extension`. In the example, the instance defines an extension named ``demo`` which is build by compiling a single source file, :file:`demo.c`. In many cases, building an extension is more complex, since additional preprocessor defines and libraries may be needed. This is demonstrated in the -example below. :: +example below. + +.. code-block:: python3 from distutils.core import setup, Extension @@ -88,7 +124,8 @@ example below. :: ext_modules = [module1]) -In this example, :func:`setup` is called with additional meta-information, which +In this example, :func:`~distutils.core.setup` is called with additional +meta-information, which is recommended when distribution packages have to be built. For the extension itself, it specifies preprocessor defines, include directories, library directories, and libraries. Depending on the compiler, distutils passes this @@ -119,8 +156,7 @@ Module maintainers should produce source packages; to do so, they run :: python setup.py sdist In some cases, additional files need to be included in a source distribution; -this is done through a :file:`MANIFEST.in` file; see the distutils documentation -for details. +this is done through a :file:`MANIFEST.in` file; see :ref:`manifest` for details. If the source distribution has been build successfully, maintainers can also create binary distributions. Depending on the platform, one of the following @@ -129,4 +165,3 @@ commands can be used to do so. :: python setup.py bdist_wininst python setup.py bdist_rpm python setup.py bdist_dumb - diff --git a/Doc/extending/embedding.rst b/Doc/extending/embedding.rst index 6cb686ab09..64033dcd97 100644 --- a/Doc/extending/embedding.rst +++ b/Doc/extending/embedding.rst @@ -58,12 +58,18 @@ perform some operation on a file. :: int main(int argc, char *argv[]) { - Py_SetProgramName(argv[0]); /* optional but recommended */ - Py_Initialize(); - PyRun_SimpleString("from time import time,ctime\n" - "print('Today is', ctime(time()))\n"); - Py_Finalize(); - return 0; + wchar_t *program = Py_DecodeLocale(argv[0], NULL); + if (program == NULL) { + fprintf(stderr, "Fatal error: cannot decode argv[0]\n"); + exit(1); + } + Py_SetProgramName(program); /* optional but recommended */ + Py_Initialize(); + PyRun_SimpleString("from time import time,ctime\n" + "print('Today is', ctime(time()))\n"); + Py_Finalize(); + PyMem_RawFree(program); + return 0; } The :c:func:`Py_SetProgramName` function should be called before @@ -149,7 +155,9 @@ script, such as: c = c + b return c -then the result should be:: +then the result should be: + +.. code-block:: shell-session $ call multiply multiply 3 2 Will compute 3 times 2 @@ -160,7 +168,7 @@ for data conversion between Python and C, and for error reporting. The interesting part with respect to embedding Python starts with :: Py_Initialize(); - pName = PyUnicode_FromString(argv[1]); + pName = PyUnicode_DecodeFSDefault(argv[1]); /* Error checking of pName left out */ pModule = PyImport_Import(pName); @@ -283,16 +291,20 @@ available). This script has several options, of which the following will be directly useful to you: * ``pythonX.Y-config --cflags`` will give you the recommended flags when - compiling:: + compiling: + + .. code-block:: shell-session - $ /opt/bin/python3.4-config --cflags - -I/opt/include/python3.4m -I/opt/include/python3.4m -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes + $ /opt/bin/python3.4-config --cflags + -I/opt/include/python3.4m -I/opt/include/python3.4m -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes * ``pythonX.Y-config --ldflags`` will give you the recommended flags when - linking:: + linking: + + .. code-block:: shell-session - $ /opt/bin/python3.4-config --ldflags - -L/opt/lib/python3.4/config-3.4m -lpthread -ldl -lutil -lm -lpython3.4m -Xlinker -export-dynamic + $ /opt/bin/python3.4-config --ldflags + -L/opt/lib/python3.4/config-3.4m -lpthread -ldl -lutil -lm -lpython3.4m -Xlinker -export-dynamic .. note:: To avoid confusion between several Python installations (and especially diff --git a/Doc/extending/extending.rst b/Doc/extending/extending.rst index ba6bfa70a2..82197075c1 100644 --- a/Doc/extending/extending.rst +++ b/Doc/extending/extending.rst @@ -27,7 +27,7 @@ your system setup; details are given in later chapters. avoid writing C extensions and preserve portability to other implementations. For example, if your use case is calling C library functions or system calls, you should consider using the :mod:`ctypes` module or the `cffi - <http://cffi.readthedocs.org>`_ library rather than writing custom C code. + <https://cffi.readthedocs.org>`_ library rather than writing custom C code. These modules let you write Python code to interface with C code and are more portable between implementations of Python than writing and compiling a C extension module. @@ -375,11 +375,17 @@ optionally followed by an import of the module:: int main(int argc, char *argv[]) { + wchar_t *program = Py_DecodeLocale(argv[0], NULL); + if (program == NULL) { + fprintf(stderr, "Fatal error: cannot decode argv[0]\n"); + exit(1); + } + /* Add a built-in module, before Py_Initialize */ PyImport_AppendInittab("spam", PyInit_spam); /* Pass argv[0] to the Python interpreter */ - Py_SetProgramName(argv[0]); + Py_SetProgramName(program); /* Initialize the Python interpreter. Required. */ Py_Initialize(); @@ -391,6 +397,10 @@ optionally followed by an import of the module:: ... + PyMem_RawFree(program); + return 0; + } + .. note:: Removing entries from ``sys.modules`` or importing compiled modules into @@ -403,6 +413,13 @@ A more substantial example module is included in the Python source distribution as :file:`Modules/xxmodule.c`. This file may be used as a template or simply read as an example. +.. note:: + + Unlike our ``spam`` example, ``xxmodule`` uses *multi-phase initialization* + (new in Python 3.5), where a PyModuleDef structure is returned from + ``PyInit_spam``, and creation of the module is left to the import machinery. + For details on multi-phase initialization, see :PEP:`489`. + .. _compilation: @@ -775,7 +792,9 @@ the format string is empty, it returns ``None``; if it contains exactly one format unit, it returns whatever object is described by that format unit. To force it to return a tuple of size 0 or one, parenthesize the format string. -Examples (to the left the call, to the right the resulting Python value):: +Examples (to the left the call, to the right the resulting Python value): + +.. code-block:: none Py_BuildValue("") None Py_BuildValue("i", 123) 123 @@ -1331,4 +1350,3 @@ code distribution). .. [#] These guarantees don't hold when you use the "old" style calling convention --- this is still found in much existing code. - diff --git a/Doc/extending/index.rst b/Doc/extending/index.rst index dd43926450..9eec8c273a 100644 --- a/Doc/extending/index.rst +++ b/Doc/extending/index.rst @@ -32,7 +32,7 @@ approaches to creating C and C++ extensions for Python. .. seealso:: - `Python Packaging User Guide: Binary Extensions <https://packaging.python.org/en/latest/extensions.html>`_ + `Python Packaging User Guide: Binary Extensions <https://packaging.python.org/en/latest/extensions/>`_ The Python Packaging User Guide not only covers several available tools that simplify the creation of binary extensions, but also discusses the various reasons why creating an extension module may be diff --git a/Doc/extending/newtypes.rst b/Doc/extending/newtypes.rst index 0884430044..a69f114fe4 100644 --- a/Doc/extending/newtypes.rst +++ b/Doc/extending/newtypes.rst @@ -52,11 +52,15 @@ The first bit that will be new is:: } noddy_NoddyObject; This is what a Noddy object will contain---in this case, nothing more than what -every Python object contains---a refcount and a pointer to a type object. -These are the fields the ``PyObject_HEAD`` macro brings in. The reason for the -macro is to standardize the layout and to enable special debugging fields in -debug builds. Note that there is no semicolon after the ``PyObject_HEAD`` -macro; one is included in the macro definition. Be wary of adding one by +every Python object contains---a field called ``ob_base`` of type +:c:type:`PyObject`. :c:type:`PyObject` in turn, contains an ``ob_refcnt`` +field and a pointer to a type object. These can be accessed using the macros +:c:macro:`Py_REFCNT` and :c:macro:`Py_TYPE` respectively. These are the fields +the :c:macro:`PyObject_HEAD` macro brings in. The reason for the macro is to +standardize the layout and to enable special debugging fields in debug builds. + +Note that there is no semicolon after the :c:macro:`PyObject_HEAD` macro; +one is included in the macro definition. Be wary of adding one by accident; it's easy to do from habit, and your compiler might not complain, but someone else's probably will! (On Windows, MSVC is known to call this an error and refuse to compile the code.) @@ -80,7 +84,7 @@ Moving on, we come to the crunch --- the type object. :: 0, /* tp_print */ 0, /* tp_getattr */ 0, /* tp_setattr */ - 0, /* tp_reserved */ + 0, /* tp_as_async */ 0, /* tp_repr */ 0, /* tp_as_number */ 0, /* tp_as_sequence */ @@ -209,7 +213,9 @@ That's it! All that remains is to build it; put the above code in a file called setup(name="noddy", version="1.0", ext_modules=[Extension("noddy", ["noddy.c"])]) -in a file called :file:`setup.py`; then typing :: +in a file called :file:`setup.py`; then typing + +.. code-block:: shell-session $ python setup.py build @@ -1513,4 +1519,3 @@ might be something like the following:: .. [#] Even in the third version, we aren't guaranteed to avoid cycles. Instances of string subclasses are allowed and string subclasses could allow cycles even if normal strings don't. - diff --git a/Doc/faq/design.rst b/Doc/faq/design.rst index 9fdf8cb949..1b6cd7e241 100644 --- a/Doc/faq/design.rst +++ b/Doc/faq/design.rst @@ -158,7 +158,7 @@ where in Python you're forced to write this:: line = f.readline() if not line: break - ... # do something with line + ... # do something with line The reason for not allowing assignment in Python expressions is a common, hard-to-find bug in those other languages, caused by this construct: @@ -190,7 +190,7 @@ generally less robust than the "while True" solution:: line = f.readline() while line: - ... # do something with line... + ... # do something with line... line = f.readline() The problem with this is that if you change your mind about exactly how you get @@ -203,7 +203,7 @@ objects using the ``for`` statement. For example, :term:`file objects <file object>` support the iterator protocol, so you can write simply:: for line in f: - ... # do something with line... + ... # do something with line... @@ -368,9 +368,9 @@ Can Python be compiled to machine code, C or some other language? Practical answer: -`Cython <http://cython.org/>`_ and `Pyrex <http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/>`_ +`Cython <http://cython.org/>`_ and `Pyrex <https://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/>`_ compile a modified version of Python with optional annotations into C -extensions. `Weave <http://docs.scipy.org/doc/scipy-dev/reference/tutorial/weave.html>`_ makes it easy to +extensions. `Weave <https://scipy.github.io/devdocs/tutorial/weave.html>`_ makes it easy to intermingle Python and C code in various ways to increase performance. `Nuitka <http://www.nuitka.net/>`_ is an up-and-coming compiler of Python into C++ code, aiming to support the full Python language. @@ -577,8 +577,10 @@ other structure). :: class ListWrapper: def __init__(self, the_list): self.the_list = the_list + def __eq__(self, other): return self.the_list == other.the_list + def __hash__(self): l = self.the_list result = 98767 - len(l)*555 @@ -619,7 +621,7 @@ it and returns it. For example, here's how to iterate over the keys of a dictionary in sorted order:: for key in sorted(mydict): - ... # do whatever with mydict[key]... + ... # do whatever with mydict[key]... How do you specify and enforce an interface spec in Python? @@ -675,11 +677,11 @@ languages. For example:: class label(Exception): pass # declare a label try: - ... - if condition: raise label() # goto label - ... + ... + if condition: raise label() # goto label + ... except label: # where to goto - pass + pass ... This doesn't allow you to jump into the middle of a loop, but that's usually diff --git a/Doc/faq/extending.rst b/Doc/faq/extending.rst index 7bb4dc2e60..3eafdf193c 100644 --- a/Doc/faq/extending.rst +++ b/Doc/faq/extending.rst @@ -42,7 +42,7 @@ on what you're trying to do. .. XXX make sure these all work `Cython <http://cython.org>`_ and its relative `Pyrex -<http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/>`_ are compilers +<https://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/>`_ are compilers that accept a slightly modified form of Python and generate the corresponding C code. Cython and Pyrex make it possible to write an extension without having to learn Python's C API. @@ -50,10 +50,10 @@ to learn Python's C API. If you need to interface to some C or C++ library for which no Python extension currently exists, you can try wrapping the library's data types and functions with a tool such as `SWIG <http://www.swig.org>`_. `SIP -<http://www.riverbankcomputing.co.uk/software/sip/intro>`__, `CXX +<https://riverbankcomputing.com/software/sip/intro>`__, `CXX <http://cxx.sourceforge.net/>`_ `Boost <http://www.boost.org/libs/python/doc/index.html>`_, or `Weave -<http://docs.scipy.org/doc/scipy-dev/reference/tutorial/weave.html>`_ are also +<https://scipy.github.io/devdocs/tutorial/weave.html>`_ are also alternatives for wrapping C++ libraries. @@ -146,7 +146,9 @@ this object to :data:`sys.stdout` and :data:`sys.stderr`. Call print_error, or just allow the standard traceback mechanism to work. Then, the output will go wherever your ``write()`` method sends it. -The easiest way to do this is to use the :class:`io.StringIO` class:: +The easiest way to do this is to use the :class:`io.StringIO` class: + +.. code-block:: pycon >>> import io, sys >>> sys.stdout = io.StringIO() @@ -156,7 +158,9 @@ The easiest way to do this is to use the :class:`io.StringIO` class:: foo hello world! -A custom object to do the same would look like this:: +A custom object to do the same would look like this: + +.. code-block:: pycon >>> import io, sys >>> class StdoutCatcher(io.TextIOBase): @@ -222,11 +226,15 @@ How do I debug an extension? When using GDB with dynamically loaded extensions, you can't set a breakpoint in your extension until your extension is loaded. -In your ``.gdbinit`` file (or interactively), add the command:: +In your ``.gdbinit`` file (or interactively), add the command: + +.. code-block:: none br _PyImport_LoadDynamicModule -Then, when you run GDB:: +Then, when you run GDB: + +.. code-block:: shell-session $ gdb /local/bin/python gdb) run myscript.py @@ -247,20 +255,6 @@ For Red Hat, install the python-devel RPM to get the necessary files. For Debian, run ``apt-get install python-dev``. -What does "SystemError: _PyImport_FixupExtension: module yourmodule not loaded" mean? -------------------------------------------------------------------------------------- - -This means that you have created an extension module named "yourmodule", but -your module init function does not initialize with that name. - -Every module init function will have a line similar to:: - - module = Py_InitModule("yourmodule", yourmodule_functions); - -If the string passed to this function is not the same name as your extension -module, the :exc:`SystemError` exception will be raised. - - How do I tell "incomplete input" from "invalid input"? ------------------------------------------------------ diff --git a/Doc/faq/general.rst b/Doc/faq/general.rst index 2221f14927..3f96700b2d 100644 --- a/Doc/faq/general.rst +++ b/Doc/faq/general.rst @@ -146,10 +146,9 @@ labeled 2.0aN precede the versions labeled 2.0bN, which precede versions labeled 2.0cN, and *those* precede 2.0. You may also find version numbers with a "+" suffix, e.g. "2.2+". These are -unreleased versions, built directly from the Subversion trunk. In practice, -after a final minor release is made, the Subversion trunk is incremented to the -next minor version, which becomes the "a0" version, -e.g. "2.4a0". +unreleased versions, built directly from the CPython development repository. In +practice, after a final minor release is made, the version is incremented to the +next minor version, which becomes the "a0" version, e.g. "2.4a0". See also the documentation for :data:`sys.version`, :data:`sys.hexversion`, and :data:`sys.version_info`. @@ -159,7 +158,7 @@ How do I obtain a copy of the Python source? -------------------------------------------- The latest Python source distribution is always available from python.org, at -https://www.python.org/download/. The latest development sources can be obtained +https://www.python.org/downloads/. The latest development sources can be obtained via anonymous Mercurial access at https://hg.python.org/cpython. The source distribution is a gzipped tar file containing the complete C source, @@ -218,7 +217,7 @@ can be found at https://www.python.org/community/lists/. How do I get a beta test version of Python? ------------------------------------------- -Alpha and beta releases are available from https://www.python.org/download/. All +Alpha and beta releases are available from https://www.python.org/downloads/. All releases are announced on the comp.lang.python and comp.lang.python.announce newsgroups and on the Python home page at https://www.python.org/; an RSS feed of news is available. @@ -271,9 +270,9 @@ Where in the world is www.python.org located? The Python project's infrastructure is located all over the world. `www.python.org <https://www.python.org>`_ is graciously hosted by `Rackspace -<http://www.rackspace.com>`_, with CDN caching provided by `Fastly +<https://www.rackspace.com>`_, with CDN caching provided by `Fastly <https://www.fastly.com>`_. `Upfront Systems -<http://www.upfrontsystems.co.za>`_ hosts `bugs.python.org +<http://www.upfrontsystems.co.za/>`_ hosts `bugs.python.org <https://bugs.python.org>`_. Many other Python services like `the Wiki <https://wiki.python.org>`_ are hosted by `Oregon State University Open Source Lab <https://osuosl.org>`_. @@ -284,7 +283,7 @@ Why is it called Python? When he began implementing Python, Guido van Rossum was also reading the published scripts from `"Monty Python's Flying Circus" -<http://en.wikipedia.org/wiki/Monty_Python>`__, a BBC comedy series from the 1970s. Van Rossum +<https://en.wikipedia.org/wiki/Monty_Python>`__, a BBC comedy series from the 1970s. Van Rossum thought he needed a name that was short, unique, and slightly mysterious, so he decided to call the language Python. @@ -313,7 +312,7 @@ guaranteed that interfaces will remain the same throughout a series of bugfix releases. The latest stable releases can always be found on the `Python download page -<https://www.python.org/download/>`_. There are two recommended production-ready +<https://www.python.org/downloads/>`_. There are two recommended production-ready versions at this point in time, because at the moment there are two branches of stable releases: 2.x and 3.x. Python 3.x may be less useful than 2.x, since currently there is more third party software available for Python 2 than for @@ -345,7 +344,7 @@ different companies and organizations. High-profile Python projects include `the Mailman mailing list manager <http://www.list.org>`_ and `the Zope application server <http://www.zope.org>`_. Several Linux distributions, most notably `Red Hat -<http://www.redhat.com>`_, have written part or all of their installer and +<https://www.redhat.com>`_, have written part or all of their installer and system administration software in Python. Companies that use Python internally include Google, Yahoo, and Lucasfilm Ltd. @@ -439,7 +438,7 @@ remember the methods for a list, they can do something like this:: >>> L [1] -With the interpreter, documentation is never far from the student as he's +With the interpreter, documentation is never far from the student as they are programming. There are also good IDEs for Python. IDLE is a cross-platform IDE for Python diff --git a/Doc/faq/gui.rst b/Doc/faq/gui.rst index 5122de1c1a..98a28c3c8a 100644 --- a/Doc/faq/gui.rst +++ b/Doc/faq/gui.rst @@ -29,15 +29,15 @@ Tkinter Standard builds of Python include an object-oriented interface to the Tcl/Tk widget set, called :ref:`tkinter <Tkinter>`. This is probably the easiest to install (since it comes included with most -`binary distributions <https://www.python.org/download/>`_ of Python) and use. +`binary distributions <https://www.python.org/downloads/>`_ of Python) and use. For more info about Tk, including pointers to the source, see the -`Tcl/Tk home page <http://www.tcl.tk>`_. Tcl/Tk is fully portable to the +`Tcl/Tk home page <https://www.tcl.tk>`_. Tcl/Tk is fully portable to the Mac OS X, Windows, and Unix platforms. wxWidgets --------- -wxWidgets (http://www.wxwidgets.org) is a free, portable GUI class +wxWidgets (https://www.wxwidgets.org) is a free, portable GUI class library written in C++ that provides a native look and feel on a number of platforms, with Windows, Mac OS X, GTK, X11, all listed as current stable targets. Language bindings are available for a number @@ -58,21 +58,21 @@ Qt --- There are bindings available for the Qt toolkit (using either `PyQt -<http://www.riverbankcomputing.co.uk/software/pyqt/intro>`_ or `PySide -<http://www.pyside.org/>`_) and for KDE (`PyKDE <https://techbase.kde.org/Development/Languages/Python>`__). +<https://riverbankcomputing.com/software/pyqt/intro>`_ or `PySide +<https://wiki.qt.io/PySide>`_) and for KDE (`PyKDE4 <https://techbase.kde.org/Languages/Python/Using_PyKDE_4>`__). PyQt is currently more mature than PySide, but you must buy a PyQt license from -`Riverbank Computing <http://www.riverbankcomputing.co.uk/software/pyqt/license>`_ +`Riverbank Computing <https://www.riverbankcomputing.com/commercial/license-faq>`_ if you want to write proprietary applications. PySide is free for all applications. Qt 4.5 upwards is licensed under the LGPL license; also, commercial licenses -are available from `The Qt Company <http://www.qt.io/licensing/>`_. +are available from `The Qt Company <https://www.qt.io/licensing/>`_. Gtk+ ---- -The `GObject introspection bindings <https://live.gnome.org/PyGObject>`_ +The `GObject introspection bindings <https://wiki.gnome.org/Projects/PyGObject>`_ for Python allow you to write GTK+ 3 applications. There is also a -`Python GTK+ 3 Tutorial <http://python-gtk-3-tutorial.readthedocs.org/en/latest/>`_. +`Python GTK+ 3 Tutorial <https://python-gtk-3-tutorial.readthedocs.org/en/latest/>`_. The older PyGtk bindings for the `Gtk+ 2 toolkit <http://www.gtk.org>`_ have been implemented by James Henstridge; see <http://www.pygtk.org>. diff --git a/Doc/faq/library.rst b/Doc/faq/library.rst index 064728ff68..b5fdfa42cd 100644 --- a/Doc/faq/library.rst +++ b/Doc/faq/library.rst @@ -257,7 +257,8 @@ all the threads to finish:: import threading, time def thread_task(name, n): - for i in range(n): print(name, i) + for i in range(n): + print(name, i) for i in range(10): T = threading.Thread(target=thread_task, args=(str(i), i)) @@ -273,7 +274,8 @@ A simple fix is to add a tiny sleep to the start of the run function:: def thread_task(name, n): time.sleep(0.001) # <--------------------! - for i in range(n): print(name, i) + for i in range(n): + print(name, i) for i in range(10): T = threading.Thread(target=thread_task, args=(str(i), i)) @@ -502,8 +504,8 @@ in big-endian format from a file:: import struct with open(filename, "rb") as f: - s = f.read(8) - x, y, z = struct.unpack(">hhl", s) + s = f.read(8) + x, y, z = struct.unpack(">hhl", s) The '>' in the format string forces big-endian data; the letter 'h' reads one "short integer" (2 bytes), and 'l' reads one "long integer" (4 bytes) from the @@ -619,7 +621,7 @@ For Win32, POSIX (Linux, BSD, etc.), Jython: For Unix, see a Usenet post by Mitch Chapman: - http://groups.google.com/groups?selm=34A04430.CF9@ohioee.com + https://groups.google.com/groups?selm=34A04430.CF9@ohioee.com Why doesn't closing sys.stdout (stdin, stderr) really close it? @@ -681,10 +683,10 @@ Yes. Here's a simple example that uses urllib.request:: import urllib.request - ### build the query string + # build the query string qs = "First=Josephine&MI=Q&Last=Public" - ### connect and send the server a path + # connect and send the server a path req = urllib.request.urlopen('http://www.some-server.out-there' '/cgi-bin/some-cgi-script', data=qs) with req: @@ -740,8 +742,9 @@ varies between systems; sometimes it is ``/usr/lib/sendmail``, sometimes ``/usr/sbin/sendmail``. The sendmail manual page will help you out. Here's some sample code:: - SENDMAIL = "/usr/sbin/sendmail" # sendmail location import os + + SENDMAIL = "/usr/sbin/sendmail" # sendmail location p = os.popen("%s -t -i" % SENDMAIL, "w") p.write("To: receiver@example.com\n") p.write("Subject: test\n") diff --git a/Doc/faq/programming.rst b/Doc/faq/programming.rst index 94b428d299..694753e5b9 100644 --- a/Doc/faq/programming.rst +++ b/Doc/faq/programming.rst @@ -28,9 +28,9 @@ graphical debugger. PythonWin is a Python IDE that includes a GUI debugger based on pdb. The Pythonwin debugger colors breakpoints and has quite a few cool features such as debugging non-Pythonwin programs. Pythonwin is available as part of the `Python -for Windows Extensions <http://sourceforge.net/projects/pywin32/>`__ project and +for Windows Extensions <https://sourceforge.net/projects/pywin32/>`__ project and as a part of the ActivePython distribution (see -http://www.activestate.com/activepython\ ). +https://www.activestate.com/activepython\ ). `Boa Constructor <http://boa-constructor.sourceforge.net/>`_ is an IDE and GUI builder that uses wxWidgets. It offers visual frame creation and manipulation, @@ -44,13 +44,13 @@ and the Scintilla editing component. Pydb is a version of the standard Python debugger pdb, modified for use with DDD (Data Display Debugger), a popular graphical debugger front end. Pydb can be found at http://bashdb.sourceforge.net/pydb/ and DDD can be found at -http://www.gnu.org/software/ddd. +https://www.gnu.org/software/ddd. There are a number of commercial Python IDEs that include graphical debuggers. They include: -* Wing IDE (http://wingware.com/) -* Komodo IDE (http://komodoide.com/) +* Wing IDE (https://wingware.com/) +* Komodo IDE (https://komodoide.com/) * PyCharm (https://www.jetbrains.com/pycharm/) @@ -63,13 +63,13 @@ PyChecker is a static analysis tool that finds bugs in Python source code and warns about code complexity and style. You can get PyChecker from http://pychecker.sourceforge.net/. -`Pylint <http://www.logilab.org/projects/pylint>`_ is another tool that checks +`Pylint <https://www.pylint.org/>`_ is another tool that checks if a module satisfies a coding standard, and also makes it possible to write plug-ins to add a custom feature. In addition to the bug checking that PyChecker performs, Pylint offers some additional features such as checking line length, whether variable names are well-formed according to your coding standard, whether declared interfaces are fully implemented, and more. -http://docs.pylint.org/ provides a full list of Pylint's features. +https://docs.pylint.org/ provides a full list of Pylint's features. How can I create a stand-alone binary from a Python script? @@ -207,7 +207,7 @@ functions), e.g.:: >>> squares = [] >>> for x in range(5): - ... squares.append(lambda: x**2) + ... squares.append(lambda: x**2) This gives you a list that contains 5 lambdas that calculate ``x**2``. You might expect that, when called, they would return, respectively, ``0``, ``1``, @@ -234,7 +234,7 @@ lambdas, so that they don't rely on the value of the global ``x``:: >>> squares = [] >>> for x in range(5): - ... squares.append(lambda n=x: n**2) + ... squares.append(lambda n=x: n**2) Here, ``n=x`` creates a new variable ``n`` local to the lambda and computed when the lambda is defined so that it has the same value that ``x`` had at @@ -539,7 +539,7 @@ desired effect in a number of ways. args['a'] = 'new-value' # args is a mutable dictionary args['b'] = args['b'] + 1 # change it in-place - args = {'a':' old-value', 'b': 99} + args = {'a': 'old-value', 'b': 99} func3(args) print(args['a'], args['b']) @@ -655,16 +655,15 @@ Essentially, assignment always binds a name to a value; The same is true of ``def`` and ``class`` statements, but in that case the value is a callable. Consider the following code:: - class A: - pass - - B = A - - a = B() - b = a - print(b) + >>> class A: + ... pass + ... + >>> B = A + >>> a = B() + >>> b = a + >>> print(b) <__main__.A object at 0x16D07CC> - print(a) + >>> print(a) <__main__.A object at 0x16D07CC> Arguably the class has a name: even though it is bound to two names and invoked @@ -839,7 +838,7 @@ How do I convert a number to a string? To convert, e.g., the number 144 to the string '144', use the built-in type constructor :func:`str`. If you want a hexadecimal or octal representation, use the built-in functions :func:`hex` or :func:`oct`. For fancy formatting, see -the :ref:`string-formatting` section, e.g. ``"{:04d}".format(144)`` yields +the :ref:`formatstrings` section, e.g. ``"{:04d}".format(144)`` yields ``'0144'`` and ``"{:.3f}".format(1.0/3.0)`` yields ``'0.333'``. @@ -1099,7 +1098,7 @@ How do I iterate over a sequence in reverse order? Use the :func:`reversed` built-in function, which is new in Python 2.4:: for x in reversed(sequence): - ... # do something with x... + ... # do something with x ... This won't touch your original sequence, but build a new copy with reversed order to iterate over. @@ -1107,7 +1106,7 @@ order to iterate over. With Python 2.3, you can use an extended slice syntax:: for x in sequence[::-1]: - ... # do something with x... + ... # do something with x ... How do you remove duplicates from a list? @@ -1115,7 +1114,7 @@ How do you remove duplicates from a list? See the Python Cookbook for a long discussion of many ways to do this: - http://code.activestate.com/recipes/52560/ + https://code.activestate.com/recipes/52560/ If you don't mind reordering the list, sort it and then scan from the end of the list, deleting duplicates as you go:: @@ -1172,16 +1171,28 @@ You probably tried to make a multidimensional array like this:: >>> A = [[None] * 2] * 3 -This looks correct if you print it:: +This looks correct if you print it: + +.. testsetup:: + + A = [[None] * 2] * 3 + +.. doctest:: >>> A [[None, None], [None, None], [None, None]] But when you assign a value, it shows up in multiple places: - >>> A[0][0] = 5 - >>> A - [[5, None], [5, None], [5, None]] +.. testsetup:: + + A = [[None] * 2] * 3 + +.. doctest:: + + >>> A[0][0] = 5 + >>> A + [[5, None], [5, None], [5, None]] The reason is that replicating a list with ``*`` doesn't create copies, it only creates references to the existing objects. The ``*3`` creates a list @@ -1201,7 +1212,7 @@ use a list comprehension:: w, h = 2, 3 A = [[None] * w for i in range(h)] -Or, you can use an extension that provides a matrix datatype; `Numeric Python +Or, you can use an extension that provides a matrix datatype; `NumPy <http://www.numpy.org/>`_ is the best known. @@ -1313,40 +1324,11 @@ I want to do a complicated sort: can you do a Schwartzian Transform in Python? The technique, attributed to Randal Schwartz of the Perl community, sorts the elements of a list by a metric which maps each element to its "sort value". In -Python, just use the ``key`` argument for the ``sort()`` method:: +Python, use the ``key`` argument for the :meth:`list.sort` method:: Isorted = L[:] Isorted.sort(key=lambda s: int(s[10:15])) -The ``key`` argument is new in Python 2.4, for older versions this kind of -sorting is quite simple to do with list comprehensions. To sort a list of -strings by their uppercase values:: - - tmp1 = [(x.upper(), x) for x in L] # Schwartzian transform - tmp1.sort() - Usorted = [x[1] for x in tmp1] - -To sort by the integer value of a subfield extending from positions 10-15 in -each string:: - - tmp2 = [(int(s[10:15]), s) for s in L] # Schwartzian transform - tmp2.sort() - Isorted = [x[1] for x in tmp2] - -For versions prior to 3.0, Isorted may also be computed by :: - - def intfield(s): - return int(s[10:15]) - - def Icmp(s1, s2): - return cmp(intfield(s1), intfield(s2)) - - Isorted = L[:] - Isorted.sort(Icmp) - -but since this method calls ``intfield()`` many times for each element of L, it -is slower than the Schwartzian Transform. - How can I sort one list by values from another list? ---------------------------------------------------- @@ -1405,7 +1387,7 @@ A method is a function on some object ``x`` that you normally call as definition:: class C: - def meth (self, arg): + def meth(self, arg): return arg * 2 + self.attribute @@ -1438,9 +1420,9 @@ that does something:: def search(obj): if isinstance(obj, Mailbox): - # ... code to search a mailbox + ... # code to search a mailbox elif isinstance(obj, Document): - # ... code to search a document + ... # code to search a document elif ... A better approach is to define a ``search()`` method on all the classes and just @@ -1448,11 +1430,11 @@ call it:: class Mailbox: def search(self): - # ... code to search a mailbox + ... # code to search a mailbox class Document: def search(self): - # ... code to search a document + ... # code to search a document obj.search() @@ -1509,7 +1491,7 @@ How do I call a method defined in a base class from a derived class that overrid Use the built-in :func:`super` function:: class Derived(Base): - def meth (self): + def meth(self): super(Derived, self).meth() For version prior to 3.0, you may be using classic classes: For a class @@ -1694,9 +1676,9 @@ address, it happens frequently that after an object is deleted from memory, the next freshly created object is allocated at the same position in memory. This is illustrated by this example: ->>> id(1000) +>>> id(1000) # doctest: +SKIP 13901272 ->>> id(2000) +>>> id(2000) # doctest: +SKIP 13901272 The two ids belong to different integer objects that are created before, and @@ -1705,9 +1687,9 @@ objects whose id you want to examine are still alive, create another reference to the object: >>> a = 1000; b = 2000 ->>> id(a) +>>> id(a) # doctest: +SKIP 13901272 ->>> id(b) +>>> id(b) # doctest: +SKIP 13891296 diff --git a/Doc/faq/windows.rst b/Doc/faq/windows.rst index 6db6637ed7..d7253436be 100644 --- a/Doc/faq/windows.rst +++ b/Doc/faq/windows.rst @@ -340,5 +340,5 @@ This is a mistake; the extension should be .TGZ. Simply rename the downloaded file to have the .TGZ extension, and WinZip will be able to handle it. (If your copy of WinZip doesn't, get a newer one from -http://www.winzip.com.) +https://www.winzip.com.) diff --git a/Doc/glossary.rst b/Doc/glossary.rst index 36832a33d4..45b794f48e 100644 --- a/Doc/glossary.rst +++ b/Doc/glossary.rst @@ -69,11 +69,33 @@ Glossary :ref:`the difference between arguments and parameters <faq-argument-vs-parameter>`, and :pep:`362`. + asynchronous context manager + An object which controls the environment seen in an + :keyword:`async with` statement by defining :meth:`__aenter__` and + :meth:`__aexit__` methods. Introduced by :pep:`492`. + + asynchronous iterable + An object, that can be used in an :keyword:`async for` statement. + Must return an :term:`asynchronous iterator` from its + :meth:`__aiter__` method. Introduced by :pep:`492`. + + asynchronous iterator + An object that implements :meth:`__aiter__` and :meth:`__anext__` + methods. ``__anext__`` must return an :term:`awaitable` object. + :keyword:`async for` resolves awaitable returned from asynchronous + iterator's :meth:`__anext__` method until it raises + :exc:`StopAsyncIteration` exception. Introduced by :pep:`492`. + attribute A value associated with an object which is referenced by name using dotted expressions. For example, if an object *o* has an attribute *a* it would be referenced as *o.a*. + awaitable + An object that can be used in an :keyword:`await` expression. Can be + a :term:`coroutine` or an object with an :meth:`__await__` method. + See also :pep:`492`. + BDFL Benevolent Dictator For Life, a.k.a. `Guido van Rossum <https://www.python.org/~guido/>`_, Python's creator. @@ -86,12 +108,21 @@ Glossary A :term:`text file` reads and writes :class:`str` objects. bytes-like object - An object that supports the :ref:`bufferobjects`, like :class:`bytes`, - :class:`bytearray` or :class:`memoryview`. Bytes-like objects can - be used for various operations that expect binary data, such as - compression, saving to a binary file or sending over a socket. - Some operations need the binary data to be mutable, in which case - not all bytes-like objects can apply. + An object that supports the :ref:`bufferobjects` and can + export a C-:term:`contiguous` buffer. This includes all :class:`bytes`, + :class:`bytearray`, and :class:`array.array` objects, as well as many + common :class:`memoryview` objects. Bytes-like objects can + be used for various operations that work with binary data; these include + compression, saving to a binary file, and sending over a socket. + + Some operations need the binary data to be mutable. The documentation + often refers to these as "read-write bytes-like objects". Example + mutable buffer objects include :class:`bytearray` and a + :class:`memoryview` of a :class:`bytearray`. + Other operations require the binary data to be stored in + immutable objects ("read-only bytes-like objects"); examples + of these include :class:`bytes` and a :class:`memoryview` + of a :class:`bytes` object. bytecode Python source code is compiled into bytecode, the internal representation @@ -139,6 +170,32 @@ Glossary statement by defining :meth:`__enter__` and :meth:`__exit__` methods. See :pep:`343`. + contiguous + .. index:: C-contiguous, Fortran contiguous + + A buffer is considered contiguous exactly if it is either + *C-contiguous* or *Fortran contiguous*. Zero-dimensional buffers are + C and Fortran contiguous. In one-dimensional arrays, the items + must be laid out in memory next to each other, in order of + increasing indexes starting from zero. In multidimensional + C-contiguous arrays, the last index varies the fastest when + visiting items in order of memory address. However, in + Fortran contiguous arrays, the first index varies the fastest. + + coroutine + Coroutines is a more generalized form of subroutines. Subroutines are + entered at one point and exited at another point. Coroutines can be + entered, exited, and resumed at many different points. They can be + implemented with the :keyword:`async def` statement. See also + :pep:`492`. + + coroutine function + A function which returns a :term:`coroutine` object. A coroutine + function may be defined with the :keyword:`async def` statement, + and may contain :keyword:`await`, :keyword:`async for`, and + :keyword:`async with` keywords. These were introduced + by :pep:`492`. + CPython The canonical implementation of the Python programming language, as distributed on `python.org <https://www.python.org>`_. The term "CPython" @@ -250,10 +307,14 @@ Glossary A synonym for :term:`file object`. finder - An object that tries to find the :term:`loader` for a module. It must - implement either a method named :meth:`find_loader` or a method named - :meth:`find_module`. See :pep:`302` and :pep:`420` for details and - :class:`importlib.abc.Finder` for an :term:`abstract base class`. + An object that tries to find the :term:`loader` for a module that is + being imported. + + Since Python 3.3, there are two types of finder: :term:`meta path finders + <meta path finder>` for use with :data:`sys.meta_path`, and :term:`path + entry finders <path entry finder>` for use with :data:`sys.path_hooks`. + + See :pep:`302`, :pep:`420` and :pep:`451` for much more detail. floor division Mathematical division that rounds down to nearest integer. The floor @@ -298,14 +359,23 @@ Glossary .. index:: single: generator generator - A function which returns an iterator. It looks like a normal function - except that it contains :keyword:`yield` statements for producing a series - of values usable in a for-loop or that can be retrieved one at a time with - the :func:`next` function. Each :keyword:`yield` temporarily suspends - processing, remembering the location execution state (including local - variables and pending try-statements). When the generator resumes, it - picks-up where it left-off (in contrast to functions which start fresh on - every invocation). + A function which returns a :term:`generator iterator`. It looks like a + normal function except that it contains :keyword:`yield` expressions + for producing a series of values usable in a for-loop or that can be + retrieved one at a time with the :func:`next` function. + + Usually refers to a generator function, but may refer to a + *generator iterator* in some contexts. In cases where the intended + meaning isn't clear, using the full terms avoids ambiguity. + + generator iterator + An object created by a :term:`generator` function. + + Each :keyword:`yield` temporarily suspends processing, remembering the + location execution state (including local variables and pending + try-statements). When the *generator iterator* resumes, it picks-up where + it left-off (in contrast to functions which start fresh on every + invocation). .. index:: single: generator expression @@ -410,6 +480,19 @@ Glossary than compiled ones, though their programs generally also run more slowly. See also :term:`interactive`. + interpreter shutdown + When asked to shut down, the Python interpreter enters a special phase + where it gradually releases all allocated resources, such as modules + and various critical internal structures. It also makes several calls + to the :term:`garbage collector <garbage collection>`. This can trigger + the execution of code in user-defined destructors or weakref callbacks. + Code executed during the shutdown phase can encounter various + exceptions as the resources it relies on may not function anymore + (common examples are library modules or the warnings machinery). + + The main reason for interpreter shutdown is that the ``__main__`` module + or the script being run has finished executing. + iterable An object capable of returning its members one at a time. Examples of iterables include all sequence types (such as :class:`list`, :class:`str`, @@ -452,12 +535,13 @@ Glossary A number of tools in Python accept key functions to control how elements are ordered or grouped. They include :func:`min`, :func:`max`, - :func:`sorted`, :meth:`list.sort`, :func:`heapq.nsmallest`, - :func:`heapq.nlargest`, and :func:`itertools.groupby`. + :func:`sorted`, :meth:`list.sort`, :func:`heapq.merge`, + :func:`heapq.nsmallest`, :func:`heapq.nlargest`, and + :func:`itertools.groupby`. There are several ways to create a key function. For example. the :meth:`str.lower` method can serve as a key function for case insensitive - sorts. Alternatively, an ad-hoc key function can be built from a + sorts. Alternatively, a key function can be built from a :keyword:`lambda` expression such as ``lambda r: (r[0], r[2])``. Also, the :mod:`operator` module provides three key function constructors: :func:`~operator.attrgetter`, :func:`~operator.itemgetter`, and @@ -512,10 +596,13 @@ Glossary :class:`collections.OrderedDict` and :class:`collections.Counter`. meta path finder - A finder returned by a search of :data:`sys.meta_path`. Meta path + A :term:`finder` returned by a search of :data:`sys.meta_path`. Meta path finders are related to, but different from :term:`path entry finders <path entry finder>`. + See :class:`importlib.abc.MetaPathFinder` for the methods that meta path + finders implement. + metaclass The class of a class. Class definitions create a class name, a class dictionary, and a list of base classes. The metaclass is responsible for @@ -538,7 +625,8 @@ Glossary method resolution order Method Resolution Order is the order in which base classes are searched for a member during lookup. See `The Python 2.3 Method Resolution Order - <https://www.python.org/download/releases/2.3/mro/>`_. + <https://www.python.org/download/releases/2.3/mro/>`_ for details of the + algorithm used by the Python interpreter since the 2.3 release. module An object that serves as an organizational unit of Python code. Modules @@ -549,7 +637,7 @@ Glossary module spec A namespace containing the import-related information used to load a - module. + module. An instance of :class:`importlib.machinery.ModuleSpec`. MRO See :term:`method resolution order`. @@ -676,6 +764,9 @@ Glossary (i.e. a :term:`path entry hook`) which knows how to locate modules given a :term:`path entry`. + See :class:`importlib.abc.PathEntryFinder` for the methods that path entry + finders implement. + path entry hook A callable on the :data:`sys.path_hook` list which returns a :term:`path entry finder` if it knows how to find modules on a specific :term:`path diff --git a/Doc/howto/argparse.rst b/Doc/howto/argparse.rst index 510d1d49db..cfe9868875 100644 --- a/Doc/howto/argparse.rst +++ b/Doc/howto/argparse.rst @@ -511,7 +511,7 @@ to count the number of occurrences of a specific optional arguments: * Sadly, our help output isn't very informative on the new ability our script has acquired, but that can always be fixed by improving the documentation for - out script (e.g. via the ``help`` keyword argument). + our script (e.g. via the ``help`` keyword argument). * That last output exposes a bug in our program. diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index e362631db6..5a69ca8aba 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -152,7 +152,9 @@ Let's dive in! For my example I'm using ``_pickle.Pickler.dump()``. 2. If the call to the ``PyArg_Parse`` function uses any of the - following format units:: + following format units: + + .. code-block:: none O& O! @@ -758,6 +760,14 @@ All Argument Clinic converters accept the following arguments: In addition, some converters accept additional arguments. Here is a list of these arguments, along with their meanings: + ``accept`` + A set of Python types (and possibly pseudo-types); + this restricts the allowable Python argument to values of these types. + (This is not a general-purpose facility; as a rule it only supports + specific lists of types as shown in the legacy converter table.) + + To accept ``None``, add ``NoneType`` to this set. + ``bitwise`` Only supported for unsigned integers. The native integer value of this Python argument will be written to the parameter without any range checking, @@ -772,39 +782,27 @@ of these arguments, along with their meanings: Only supported for strings. Specifies the encoding to use when converting this string from a Python str (Unicode) value into a C ``char *`` value. - ``length`` - Only supported for strings. If true, requests that the length of the - string be passed in to the impl function, just after the string parameter, - in a parameter named ``<parameter_name>_length``. - - ``nullable`` - Only supported for strings. If true, this parameter may also be set to - ``None``, in which case the C parameter will be set to ``NULL``. ``subclass_of`` Only supported for the ``object`` converter. Requires that the Python value be a subclass of a Python type, as expressed in C. - ``types`` - Only supported for the ``object`` (and ``self``) converter. Specifies + ``type`` + Only supported for the ``object`` and ``self`` converters. Specifies the C type that will be used to declare the variable. Default value is ``"PyObject *"``. - ``types`` - A string containing a list of Python types (and possibly pseudo-types); - this restricts the allowable Python argument to values of these types. - (This is not a general-purpose facility; as a rule it only supports - specific lists of types as shown in the legacy converter table.) - ``zeroes`` Only supported for strings. If true, embedded NUL bytes (``'\\0'``) are - permitted inside the value. + permitted inside the value. The length of the string will be passed in + to the impl function, just after the string parameter, as a parameter named + ``<parameter_name>_length``. Please note, not every possible combination of arguments will work. -Often these arguments are implemented internally by specific ``PyArg_ParseTuple`` +Usually these arguments are implemented by specific ``PyArg_ParseTuple`` *format units*, with specific behavior. For example, currently you cannot -call ``str`` and pass in ``zeroes=True`` without also specifying an ``encoding``; -although it's perfectly reasonable to think this would work, these semantics don't +call ``unsigned_short`` without also specifying ``bitwise=True``. +Although it's perfectly reasonable to think this would work, these semantics don't map to any existing format unit. So Argument Clinic doesn't support it. (Or, at least, not yet.) @@ -816,13 +814,13 @@ on the right is the text you'd replace it with. ``'B'`` ``unsigned_char(bitwise=True)`` ``'b'`` ``unsigned_char`` ``'c'`` ``char`` -``'C'`` ``int(types='str')`` +``'C'`` ``int(accept={str})`` ``'d'`` ``double`` ``'D'`` ``Py_complex`` -``'es#'`` ``str(encoding='name_of_encoding', length=True, zeroes=True)`` ``'es'`` ``str(encoding='name_of_encoding')`` -``'et#'`` ``str(encoding='name_of_encoding', types='bytes bytearray str', length=True)`` -``'et'`` ``str(encoding='name_of_encoding', types='bytes bytearray str')`` +``'es#'`` ``str(encoding='name_of_encoding', zeroes=True)`` +``'et'`` ``str(encoding='name_of_encoding', accept={bytes, bytearray, str})`` +``'et#'`` ``str(encoding='name_of_encoding', accept={bytes, bytearray, str}, zeroes=True)`` ``'f'`` ``float`` ``'h'`` ``short`` ``'H'`` ``unsigned_short(bitwise=True)`` @@ -830,29 +828,30 @@ on the right is the text you'd replace it with. ``'I'`` ``unsigned_int(bitwise=True)`` ``'k'`` ``unsigned_long(bitwise=True)`` ``'K'`` ``unsigned_PY_LONG_LONG(bitwise=True)`` +``'l'`` ``long`` ``'L'`` ``PY_LONG_LONG`` ``'n'`` ``Py_ssize_t`` +``'O'`` ``object`` ``'O!'`` ``object(subclass_of='&PySomething_Type')`` ``'O&'`` ``object(converter='name_of_c_function')`` -``'O'`` ``object`` ``'p'`` ``bool`` -``'s#'`` ``str(length=True)`` ``'S'`` ``PyBytesObject`` ``'s'`` ``str`` -``'s*'`` ``Py_buffer(types='str bytes bytearray buffer')`` -``'u#'`` ``Py_UNICODE(length=True)`` -``'u'`` ``Py_UNICODE`` +``'s#'`` ``str(zeroes=True)`` +``'s*'`` ``Py_buffer(accept={buffer, str})`` ``'U'`` ``unicode`` -``'w*'`` ``Py_buffer(types='bytearray rwbuffer')`` -``'y#'`` ``str(types='bytes', length=True)`` +``'u'`` ``Py_UNICODE`` +``'u#'`` ``Py_UNICODE(zeroes=True)`` +``'w*'`` ``Py_buffer(accept={rwbuffer})`` ``'Y'`` ``PyByteArrayObject`` -``'y'`` ``str(types='bytes')`` +``'y'`` ``str(accept={bytes})`` +``'y#'`` ``str(accept={robuffer}, zeroes=True)`` ``'y*'`` ``Py_buffer`` -``'Z#'`` ``Py_UNICODE(nullable=True, length=True)`` -``'z#'`` ``str(nullable=True, length=True)`` -``'Z'`` ``Py_UNICODE(nullable=True)`` -``'z'`` ``str(nullable=True)`` -``'z*'`` ``Py_buffer(types='str bytes bytearray buffer', nullable=True)`` +``'Z'`` ``Py_UNICODE(accept={str, NoneType})`` +``'Z#'`` ``Py_UNICODE(accept={str, NoneType}, zeroes=True)`` +``'z'`` ``str(accept={str, NoneType})`` +``'z#'`` ``str(accept={str, NoneType}, zeroes=True)`` +``'z*'`` ``Py_buffer(accept={buffer, str, NoneType})`` ========= ================================================================================= As an example, here's our sample ``pickle.Pickler.dump`` using the proper @@ -1252,18 +1251,18 @@ Here's the simplest example of a custom converter, from ``Modules/zlibmodule.c`` /*[python input] - class uint_converter(CConverter): - type = 'unsigned int' - converter = 'uint_converter' + class ssize_t_converter(CConverter): + type = 'Py_ssize_t' + converter = 'ssize_t_converter' [python start generated code]*/ - /*[python end generated code: checksum=da39a3ee5e6b4b0d3255bfef95601890afd80709]*/ + /*[python end generated code: output=da39a3ee5e6b4b0d input=35521e4e733823c7]*/ -This block adds a converter to Argument Clinic named ``uint``. Parameters -declared as ``uint`` will be declared as type ``unsigned int``, and will -be parsed by the ``'O&'`` format unit, which will call the ``uint_converter`` -converter function. -``uint`` variables automatically support default values. +This block adds a converter to Argument Clinic named ``ssize_t``. Parameters +declared as ``ssize_t`` will be declared as type ``Py_ssize_t``, and will +be parsed by the ``'O&'`` format unit, which will call the +``ssize_t_converter`` converter function. ``ssize_t`` variables +automatically support default values. More sophisticated custom converters can insert custom C code to handle initialization and cleanup. @@ -1586,7 +1585,7 @@ called ``preserve``:: preserve -This tells Clinic that the current contents of the output should be kept, unmodifed. +This tells Clinic that the current contents of the output should be kept, unmodified. This is used internally by Clinic when dumping output into ``file`` files; wrapping it in a Clinic block lets Clinic use its existing checksum functionality to ensure the file was not modified by hand before it gets overwritten. diff --git a/Doc/howto/cporting.rst b/Doc/howto/cporting.rst index d7a7086302..27e7e6f6e0 100644 --- a/Doc/howto/cporting.rst +++ b/Doc/howto/cporting.rst @@ -161,7 +161,7 @@ simple example demonstrates how. :: #define INITERROR return NULL - PyObject * + PyMODINIT_FUNC PyInit_myextension(void) #else diff --git a/Doc/howto/curses.rst b/Doc/howto/curses.rst index 87a5cab142..188a5cf223 100644 --- a/Doc/howto/curses.rst +++ b/Doc/howto/curses.rst @@ -545,7 +545,7 @@ learn more about submitting patches to Python. a lengthy tutorial for C programmers. * `The ncurses man page <http://linux.die.net/man/3/ncurses>`_ * `The ncurses FAQ <http://invisible-island.net/ncurses/ncurses.faq.html>`_ -* `"Use curses... don't swear" <http://www.youtube.com/watch?v=eN1eZtjLEnU>`_: +* `"Use curses... don't swear" <https://www.youtube.com/watch?v=eN1eZtjLEnU>`_: video of a PyCon 2013 talk on controlling terminals using curses or Urwid. * `"Console Applications with Urwid" <http://www.pyvideo.org/video/1568/console-applications-with-urwid>`_: video of a PyCon CA 2012 talk demonstrating some applications written using diff --git a/Doc/howto/descriptor.rst b/Doc/howto/descriptor.rst index 530f34b88f..d370eb5572 100644 --- a/Doc/howto/descriptor.rst +++ b/Doc/howto/descriptor.rst @@ -104,7 +104,7 @@ like:: "Emulate type_getattro() in Objects/typeobject.c" v = object.__getattribute__(self, key) if hasattr(v, '__get__'): - return v.__get__(None, self) + return v.__get__(None, self) return v The important points to remember are: @@ -163,9 +163,9 @@ descriptor is useful for monitoring just a few chosen attributes:: self.val = val >>> class MyClass(object): - x = RevealAccess(10, 'var "x"') - y = 5 - + ... x = RevealAccess(10, 'var "x"') + ... y = 5 + ... >>> m = MyClass() >>> m.x Retrieving var "x" @@ -287,15 +287,15 @@ this:: Running the interpreter shows how the function descriptor works in practice:: >>> class D(object): - def f(self, x): - return x - + ... def f(self, x): + ... return x + ... >>> d = D() - >>> D.__dict__['f'] # Stored internally as a function + >>> D.__dict__['f'] # Stored internally as a function <function f at 0x00C45070> - >>> D.f # Get from a class becomes an unbound method + >>> D.f # Get from a class becomes an unbound method <unbound method D.f> - >>> d.f # Get from an instance becomes a bound method + >>> d.f # Get from an instance becomes a bound method <bound method D.f of <__main__.D object at 0x00B18C90>> The output suggests that bound and unbound methods are two different types. @@ -358,10 +358,10 @@ Since staticmethods return the underlying function with no changes, the example calls are unexciting:: >>> class E(object): - def f(x): - print(x) - f = staticmethod(f) - + ... def f(x): + ... print(x) + ... f = staticmethod(f) + ... >>> print(E.f(3)) 3 >>> print(E().f(3)) @@ -371,23 +371,23 @@ Using the non-data descriptor protocol, a pure Python version of :func:`staticmethod` would look like this:: class StaticMethod(object): - "Emulate PyStaticMethod_Type() in Objects/funcobject.c" + "Emulate PyStaticMethod_Type() in Objects/funcobject.c" - def __init__(self, f): - self.f = f + def __init__(self, f): + self.f = f - def __get__(self, obj, objtype=None): - return self.f + def __get__(self, obj, objtype=None): + return self.f Unlike static methods, class methods prepend the class reference to the argument list before calling the function. This format is the same for whether the caller is an object or a class:: >>> class E(object): - def f(klass, x): - return klass.__name__, x - f = classmethod(f) - + ... def f(klass, x): + ... return klass.__name__, x + ... f = classmethod(f) + ... >>> print(E.f(3)) ('E', 3) >>> print(E().f(3)) @@ -419,15 +419,15 @@ Using the non-data descriptor protocol, a pure Python version of :func:`classmethod` would look like this:: class ClassMethod(object): - "Emulate PyClassMethod_Type() in Objects/funcobject.c" + "Emulate PyClassMethod_Type() in Objects/funcobject.c" - def __init__(self, f): - self.f = f + def __init__(self, f): + self.f = f - def __get__(self, obj, klass=None): - if klass is None: - klass = type(obj) - def newfunc(*args): - return self.f(klass, *args) - return newfunc + def __get__(self, obj, klass=None): + if klass is None: + klass = type(obj) + def newfunc(*args): + return self.f(klass, *args) + return newfunc diff --git a/Doc/howto/functional.rst b/Doc/howto/functional.rst index 1969b32108..6e21f93544 100644 --- a/Doc/howto/functional.rst +++ b/Doc/howto/functional.rst @@ -332,7 +332,7 @@ substring. List comprehensions and generator expressions (short form: "listcomps" and "genexps") are a concise notation for such operations, borrowed from the -functional programming language Haskell (http://www.haskell.org/). You can strip +functional programming language Haskell (https://www.haskell.org/). You can strip all the whitespace from a stream of strings with the following code:: line_list = [' line 1\n', 'line 2 \n', ...] @@ -395,14 +395,14 @@ equivalent to the following Python code:: continue # Skip this element for expr2 in sequence2: if not (condition2): - continue # Skip this element + continue # Skip this element ... for exprN in sequenceN: - if not (conditionN): - continue # Skip this element + if not (conditionN): + continue # Skip this element - # Output the value of - # the expression. + # Output the value of + # the expression. This means that when there are multiple ``for...in`` clauses but no ``if`` clauses, the length of the resulting output will be equal to the product of the @@ -481,10 +481,10 @@ Here's a sample usage of the ``generate_ints()`` generator: You could equally write ``for i in generate_ints(5)``, or ``a,b,c = generate_ints(3)``. -Inside a generator function, ``return value`` is semantically equivalent to -``raise StopIteration(value)``. If no value is returned or the bottom of the -function is reached, the procession of values ends and the generator cannot -return any further values. +Inside a generator function, ``return value`` causes ``StopIteration(value)`` +to be raised from the :meth:`~generator.__next__` method. Once this happens, or +the bottom of the function is reached, the procession of values ends and the +generator cannot yield any further values. You could achieve the effect of generators manually by writing your own class and storing all the local variables of the generator as instance variables. For @@ -716,7 +716,7 @@ returns them in a tuple:: It doesn't construct an in-memory list and exhaust all the input iterators before returning; instead tuples are constructed and returned only if they're requested. (The technical term for this behaviour is `lazy evaluation -<http://en.wikipedia.org/wiki/Lazy_evaluation>`__.) +<https://en.wikipedia.org/wiki/Lazy_evaluation>`__.) This iterator is intended to be used with iterables that are all of the same length. If the iterables are of different lengths, the resulting stream will be @@ -1199,7 +1199,7 @@ General **Structure and Interpretation of Computer Programs**, by Harold Abelson and Gerald Jay Sussman with Julie Sussman. Full text at -http://mitpress.mit.edu/sicp/. In this classic textbook of computer science, +https://mitpress.mit.edu/sicp/. In this classic textbook of computer science, chapters 2 and 3 discuss the use of sequences and streams to organize the data flow inside a program. The book uses Scheme for its examples, but many of the design approaches described in these chapters are applicable to functional-style @@ -1208,12 +1208,12 @@ Python code. http://www.defmacro.org/ramblings/fp.html: A general introduction to functional programming that uses Java examples and has a lengthy historical introduction. -http://en.wikipedia.org/wiki/Functional_programming: General Wikipedia entry +https://en.wikipedia.org/wiki/Functional_programming: General Wikipedia entry describing functional programming. -http://en.wikipedia.org/wiki/Coroutine: Entry for coroutines. +https://en.wikipedia.org/wiki/Coroutine: Entry for coroutines. -http://en.wikipedia.org/wiki/Currying: Entry for the concept of currying. +https://en.wikipedia.org/wiki/Currying: Entry for the concept of currying. Python-specific --------------- @@ -1225,9 +1225,9 @@ Text Processing". Mertz also wrote a 3-part series of articles on functional programming for IBM's DeveloperWorks site; see -`part 1 <http://www.ibm.com/developerworks/linux/library/l-prog/index.html>`__, -`part 2 <http://www.ibm.com/developerworks/linux/library/l-prog2/index.html>`__, and -`part 3 <http://www.ibm.com/developerworks/linux/library/l-prog3/index.html>`__, +`part 1 <https://www.ibm.com/developerworks/linux/library/l-prog/index.html>`__, +`part 2 <https://www.ibm.com/developerworks/linux/library/l-prog2/index.html>`__, and +`part 3 <https://www.ibm.com/developerworks/linux/library/l-prog3/index.html>`__, Python documentation diff --git a/Doc/howto/index.rst b/Doc/howto/index.rst index 2c9d69910a..de659505f7 100644 --- a/Doc/howto/index.rst +++ b/Doc/howto/index.rst @@ -25,7 +25,6 @@ Currently, the HOWTOs are: sorting.rst unicode.rst urllib2.rst - webservers.rst argparse.rst ipaddress.rst clinic.rst diff --git a/Doc/howto/logging-cookbook.rst b/Doc/howto/logging-cookbook.rst index af888c2b24..de0d304b86 100644 --- a/Doc/howto/logging-cookbook.rst +++ b/Doc/howto/logging-cookbook.rst @@ -63,6 +63,7 @@ Here is the auxiliary module:: def __init__(self): self.logger = logging.getLogger('spam_application.auxiliary.Auxiliary') self.logger.info('creating an instance of Auxiliary') + def do_something(self): self.logger.info('doing something') a = 1 + 1 @@ -94,6 +95,61 @@ The output looks like this:: 2005-03-23 23:47:11,673 - spam_application - INFO - done with auxiliary_module.some_function() +Logging from multiple threads +----------------------------- + +Logging from multiple threads requires no special effort. The following example +shows logging from the main (initial) thread and another thread:: + + import logging + import threading + import time + + def worker(arg): + while not arg['stop']: + logging.debug('Hi from myfunc') + time.sleep(0.5) + + def main(): + logging.basicConfig(level=logging.DEBUG, format='%(relativeCreated)6d %(threadName)s %(message)s') + info = {'stop': False} + thread = threading.Thread(target=worker, args=(info,)) + thread.start() + while True: + try: + logging.debug('Hello from main') + time.sleep(0.75) + except KeyboardInterrupt: + info['stop'] = True + break + thread.join() + + if __name__ == '__main__': + main() + +When run, the script should print something like the following:: + + 0 Thread-1 Hi from myfunc + 3 MainThread Hello from main + 505 Thread-1 Hi from myfunc + 755 MainThread Hello from main + 1007 Thread-1 Hi from myfunc + 1507 MainThread Hello from main + 1508 Thread-1 Hi from myfunc + 2010 Thread-1 Hi from myfunc + 2258 MainThread Hello from main + 2512 Thread-1 Hi from myfunc + 3009 MainThread Hello from main + 3013 Thread-1 Hi from myfunc + 3515 Thread-1 Hi from myfunc + 3761 MainThread Hello from main + 4017 Thread-1 Hi from myfunc + 4513 MainThread Hello from main + 4518 Thread-1 Hi from myfunc + +This shows the logging output interspersed as one might expect. This approach +works for more threads than shown here, of course. + Multiple handlers and formatters -------------------------------- @@ -305,7 +361,7 @@ classes, which would eat up one thread per handler for no particular benefit. An example of using these two classes follows (imports omitted):: - que = queue.Queue(-1) # no limit on size + que = queue.Queue(-1) # no limit on size queue_handler = QueueHandler(que) handler = logging.StreamHandler() listener = QueueListener(que, handler) @@ -321,10 +377,21 @@ An example of using these two classes follows (imports omitted):: root.warning('Look out!') listener.stop() -which, when run, will produce:: +which, when run, will produce: + +.. code-block:: none MainThread: Look out! +.. versionchanged:: 3.5 + Prior to Python 3.5, the :class:`QueueListener` always passed every message + received from the queue to every handler it was initialized with. (This was + because it was assumed that level filtering was all done on the other side, + where the queue is filled.) From 3.5 onwards, this behaviour can be changed + by passing a keyword argument ``respect_handler_level=True`` to the + listener's constructor. When this is done, the listener compares the level + of each message with the handler's level, and only passes a message to a + handler if it's appropriate to do so. .. _network-logging: @@ -592,21 +659,21 @@ script:: return True if __name__ == '__main__': - levels = (logging.DEBUG, logging.INFO, logging.WARNING, logging.ERROR, logging.CRITICAL) - logging.basicConfig(level=logging.DEBUG, - format='%(asctime)-15s %(name)-5s %(levelname)-8s IP: %(ip)-15s User: %(user)-8s %(message)s') - a1 = logging.getLogger('a.b.c') - a2 = logging.getLogger('d.e.f') - - f = ContextFilter() - a1.addFilter(f) - a2.addFilter(f) - a1.debug('A debug message') - a1.info('An info message with %s', 'some parameters') - for x in range(10): - lvl = choice(levels) - lvlname = logging.getLevelName(lvl) - a2.log(lvl, 'A message at %s level with %d %s', lvlname, 2, 'parameters') + levels = (logging.DEBUG, logging.INFO, logging.WARNING, logging.ERROR, logging.CRITICAL) + logging.basicConfig(level=logging.DEBUG, + format='%(asctime)-15s %(name)-5s %(levelname)-8s IP: %(ip)-15s User: %(user)-8s %(message)s') + a1 = logging.getLogger('a.b.c') + a2 = logging.getLogger('d.e.f') + + f = ContextFilter() + a1.addFilter(f) + a2.addFilter(f) + a1.debug('A debug message') + a1.info('An info message with %s', 'some parameters') + for x in range(10): + lvl = choice(levels) + lvlname = logging.getLevelName(lvl) + a2.log(lvl, 'A message at %s level with %d %s', lvlname, 2, 'parameters') which, when run, produces something like:: @@ -700,10 +767,10 @@ the basis for code meeting your own specific requirements:: while True: try: record = queue.get() - if record is None: # We send this as a sentinel to tell the listener to quit. + if record is None: # We send this as a sentinel to tell the listener to quit. break logger = logging.getLogger(record.name) - logger.handle(record) # No level or filter logic applied - just do it! + logger.handle(record) # No level or filter logic applied - just do it! except Exception: import sys, traceback print('Whoops! Problem:', file=sys.stderr) @@ -726,10 +793,11 @@ the basis for code meeting your own specific requirements:: # Note that on Windows you can't rely on fork semantics, so each process # will run the logging configuration code when it starts. def worker_configurer(queue): - h = logging.handlers.QueueHandler(queue) # Just the one handler needed + h = logging.handlers.QueueHandler(queue) # Just the one handler needed root = logging.getLogger() root.addHandler(h) - root.setLevel(logging.DEBUG) # send all messages, for demo; no other level or filter logic applied. + # send all messages, for demo; no other level or filter logic applied. + root.setLevel(logging.DEBUG) # This is the worker process top-level loop, which just logs ten events with # random intervening delays before terminating. @@ -757,7 +825,7 @@ the basis for code meeting your own specific requirements:: workers = [] for i in range(10): worker = multiprocessing.Process(target=worker_process, - args=(queue, worker_configurer)) + args=(queue, worker_configurer)) workers.append(worker) worker.start() for w in workers: @@ -1181,12 +1249,12 @@ You can use a :class:`QueueHandler` subclass to send messages to other kinds of queues, for example a ZeroMQ 'publish' socket. In the example below,the socket is created separately and passed to the handler (as its 'queue'):: - import zmq # using pyzmq, the Python binding for ZeroMQ - import json # for serializing records portably + import zmq # using pyzmq, the Python binding for ZeroMQ + import json # for serializing records portably ctx = zmq.Context() - sock = zmq.Socket(ctx, zmq.PUB) # or zmq.PUSH, or other suitable value - sock.bind('tcp://*:5556') # or wherever + sock = zmq.Socket(ctx, zmq.PUB) # or zmq.PUSH, or other suitable value + sock.bind('tcp://*:5556') # or wherever class ZeroMQSocketHandler(QueueHandler): def enqueue(self, record): @@ -1224,7 +1292,7 @@ of queues, for example a ZeroMQ 'subscribe' socket. Here's an example:: def __init__(self, uri, *handlers, **kwargs): self.ctx = kwargs.get('ctx') or zmq.Context() socket = zmq.Socket(self.ctx, zmq.SUB) - socket.setsockopt(zmq.SUBSCRIBE, '') # subscribe to everything + socket.setsockopt(zmq.SUBSCRIBE, '') # subscribe to everything socket.connect(uri) def dequeue(self): @@ -1252,7 +1320,7 @@ An example dictionary-based configuration ----------------------------------------- Below is an example of a logging configuration dictionary - it's taken from -the `documentation on the Django project <https://docs.djangoproject.com/en/1.3/topics/logging/#configuring-logging>`_. +the `documentation on the Django project <https://docs.djangoproject.com/en/1.9/topics/logging/#configuring-logging>`_. This dictionary is passed to :func:`~config.dictConfig` to put the configuration into effect:: LOGGING = { @@ -1308,7 +1376,7 @@ This dictionary is passed to :func:`~config.dictConfig` to put the configuration } For more information about this configuration, you can see the `relevant -section <https://docs.djangoproject.com/en/1.6/topics/logging/#configuring-logging>`_ +section <https://docs.djangoproject.com/en/1.9/topics/logging/#configuring-logging>`_ of the Django documentation. .. _cookbook-rotator-namer: @@ -1570,11 +1638,11 @@ works:: Inserting a BOM into messages sent to a SysLogHandler ----------------------------------------------------- -`RFC 5424 <http://tools.ietf.org/html/rfc5424>`_ requires that a +`RFC 5424 <https://tools.ietf.org/html/rfc5424>`_ requires that a Unicode message be sent to a syslog daemon as a set of bytes which have the following structure: an optional pure-ASCII component, followed by a UTF-8 Byte Order Mark (BOM), followed by Unicode encoded using UTF-8. (See the `relevant -section of the specification <http://tools.ietf.org/html/rfc5424#section-6>`_.) +section of the specification <https://tools.ietf.org/html/rfc5424#section-6>`_.) In Python 3.1, code was added to :class:`~logging.handlers.SysLogHandler` to insert a BOM into the message, but @@ -1680,7 +1748,7 @@ as in the following complete example:: def main(): logging.basicConfig(level=logging.INFO, format='%(message)s') - logging.info(_('message 1', set_value=set([1, 2, 3]), snowman='\u2603')) + logging.info(_('message 1', set_value={1, 2, 3}, snowman='\u2603')) if __name__ == '__main__': main() @@ -1794,7 +1862,9 @@ script, ``chowntest.py``:: logger = logging.getLogger('mylogger') logger.debug('A debug message') -To run this, you will probably need to run as ``root``:: +To run this, you will probably need to run as ``root``: + +.. code-block:: shell-session $ sudo python3.3 chowntest.py $ cat chowntest.log @@ -2052,7 +2122,7 @@ class, as shown in the following example:: Format an exception so that it prints on a single line. """ result = super(OneLineExceptionFormatter, self).formatException(exc_info) - return repr(result) # or format into one line however you want to + return repr(result) # or format into one line however you want to def format(self, record): s = super(OneLineExceptionFormatter, self).format(record) @@ -2167,7 +2237,7 @@ flushing behavior. The example script has a simple function, ``foo``, which just cycles through all the logging levels, writing to ``sys.stderr`` to say what level it's about -to log at, and then actually logging a message that that level. You can pass a +to log at, and then actually logging a message at that level. You can pass a parameter to ``foo`` which, if true, will log at ERROR and CRITICAL levels - otherwise, it only logs at DEBUG, INFO and WARNING levels. @@ -2345,3 +2415,111 @@ When this script is run, it should print something like:: showing how the time is formatted both as local time and UTC, one for each handler. + + +.. _context-manager: + +Using a context manager for selective logging +--------------------------------------------- + +There are times when it would be useful to temporarily change the logging +configuration and revert it back after doing something. For this, a context +manager is the most obvious way of saving and restoring the logging context. +Here is a simple example of such a context manager, which allows you to +optionally change the logging level and add a logging handler purely in the +scope of the context manager:: + + import logging + import sys + + class LoggingContext(object): + def __init__(self, logger, level=None, handler=None, close=True): + self.logger = logger + self.level = level + self.handler = handler + self.close = close + + def __enter__(self): + if self.level is not None: + self.old_level = self.logger.level + self.logger.setLevel(self.level) + if self.handler: + self.logger.addHandler(self.handler) + + def __exit__(self, et, ev, tb): + if self.level is not None: + self.logger.setLevel(self.old_level) + if self.handler: + self.logger.removeHandler(self.handler) + if self.handler and self.close: + self.handler.close() + # implicit return of None => don't swallow exceptions + +If you specify a level value, the logger's level is set to that value in the +scope of the with block covered by the context manager. If you specify a +handler, it is added to the logger on entry to the block and removed on exit +from the block. You can also ask the manager to close the handler for you on +block exit - you could do this if you don't need the handler any more. + +To illustrate how it works, we can add the following block of code to the +above:: + + if __name__ == '__main__': + logger = logging.getLogger('foo') + logger.addHandler(logging.StreamHandler()) + logger.setLevel(logging.INFO) + logger.info('1. This should appear just once on stderr.') + logger.debug('2. This should not appear.') + with LoggingContext(logger, level=logging.DEBUG): + logger.debug('3. This should appear once on stderr.') + logger.debug('4. This should not appear.') + h = logging.StreamHandler(sys.stdout) + with LoggingContext(logger, level=logging.DEBUG, handler=h, close=True): + logger.debug('5. This should appear twice - once on stderr and once on stdout.') + logger.info('6. This should appear just once on stderr.') + logger.debug('7. This should not appear.') + +We initially set the logger's level to ``INFO``, so message #1 appears and +message #2 doesn't. We then change the level to ``DEBUG`` temporarily in the +following ``with`` block, and so message #3 appears. After the block exits, the +logger's level is restored to ``INFO`` and so message #4 doesn't appear. In the +next ``with`` block, we set the level to ``DEBUG`` again but also add a handler +writing to ``sys.stdout``. Thus, message #5 appears twice on the console (once +via ``stderr`` and once via ``stdout``). After the ``with`` statement's +completion, the status is as it was before so message #6 appears (like message +#1) whereas message #7 doesn't (just like message #2). + +If we run the resulting script, the result is as follows: + +.. code-block:: shell-session + + $ python logctx.py + 1. This should appear just once on stderr. + 3. This should appear once on stderr. + 5. This should appear twice - once on stderr and once on stdout. + 5. This should appear twice - once on stderr and once on stdout. + 6. This should appear just once on stderr. + +If we run it again, but pipe ``stderr`` to ``/dev/null``, we see the following, +which is the only message written to ``stdout``: + +.. code-block:: shell-session + + $ python logctx.py 2>/dev/null + 5. This should appear twice - once on stderr and once on stdout. + +Once again, but piping ``stdout`` to ``/dev/null``, we get: + +.. code-block:: shell-session + + $ python logctx.py >/dev/null + 1. This should appear just once on stderr. + 3. This should appear once on stderr. + 5. This should appear twice - once on stderr and once on stdout. + 6. This should appear just once on stderr. + +In this case, the message #5 printed to ``stdout`` doesn't appear, as expected. + +Of course, the approach described here can be generalised, for example to attach +logging filters temporarily. Note that the above code works in Python 2 as well +as Python 3. diff --git a/Doc/howto/logging.rst b/Doc/howto/logging.rst index 4ce14f98ad..82d130823c 100644 --- a/Doc/howto/logging.rst +++ b/Doc/howto/logging.rst @@ -103,10 +103,12 @@ A simple example A very simple example is:: import logging - logging.warning('Watch out!') # will print a message to the console - logging.info('I told you so') # will not print anything + logging.warning('Watch out!') # will print a message to the console + logging.info('I told you so') # will not print anything -If you type these lines into a script and run it, you'll see:: +If you type these lines into a script and run it, you'll see: + +.. code-block:: none WARNING:root:Watch out! @@ -230,7 +232,9 @@ append the variable data as arguments. For example:: import logging logging.warning('%s before you %s', 'Look', 'leap!') -will display:: +will display: + +.. code-block:: none WARNING:root:Look before you leap! @@ -310,7 +314,7 @@ favourite beverage and carry on. If your logging needs are simple, then use the above examples to incorporate logging into your own scripts, and if you run into problems or don't understand something, please post a question on the comp.lang.python Usenet -group (available at http://groups.google.com/group/comp.lang.python) and you +group (available at https://groups.google.com/group/comp.lang.python) and you should receive help before too long. Still here? You can carry on reading the next few sections, which provide a @@ -594,7 +598,9 @@ logger, a console handler, and a simple formatter using Python code:: logger.error('error message') logger.critical('critical message') -Running this module from the command line produces the following output:: +Running this module from the command line produces the following output: + +.. code-block:: shell-session $ python simple_logging_module.py 2005-03-19 15:10:26,618 - simple_example - DEBUG - debug message @@ -653,7 +659,9 @@ Here is the logging.conf file:: format=%(asctime)s - %(name)s - %(levelname)s - %(message)s datefmt= -The output is nearly identical to that of the non-config-file-based example:: +The output is nearly identical to that of the non-config-file-based example: + +.. code-block:: shell-session $ python simple_logging_config.py 2005-03-19 15:38:55,977 - simpleExample - DEBUG - debug message @@ -1073,4 +1081,3 @@ take up any memory. Useful handlers included with the logging module. :ref:`A logging cookbook <logging-cookbook>` - diff --git a/Doc/howto/pyporting.rst b/Doc/howto/pyporting.rst index 5e875cdee2..c479f2264e 100644 --- a/Doc/howto/pyporting.rst +++ b/Doc/howto/pyporting.rst @@ -207,13 +207,12 @@ that's ``str``/``bytes`` in Python 2 and ``bytes`` in Python 3). The following table lists the **unique** methods of each data type across Python 2 & 3 (e.g., the ``decode()`` method is usable on the equivalent binary data type in either Python 2 or 3, but it can't be used by the text data type consistently -between Python 2 and 3 because ``str`` in Python 3 doesn't have the method). +between Python 2 and 3 because ``str`` in Python 3 doesn't have the method). Do +note that as of Python 3.5 the ``__mod__`` method was added to the bytes type. ======================== ===================== **Text data** **Binary data** ------------------------ --------------------- -__mod__ (``%`` operator) ------------------------- --------------------- \ decode ------------------------ --------------------- encode @@ -244,8 +243,8 @@ bothered to add the ``b`` mode when opening a binary file (e.g., ``rb`` for binary reading). Under Python 3, binary files and text files are clearly distinct and mutually incompatible; see the :mod:`io` module for details. Therefore, you **must** make a decision of whether a file will be used for -binary access (allowing to read and/or write binary data) or text access -(allowing to read and/or write text data). You should also use :func:`io.open` +binary access (allowing binary data to be read and/or written) or text access +(allowing text data to be read and/or written). You should also use :func:`io.open` for opening files instead of the built-in :func:`open` function as the :mod:`io` module is consistent from Python 2 to 3 while the built-in :func:`open` function is not (in Python 3 it's actually :func:`io.open`). @@ -283,6 +282,50 @@ To summarize: appropriate #. Be careful when indexing binary data + +Use feature detection instead of version detection +++++++++++++++++++++++++++++++++++++++++++++++++++ +Inevitably you will have code that has to choose what to do based on what +version of Python is running. The best way to do this is with feature detection +of whether the version of Python you're running under supports what you need. +If for some reason that doesn't work then you should make the version check is +against Python 2 and not Python 3. To help explain this, let's look at an +example. + +Let's pretend that you need access to a feature of importlib_ that +is available in Python's standard library since Python 3.3 and available for +Python 2 through importlib2_ on PyPI. You might be tempted to write code to +access e.g. the ``importlib.abc`` module by doing the following:: + + import sys + + if sys.version[0] == 3: + from importlib import abc + else: + from importlib2 import abc + +The problem with this code is what happens when Python 4 comes out? It would +be better to treat Python 2 as the exceptional case instead of Python 3 and +assume that future Python versions will be more compatible with Python 3 than +Python 2:: + + import sys + + if sys.version[0] > 2: + from importlib import abc + else: + from importlib2 import abc + +The best solution, though, is to do no version detection at all and instead rely +on feature detection. That avoids any potential issues of getting the version +detection wrong and helps keep you future-compatible:: + + try: + from importlib import abc + except ImportError: + from importlib2 import abc + + Prevent compatibility regressions --------------------------------- @@ -347,11 +390,13 @@ your tests under multiple Python interpreters is tox_. You can then integrate tox with your continuous integration system so that you never accidentally break Python 2 or 3 support. -You may also want to use use the ``-bb`` flag with the Python 3 interpreter to -trigger an exception when you are comparing bytes to strings. Usually it's -simply ``False``, but if you made a mistake in your separation of text/binary -data handling you may be accidentally comparing text and binary data. This flag -will raise an exception when that occurs to help track down such cases. +You may also want to use the ``-bb`` flag with the Python 3 interpreter to +trigger an exception when you are comparing bytes to strings or bytes to an int +(the latter is available starting in Python 3.5). By default type-differing +comparisons simply return ``False``, but if you made a mistake in your +separation of text/binary data handling or indexing on bytes you wouldn't easily +find the mistake. This flag will raise an exception when these kinds of +comparisons occur, making the mistake much easier to track down. And that's mostly it! At this point your code base is compatible with both Python 2 and 3 simultaneously. Your testing will also be set up so that you @@ -380,10 +425,12 @@ supported by Python 2. You should also update the classifiers in your .. _cheat sheet: http://python-future.org/compatible_idioms.html .. _coverage.py: https://pypi.python.org/pypi/coverage .. _Futurize: http://python-future.org/automatic_conversion.html -.. _Modernize: http://python-modernize.readthedocs.org/en/latest/ +.. _importlib: https://docs.python.org/3/library/importlib.html#module-importlib +.. _importlib2: https://pypi.python.org/pypi/importlib2 +.. _Modernize: https://python-modernize.readthedocs.org/en/latest/ .. _Porting to Python 3: http://python3porting.com/ .. _Pylint: https://pypi.python.org/pypi/pylint -.. _Python 3 Q & A: http://ncoghlan-devs-python-notes.readthedocs.org/en/latest/python3/questions_and_answers.html +.. _Python 3 Q & A: https://ncoghlan-devs-python-notes.readthedocs.org/en/latest/python3/questions_and_answers.html .. _python-future: http://python-future.org/ .. _python-porting: https://mail.python.org/mailman/listinfo/python-porting diff --git a/Doc/howto/regex.rst b/Doc/howto/regex.rst index 9ae04d718d..d9b7c9091d 100644 --- a/Doc/howto/regex.rst +++ b/Doc/howto/regex.rst @@ -74,7 +74,9 @@ of the RE by repeating them or changing their meaning. Much of this document is devoted to discussing various metacharacters and what they do. Here's a complete list of the metacharacters; their meanings will be discussed -in the rest of this HOWTO. :: +in the rest of this HOWTO. + +.. code-block:: none . ^ $ * + ? { } [ ] \ | ( ) @@ -178,7 +180,7 @@ are usually not written to match that much data. Repetitions such as ``*`` are :dfn:`greedy`; when repeating a RE, the matching engine will try to repeat it as many times as possible. If later portions of the pattern don't match, the matching engine will then back up and try again with -few repetitions. +fewer repetitions. A step-by-step example will make this more obvious. Let's consider the expression ``a[bcd]*b``. This matches the letter ``'a'``, zero or more letters @@ -374,9 +376,7 @@ module. If you have :mod:`tkinter` available, you may also want to look at :source:`Tools/demo/redemo.py`, a demonstration program included with the Python distribution. It allows you to enter REs and strings, and displays whether the RE matches or fails. :file:`redemo.py` can be quite useful when -trying to debug a complicated RE. Phil Schwartz's `Kodos -<http://kodos.sourceforge.net/>`_ is also an interactive tool for developing and -testing RE patterns. +trying to debug a complicated RE. This HOWTO uses the standard Python interpreter for its examples. First, run the Python interpreter, import the :mod:`re` module, and compile a RE:: @@ -1004,17 +1004,18 @@ confusing. A negative lookahead cuts through all this confusion: -``.*[.](?!bat$).*$`` The negative lookahead means: if the expression ``bat`` +``.*[.](?!bat$)[^.]*$`` The negative lookahead means: if the expression ``bat`` doesn't match at this point, try the rest of the pattern; if ``bat$`` does match, the whole pattern will fail. The trailing ``$`` is required to ensure that something like ``sample.batch``, where the extension only starts with -``bat``, will be allowed. +``bat``, will be allowed. The ``[^.]*`` makes sure that the pattern works +when there are multiple dots in the filename. Excluding another filename extension is now easy; simply add it as an alternative inside the assertion. The following pattern excludes filenames that end in either ``bat`` or ``exe``: -``.*[.](?!bat$|exe$).*$`` +``.*[.](?!bat$|exe$)[^.]*$`` Modifying Strings @@ -1114,19 +1115,19 @@ which can be either a string or a function, and the string to be processed. Here's a simple example of using the :meth:`sub` method. It replaces colour names with the word ``colour``:: - >>> p = re.compile( '(blue|white|red)') - >>> p.sub( 'colour', 'blue socks and red shoes') + >>> p = re.compile('(blue|white|red)') + >>> p.sub('colour', 'blue socks and red shoes') 'colour socks and colour shoes' - >>> p.sub( 'colour', 'blue socks and red shoes', count=1) + >>> p.sub('colour', 'blue socks and red shoes', count=1) 'colour socks and red shoes' The :meth:`subn` method does the same work, but returns a 2-tuple containing the new string value and the number of replacements that were performed:: - >>> p = re.compile( '(blue|white|red)') - >>> p.subn( 'colour', 'blue socks and red shoes') + >>> p = re.compile('(blue|white|red)') + >>> p.subn('colour', 'blue socks and red shoes') ('colour socks and colour shoes', 2) - >>> p.subn( 'colour', 'no colours at all') + >>> p.subn('colour', 'no colours at all') ('no colours at all', 0) Empty matches are replaced only when they're not adjacent to a previous match. @@ -1138,7 +1139,7 @@ Empty matches are replaced only when they're not adjacent to a previous match. If *replacement* is a string, any backslash escapes in it are processed. That is, ``\n`` is converted to a single newline character, ``\r`` is converted to a -carriage return, and so forth. Unknown escapes such as ``\j`` are left alone. +carriage return, and so forth. Unknown escapes such as ``\&`` are left alone. Backreferences, such as ``\6``, are replaced with the substring matched by the corresponding group in the RE. This lets you incorporate portions of the original text in the resulting replacement string. diff --git a/Doc/howto/sockets.rst b/Doc/howto/sockets.rst index 04394d49f0..bc71d85a83 100644 --- a/Doc/howto/sockets.rst +++ b/Doc/howto/sockets.rst @@ -106,7 +106,7 @@ mainloop of the web server:: There's actually 3 general ways in which this loop could work - dispatching a thread to handle ``clientsocket``, create a new process to handle ``clientsocket``, or restructure this app to use non-blocking sockets, and -mulitplex between our "server" socket and any active ``clientsocket``\ s using +multiplex between our "server" socket and any active ``clientsocket``\ s using ``select``. More about that later. The important thing to understand now is this: this is *all* a "server" socket does. It doesn't send any data. It doesn't receive any data. It just produces "client" sockets. Each ``clientsocket`` is diff --git a/Doc/howto/sorting.rst b/Doc/howto/sorting.rst index f2e64ee98b..0334b26657 100644 --- a/Doc/howto/sorting.rst +++ b/Doc/howto/sorting.rst @@ -127,7 +127,7 @@ Sort Stability and Complex Sorts ================================ Sorts are guaranteed to be `stable -<http://en.wikipedia.org/wiki/Sorting_algorithm#Stability>`_\. That means that +<https://en.wikipedia.org/wiki/Sorting_algorithm#Stability>`_\. That means that when multiple records have the same key, their original order is preserved. >>> data = [('red', 1), ('blue', 1), ('red', 2), ('blue', 2)] @@ -145,7 +145,7 @@ ascending *age*, do the *age* sort first and then sort again using *grade*: >>> sorted(s, key=attrgetter('grade'), reverse=True) # now sort on primary key, descending [('dave', 'B', 10), ('jane', 'B', 12), ('john', 'A', 15)] -The `Timsort <http://en.wikipedia.org/wiki/Timsort>`_ algorithm used in Python +The `Timsort <https://en.wikipedia.org/wiki/Timsort>`_ algorithm used in Python does multiple sorts efficiently because it can take advantage of any ordering already present in a dataset. @@ -184,7 +184,7 @@ decorated list, but including it gives two benefits: directly. Another name for this idiom is -`Schwartzian transform <http://en.wikipedia.org/wiki/Schwartzian_transform>`_\, +`Schwartzian transform <https://en.wikipedia.org/wiki/Schwartzian_transform>`_\, after Randal L. Schwartz, who popularized it among Perl programmers. Now that Python sorting provides key-functions, this technique is not often needed. @@ -262,7 +262,11 @@ Odd and Ends twice: >>> data = [('red', 1), ('blue', 1), ('red', 2), ('blue', 2)] - >>> assert sorted(data, reverse=True) == list(reversed(sorted(reversed(data)))) + >>> standard_way = sorted(data, key=itemgetter(0), reverse=True) + >>> double_reversed = list(reversed(sorted(reversed(data), key=itemgetter(0)))) + >>> assert standard_way == double_reversed + >>> standard_way + [('red', 1), ('red', 2), ('blue', 1), ('blue', 2)] * The sort routines are guaranteed to use :meth:`__lt__` when making comparisons between two objects. So, it is easy to add a standard sort order to a class by diff --git a/Doc/howto/unicode.rst b/Doc/howto/unicode.rst index b49ac391ce..fbeb11aac9 100644 --- a/Doc/howto/unicode.rst +++ b/Doc/howto/unicode.rst @@ -73,7 +73,7 @@ revision of Unicode. precise historical details aren't necessary for understanding how to use Unicode effectively, but if you're curious, consult the Unicode consortium site listed in the References or -the `Wikipedia entry for Unicode <http://en.wikipedia.org/wiki/Unicode#History>`_ +the `Wikipedia entry for Unicode <https://en.wikipedia.org/wiki/Unicode#History>`_ for more information.) @@ -214,7 +214,7 @@ difficult reading. `A chronology <http://www.unicode.org/history/>`_ of the origin and development of Unicode is also available on the site. To help understand the standard, Jukka Korpela has written `an introductory -guide <http://www.cs.tut.fi/~jkorpela/unicode/guide.html>`_ to reading the +guide <https://www.cs.tut.fi/~jkorpela/unicode/guide.html>`_ to reading the Unicode character tables. Another `good introductory article <http://www.joelonsoftware.com/articles/Unicode.html>`_ @@ -223,8 +223,8 @@ If this introduction didn't make things clear to you, you should try reading this alternate article before continuing. Wikipedia entries are often helpful; see the entries for "`character encoding -<http://en.wikipedia.org/wiki/Character_encoding>`_" and `UTF-8 -<http://en.wikipedia.org/wiki/UTF-8>`_, for example. +<https://en.wikipedia.org/wiki/Character_encoding>`_" and `UTF-8 +<https://en.wikipedia.org/wiki/UTF-8>`_, for example. Python's Unicode Support @@ -280,8 +280,9 @@ and optionally an *errors* argument. The *errors* argument specifies the response when the input string can't be converted according to the encoding's rules. Legal values for this argument are ``'strict'`` (raise a :exc:`UnicodeDecodeError` exception), ``'replace'`` (use -``U+FFFD``, ``REPLACEMENT CHARACTER``), or ``'ignore'`` (just leave the -character out of the Unicode result). +``U+FFFD``, ``REPLACEMENT CHARACTER``), ``'ignore'`` (just leave the +character out of the Unicode result), or ``'backslashreplace'`` (inserts a +``\xNN`` escape sequence). The following examples show the differences:: >>> b'\x80abc'.decode("utf-8", "strict") #doctest: +NORMALIZE_WHITESPACE @@ -291,12 +292,11 @@ The following examples show the differences:: invalid start byte >>> b'\x80abc'.decode("utf-8", "replace") '\ufffdabc' + >>> b'\x80abc'.decode("utf-8", "backslashreplace") + '\\x80abc' >>> b'\x80abc'.decode("utf-8", "ignore") 'abc' -(In this code example, the Unicode replacement character has been replaced by -a question mark because it may not be displayed on some systems.) - Encodings are specified as strings containing the encoding's name. Python 3.2 comes with roughly 100 different encodings; see the Python Library Reference at :ref:`standard-encodings` for a list. Some encodings have multiple names; for @@ -325,8 +325,9 @@ The *errors* parameter is the same as the parameter of the :meth:`~bytes.decode` method but supports a few more possible handlers. As well as ``'strict'``, ``'ignore'``, and ``'replace'`` (which in this case inserts a question mark instead of the unencodable character), there is -also ``'xmlcharrefreplace'`` (inserts an XML character reference) and -``backslashreplace`` (inserts a ``\uNNNN`` escape sequence). +also ``'xmlcharrefreplace'`` (inserts an XML character reference), +``backslashreplace`` (inserts a ``\uNNNN`` escape sequence) and +``namereplace`` (inserts a ``\N{...}`` escape sequence). The following example shows the different results:: @@ -346,6 +347,8 @@ The following example shows the different results:: b'ꀀabcd޴' >>> u.encode('ascii', 'backslashreplace') b'\\ua000abcd\\u07b4' + >>> u.encode('ascii', 'namereplace') + b'\\N{YI SYLLABLE IT}abcd\\u07b4' The low-level routines for registering and accessing the available encodings are found in the :mod:`codecs` module. Implementing new @@ -610,7 +613,9 @@ program:: print(os.listdir(b'.')) print(os.listdir('.')) -will produce the following output:: +will produce the following output: + +.. code-block:: shell-session amk:~$ python t.py [b'filename\xe4\x94\x80abc', ...] @@ -684,7 +689,7 @@ with the ``surrogateescape`` error handler:: # make changes to the string 'data' with open(fname + '.new', 'w', - encoding="ascii", errors="surrogateescape") as f: + encoding="ascii", errors="surrogateescape") as f: f.write(data) The ``surrogateescape`` error handler will decode any non-ASCII bytes diff --git a/Doc/howto/urllib2.rst b/Doc/howto/urllib2.rst index 4a67b30d89..d2c799196b 100644 --- a/Doc/howto/urllib2.rst +++ b/Doc/howto/urllib2.rst @@ -64,7 +64,7 @@ you can do so via the :func:`~urllib.request.urlretrieve` function:: html = open(local_filename) Many uses of urllib will be that simple (note that instead of an 'http:' URL we -could have used an URL starting with 'ftp:', 'file:', etc.). However, it's the +could have used a URL starting with 'ftp:', 'file:', etc.). However, it's the purpose of this tutorial to explain the more complicated cases, concentrating on HTTP. @@ -122,7 +122,7 @@ library. :: Note that other encodings are sometimes required (e.g. for file upload from HTML forms - see `HTML Specification, Form Submission -<http://www.w3.org/TR/REC-html40/interact/forms.html#h-17.13>`_ for more +<https://www.w3.org/TR/REC-html40/interact/forms.html#h-17.13>`_ for more details). If you do not pass the ``data`` argument, urllib uses a **GET** request. One @@ -175,10 +175,10 @@ Explorer [#]_. :: url = 'http://www.someserver.com/cgi-bin/register.cgi' user_agent = 'Mozilla/5.0 (Windows NT 6.1; Win64; x64)' - values = {'name' : 'Michael Foord', - 'location' : 'Northampton', - 'language' : 'Python' } - headers = { 'User-Agent' : user_agent } + values = {'name': 'Michael Foord', + 'location': 'Northampton', + 'language': 'Python' } + headers = {'User-Agent': user_agent} data = urllib.parse.urlencode(values) data = data.encode('ascii') @@ -215,7 +215,7 @@ e.g. :: >>> req = urllib.request.Request('http://www.pretend_server.org') >>> try: urllib.request.urlopen(req) ... except urllib.error.URLError as e: - ... print(e.reason) #doctest: +SKIP + ... print(e.reason) #doctest: +SKIP ... (4, 'getaddrinfo failed') @@ -372,7 +372,7 @@ Number 2 :: from urllib.request import Request, urlopen - from urllib.error import URLError + from urllib.error import URLError req = Request(someurl) try: response = urlopen(req) @@ -403,7 +403,7 @@ fetched, particularly the headers sent by the server. It is currently an :class:`http.client.HTTPMessage` instance. Typical headers include 'Content-length', 'Content-type', and so on. See the -`Quick Reference to HTTP Headers <http://www.cs.tut.fi/~jkorpela/http.html>`_ +`Quick Reference to HTTP Headers <https://www.cs.tut.fi/~jkorpela/http.html>`_ for a useful listing of HTTP headers with brief explanations of their meaning and use. @@ -514,7 +514,7 @@ component and the hostname and optionally the port number) e.g. "http://example.com/" *or* an "authority" (i.e. the hostname, optionally including the port number) e.g. "example.com" or "example.com:8080" (the latter example includes a port number). The authority, if present, must -NOT contain the "userinfo" component - for example "joe@password:example.com" is +NOT contain the "userinfo" component - for example "joe:password@example.com" is not correct. @@ -540,8 +540,8 @@ setting up a `Basic Authentication`_ handler: :: .. note:: - `HTTP_PROXY`` will be ignored if a variable ``REQUEST_METHOD`` is set; see - the documentation on :func:`~urllib.request.getproxies`. + ``HTTP_PROXY`` will be ignored if a variable ``REQUEST_METHOD`` is set; see + the documentation on :func:`~urllib.request.getproxies`. Sockets and Layers @@ -591,5 +591,5 @@ This document was reviewed and revised by John Lee. scripts with a localhost server, I have to prevent urllib from using the proxy. .. [#] urllib opener for SSL proxy (CONNECT method): `ASPN Cookbook Recipe - <http://code.activestate.com/recipes/456195/>`_. + <https://code.activestate.com/recipes/456195/>`_. diff --git a/Doc/howto/webservers.rst b/Doc/howto/webservers.rst deleted file mode 100644 index 9e9b69d9eb..0000000000 --- a/Doc/howto/webservers.rst +++ /dev/null @@ -1,731 +0,0 @@ -******************************* - HOWTO Use Python in the web -******************************* - -:Author: Marek Kubica - -.. topic:: Abstract - - This document shows how Python fits into the web. It presents some ways - to integrate Python with a web server, and general practices useful for - developing web sites. - - -Programming for the Web has become a hot topic since the rise of "Web 2.0", -which focuses on user-generated content on web sites. It has always been -possible to use Python for creating web sites, but it was a rather tedious task. -Therefore, many frameworks and helper tools have been created to assist -developers in creating faster and more robust sites. This HOWTO describes -some of the methods used to combine Python with a web server to create -dynamic content. It is not meant as a complete introduction, as this topic is -far too broad to be covered in one single document. However, a short overview -of the most popular libraries is provided. - -.. seealso:: - - While this HOWTO tries to give an overview of Python in the web, it cannot - always be as up to date as desired. Web development in Python is rapidly - moving forward, so the wiki page on `Web Programming - <https://wiki.python.org/moin/WebProgramming>`_ may be more in sync with - recent development. - - -The Low-Level View -================== - -When a user enters a web site, their browser makes a connection to the site's -web server (this is called the *request*). The server looks up the file in the -file system and sends it back to the user's browser, which displays it (this is -the *response*). This is roughly how the underlying protocol, HTTP, works. - -Dynamic web sites are not based on files in the file system, but rather on -programs which are run by the web server when a request comes in, and which -*generate* the content that is returned to the user. They can do all sorts of -useful things, like display the postings of a bulletin board, show your email, -configure software, or just display the current time. These programs can be -written in any programming language the server supports. Since most servers -support Python, it is easy to use Python to create dynamic web sites. - -Most HTTP servers are written in C or C++, so they cannot execute Python code -directly -- a bridge is needed between the server and the program. These -bridges, or rather interfaces, define how programs interact with the server. -There have been numerous attempts to create the best possible interface, but -there are only a few worth mentioning. - -Not every web server supports every interface. Many web servers only support -old, now-obsolete interfaces; however, they can often be extended using -third-party modules to support newer ones. - - -Common Gateway Interface ------------------------- - -This interface, most commonly referred to as "CGI", is the oldest, and is -supported by nearly every web server out of the box. Programs using CGI to -communicate with their web server need to be started by the server for every -request. So, every request starts a new Python interpreter -- which takes some -time to start up -- thus making the whole interface only usable for low load -situations. - -The upside of CGI is that it is simple -- writing a Python program which uses -CGI is a matter of about three lines of code. This simplicity comes at a -price: it does very few things to help the developer. - -Writing CGI programs, while still possible, is no longer recommended. With -:ref:`WSGI <WSGI>`, a topic covered later in this document, it is possible to write -programs that emulate CGI, so they can be run as CGI if no better option is -available. - -.. seealso:: - - The Python standard library includes some modules that are helpful for - creating plain CGI programs: - - * :mod:`cgi` -- Handling of user input in CGI scripts - * :mod:`cgitb` -- Displays nice tracebacks when errors happen in CGI - applications, instead of presenting a "500 Internal Server Error" message - - The Python wiki features a page on `CGI scripts - <https://wiki.python.org/moin/CgiScripts>`_ with some additional information - about CGI in Python. - - -Simple script for testing CGI -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -To test whether your web server works with CGI, you can use this short and -simple CGI program:: - - #!/usr/bin/env python - # -*- coding: UTF-8 -*- - - # enable debugging - import cgitb - cgitb.enable() - - print("Content-Type: text/plain;charset=utf-8") - print() - - print("Hello World!") - -Depending on your web server configuration, you may need to save this code with -a ``.py`` or ``.cgi`` extension. Additionally, this file may also need to be -in a ``cgi-bin`` folder, for security reasons. - -You might wonder what the ``cgitb`` line is about. This line makes it possible -to display a nice traceback instead of just crashing and displaying an "Internal -Server Error" in the user's browser. This is useful for debugging, but it might -risk exposing some confidential data to the user. You should not use ``cgitb`` -in production code for this reason. You should *always* catch exceptions, and -display proper error pages -- end-users don't like to see nondescript "Internal -Server Errors" in their browsers. - - -Setting up CGI on your own server -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -If you don't have your own web server, this does not apply to you. You can -check whether it works as-is, and if not you will need to talk to the -administrator of your web server. If it is a big host, you can try filing a -ticket asking for Python support. - -If you are your own administrator or want to set up CGI for testing purposes on -your own computers, you have to configure it by yourself. There is no single -way to configure CGI, as there are many web servers with different -configuration options. Currently the most widely used free web server is -`Apache HTTPd <http://httpd.apache.org/>`_, or Apache for short. Apache can be -easily installed on nearly every system using the system's package management -tool. `lighttpd <http://www.lighttpd.net>`_ is another alternative and is -said to have better performance. On many systems this server can also be -installed using the package management tool, so manually compiling the web -server may not be needed. - -* On Apache you can take a look at the `Dynamic Content with CGI - <http://httpd.apache.org/docs/2.2/howto/cgi.html>`_ tutorial, where everything - is described. Most of the time it is enough just to set ``+ExecCGI``. The - tutorial also describes the most common gotchas that might arise. - -* On lighttpd you need to use the `CGI module - <http://redmine.lighttpd.net/projects/lighttpd/wiki/Docs_ModCGI>`_\ , which can be configured - in a straightforward way. It boils down to setting ``cgi.assign`` properly. - - -Common problems with CGI scripts -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Using CGI sometimes leads to small annoyances while trying to get these -scripts to run. Sometimes a seemingly correct script does not work as -expected, the cause being some small hidden problem that's difficult to spot. - -Some of these potential problems are: - -* The Python script is not marked as executable. When CGI scripts are not - executable most web servers will let the user download it, instead of - running it and sending the output to the user. For CGI scripts to run - properly on Unix-like operating systems, the ``+x`` bit needs to be set. - Using ``chmod a+x your_script.py`` may solve this problem. - -* On a Unix-like system, The line endings in the program file must be Unix - style line endings. This is important because the web server checks the - first line of the script (called shebang) and tries to run the program - specified there. It gets easily confused by Windows line endings (Carriage - Return & Line Feed, also called CRLF), so you have to convert the file to - Unix line endings (only Line Feed, LF). This can be done automatically by - uploading the file via FTP in text mode instead of binary mode, but the - preferred way is just telling your editor to save the files with Unix line - endings. Most editors support this. - -* Your web server must be able to read the file, and you need to make sure the - permissions are correct. On unix-like systems, the server often runs as user - and group ``www-data``, so it might be worth a try to change the file - ownership, or making the file world readable by using ``chmod a+r - your_script.py``. - -* The web server must know that the file you're trying to access is a CGI script. - Check the configuration of your web server, as it may be configured - to expect a specific file extension for CGI scripts. - -* On Unix-like systems, the path to the interpreter in the shebang - (``#!/usr/bin/env python``) must be correct. This line calls - ``/usr/bin/env`` to find Python, but it will fail if there is no - ``/usr/bin/env``, or if Python is not in the web server's path. If you know - where your Python is installed, you can also use that full path. The - commands ``whereis python`` and ``type -p python`` could help you find - where it is installed. Once you know the path, you can change the shebang - accordingly: ``#!/usr/bin/python``. - -* The file must not contain a BOM (Byte Order Mark). The BOM is meant for - determining the byte order of UTF-16 and UTF-32 encodings, but some editors - write this also into UTF-8 files. The BOM interferes with the shebang line, - so be sure to tell your editor not to write the BOM. - -* If the web server is using :ref:`mod-python`, ``mod_python`` may be having - problems. ``mod_python`` is able to handle CGI scripts by itself, but it can - also be a source of issues. - - -.. _mod-python: - -mod_python ----------- - -People coming from PHP often find it hard to grasp how to use Python in the web. -Their first thought is mostly `mod_python <http://modpython.org/>`_\ , -because they think that this is the equivalent to ``mod_php``. Actually, there -are many differences. What ``mod_python`` does is embed the interpreter into -the Apache process, thus speeding up requests by not having to start a Python -interpreter for each request. On the other hand, it is not "Python intermixed -with HTML" in the way that PHP is often intermixed with HTML. The Python -equivalent of that is a template engine. ``mod_python`` itself is much more -powerful and provides more access to Apache internals. It can emulate CGI, -work in a "Python Server Pages" mode (similar to JSP) which is "HTML -intermingled with Python", and it has a "Publisher" which designates one file -to accept all requests and decide what to do with them. - -``mod_python`` does have some problems. Unlike the PHP interpreter, the Python -interpreter uses caching when executing files, so changes to a file will -require the web server to be restarted. Another problem is the basic concept --- Apache starts child processes to handle the requests, and unfortunately -every child process needs to load the whole Python interpreter even if it does -not use it. This makes the whole web server slower. Another problem is that, -because ``mod_python`` is linked against a specific version of ``libpython``, -it is not possible to switch from an older version to a newer (e.g. 2.4 to 2.5) -without recompiling ``mod_python``. ``mod_python`` is also bound to the Apache -web server, so programs written for ``mod_python`` cannot easily run on other -web servers. - -These are the reasons why ``mod_python`` should be avoided when writing new -programs. In some circumstances it still might be a good idea to use -``mod_python`` for deployment, but WSGI makes it possible to run WSGI programs -under ``mod_python`` as well. - - -FastCGI and SCGI ----------------- - -FastCGI and SCGI try to solve the performance problem of CGI in another way. -Instead of embedding the interpreter into the web server, they create -long-running background processes. There is still a module in the web server -which makes it possible for the web server to "speak" with the background -process. As the background process is independent of the server, it can be -written in any language, including Python. The language just needs to have a -library which handles the communication with the webserver. - -The difference between FastCGI and SCGI is very small, as SCGI is essentially -just a "simpler FastCGI". As the web server support for SCGI is limited, -most people use FastCGI instead, which works the same way. Almost everything -that applies to SCGI also applies to FastCGI as well, so we'll only cover -the latter. - -These days, FastCGI is never used directly. Just like ``mod_python``, it is only -used for the deployment of WSGI applications. - - -Setting up FastCGI -^^^^^^^^^^^^^^^^^^ - -Each web server requires a specific module. - -* Apache has both `mod_fastcgi <http://www.fastcgi.com/drupal/>`_ and `mod_fcgid - <http://httpd.apache.org/mod_fcgid/>`_. ``mod_fastcgi`` is the original one, but it - has some licensing issues, which is why it is sometimes considered non-free. - ``mod_fcgid`` is a smaller, compatible alternative. One of these modules needs - to be loaded by Apache. - -* lighttpd ships its own `FastCGI module - <http://redmine.lighttpd.net/projects/lighttpd/wiki/Docs_ModFastCGI>`_ as well as an - `SCGI module <http://redmine.lighttpd.net/projects/lighttpd/wiki/Docs_ModSCGI>`_. - -* `nginx <http://nginx.org/>`_ also supports `FastCGI - <http://wiki.nginx.org/NginxSimplePythonFCGI>`_. - -Once you have installed and configured the module, you can test it with the -following WSGI-application:: - - #!/usr/bin/env python - # -*- coding: UTF-8 -*- - - import sys, os - from html import escape - from flup.server.fcgi import WSGIServer - - def app(environ, start_response): - start_response('200 OK', [('Content-Type', 'text/html')]) - - yield '<h1>FastCGI Environment</h1>' - yield '<table>' - for k, v in sorted(environ.items()): - yield '<tr><th>{0}</th><td>{1}</td></tr>'.format( - escape(k), escape(v)) - yield '</table>' - - WSGIServer(app).run() - -This is a simple WSGI application, but you need to install `flup -<https://pypi.python.org/pypi/flup/1.0>`_ first, as flup handles the low level -FastCGI access. - -.. seealso:: - - There is some documentation on `setting up Django with FastCGI - <https://docs.djangoproject.com/en/dev/howto/deployment/fastcgi/>`_, most of - which can be reused for other WSGI-compliant frameworks and libraries. - Only the ``manage.py`` part has to be changed, the example used here can be - used instead. Django does more or less the exact same thing. - - -mod_wsgi --------- - -`mod_wsgi <http://code.google.com/p/modwsgi/>`_ is an attempt to get rid of the -low level gateways. Given that FastCGI, SCGI, and mod_python are mostly used to -deploy WSGI applications, mod_wsgi was started to directly embed WSGI applications -into the Apache web server. mod_wsgi is specifically designed to host WSGI -applications. It makes the deployment of WSGI applications much easier than -deployment using other low level methods, which need glue code. The downside -is that mod_wsgi is limited to the Apache web server; other servers would need -their own implementations of mod_wsgi. - -mod_wsgi supports two modes: embedded mode, in which it integrates with the -Apache process, and daemon mode, which is more FastCGI-like. Unlike FastCGI, -mod_wsgi handles the worker-processes by itself, which makes administration -easier. - - -.. _WSGI: - -Step back: WSGI -=============== - -WSGI has already been mentioned several times, so it has to be something -important. In fact it really is, and now it is time to explain it. - -The *Web Server Gateway Interface*, or WSGI for short, is defined in -:pep:`333` and is currently the best way to do Python web programming. While -it is great for programmers writing frameworks, a normal web developer does not -need to get in direct contact with it. When choosing a framework for web -development it is a good idea to choose one which supports WSGI. - -The big benefit of WSGI is the unification of the application programming -interface. When your program is compatible with WSGI -- which at the outer -level means that the framework you are using has support for WSGI -- your -program can be deployed via any web server interface for which there are WSGI -wrappers. You do not need to care about whether the application user uses -mod_python or FastCGI or mod_wsgi -- with WSGI your application will work on -any gateway interface. The Python standard library contains its own WSGI -server, :mod:`wsgiref`, which is a small web server that can be used for -testing. - -A really great WSGI feature is middleware. Middleware is a layer around your -program which can add various functionality to it. There is quite a bit of -`middleware <http://www.wsgi.org/en/latest/libraries.html>`_ already -available. For example, instead of writing your own session management (HTTP -is a stateless protocol, so to associate multiple HTTP requests with a single -user your application must create and manage such state via a session), you can -just download middleware which does that, plug it in, and get on with coding -the unique parts of your application. The same thing with compression -- there -is existing middleware which handles compressing your HTML using gzip to save -on your server's bandwidth. Authentication is another a problem easily solved -using existing middleware. - -Although WSGI may seem complex, the initial phase of learning can be very -rewarding because WSGI and the associated middleware already have solutions to -many problems that might arise while developing web sites. - - -WSGI Servers ------------- - -The code that is used to connect to various low level gateways like CGI or -mod_python is called a *WSGI server*. One of these servers is ``flup``, which -supports FastCGI and SCGI, as well as `AJP -<http://en.wikipedia.org/wiki/Apache_JServ_Protocol>`_. Some of these servers -are written in Python, as ``flup`` is, but there also exist others which are -written in C and can be used as drop-in replacements. - -There are many servers already available, so a Python web application -can be deployed nearly anywhere. This is one big advantage that Python has -compared with other web technologies. - -.. seealso:: - - A good overview of WSGI-related code can be found in the `WSGI homepage - <http://www.wsgi.org/en/latest/index.html>`_, which contains an extensive list of `WSGI servers - <http://www.wsgi.org/en/latest/servers.html>`_ which can be used by *any* application - supporting WSGI. - - You might be interested in some WSGI-supporting modules already contained in - the standard library, namely: - - * :mod:`wsgiref` -- some tiny utilities and servers for WSGI - - -Case study: MoinMoin --------------------- - -What does WSGI give the web application developer? Let's take a look at -an application that's been around for a while, which was written in -Python without using WSGI. - -One of the most widely used wiki software packages is `MoinMoin -<http://moinmo.in/>`_. It was created in 2000, so it predates WSGI by about -three years. Older versions needed separate code to run on CGI, mod_python, -FastCGI and standalone. - -It now includes support for WSGI. Using WSGI, it is possible to deploy -MoinMoin on any WSGI compliant server, with no additional glue code. -Unlike the pre-WSGI versions, this could include WSGI servers that the -authors of MoinMoin know nothing about. - - -Model-View-Controller -===================== - -The term *MVC* is often encountered in statements such as "framework *foo* -supports MVC". MVC is more about the overall organization of code, rather than -any particular API. Many web frameworks use this model to help the developer -bring structure to their program. Bigger web applications can have lots of -code, so it is a good idea to have an effective structure right from the beginning. -That way, even users of other frameworks (or even other languages, since MVC is -not Python-specific) can easily understand the code, given that they are -already familiar with the MVC structure. - -MVC stands for three components: - -* The *model*. This is the data that will be displayed and modified. In - Python frameworks, this component is often represented by the classes used by - an object-relational mapper. - -* The *view*. This component's job is to display the data of the model to the - user. Typically this component is implemented via templates. - -* The *controller*. This is the layer between the user and the model. The - controller reacts to user actions (like opening some specific URL), tells - the model to modify the data if necessary, and tells the view code what to - display, - -While one might think that MVC is a complex design pattern, in fact it is not. -It is used in Python because it has turned out to be useful for creating clean, -maintainable web sites. - -.. note:: - - While not all Python frameworks explicitly support MVC, it is often trivial - to create a web site which uses the MVC pattern by separating the data logic - (the model) from the user interaction logic (the controller) and the - templates (the view). That's why it is important not to write unnecessary - Python code in the templates -- it works against the MVC model and creates - chaos in the code base, making it harder to understand and modify. - -.. seealso:: - - The English Wikipedia has an article about the `Model-View-Controller pattern - <http://en.wikipedia.org/wiki/Model-view-controller>`_. It includes a long - list of web frameworks for various programming languages. - - -Ingredients for Websites -======================== - -Websites are complex constructs, so tools have been created to help web -developers make their code easier to write and more maintainable. Tools like -these exist for all web frameworks in all languages. Developers are not forced -to use these tools, and often there is no "best" tool. It is worth learning -about the available tools because they can greatly simplify the process of -developing a web site. - - -.. seealso:: - - There are far more components than can be presented here. The Python wiki - has a page about these components, called - `Web Components <https://wiki.python.org/moin/WebComponents>`_. - - -Templates ---------- - -Mixing of HTML and Python code is made possible by a few libraries. While -convenient at first, it leads to horribly unmaintainable code. That's why -templates exist. Templates are, in the simplest case, just HTML files with -placeholders. The HTML is sent to the user's browser after filling in the -placeholders. - -Python already includes a way to build simple templates:: - - # a simple template - template = "<html><body><h1>Hello {who}!</h1></body></html>" - print(template.format(who="Reader")) - -To generate complex HTML based on non-trivial model data, conditional -and looping constructs like Python's *for* and *if* are generally needed. -*Template engines* support templates of this complexity. - -There are a lot of template engines available for Python which can be used with -or without a `framework`_. Some of these define a plain-text programming -language which is easy to learn, partly because it is limited in scope. -Others use XML, and the template output is guaranteed to be always be valid -XML. There are many other variations. - -Some `frameworks`_ ship their own template engine or recommend one in -particular. In the absence of a reason to use a different template engine, -using the one provided by or recommended by the framework is a good idea. - -Popular template engines include: - - * `Mako <http://www.makotemplates.org/>`_ - * `Genshi <http://genshi.edgewall.org/>`_ - * `Jinja <http://jinja.pocoo.org/>`_ - -.. seealso:: - - There are many template engines competing for attention, because it is - pretty easy to create them in Python. The page `Templating - <https://wiki.python.org/moin/Templating>`_ in the wiki lists a big, - ever-growing number of these. The three listed above are considered "second - generation" template engines and are a good place to start. - - -Data persistence ----------------- - -*Data persistence*, while sounding very complicated, is just about storing data. -This data might be the text of blog entries, the postings on a bulletin board or -the text of a wiki page. There are, of course, a number of different ways to store -information on a web server. - -Often, relational database engines like `MySQL <http://www.mysql.com/>`_ or -`PostgreSQL <http://www.postgresql.org/>`_ are used because of their good -performance when handling very large databases consisting of millions of -entries. There is also a small database engine called `SQLite -<http://www.sqlite.org/>`_, which is bundled with Python in the :mod:`sqlite3` -module, and which uses only one file. It has no other dependencies. For -smaller sites SQLite is just enough. - -Relational databases are *queried* using a language called `SQL -<http://en.wikipedia.org/wiki/SQL>`_. Python programmers in general do not -like SQL too much, as they prefer to work with objects. It is possible to save -Python objects into a database using a technology called `ORM -<http://en.wikipedia.org/wiki/Object-relational_mapping>`_ (Object Relational -Mapping). ORM translates all object-oriented access into SQL code under the -hood, so the developer does not need to think about it. Most `frameworks`_ use -ORMs, and it works quite well. - -A second possibility is storing data in normal, plain text files (some -times called "flat files"). This is very easy for simple sites, -but can be difficult to get right if the web site is performing many -updates to the stored data. - -A third possibility are object oriented databases (also called "object -databases"). These databases store the object data in a form that closely -parallels the way the objects are structured in memory during program -execution. (By contrast, ORMs store the object data as rows of data in tables -and relations between those rows.) Storing the objects directly has the -advantage that nearly all objects can be saved in a straightforward way, unlike -in relational databases where some objects are very hard to represent. - -`Frameworks`_ often give hints on which data storage method to choose. It is -usually a good idea to stick to the data store recommended by the framework -unless the application has special requirements better satisfied by an -alternate storage mechanism. - -.. seealso:: - - * `Persistence Tools <https://wiki.python.org/moin/PersistenceTools>`_ lists - possibilities on how to save data in the file system. Some of these - modules are part of the standard library - - * `Database Programming <https://wiki.python.org/moin/DatabaseProgramming>`_ - helps with choosing a method for saving data - - * `SQLAlchemy <http://www.sqlalchemy.org/>`_, the most powerful OR-Mapper - for Python, and `Elixir <http://elixir.ematia.de/>`_, which makes - SQLAlchemy easier to use - - * `SQLObject <http://www.sqlobject.org/>`_, another popular OR-Mapper - - * `ZODB <https://launchpad.net/zodb>`_ and `Durus - <http://www.mems-exchange.org/software/durus/>`_, two object oriented - databases - - -.. _framework: - -Frameworks -========== - -The process of creating code to run web sites involves writing code to provide -various services. The code to provide a particular service often works the -same way regardless of the complexity or purpose of the web site in question. -Abstracting these common solutions into reusable code produces what are called -"frameworks" for web development. Perhaps the most well-known framework for -web development is Ruby on Rails, but Python has its own frameworks. Some of -these were partly inspired by Rails, or borrowed ideas from Rails, but many -existed a long time before Rails. - -Originally Python web frameworks tended to incorporate all of the services -needed to develop web sites as a giant, integrated set of tools. No two web -frameworks were interoperable: a program developed for one could not be -deployed on a different one without considerable re-engineering work. This led -to the development of "minimalist" web frameworks that provided just the tools -to communicate between the Python code and the http protocol, with all other -services to be added on top via separate components. Some ad hoc standards -were developed that allowed for limited interoperability between frameworks, -such as a standard that allowed different template engines to be used -interchangeably. - -Since the advent of WSGI, the Python web framework world has been evolving -toward interoperability based on the WSGI standard. Now many web frameworks, -whether "full stack" (providing all the tools one needs to deploy the most -complex web sites) or minimalist, or anything in between, are built from -collections of reusable components that can be used with more than one -framework. - -The majority of users will probably want to select a "full stack" framework -that has an active community. These frameworks tend to be well documented, -and provide the easiest path to producing a fully functional web site in -minimal time. - - -Some notable frameworks ------------------------ - -There are an incredible number of frameworks, so they cannot all be covered -here. Instead we will briefly touch on some of the most popular. - - -Django -^^^^^^ - -`Django <https://www.djangoproject.com/>`_ is a framework consisting of several -tightly coupled elements which were written from scratch and work together very -well. It includes an ORM which is quite powerful while being simple to use, -and has a great online administration interface which makes it possible to edit -the data in the database with a browser. The template engine is text-based and -is designed to be usable for page designers who cannot write Python. It -supports template inheritance and filters (which work like Unix pipes). Django -has many handy features bundled, such as creation of RSS feeds or generic views, -which make it possible to create web sites almost without writing any Python code. - -It has a big, international community, the members of which have created many -web sites. There are also a lot of add-on projects which extend Django's normal -functionality. This is partly due to Django's well written `online -documentation <https://docs.djangoproject.com/>`_ and the `Django book -<http://www.djangobook.com/>`_. - - -.. note:: - - Although Django is an MVC-style framework, it names the elements - differently, which is described in the `Django FAQ - <https://docs.djangoproject.com/en/dev/faq/general/#django-appears-to-be-a-mvc-framework-but-you-call-the-controller-the-view-and-the-view-the-template-how-come-you-don-t-use-the-standard-names>`_. - - -TurboGears -^^^^^^^^^^ - -Another popular web framework for Python is `TurboGears -<http://www.turbogears.org/>`_. TurboGears takes the approach of using already -existing components and combining them with glue code to create a seamless -experience. TurboGears gives the user flexibility in choosing components. For -example the ORM and template engine can be changed to use packages different -from those used by default. - -The documentation can be found in the `TurboGears wiki -<http://docs.turbogears.org/>`_, where links to screencasts can be found. -TurboGears has also an active user community which can respond to most related -questions. There is also a `TurboGears book <http://turbogearsbook.com/>`_ -published, which is a good starting point. - -The newest version of TurboGears, version 2.0, moves even further in direction -of WSGI support and a component-based architecture. TurboGears 2 is based on -the WSGI stack of another popular component-based web framework, `Pylons -<http://www.pylonsproject.org/>`_. - - -Zope -^^^^ - -The Zope framework is one of the "old original" frameworks. Its current -incarnation in Zope2 is a tightly integrated full-stack framework. One of its -most interesting feature is its tight integration with a powerful object -database called the `ZODB <https://launchpad.net/zodb>`_ (Zope Object Database). -Because of its highly integrated nature, Zope wound up in a somewhat isolated -ecosystem: code written for Zope wasn't very usable outside of Zope, and -vice-versa. To solve this problem the Zope 3 effort was started. Zope 3 -re-engineers Zope as a set of more cleanly isolated components. This effort -was started before the advent of the WSGI standard, but there is WSGI support -for Zope 3 from the `Repoze <http://repoze.org/>`_ project. Zope components -have many years of production use behind them, and the Zope 3 project gives -access to these components to the wider Python community. There is even a -separate framework based on the Zope components: `Grok -<http://grok.zope.org/>`_. - -Zope is also the infrastructure used by the `Plone <https://plone.org/>`_ content -management system, one of the most powerful and popular content management -systems available. - - -Other notable frameworks -^^^^^^^^^^^^^^^^^^^^^^^^ - -Of course these are not the only frameworks that are available. There are -many other frameworks worth mentioning. - -Another framework that's already been mentioned is `Pylons`_. Pylons is much -like TurboGears, but with an even stronger emphasis on flexibility, which comes -at the cost of being more difficult to use. Nearly every component can be -exchanged, which makes it necessary to use the documentation of every single -component, of which there are many. Pylons builds upon `Paste -<http://pythonpaste.org/>`_, an extensive set of tools which are handy for WSGI. - -And that's still not everything. The most up-to-date information can always be -found in the Python wiki. - -.. seealso:: - - The Python wiki contains an extensive list of `web frameworks - <https://wiki.python.org/moin/WebFrameworks>`_. - - Most frameworks also have their own mailing lists and IRC channels, look out - for these on the projects' web sites. diff --git a/Doc/includes/email-alternative-new-api.py b/Doc/includes/email-alternative-new-api.py index c1255a68df..321f727c31 100644 --- a/Doc/includes/email-alternative-new-api.py +++ b/Doc/includes/email-alternative-new-api.py @@ -9,9 +9,9 @@ from email.utils import make_msgid # Create the base text message. msg = EmailMessage() msg['Subject'] = "Ayons asperges pour le déjeuner" -msg['From'] = Address("Pepé Le Pew", "pepe@example.com") -msg['To'] = (Address("Penelope Pussycat", "penelope@example.com"), - Address("Fabrette Pussycat", "fabrette@example.com")) +msg['From'] = Address("Pepé Le Pew", "pepe", "example.com") +msg['To'] = (Address("Penelope Pussycat", "penelope", "example.com"), + Address("Fabrette Pussycat", "fabrette", "example.com")) msg.set_content("""\ Salut! diff --git a/Doc/includes/noddy.c b/Doc/includes/noddy.c index 8f79fcf6bf..19a27a89e8 100644 --- a/Doc/includes/noddy.c +++ b/Doc/includes/noddy.c @@ -38,7 +38,7 @@ static PyModuleDef noddymodule = { }; PyMODINIT_FUNC -PyInit_noddy(void) +PyInit_noddy(void) { PyObject* m; diff --git a/Doc/includes/run-func.c b/Doc/includes/run-func.c index 1c9860d7a8..986d670319 100644 --- a/Doc/includes/run-func.c +++ b/Doc/includes/run-func.c @@ -13,7 +13,7 @@ main(int argc, char *argv[]) } Py_Initialize(); - pName = PyUnicode_FromString(argv[1]); + pName = PyUnicode_DecodeFSDefault(argv[1]); /* Error checking of pName left out */ pModule = PyImport_Import(pName); diff --git a/Doc/includes/typestruct.h b/Doc/includes/typestruct.h index 726500980e..9f47899a19 100644 --- a/Doc/includes/typestruct.h +++ b/Doc/includes/typestruct.h @@ -9,7 +9,8 @@ typedef struct _typeobject { printfunc tp_print; getattrfunc tp_getattr; setattrfunc tp_setattr; - void *tp_reserved; /* formerly known as tp_compare */ + PyAsyncMethods *tp_as_async; /* formerly known as tp_compare (Python 2) + or tp_reserved (Python 3) */ reprfunc tp_repr; /* Method suites for standard classes */ diff --git a/Doc/install/index.rst b/Doc/install/index.rst index e63b6cd3d3..b22fc5958c 100644 --- a/Doc/install/index.rst +++ b/Doc/install/index.rst @@ -10,6 +10,11 @@ .. TODO: Fill in XXX comments +.. seealso:: + + :ref:`installing-index` + The up to date module installation documentations + .. The audience for this document includes people who don't know anything about Python and aren't about to learn the language just in order to install and maintain it for their users, i.e. system administrators. @@ -361,7 +366,7 @@ And here are the values used on Windows: Type of file Installation directory =============== =========================================================== modules :file:`{userbase}\\Python{XY}\\site-packages` -scripts :file:`{userbase}\\Scripts` +scripts :file:`{userbase}\\Python{XY}\\Scripts` data :file:`{userbase}` C headers :file:`{userbase}\\Python{XY}\\Include\\{distname}` =============== =========================================================== @@ -865,12 +870,12 @@ config file will apply. (Or if other commands that derive values from it are run, they will use the values in the config file.) You can find out the complete list of options for any command using the -:option:`--help` option, e.g.:: +:option:`!--help` option, e.g.:: python setup.py build --help and you can find out the complete list of global options by using -:option:`--help` without a command:: +:option:`!--help` without a command:: python setup.py --help @@ -927,7 +932,7 @@ Let's examine each of the fields in turn. to be in Objective C. * *cpparg* is an argument for the C preprocessor, and is anything starting with - :option:`-I`, :option:`-D`, :option:`-U` or :option:`-C`. + :option:`!-I`, :option:`-D`, :option:`!-U` or :option:`-C`. * *library* is anything ending in :file:`.a` or beginning with :option:`-l` or :option:`-L`. @@ -1012,7 +1017,7 @@ section :ref:`inst-config-files`.) .. seealso:: - `C++Builder Compiler <http://www.embarcadero.com/downloads>`_ + `C++Builder Compiler <https://www.embarcadero.com/products>`_ Information about the free C++ compiler from Borland, including links to the download pages. @@ -1055,7 +1060,7 @@ These compilers require some special libraries. This task is more complex than for Borland's C++, because there is no program to convert the library. First you have to create a list of symbols which the Python DLL exports. (You can find a good program for this task at -http://sourceforge.net/projects/mingw/files/MinGW/Extension/pexports/). +https://sourceforge.net/projects/mingw/files/MinGW/Extension/pexports/). .. I don't understand what the next line means. --amk .. (inclusive the references on data structures.) @@ -1093,7 +1098,7 @@ normal libraries do. .. [#] This also means you could replace all existing COFF-libraries with OMF-libraries of the same name. -.. [#] Check http://www.sourceware.org/cygwin/ and http://www.mingw.org/ for more +.. [#] Check https://www.sourceware.org/cygwin/ and http://www.mingw.org/ for more information .. [#] Then you have no POSIX emulation available, but you also don't need diff --git a/Doc/installing/index.rst b/Doc/installing/index.rst index 973c689861..1ef314999b 100644 --- a/Doc/installing/index.rst +++ b/Doc/installing/index.rst @@ -48,7 +48,7 @@ Key terms repository of open source licensed packages made available for use by other Python users * the `Python Packaging Authority - <https://packaging.python.org/en/latest/future.html>`__ are the group of + <https://www.pypa.io/en/latest/>`__ are the group of developers and documentation authors responsible for the maintenance and evolution of the standard packaging tools and the associated metadata and file format standards. They maintain a variety of tools, documentation @@ -84,10 +84,12 @@ dependencies from the Python Packaging Index:: Python. It's also possible to specify an exact or minimum version directly on the -command line:: +command line. When using comparator operators such as ``>``, ``<`` or some other +special character which get interpreted by shell, the package name and the +version should be enclosed within double quotes:: python -m pip install SomePackage==1.0.4 # specific version - python -m pip install 'SomePackage>=1.0.4' # minimum version + python -m pip install "SomePackage>=1.0.4" # minimum version Normally, if a suitable module is already installed, attempting to install it again will have no effect. Upgrading existing modules must be requested @@ -104,7 +106,7 @@ into an active virtual environment uses the commands shown above. .. seealso:: `Python Packaging User Guide: Installing Python Distribution Packages - <https://packaging.python.org/en/latest/installing.html#installing-python-distribution-packages>`__ + <https://packaging.python.org/en/latest/installing/>`__ How do I ...? @@ -121,8 +123,8 @@ User Guide. .. seealso:: - `Python Packaging User Guide: Setup for Installing Distribution Packages - <https://packaging.python.org/en/latest/installing.html#setup-for-installing-distribution-packages>`__ + `Python Packaging User Guide: Requirements for Installing Packages + <https://packaging.python.org/en/latest/installing/#requirements-for-installing-packages>`__ .. installing-per-user-installation: @@ -141,13 +143,13 @@ A number of scientific Python packages have complex binary dependencies, and aren't currently easy to install using ``pip`` directly. At this point in time, it will often be easier for users to install these packages by `other means -<https://packaging.python.org/en/latest/science.html>`__ +<https://packaging.python.org/en/latest/science/>`__ rather than attempting to install them with ``pip``. .. seealso:: `Python Packaging User Guide: Installing Scientific Packages - <https://packaging.python.org/en/latest/science.html>`__ + <https://packaging.python.org/en/latest/science/>`__ ... work with multiple versions of Python installed in parallel? @@ -177,7 +179,7 @@ switch:: Once the Development & Deployment part of PPUG is fleshed out, some of those sections should be linked from new questions here (most notably, we should have a question about avoiding depending on PyPI that links to - https://packaging.python.org/en/latest/deployment.html#pypi-mirrors-and-caches) + https://packaging.python.org/en/latest/mirrors/) Common installation issues @@ -210,11 +212,11 @@ as users are more regularly able to install pre-built extensions rather than needing to build them themselves. Some of the solutions for installing `scientific software -<https://packaging.python.org/en/latest/science.html>`__ +<https://packaging.python.org/en/latest/science/>`__ that is not yet available as pre-built ``wheel`` files may also help with obtaining other binary extensions without needing to build them locally. .. seealso:: `Python Packaging User Guide: Binary Extensions - <https://packaging.python.org/en/latest/extensions.html>`__ + <https://packaging.python.org/en/latest/extensions/>`__ diff --git a/Doc/library/2to3.rst b/Doc/library/2to3.rst index 31f681d7e0..ec59679b35 100644 --- a/Doc/library/2to3.rst +++ b/Doc/library/2to3.rst @@ -33,14 +33,18 @@ Here is a sample Python 2.x source file, :file:`example.py`:: name = raw_input() greet(name) -It can be converted to Python 3.x code via 2to3 on the command line:: +It can be converted to Python 3.x code via 2to3 on the command line: + +.. code-block:: shell-session $ 2to3 example.py A diff against the original source file is printed. 2to3 can also write the needed modifications right back to the source file. (A backup of the original file is made unless :option:`-n` is also given.) Writing the changes back is -enabled with the :option:`-w` flag:: +enabled with the :option:`-w` flag: + +.. code-block:: shell-session $ 2to3 -w example.py @@ -55,19 +59,25 @@ After transformation, :file:`example.py` looks like this:: Comments and exact indentation are preserved throughout the translation process. By default, 2to3 runs a set of :ref:`predefined fixers <2to3-fixers>`. The -:option:`-l` flag lists all available fixers. An explicit set of fixers to run -can be given with :option:`-f`. Likewise the :option:`-x` explicitly disables a -fixer. The following example runs only the ``imports`` and ``has_key`` fixers:: +:option:`!-l` flag lists all available fixers. An explicit set of fixers to run +can be given with :option:`-f`. Likewise the :option:`!-x` explicitly disables a +fixer. The following example runs only the ``imports`` and ``has_key`` fixers: + +.. code-block:: shell-session $ 2to3 -f imports -f has_key example.py -This command runs every fixer except the ``apply`` fixer:: +This command runs every fixer except the ``apply`` fixer: + +.. code-block:: shell-session $ 2to3 -x apply example.py Some fixers are *explicit*, meaning they aren't run by default and must be listed on the command line to be run. Here, in addition to the default fixers, -the ``idioms`` fixer is run:: +the ``idioms`` fixer is run: + +.. code-block:: shell-session $ 2to3 -f all -f idioms example.py @@ -78,12 +88,12 @@ but 2to3 cannot fix automatically. In this case, 2to3 will print a warning beneath the diff for a file. You should address the warning in order to have compliant 3.x code. -2to3 can also refactor doctests. To enable this mode, use the :option:`-d` +2to3 can also refactor doctests. To enable this mode, use the :option:`!-d` flag. Note that *only* doctests will be refactored. This also doesn't require the module to be valid Python. For example, doctest like examples in a reST document could also be refactored with this option. -The :option:`-v` option enables output of more information on the translation +The :option:`!-v` option enables output of more information on the translation process. Since some print statements can be parsed as function calls or statements, 2to3 @@ -102,18 +112,20 @@ when not overwriting the input files. .. versionadded:: 3.2.3 The :option:`-o` option was added. -The :option:`-W` or :option:`--write-unchanged-files` flag tells 2to3 to always +The :option:`!-W` or :option:`--write-unchanged-files` flag tells 2to3 to always write output files even if no changes were required to the file. This is most useful with :option:`-o` so that an entire Python source tree is copied with translation from one directory to another. This option implies the :option:`-w` flag as it would not make sense otherwise. .. versionadded:: 3.2.3 - The :option:`-W` flag was added. + The :option:`!-W` flag was added. The :option:`--add-suffix` option specifies a string to append to all output filenames. The :option:`-n` flag is required when specifying this as backups -are not necessary when writing to different filenames. Example:: +are not necessary when writing to different filenames. Example: + +.. code-block:: shell-session $ 2to3 -n -W --add-suffix=3 example.py @@ -122,7 +134,9 @@ Will cause a converted file named ``example.py3`` to be written. .. versionadded:: 3.2.3 The :option:`--add-suffix` option was added. -To translate an entire project from one directory tree to another use:: +To translate an entire project from one directory tree to another use: + +.. code-block:: shell-session $ 2to3 --output-dir=python3-version/mycode -W -n python2-version/mycode @@ -449,10 +463,14 @@ and off individually. They are described here in more detail. .. module:: lib2to3 :synopsis: the 2to3 library + .. moduleauthor:: Guido van Rossum .. moduleauthor:: Collin Winter .. moduleauthor:: Benjamin Peterson <benjamin@python.org> +**Source code:** :source:`Lib/lib2to3/` + +-------------- .. note:: diff --git a/Doc/library/__future__.rst b/Doc/library/__future__.rst index 72f2963a2c..73d8b6b7e8 100644 --- a/Doc/library/__future__.rst +++ b/Doc/library/__future__.rst @@ -87,6 +87,9 @@ language using this mechanism: | unicode_literals | 2.6.0a2 | 3.0 | :pep:`3112`: | | | | | *Bytes literals in Python 3000* | +------------------+-------------+--------------+---------------------------------------------+ +| generator_stop | 3.5.0b1 | 3.7 | :pep:`479`: | +| | | | *StopIteration handling inside generators* | ++------------------+-------------+--------------+---------------------------------------------+ .. seealso:: diff --git a/Doc/library/__main__.rst b/Doc/library/__main__.rst index a46993d55e..a64faf1bbe 100644 --- a/Doc/library/__main__.rst +++ b/Doc/library/__main__.rst @@ -5,6 +5,8 @@ .. module:: __main__ :synopsis: The environment where the top-level script is run. +-------------- + ``'__main__'`` is the name of the scope in which top-level code executes. A module's __name__ is set equal to ``'__main__'`` when read from standard input, a script, or from an interactive prompt. diff --git a/Doc/library/_thread.rst b/Doc/library/_thread.rst index 7122861c45..0d2d818f5f 100644 --- a/Doc/library/_thread.rst +++ b/Doc/library/_thread.rst @@ -4,13 +4,14 @@ .. module:: _thread :synopsis: Low-level threading API. - .. index:: single: light-weight processes single: processes, light-weight single: binary semaphores single: semaphores, binary +-------------- + This module provides low-level primitives for working with multiple threads (also called :dfn:`light-weight processes` or :dfn:`tasks`) --- multiple threads of control sharing their global data space. For synchronization, simple locks diff --git a/Doc/library/abc.rst b/Doc/library/abc.rst index 7a73704bf6..966003bd45 100644 --- a/Doc/library/abc.rst +++ b/Doc/library/abc.rst @@ -3,6 +3,7 @@ .. module:: abc :synopsis: Abstract base classes according to PEP 3119. + .. moduleauthor:: Guido van Rossum .. sectionauthor:: Georg Brandl .. much of the content adapted from docstrings diff --git a/Doc/library/aifc.rst b/Doc/library/aifc.rst index 6fbcf286cf..23a96207d0 100644 --- a/Doc/library/aifc.rst +++ b/Doc/library/aifc.rst @@ -4,14 +4,13 @@ .. module:: aifc :synopsis: Read and write audio files in AIFF or AIFC format. +**Source code:** :source:`Lib/aifc.py` .. index:: single: Audio Interchange File Format single: AIFF single: AIFF-C -**Source code:** :source:`Lib/aifc.py` - -------------- This module provides support for reading and writing AIFF and AIFF-C files. diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst index d907203a02..995c4ee28c 100644 --- a/Doc/library/argparse.rst +++ b/Doc/library/argparse.rst @@ -3,6 +3,7 @@ .. module:: argparse :synopsis: Command-line option and argument parsing library. + .. moduleauthor:: Steven Bethard <steven.bethard@gmail.com> .. sectionauthor:: Steven Bethard <steven.bethard@gmail.com> @@ -35,16 +36,18 @@ produces either the sum or the max:: parser = argparse.ArgumentParser(description='Process some integers.') parser.add_argument('integers', metavar='N', type=int, nargs='+', - help='an integer for the accumulator') + help='an integer for the accumulator') parser.add_argument('--sum', dest='accumulate', action='store_const', - const=sum, default=max, - help='sum the integers (default: find the max)') + const=sum, default=max, + help='sum the integers (default: find the max)') args = parser.parse_args() print(args.accumulate(args.integers)) Assuming the Python code above is saved into a file called ``prog.py``, it can -be run at the command line and provides useful help messages:: +be run at the command line and provides useful help messages: + +.. code-block:: shell-session $ python prog.py -h usage: prog.py [-h] [--sum] N [N ...] @@ -59,7 +62,9 @@ be run at the command line and provides useful help messages:: --sum sum the integers (default: find the max) When run with the appropriate arguments, it prints either the sum or the max of -the command-line integers:: +the command-line integers: + +.. code-block:: shell-session $ python prog.py 1 2 3 4 4 @@ -67,7 +72,9 @@ the command-line integers:: $ python prog.py 1 2 3 4 --sum 10 -If invalid arguments are passed in, it will issue an error:: +If invalid arguments are passed in, it will issue an error: + +.. code-block:: shell-session $ python prog.py a b c usage: prog.py [-h] [--sum] N [N ...] @@ -135,7 +142,7 @@ ArgumentParser objects formatter_class=argparse.HelpFormatter, \ prefix_chars='-', fromfile_prefix_chars=None, \ argument_default=None, conflict_handler='error', \ - add_help=True) + add_help=True, allow_abbrev=True) Create a new :class:`ArgumentParser` object. All parameters should be passed as keyword arguments. Each parameter has its own more detailed description @@ -169,6 +176,12 @@ ArgumentParser objects * add_help_ - Add a -h/--help option to the parser (default: ``True``) + * allow_abbrev_ - Allows long options to be abbreviated if the + abbreviation is unambiguous. (default: ``True``) + + .. versionchanged:: 3.5 + *allow_abbrev* parameter was added. + The following sections describe how each of these are used. @@ -187,7 +200,9 @@ invoked on the command line. For example, consider a file named args = parser.parse_args() The help for this program will display ``myprogram.py`` as the program name -(regardless of where the program was invoked from):: +(regardless of where the program was invoked from): + +.. code-block:: shell-session $ python myprogram.py --help usage: myprogram.py [-h] [--foo FOO] @@ -482,7 +497,7 @@ specified characters will be treated as files, and will be replaced by the arguments they contain. For example:: >>> with open('args.txt', 'w') as fp: - ... fp.write('-f\nbar') + ... fp.write('-f\nbar') >>> parser = argparse.ArgumentParser(fromfile_prefix_chars='@') >>> parser.add_argument('-f') >>> parser.parse_args(['-f', 'foo', '@args.txt']) @@ -518,6 +533,26 @@ calls, we supply ``argument_default=SUPPRESS``:: >>> parser.parse_args([]) Namespace() +.. _allow_abbrev: + +allow_abbrev +^^^^^^^^^^^^ + +Normally, when you pass an argument list to the +:meth:`~ArgumentParser.parse_args` method of an :class:`ArgumentParser`, +it :ref:`recognizes abbreviations <prefix-matching>` of long options. + +This feature can be disabled by setting ``allow_abbrev`` to ``False``:: + + >>> parser = argparse.ArgumentParser(prog='PROG', allow_abbrev=False) + >>> parser.add_argument('--foobar', action='store_true') + >>> parser.add_argument('--foonley', action='store_false') + >>> parser.parse_args(['--foon']) + usage: PROG [-h] [--foobar] [--foonley] + PROG: error: unrecognized arguments: --foon + +.. versionadded:: 3.5 + conflict_handler ^^^^^^^^^^^^^^^^ @@ -569,7 +604,9 @@ the parser's help message. For example, consider a file named args = parser.parse_args() If ``-h`` or ``--help`` is supplied at the command line, the ArgumentParser -help will be printed:: +help will be printed: + +.. code-block:: shell-session $ python myprogram.py --help usage: myprogram.py [-h] [--foo FOO] @@ -694,24 +731,25 @@ how the command-line arguments should be handled. The supplied actions are: Namespace(foo='1') * ``'store_const'`` - This stores the value specified by the const_ keyword - argument. (Note that the const_ keyword argument defaults to the rather - unhelpful ``None``.) The ``'store_const'`` action is most commonly used with + argument. The ``'store_const'`` action is most commonly used with optional arguments that specify some sort of flag. For example:: >>> parser = argparse.ArgumentParser() >>> parser.add_argument('--foo', action='store_const', const=42) - >>> parser.parse_args('--foo'.split()) + >>> parser.parse_args(['--foo']) Namespace(foo=42) -* ``'store_true'`` and ``'store_false'`` - These store the values ``True`` and - ``False`` respectively. These are special cases of ``'store_const'``. For - example:: +* ``'store_true'`` and ``'store_false'`` - These are special cases of + ``'store_const'`` used for storing the values ``True`` and ``False`` + respectively. In addition, they create default values of ``False`` and + ``True`` respectively. For example:: >>> parser = argparse.ArgumentParser() >>> parser.add_argument('--foo', action='store_true') >>> parser.add_argument('--bar', action='store_false') + >>> parser.add_argument('--baz', action='store_false') >>> parser.parse_args('--foo --bar'.split()) - Namespace(bar=False, foo=True) + Namespace(foo=True, bar=False, baz=True) * ``'append'`` - This stores a list, and appends each argument value to the list. This is useful to allow an option to be specified multiple times. @@ -739,7 +777,7 @@ how the command-line arguments should be handled. The supplied actions are: >>> parser = argparse.ArgumentParser() >>> parser.add_argument('--verbose', '-v', action='count') - >>> parser.parse_args('-vvv'.split()) + >>> parser.parse_args(['-vvv']) Namespace(verbose=3) * ``'help'`` - This prints a complete help message for all the options in the @@ -814,11 +852,11 @@ values are: >>> parser = argparse.ArgumentParser() >>> parser.add_argument('--foo', nargs='?', const='c', default='d') >>> parser.add_argument('bar', nargs='?', default='d') - >>> parser.parse_args('XX --foo YY'.split()) + >>> parser.parse_args(['XX', '--foo', 'YY']) Namespace(bar='XX', foo='YY') - >>> parser.parse_args('XX --foo'.split()) + >>> parser.parse_args(['XX', '--foo']) Namespace(bar='XX', foo='c') - >>> parser.parse_args(''.split()) + >>> parser.parse_args([]) Namespace(bar='d', foo='d') One of the more common uses of ``nargs='?'`` is to allow optional input and @@ -854,9 +892,9 @@ values are: >>> parser = argparse.ArgumentParser(prog='PROG') >>> parser.add_argument('foo', nargs='+') - >>> parser.parse_args('a b'.split()) + >>> parser.parse_args(['a', 'b']) Namespace(foo=['a', 'b']) - >>> parser.parse_args(''.split()) + >>> parser.parse_args([]) usage: PROG [-h] foo [foo ...] PROG: error: too few arguments @@ -895,7 +933,8 @@ the various :class:`ArgumentParser` actions. The two most common uses of it are command-line argument following it, the value of ``const`` will be assumed instead. See the nargs_ description for examples. -The ``const`` keyword argument defaults to ``None``. +With the ``'store_const'`` and ``'append_const'`` actions, the ``const`` +keyword argument must be given. For other actions, it defaults to ``None``. default @@ -910,9 +949,9 @@ was not present at the command line:: >>> parser = argparse.ArgumentParser() >>> parser.add_argument('--foo', default=42) - >>> parser.parse_args('--foo 2'.split()) + >>> parser.parse_args(['--foo', '2']) Namespace(foo='2') - >>> parser.parse_args(''.split()) + >>> parser.parse_args([]) Namespace(foo=42) If the ``default`` value is a string, the parser parses the value as if it @@ -931,9 +970,9 @@ is used when no command-line argument was present:: >>> parser = argparse.ArgumentParser() >>> parser.add_argument('foo', nargs='?', default=42) - >>> parser.parse_args('a'.split()) + >>> parser.parse_args(['a']) Namespace(foo='a') - >>> parser.parse_args(''.split()) + >>> parser.parse_args([]) Namespace(foo=42) @@ -990,9 +1029,9 @@ the converted value:: ... >>> parser = argparse.ArgumentParser(prog='PROG') >>> parser.add_argument('foo', type=perfect_square) - >>> parser.parse_args('9'.split()) + >>> parser.parse_args(['9']) Namespace(foo=9) - >>> parser.parse_args('7'.split()) + >>> parser.parse_args(['7']) usage: PROG [-h] foo PROG: error: argument foo: '7' is not a perfect square @@ -1001,9 +1040,9 @@ simply check against a range of values:: >>> parser = argparse.ArgumentParser(prog='PROG') >>> parser.add_argument('foo', type=int, choices=range(5, 10)) - >>> parser.parse_args('7'.split()) + >>> parser.parse_args(['7']) Namespace(foo=7) - >>> parser.parse_args('11'.split()) + >>> parser.parse_args(['11']) usage: PROG [-h] {5,6,7,8,9} PROG: error: argument foo: invalid choice: 11 (choose from 5, 6, 7, 8, 9) @@ -1081,10 +1120,10 @@ argument:: >>> parser = argparse.ArgumentParser(prog='frobble') >>> parser.add_argument('--foo', action='store_true', - ... help='foo the bars before frobbling') + ... help='foo the bars before frobbling') >>> parser.add_argument('bar', nargs='+', - ... help='one of the bars to be frobbled') - >>> parser.parse_args('-h'.split()) + ... help='one of the bars to be frobbled') + >>> parser.parse_args(['-h']) usage: frobble [-h] [--foo] bar [bar ...] positional arguments: @@ -1101,7 +1140,7 @@ specifiers include the program name, ``%(prog)s`` and most keyword arguments to >>> parser = argparse.ArgumentParser(prog='frobble') >>> parser.add_argument('bar', nargs='?', type=int, default=42, - ... help='the bar to %(prog)s (default: %(default)s)') + ... help='the bar to %(prog)s (default: %(default)s)') >>> parser.print_help() usage: frobble [-h] [bar] @@ -1202,7 +1241,7 @@ attribute is determined by the ``dest`` keyword argument of >>> parser = argparse.ArgumentParser() >>> parser.add_argument('bar') - >>> parser.parse_args('XXX'.split()) + >>> parser.parse_args(['XXX']) Namespace(bar='XXX') For optional argument actions, the value of ``dest`` is normally inferred from @@ -1299,22 +1338,22 @@ option and its value are passed as two separate arguments:: >>> parser = argparse.ArgumentParser(prog='PROG') >>> parser.add_argument('-x') >>> parser.add_argument('--foo') - >>> parser.parse_args('-x X'.split()) + >>> parser.parse_args(['-x', 'X']) Namespace(foo=None, x='X') - >>> parser.parse_args('--foo FOO'.split()) + >>> parser.parse_args(['--foo', 'FOO']) Namespace(foo='FOO', x=None) For long options (options with names longer than a single character), the option and value can also be passed as a single command-line argument, using ``=`` to separate them:: - >>> parser.parse_args('--foo=FOO'.split()) + >>> parser.parse_args(['--foo=FOO']) Namespace(foo='FOO', x=None) For short options (options only one character long), the option and its value can be concatenated:: - >>> parser.parse_args('-xX'.split()) + >>> parser.parse_args(['-xX']) Namespace(foo=None, x='X') Several short options can be joined together, using only a single ``-`` prefix, @@ -1324,7 +1363,7 @@ as long as only the last option (or none of them) requires a value:: >>> parser.add_argument('-x', action='store_true') >>> parser.add_argument('-y', action='store_true') >>> parser.add_argument('-z') - >>> parser.parse_args('-xyzZ'.split()) + >>> parser.parse_args(['-xyzZ']) Namespace(x=True, y=True, z='Z') @@ -1410,9 +1449,9 @@ argument:: Argument abbreviations (prefix matching) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -The :meth:`~ArgumentParser.parse_args` method allows long options to be -abbreviated to a prefix, if the abbreviation is unambiguous (the prefix matches -a unique option):: +The :meth:`~ArgumentParser.parse_args` method :ref:`by default <allow_abbrev>` +allows long options to be abbreviated to a prefix, if the abbreviation is +unambiguous (the prefix matches a unique option):: >>> parser = argparse.ArgumentParser(prog='PROG') >>> parser.add_argument('-bacon') @@ -1426,6 +1465,7 @@ a unique option):: PROG: error: ambiguous option: -ba could match -badger, -bacon An error is produced for arguments that could produce more than one options. +This feature can be disabled by setting :ref:`allow_abbrev` to ``False``. Beyond ``sys.argv`` @@ -1439,13 +1479,13 @@ interactive prompt:: >>> parser = argparse.ArgumentParser() >>> parser.add_argument( ... 'integers', metavar='int', type=int, choices=range(10), - ... nargs='+', help='an integer in the range 0..9') + ... nargs='+', help='an integer in the range 0..9') >>> parser.add_argument( ... '--sum', dest='accumulate', action='store_const', const=sum, - ... default=max, help='sum the integers (default: find the max)') + ... default=max, help='sum the integers (default: find the max)') >>> parser.parse_args(['1', '2', '3', '4']) Namespace(accumulate=<built-in function max>, integers=[1, 2, 3, 4]) - >>> parser.parse_args('1 2 3 4 --sum'.split()) + >>> parser.parse_args(['1', '2', '3', '4', '--sum']) Namespace(accumulate=<built-in function sum>, integers=[1, 2, 3, 4]) diff --git a/Doc/library/array.rst b/Doc/library/array.rst index f1ab959572..24f3f62ae7 100644 --- a/Doc/library/array.rst +++ b/Doc/library/array.rst @@ -4,9 +4,10 @@ .. module:: array :synopsis: Space efficient arrays of uniformly typed numeric values. - .. index:: single: arrays +-------------- + This module defines an object type which can compactly represent an array of basic values: characters, integers, floating point numbers. Arrays are sequence types and behave very much like lists, except that the type of objects stored in @@ -91,7 +92,7 @@ Array objects support the ordinary sequence operations of indexing, slicing, concatenation, and multiplication. When using slice assignment, the assigned value must be an array object with the same type code; in all other cases, :exc:`TypeError` is raised. Array objects also implement the buffer interface, -and may be used wherever :term:`bytes-like object`\ s are supported. +and may be used wherever :term:`bytes-like objects <bytes-like object>` are supported. The following data items and methods are also supported: @@ -271,7 +272,7 @@ Examples:: Packing and unpacking of External Data Representation (XDR) data as used in some remote procedure call systems. - `The Numerical Python Documentation <http://docs.scipy.org/doc/>`_ + `The Numerical Python Documentation <https://docs.scipy.org/doc/>`_ The Numeric Python extension (NumPy) defines another array type; see http://www.numpy.org/ for further information about Numerical Python. diff --git a/Doc/library/ast.rst b/Doc/library/ast.rst index 8c3b7e4ce4..8d4ae2cc61 100644 --- a/Doc/library/ast.rst +++ b/Doc/library/ast.rst @@ -99,6 +99,7 @@ Abstract Grammar The abstract grammar is currently defined as follows: .. literalinclude:: ../../Parser/Python.asdl + :language: none :mod:`ast` Helpers diff --git a/Doc/library/asynchat.rst b/Doc/library/asynchat.rst index c6fa06162e..ae72d26122 100644 --- a/Doc/library/asynchat.rst +++ b/Doc/library/asynchat.rst @@ -3,6 +3,7 @@ .. module:: asynchat :synopsis: Support for asynchronous command/response protocols. + .. moduleauthor:: Sam Rushing <rushing@nightmare.com> .. sectionauthor:: Steve Holden <sholden@holdenweb.com> @@ -147,40 +148,6 @@ connection requests. by the channel after :meth:`found_terminator` is called. -asynchat - Auxiliary Classes ------------------------------------------- - -.. class:: fifo(list=None) - - A :class:`fifo` holding data which has been pushed by the application but - not yet popped for writing to the channel. A :class:`fifo` is a list used - to hold data and/or producers until they are required. If the *list* - argument is provided then it should contain producers or data items to be - written to the channel. - - - .. method:: is_empty() - - Returns ``True`` if and only if the fifo is empty. - - - .. method:: first() - - Returns the least-recently :meth:`push`\ ed item from the fifo. - - - .. method:: push(data) - - Adds the given data (which may be a string or a producer object) to the - producer fifo. - - - .. method:: pop() - - If the fifo is not empty, returns ``True, first()``, deleting the popped - item. Returns ``False, None`` for an empty fifo. - - .. _asynchat-example: asynchat Example @@ -236,7 +203,7 @@ any extraneous data sent by the web client are ignored. :: self.set_terminator(None) self.handle_request() elif not self.handling: - self.set_terminator(None) # browsers sometimes over-send + self.set_terminator(None) # browsers sometimes over-send self.cgi_data = parse(self.headers, b"".join(self.ibuffer)) self.handling = True self.ibuffer = [] diff --git a/Doc/library/asyncio-dev.rst b/Doc/library/asyncio-dev.rst index 1d1f795294..b9557af32c 100644 --- a/Doc/library/asyncio-dev.rst +++ b/Doc/library/asyncio-dev.rst @@ -106,6 +106,9 @@ directly its :meth:`Future.cancel` method, but:: loop.call_soon_threadsafe(fut.cancel) +To handle signals and to execute subprocesses, the event loop must be run in +the main thread. + To schedule a coroutine object from a different thread, the :func:`run_coroutine_threadsafe` function should be used. It returns a :class:`concurrent.futures.Future` to access the result:: @@ -113,9 +116,6 @@ To schedule a coroutine object from a different thread, the future = asyncio.run_coroutine_threadsafe(coro_func(), loop) result = future.result(timeout) # Wait for the result with a timeout -To handle signals and to execute subprocesses, the event loop must be run in -the main thread. - The :meth:`BaseEventLoop.run_in_executor` method can be used with a thread pool executor to execute a callback in different thread to not block the thread of the event loop. @@ -168,10 +168,10 @@ Detect coroutine objects never scheduled ---------------------------------------- When a coroutine function is called and its result is not passed to -:func:`async` or to the :meth:`BaseEventLoop.create_task` method, the execution -of the coroutine object will never be scheduled which is probably a bug. -:ref:`Enable the debug mode of asyncio <asyncio-debug-mode>` to :ref:`log a -warning <asyncio-logger>` to detect it. +:func:`ensure_future` or to the :meth:`BaseEventLoop.create_task` method, +the execution of the coroutine object will never be scheduled which is +probably a bug. :ref:`Enable the debug mode of asyncio <asyncio-debug-mode>` +to :ref:`log a warning <asyncio-logger>` to detect it. Example with the bug:: @@ -190,7 +190,7 @@ Output in debug mode:: File "test.py", line 7, in <module> test() -The fix is to call the :func:`async` function or the +The fix is to call the :func:`ensure_future` function or the :meth:`BaseEventLoop.create_task` method with the coroutine object. .. seealso:: @@ -321,14 +321,18 @@ operations:: print("Pending tasks at exit: %s" % asyncio.Task.all_tasks(loop)) loop.close() -Expected output:: +Expected output: + +.. code-block:: none (1) create file (2) write into file (3) close file Pending tasks at exit: set() -Actual output:: +Actual output: + +.. code-block:: none (3) close file (2) write into file @@ -369,13 +373,17 @@ Pending task destroyed If a pending task is destroyed, the execution of its wrapped :ref:`coroutine <coroutine>` did not complete. It is probably a bug and so a warning is logged. -Example of log:: +Example of log: + +.. code-block:: none Task was destroyed but it is pending! task: <Task pending coro=<kill_me() done, defined at test.py:5> wait_for=<Future pending cb=[Task._wakeup()]>> :ref:`Enable the debug mode of asyncio <asyncio-debug-mode>` to get the -traceback where the task was created. Example of log in debug mode:: +traceback where the task was created. Example of log in debug mode: + +.. code-block:: none Task was destroyed but it is pending! source_traceback: Object created at (most recent call last): diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst index 96468ae567..7ec3aa106e 100644 --- a/Doc/library/asyncio-eventloop.rst +++ b/Doc/library/asyncio-eventloop.rst @@ -38,14 +38,14 @@ Run an event loop that callbacks scheduled by callbacks will not run in that case; they will run the next time :meth:`run_forever` is called. - .. versionchanged:: 3.4.4 + .. versionchanged:: 3.5.1 .. method:: BaseEventLoop.run_until_complete(future) Run until the :class:`Future` is done. If the argument is a :ref:`coroutine object <coroutine>`, it is wrapped by - :func:`async`. + :func:`ensure_future`. Return the Future's result, or raise its exception. @@ -60,7 +60,7 @@ Run an event loop This causes :meth:`run_forever` to exit at the next suitable opportunity (see there for more details). - .. versionchanged:: 3.4.4 + .. versionchanged:: 3.5.1 .. method:: BaseEventLoop.is_closed() @@ -179,6 +179,20 @@ a different clock than :func:`time.time`. The :func:`asyncio.sleep` function. +Futures +------- + +.. method:: BaseEventLoop.create_future() + + Create an :class:`asyncio.Future` object attached to the loop. + + This is a preferred way to create futures in asyncio, as event + loop implementations can provide alternative implementations + of the Future class (with better performance or instrumentation). + + .. versionadded:: 3.5.2 + + Tasks ----- @@ -253,7 +267,7 @@ Creating connections a class. For example, if you want to use a pre-created protocol instance, you can pass ``lambda: my_protocol``. - Options allowing to change how the connection is created: + Options that change how the connection is created: * *ssl*: if given and not false, a SSL/TLS transport is created (by default a plain TCP transport is created). If *ssl* is @@ -285,7 +299,9 @@ Creating connections to bind the socket to locally. The *local_host* and *local_port* are looked up using getaddrinfo(), similarly to *host* and *port*. - On Windows with :class:`ProactorEventLoop`, SSL/TLS is not supported. + .. versionchanged:: 3.5 + + On Windows with :class:`ProactorEventLoop`, SSL/TLS is now supported. .. seealso:: @@ -409,14 +425,16 @@ Creating listening connections This method is a :ref:`coroutine <coroutine>`. - On Windows with :class:`ProactorEventLoop`, SSL/TLS is not supported. + .. versionchanged:: 3.5 + + On Windows with :class:`ProactorEventLoop`, SSL/TLS is now supported. .. seealso:: The function :func:`start_server` creates a (:class:`StreamReader`, :class:`StreamWriter`) pair and calls back a function with this pair. - .. versionchanged:: 3.4.4 + .. versionchanged:: 3.5.1 The *host* parameter can now be a sequence of strings. @@ -473,7 +491,10 @@ Low-level socket operations .. coroutinemethod:: BaseEventLoop.sock_recv(sock, nbytes) - Receive data from the socket. The return value is a bytes object + Receive data from the socket. Modeled after blocking + :meth:`socket.socket.recv` method. + + The return value is a bytes object representing the data received. The maximum amount of data to be received at once is specified by *nbytes*. @@ -482,13 +503,12 @@ Low-level socket operations This method is a :ref:`coroutine <coroutine>`. - .. seealso:: - - The :meth:`socket.socket.recv` method. - .. coroutinemethod:: BaseEventLoop.sock_sendall(sock, data) - Send data to the socket. The socket must be connected to a remote socket. + Send data to the socket. Modeled after blocking + :meth:`socket.socket.sendall` method. + + The socket must be connected to a remote socket. This method continues to send data from *data* until either all data has been sent or an error occurs. ``None`` is returned on success. On error, an exception is raised, and there is no way to determine how much data, if @@ -499,35 +519,35 @@ Low-level socket operations This method is a :ref:`coroutine <coroutine>`. - .. seealso:: - - The :meth:`socket.socket.sendall` method. - .. coroutinemethod:: BaseEventLoop.sock_connect(sock, address) - Connect to a remote socket at *address*. - - The *address* must be already resolved to avoid the trap of hanging the - entire event loop when the address requires doing a DNS lookup. For - example, it must be an IP address, not an hostname, for - :py:data:`~socket.AF_INET` and :py:data:`~socket.AF_INET6` address families. - Use :meth:`getaddrinfo` to resolve the hostname asynchronously. + Connect to a remote socket at *address*. Modeled after + blocking :meth:`socket.socket.connect` method. With :class:`SelectorEventLoop` event loop, the socket *sock* must be non-blocking. This method is a :ref:`coroutine <coroutine>`. + .. versionchanged:: 3.5.2 + ``address`` no longer needs to be resolved. ``sock_connect`` + will try to check if the *address* is already resolved by calling + :func:`socket.inet_pton`. If not, + :meth:`BaseEventLoop.getaddrinfo` will be used to resolve the + *address*. + .. seealso:: - The :meth:`BaseEventLoop.create_connection` method, the - :func:`open_connection` function and the :meth:`socket.socket.connect` - method. + :meth:`BaseEventLoop.create_connection` + and :func:`asyncio.open_connection() <open_connection>`. .. coroutinemethod:: BaseEventLoop.sock_accept(sock) - Accept a connection. The socket must be bound to an address and listening + Accept a connection. Modeled after blocking + :meth:`socket.socket.accept`. + + The socket must be bound to an address and listening for connections. The return value is a pair ``(conn, address)`` where *conn* is a *new* socket object usable to send and receive data on the connection, and *address* is the address bound to the socket on the other end of the @@ -539,8 +559,7 @@ Low-level socket operations .. seealso:: - The :meth:`BaseEventLoop.create_server` method, the :func:`start_server` - function and the :meth:`socket.socket.accept` method. + :meth:`BaseEventLoop.create_server` and :func:`start_server`. Resolve host name @@ -650,7 +669,7 @@ pool of processes). By default, an event loop uses a thread pool executor Error Handling API ------------------ -Allows to customize how exceptions are handled in the event loop. +Allows customizing how exceptions are handled in the event loop. .. method:: BaseEventLoop.set_exception_handler(handler) @@ -665,6 +684,13 @@ Allows to customize how exceptions are handled in the event loop. will be a ``dict`` object (see :meth:`call_exception_handler` documentation for details about context). +.. method:: BaseEventLoop.get_exception_handler() + + Return the exception handler, or ``None`` if the default one + is in use. + + .. versionadded:: 3.5.2 + .. method:: BaseEventLoop.default_exception_handler(context) Default exception handler. @@ -735,11 +761,11 @@ Server Stop serving: close listening sockets and set the :attr:`sockets` attribute to ``None``. - The sockets that represent existing incoming client connections are - leaved open. + The sockets that represent existing incoming client connections are left + open. - The server is closed asynchonously, use the :meth:`wait_closed` coroutine - to wait until the server is closed. + The server is closed asynchronously, use the :meth:`wait_closed` + coroutine to wait until the server is closed. .. coroutinemethod:: wait_closed() diff --git a/Doc/library/asyncio-eventloops.rst b/Doc/library/asyncio-eventloops.rst index f42f65a796..b8f29d78dd 100644 --- a/Doc/library/asyncio-eventloops.rst +++ b/Doc/library/asyncio-eventloops.rst @@ -41,7 +41,7 @@ asyncio currently provides two implementations of event loops: On Windows, only sockets are supported (ex: pipes are not supported): see the `MSDN documentation of select - <http://msdn.microsoft.com/en-us/library/windows/desktop/ms740141%28v=vs.85%29.aspx>`_. + <https://msdn.microsoft.com/en-us/library/windows/desktop/ms740141%28v=vs.85%29.aspx>`_. .. class:: ProactorEventLoop @@ -53,7 +53,7 @@ asyncio currently provides two implementations of event loops: .. seealso:: `MSDN documentation on I/O Completion Ports - <http://msdn.microsoft.com/en-us/library/windows/desktop/aa365198%28v=vs.85%29.aspx>`_. + <https://msdn.microsoft.com/en-us/library/windows/desktop/aa365198%28v=vs.85%29.aspx>`_. Example to use a :class:`ProactorEventLoop` on Windows:: @@ -100,8 +100,6 @@ Common limits of Windows event loops: :class:`ProactorEventLoop` specific limits: -- SSL is not supported: :meth:`~BaseEventLoop.create_connection` and - :meth:`~BaseEventLoop.create_server` cannot be used with SSL for example - :meth:`~BaseEventLoop.create_datagram_endpoint` (UDP) is not supported - :meth:`~BaseEventLoop.add_reader` and :meth:`~BaseEventLoop.add_writer` are not supported @@ -109,9 +107,13 @@ Common limits of Windows event loops: The resolution of the monotonic clock on Windows is usually around 15.6 msec. The best resolution is 0.5 msec. The resolution depends on the hardware (availability of `HPET -<http://en.wikipedia.org/wiki/High_Precision_Event_Timer>`_) and on the Windows +<https://en.wikipedia.org/wiki/High_Precision_Event_Timer>`_) and on the Windows configuration. See :ref:`asyncio delayed calls <asyncio-delayed-calls>`. +.. versionchanged:: 3.5 + + :class:`ProactorEventLoop` now supports SSL. + Mac OS X ^^^^^^^^ diff --git a/Doc/library/asyncio-protocol.rst b/Doc/library/asyncio-protocol.rst index f9298b2d86..23d34d05c0 100644 --- a/Doc/library/asyncio-protocol.rst +++ b/Doc/library/asyncio-protocol.rst @@ -45,7 +45,7 @@ BaseTransport Return ``True`` if the transport is closing or is closed. - .. versionadded:: 3.4.4 + .. versionadded:: 3.5.1 .. method:: get_extra_info(name, default=None) @@ -87,7 +87,7 @@ BaseTransport - ``'subprocess'``: :class:`subprocess.Popen` instance - .. versionchanged:: 3.4.4 + .. versionchanged:: 3.5.1 ``'ssl_object'`` info was added to SSL sockets. @@ -458,9 +458,9 @@ buffer size reaches the low-water mark. Coroutines and protocols ------------------------ -Coroutines can be scheduled in a protocol method using :func:`async`, but there -is no guarantee made about the execution order. Protocols are not aware of -coroutines created in protocol methods and so will not wait for them. +Coroutines can be scheduled in a protocol method using :func:`ensure_future`, +but there is no guarantee made about the execution order. Protocols are not +aware of coroutines created in protocol methods and so will not wait for them. To have a reliable execution order, use :ref:`stream objects <asyncio-streams>` in a coroutine with ``yield from``. For example, the :meth:`StreamWriter.drain` diff --git a/Doc/library/asyncio-queue.rst b/Doc/library/asyncio-queue.rst index 33706726a3..f11c09ac29 100644 --- a/Doc/library/asyncio-queue.rst +++ b/Doc/library/asyncio-queue.rst @@ -8,7 +8,6 @@ Queues: * :class:`Queue` * :class:`PriorityQueue` * :class:`LifoQueue` -* :class:`JoinableQueue` asyncio queue API was designed to be close to classes of the :mod:`queue` module (:class:`~queue.Queue`, :class:`~queue.PriorityQueue`, @@ -144,16 +143,6 @@ LifoQueue first. -JoinableQueue -^^^^^^^^^^^^^ - -.. class:: JoinableQueue - - Deprecated alias for :class:`Queue`. - - .. deprecated:: 3.4.4 - - Exceptions ^^^^^^^^^^ diff --git a/Doc/library/asyncio-stream.rst b/Doc/library/asyncio-stream.rst index 52b93f9ad9..08fe07156a 100644 --- a/Doc/library/asyncio-stream.rst +++ b/Doc/library/asyncio-stream.rst @@ -11,7 +11,7 @@ Stream functions .. note:: - The top-level functions in this module are meant convenience wrappers + The top-level functions in this module are meant as convenience wrappers only; there's really nothing special there, and if they don't do exactly what you want, feel free to copy their code. @@ -142,6 +142,30 @@ StreamReader This method is a :ref:`coroutine <coroutine>`. + .. coroutinemethod:: readuntil(separator=b'\n') + + Read data from the stream until ``separator`` is found. + + On success, the data and separator will be removed from the + internal buffer (consumed). Returned data will include the + separator at the end. + + Configured stream limit is used to check result. Limit sets the + maximal length of data that can be returned, not counting the + separator. + + If an EOF occurs and the complete separator is still not found, + an :exc:`IncompleteReadError` exception will be + raised, and the internal buffer will be reset. The + :attr:`IncompleteReadError.partial` attribute may contain the + separator partially. + + If the data cannot be read because of over limit, a + :exc:`LimitOverrunError` exception will be raised, and the data + will be left in the internal buffer, so it can be read again. + + .. versionadded:: 3.5.2 + .. method:: at_eof() Return ``True`` if the buffer is empty and :meth:`feed_eof` was called. @@ -223,7 +247,7 @@ StreamReaderProtocol .. class:: StreamReaderProtocol(stream_reader, client_connected_cb=None, loop=None) Trivial helper class to adapt between :class:`Protocol` and - :class:`StreamReader`. Sublclass of :class:`Protocol`. + :class:`StreamReader`. Subclass of :class:`Protocol`. *stream_reader* is a :class:`StreamReader` instance, *client_connected_cb* is an optional function called with (stream_reader, stream_writer) when a @@ -251,6 +275,18 @@ IncompleteReadError Read bytes string before the end of stream was reached (:class:`bytes`). +LimitOverrunError +================= + +.. exception:: LimitOverrunError + + Reached the buffer limit while looking for a separator. + + .. attribute:: consumed + + Total number of to be consumed bytes. + + Stream examples =============== diff --git a/Doc/library/asyncio-subprocess.rst b/Doc/library/asyncio-subprocess.rst index 21dae54e38..51ce4278da 100644 --- a/Doc/library/asyncio-subprocess.rst +++ b/Doc/library/asyncio-subprocess.rst @@ -51,7 +51,7 @@ Create a subprocess: high-level API using Process It is the application's responsibility to ensure that all whitespace and metacharacters are quoted appropriately to avoid `shell injection - <http://en.wikipedia.org/wiki/Shell_injection#Shell_injection>`_ + <https://en.wikipedia.org/wiki/Shell_injection#Shell_injection>`_ vulnerabilities. The :func:`shlex.quote` function can be used to properly escape whitespace and shell metacharacters in strings that are going to be used to construct shell commands. @@ -134,7 +134,7 @@ Run subprocesses asynchronously using the :mod:`subprocess` module. It is the application's responsibility to ensure that all whitespace and metacharacters are quoted appropriately to avoid `shell injection - <http://en.wikipedia.org/wiki/Shell_injection#Shell_injection>`_ + <https://en.wikipedia.org/wiki/Shell_injection#Shell_injection>`_ vulnerabilities. The :func:`shlex.quote` function can be used to properly escape whitespace and shell metacharacters in strings that are going to be used to construct shell commands. diff --git a/Doc/library/asyncio-sync.rst b/Doc/library/asyncio-sync.rst index ad3b523f98..0909352152 100644 --- a/Doc/library/asyncio-sync.rst +++ b/Doc/library/asyncio-sync.rst @@ -52,7 +52,7 @@ Lock :meth:`acquire` is a coroutine and should be called with ``yield from``. Locks also support the context management protocol. ``(yield from lock)`` - should be used as context manager expression. + should be used as the context manager expression. This class is :ref:`not thread safe <asyncio-multithreading>`. @@ -71,14 +71,14 @@ Lock lock = Lock() ... with (yield from lock): - ... + ... Lock objects can be tested for locking state:: if not lock.locked(): - yield from lock + yield from lock else: - # lock is acquired + # lock is acquired ... .. method:: locked() diff --git a/Doc/library/asyncio-task.rst b/Doc/library/asyncio-task.rst index 76f084a267..de6ee58e92 100644 --- a/Doc/library/asyncio-task.rst +++ b/Doc/library/asyncio-task.rst @@ -8,17 +8,23 @@ Tasks and coroutines Coroutines ---------- -A coroutine is a generator that follows certain conventions. For -documentation purposes, all coroutines should be decorated with -``@asyncio.coroutine``, but this cannot be strictly enforced. - -Coroutines use the ``yield from`` syntax introduced in :pep:`380`, +Coroutines used with :mod:`asyncio` may be implemented using the +:keyword:`async def` statement, or by using :term:`generators <generator>`. +The :keyword:`async def` type of coroutine was added in Python 3.5, and +is recommended if there is no need to support older Python versions. + +Generator-based coroutines should be decorated with :func:`@asyncio.coroutine +<asyncio.coroutine>`, although this is not strictly enforced. +The decorator enables compatibility with :keyword:`async def` coroutines, +and also serves as documentation. Generator-based +coroutines use the ``yield from`` syntax introduced in :pep:`380`, instead of the original ``yield`` syntax. The word "coroutine", like the word "generator", is used for two different (though related) concepts: -- The function that defines a coroutine (a function definition +- The function that defines a coroutine + (a function definition using :keyword:`async def` or decorated with ``@asyncio.coroutine``). If disambiguation is needed we will call this a *coroutine function* (:func:`iscoroutinefunction` returns ``True``). @@ -30,29 +36,30 @@ different (though related) concepts: Things a coroutine can do: -- ``result = yield from future`` -- suspends the coroutine until the +- ``result = await future`` or ``result = yield from future`` -- + suspends the coroutine until the future is done, then returns the future's result, or raises an exception, which will be propagated. (If the future is cancelled, it will raise a ``CancelledError`` exception.) Note that tasks are futures, and everything said about futures also applies to tasks. -- ``result = yield from coroutine`` -- wait for another coroutine to +- ``result = await coroutine`` or ``result = yield from coroutine`` -- + wait for another coroutine to produce a result (or raise an exception, which will be propagated). The ``coroutine`` expression must be a *call* to another coroutine. - ``return expression`` -- produce a result to the coroutine that is - waiting for this one using ``yield from``. + waiting for this one using :keyword:`await` or ``yield from``. - ``raise exception`` -- raise an exception in the coroutine that is - waiting for this one using ``yield from``. + waiting for this one using :keyword:`await` or ``yield from``. -Calling a coroutine does not start its code running -- it is just a -generator, and the coroutine object returned by the call is really a -generator object, which doesn't do anything until you iterate over it. -In the case of a coroutine object, there are two basic ways to start -it running: call ``yield from coroutine`` from another coroutine +Calling a coroutine does not start its code running -- +the coroutine object returned by the call doesn't do anything until you +schedule its execution. There are two basic ways to start it running: +call ``await coroutine`` or ``yield from coroutine`` from another coroutine (assuming the other coroutine is already running!), or schedule its execution -using the :func:`async` function or the :meth:`BaseEventLoop.create_task` +using the :func:`ensure_future` function or the :meth:`BaseEventLoop.create_task` method. @@ -60,9 +67,15 @@ Coroutines (and tasks) can only run when the event loop is running. .. decorator:: coroutine - Decorator to mark coroutines. + Decorator to mark generator-based coroutines. This enables + the generator use :keyword:`!yield from` to call :keyword:`async + def` coroutines, and also enables the generator to be called by + :keyword:`async def` coroutines, for instance using an + :keyword:`await` expression. + + There is no need to decorate :keyword:`async def` coroutines themselves. - If the coroutine is not yielded from before it is destroyed, an error + If the generator is not yielded from before it is destroyed, an error message is logged. See :ref:`Detect coroutines never scheduled <asyncio-coroutine-not-scheduled>`. @@ -72,7 +85,7 @@ Coroutines (and tasks) can only run when the event loop is running. even if they are plain Python functions returning a :class:`Future`. This is intentional to have a freedom of tweaking the implementation of these functions in the future. If such a function is needed to be - used in a callback-style code, wrap its result with :func:`async`. + used in a callback-style code, wrap its result with :func:`ensure_future`. .. _asyncio-hello-world-coroutine: @@ -84,8 +97,7 @@ Example of coroutine displaying ``"Hello World"``:: import asyncio - @asyncio.coroutine - def hello_world(): + async def hello_world(): print("Hello World!") loop = asyncio.get_event_loop() @@ -111,20 +123,30 @@ using the :meth:`sleep` function:: import asyncio import datetime - @asyncio.coroutine - def display_date(loop): + async def display_date(loop): end_time = loop.time() + 5.0 while True: print(datetime.datetime.now()) if (loop.time() + 1.0) >= end_time: break - yield from asyncio.sleep(1) + await asyncio.sleep(1) loop = asyncio.get_event_loop() # Blocking call which returns when the display_date() coroutine is done loop.run_until_complete(display_date(loop)) loop.close() +The same coroutine implemented using a generator:: + + @asyncio.coroutine + def display_date(loop): + end_time = loop.time() + 5.0 + while True: + print(datetime.datetime.now()) + if (loop.time() + 1.0) >= end_time: + break + yield from asyncio.sleep(1) + .. seealso:: The :ref:`display the current date with call_later() @@ -139,15 +161,13 @@ Example chaining coroutines:: import asyncio - @asyncio.coroutine - def compute(x, y): + async def compute(x, y): print("Compute %s + %s ..." % (x, y)) - yield from asyncio.sleep(1.0) + await asyncio.sleep(1.0) return x + y - @asyncio.coroutine - def print_sum(x, y): - result = yield from compute(x, y) + async def print_sum(x, y): + result = await compute(x, y) print("%s + %s = %s" % (x, y, result)) loop = asyncio.get_event_loop() @@ -375,7 +395,7 @@ Task <coroutine>` did not complete. It is probably a bug and a warning is logged: see :ref:`Pending task destroyed <asyncio-pending-task-destroyed>`. - Don't directly create :class:`Task` instances: use the :func:`async` + Don't directly create :class:`Task` instances: use the :func:`ensure_future` function or the :meth:`BaseEventLoop.create_task` method. This class is :ref:`not thread safe <asyncio-multithreading>`. @@ -490,7 +510,7 @@ Task functions .. note:: - In the functions below, the optional *loop* argument allows to explicitly set + In the functions below, the optional *loop* argument allows explicitly setting the event loop object used by the underlying task or coroutine. If it's not provided, the default event loop is used. @@ -521,6 +541,9 @@ Task functions .. versionadded:: 3.4.4 + .. versionchanged:: 3.5.1 + The function accepts any :term:`awaitable` object. + .. seealso:: The :meth:`BaseEventLoop.create_task` method. @@ -552,12 +575,14 @@ Task functions .. function:: iscoroutine(obj) - Return ``True`` if *obj* is a :ref:`coroutine object <coroutine>`. + Return ``True`` if *obj* is a :ref:`coroutine object <coroutine>`, + which may be based on a generator or an :keyword:`async def` coroutine. -.. function:: iscoroutinefunction(obj) +.. function:: iscoroutinefunction(func) - Return ``True`` if *func* is a decorated :ref:`coroutine function - <coroutine>`. + Return ``True`` if *func* is determined to be a :ref:`coroutine function + <coroutine>`, which may be a decorated generator function or an + :keyword:`async def` function. .. function:: run_coroutine_threadsafe(coro, loop) @@ -593,10 +618,11 @@ Task functions .. note:: - Unlike the functions above, :func:`run_coroutine_threadsafe` requires the - *loop* argument to be passed explicitely. + Unlike other functions from the module, + :func:`run_coroutine_threadsafe` requires the *loop* argument to + be passed explicitly. - .. versionadded:: 3.4.4, 3.5.1 + .. versionadded:: 3.5.1 .. coroutinefunction:: sleep(delay, result=None, \*, loop=None) @@ -636,18 +662,6 @@ Task functions except CancelledError: res = None -.. function:: timeout(timeout, \*, loop=None) - - Return a context manager that cancels a block on *timeout* expiring:: - - with timeout(1.5): - yield from inner() - - 1. If ``inner()`` is executed faster than in ``1.5`` seconds - nothing happens. - 2. Otherwise ``inner()`` is cancelled internally but - :exc:`asyncio.TimeoutError` is raised outside of - context manager scope. .. coroutinefunction:: wait(futures, \*, loop=None, timeout=None,\ return_when=ALL_COMPLETED) @@ -715,5 +729,3 @@ Task functions .. versionchanged:: 3.4.3 If the wait is cancelled, the future *fut* is now also cancelled. - - diff --git a/Doc/library/asyncio.rst b/Doc/library/asyncio.rst index 9b4d65e5da..f764c683b1 100644 --- a/Doc/library/asyncio.rst +++ b/Doc/library/asyncio.rst @@ -4,6 +4,10 @@ .. module:: asyncio :synopsis: Asynchronous I/O, event loop, coroutines and tasks. +.. versionadded:: 3.4 + +**Source code:** :source:`Lib/asyncio/` + .. note:: The asyncio package has been included in the standard library on a @@ -11,10 +15,6 @@ changes (up to and including removal of the module) may occur if deemed necessary by the core developers. -.. versionadded:: 3.4 - -**Source code:** :source:`Lib/asyncio/` - -------------- This module provides infrastructure for writing single-threaded concurrent diff --git a/Doc/library/asyncore.rst b/Doc/library/asyncore.rst index 917d0448c2..61061be34e 100644 --- a/Doc/library/asyncore.rst +++ b/Doc/library/asyncore.rst @@ -4,6 +4,7 @@ .. module:: asyncore :synopsis: A base class for developing asynchronous socket handling services. + .. moduleauthor:: Sam Rushing <rushing@nightmare.com> .. sectionauthor:: Christopher Petrilli <petrilli@amber.org> .. sectionauthor:: Steve Holden <sholden@holdenweb.com> @@ -315,8 +316,8 @@ implement its socket handling:: self.buffer = self.buffer[sent:] - client = HTTPClient('www.python.org', '/') - asyncore.loop() + client = HTTPClient('www.python.org', '/') + asyncore.loop() .. _asyncore-example-2: diff --git a/Doc/library/atexit.rst b/Doc/library/atexit.rst index dbdd81e6d0..1d84d4587f 100644 --- a/Doc/library/atexit.rst +++ b/Doc/library/atexit.rst @@ -3,9 +3,11 @@ .. module:: atexit :synopsis: Register and execute cleanup functions. + .. moduleauthor:: Skip Montanaro <skip@pobox.com> .. sectionauthor:: Skip Montanaro <skip@pobox.com> +-------------- The :mod:`atexit` module defines functions to register and unregister cleanup functions. Functions thus registered are automatically executed upon normal diff --git a/Doc/library/audioop.rst b/Doc/library/audioop.rst index ce127aa06a..bad9da2ec6 100644 --- a/Doc/library/audioop.rst +++ b/Doc/library/audioop.rst @@ -4,10 +4,11 @@ .. module:: audioop :synopsis: Manipulate raw audio data. +-------------- The :mod:`audioop` module contains some useful operations on sound fragments. It operates on sound fragments consisting of signed integer samples 8, 16, 24 -or 32 bits wide, stored in :term:`bytes-like object`\ s. All scalar items are +or 32 bits wide, stored in :term:`bytes-like objects <bytes-like object>`. All scalar items are integers, unless specified otherwise. .. versionchanged:: 3.4 @@ -276,6 +277,6 @@ sample and subtract the whole output sample from the input sample:: # out_test) prefill = '\0'*(pos+ipos)*2 postfill = '\0'*(len(inputdata)-len(prefill)-len(outputdata)) - outputdata = prefill + audioop.mul(outputdata,2,-factor) + postfill + outputdata = prefill + audioop.mul(outputdata, 2, -factor) + postfill return audioop.add(inputdata, outputdata, 2) diff --git a/Doc/library/base64.rst b/Doc/library/base64.rst index eba4b36e8a..080d9d77ec 100644 --- a/Doc/library/base64.rst +++ b/Doc/library/base64.rst @@ -5,11 +5,14 @@ :synopsis: RFC 3548: Base16, Base32, Base64 Data Encodings; Base85 and Ascii85 +**Source code:** :source:`Lib/base64.py` .. index:: pair: base64; encoding single: MIME; base64 encoding +-------------- + This module provides functions for encoding binary data to printable ASCII characters and decoding such encodings back to binary data. It provides encoding and decoding functions for the encodings specified in @@ -21,88 +24,103 @@ safely sent by email, used as parts of URLs, or included as part of an HTTP POST request. The encoding algorithm is not the same as the :program:`uuencode` program. -There are two :rfc:`3548` interfaces provided by this module. The modern -interface supports encoding and decoding ASCII byte string objects using all -three :rfc:`3548` defined alphabets (normal, URL-safe, and filesystem-safe). -Additionally, the decoding functions of the modern interface also accept -Unicode strings containing only ASCII characters. The legacy interface provides -for encoding and decoding to and from file-like objects as well as byte -strings, but only using the Base64 standard alphabet. +There are two interfaces provided by this module. The modern interface +supports encoding :term:`bytes-like objects <bytes-like object>` to ASCII +:class:`bytes`, and decoding :term:`bytes-like objects <bytes-like object>` or +strings containing ASCII to :class:`bytes`. Both base-64 alphabets +defined in :rfc:`3548` (normal, and URL- and filesystem-safe) are supported. + +The legacy interface does not support decoding from strings, but it does +provide functions for encoding and decoding to and from :term:`file objects +<file object>`. It only supports the Base64 standard alphabet, and it adds +newlines every 76 characters as per :rfc:`2045`. Note that if you are looking +for :rfc:`2045` support you probably want to be looking at the :mod:`email` +package instead. + .. versionchanged:: 3.3 ASCII-only Unicode strings are now accepted by the decoding functions of the modern interface. .. versionchanged:: 3.4 - Any :term:`bytes-like object`\ s are now accepted by all + Any :term:`bytes-like objects <bytes-like object>` are now accepted by all encoding and decoding functions in this module. Ascii85/Base85 support added. The modern interface provides: .. function:: b64encode(s, altchars=None) - Encode a byte string using Base64. + Encode the :term:`bytes-like object` *s* using Base64 and return the encoded + :class:`bytes`. - *s* is the string to encode. Optional *altchars* must be a string of at least + Optional *altchars* must be a :term:`bytes-like object` of at least length 2 (additional characters are ignored) which specifies an alternative alphabet for the ``+`` and ``/`` characters. This allows an application to e.g. generate URL or filesystem safe Base64 strings. The default is ``None``, for which the standard Base64 alphabet is used. - The encoded byte string is returned. - .. function:: b64decode(s, altchars=None, validate=False) - Decode a Base64 encoded byte string. + Decode the Base64 encoded :term:`bytes-like object` or ASCII string + *s* and return the decoded :class:`bytes`. - *s* is the byte string to decode. Optional *altchars* must be a string of + Optional *altchars* must be a :term:`bytes-like object` or ASCII string of at least length 2 (additional characters are ignored) which specifies the alternative alphabet used instead of the ``+`` and ``/`` characters. - The decoded string is returned. A :exc:`binascii.Error` exception is raised + A :exc:`binascii.Error` exception is raised if *s* is incorrectly padded. - If *validate* is ``False`` (the default), non-base64-alphabet characters are + If *validate* is ``False`` (the default), characters that are neither + in the normal base-64 alphabet nor the alternative alphabet are discarded prior to the padding check. If *validate* is ``True``, - non-base64-alphabet characters in the input result in a + these non-alphabet characters in the input result in a :exc:`binascii.Error`. .. function:: standard_b64encode(s) - Encode byte string *s* using the standard Base64 alphabet. + Encode :term:`bytes-like object` *s* using the standard Base64 alphabet + and return the encoded :class:`bytes`. .. function:: standard_b64decode(s) - Decode byte string *s* using the standard Base64 alphabet. + Decode :term:`bytes-like object` or ASCII string *s* using the standard + Base64 alphabet and return the decoded :class:`bytes`. .. function:: urlsafe_b64encode(s) - Encode byte string *s* using a URL-safe alphabet, which substitutes ``-`` instead of - ``+`` and ``_`` instead of ``/`` in the standard Base64 alphabet. The result + Encode :term:`bytes-like object` *s* using the + URL- and filesystem-safe alphabet, which + substitutes ``-`` instead of ``+`` and ``_`` instead of ``/`` in the + standard Base64 alphabet, and return the encoded :class:`bytes`. The result can still contain ``=``. .. function:: urlsafe_b64decode(s) - Decode byte string *s* using a URL-safe alphabet, which substitutes ``-`` instead of - ``+`` and ``_`` instead of ``/`` in the standard Base64 alphabet. + Decode :term:`bytes-like object` or ASCII string *s* + using the URL- and filesystem-safe + alphabet, which substitutes ``-`` instead of ``+`` and ``_`` instead of + ``/`` in the standard Base64 alphabet, and return the decoded + :class:`bytes`. .. function:: b32encode(s) - Encode a byte string using Base32. *s* is the string to encode. The encoded string - is returned. + Encode the :term:`bytes-like object` *s* using Base32 and return the + encoded :class:`bytes`. .. function:: b32decode(s, casefold=False, map01=None) - Decode a Base32 encoded byte string. + Decode the Base32 encoded :term:`bytes-like object` or ASCII string *s* and + return the decoded :class:`bytes`. - *s* is the byte string to decode. Optional *casefold* is a flag specifying + Optional *casefold* is a flag specifying whether a lowercase alphabet is acceptable as input. For security purposes, the default is ``False``. @@ -113,46 +131,45 @@ The modern interface provides: digit 0 is always mapped to the letter O). For security purposes the default is ``None``, so that 0 and 1 are not allowed in the input. - The decoded byte string is returned. A :exc:`binascii.Error` is raised if *s* is + A :exc:`binascii.Error` is raised if *s* is incorrectly padded or if there are non-alphabet characters present in the - string. + input. .. function:: b16encode(s) - Encode a byte string using Base16. - - *s* is the string to encode. The encoded byte string is returned. + Encode the :term:`bytes-like object` *s* using Base16 and return the + encoded :class:`bytes`. .. function:: b16decode(s, casefold=False) - Decode a Base16 encoded byte string. + Decode the Base16 encoded :term:`bytes-like object` or ASCII string *s* and + return the decoded :class:`bytes`. - *s* is the string to decode. Optional *casefold* is a flag specifying whether a + Optional *casefold* is a flag specifying whether a lowercase alphabet is acceptable as input. For security purposes, the default is ``False``. - The decoded byte string is returned. A :exc:`TypeError` is raised if *s* were + A :exc:`binascii.Error` is raised if *s* is incorrectly padded or if there are non-alphabet characters present in the - string. - + input. -.. function:: a85encode(s, *, foldspaces=False, wrapcol=0, pad=False, adobe=False) - Encode a byte string using Ascii85. +.. function:: a85encode(b, *, foldspaces=False, wrapcol=0, pad=False, adobe=False) - *s* is the string to encode. The encoded byte string is returned. + Encode the :term:`bytes-like object` *b* using Ascii85 and return the + encoded :class:`bytes`. *foldspaces* is an optional flag that uses the special short sequence 'y' instead of 4 consecutive spaces (ASCII 0x20) as supported by 'btoa'. This feature is not supported by the "standard" Ascii85 encoding. - *wrapcol* controls whether the output should have newline (``'\n'``) + *wrapcol* controls whether the output should have newline (``b'\n'``) characters added to it. If this is non-zero, each output line will be at most this many characters long. - *pad* controls whether the input string is padded to a multiple of 4 + *pad* controls whether the input is padded to a multiple of 4 before encoding. Note that the ``btoa`` implementation always pads. *adobe* controls whether the encoded byte sequence is framed with ``<~`` @@ -161,11 +178,10 @@ The modern interface provides: .. versionadded:: 3.4 -.. function:: a85decode(s, *, foldspaces=False, adobe=False, ignorechars=b' \\t\\n\\r\\v') +.. function:: a85decode(b, *, foldspaces=False, adobe=False, ignorechars=b' \\t\\n\\r\\v') - Decode an Ascii85 encoded byte string. - - *s* is the byte string to decode. + Decode the Ascii85 encoded :term:`bytes-like object` or ASCII string *b* and + return the decoded :class:`bytes`. *foldspaces* is a flag that specifies whether the 'y' short sequence should be accepted as shorthand for 4 consecutive spaces (ASCII 0x20). @@ -174,27 +190,29 @@ The modern interface provides: *adobe* controls whether the input sequence is in Adobe Ascii85 format (i.e. is framed with <~ and ~>). - *ignorechars* should be a byte string containing characters to ignore + *ignorechars* should be a :term:`bytes-like object` or ASCII string + containing characters to ignore from the input. This should only contain whitespace characters, and by default contains all whitespace characters in ASCII. .. versionadded:: 3.4 -.. function:: b85encode(s, pad=False) +.. function:: b85encode(b, pad=False) - Encode a byte string using base85, as used in e.g. git-style binary - diffs. + Encode the :term:`bytes-like object` *b* using base85 (as used in e.g. + git-style binary diffs) and return the encoded :class:`bytes`. - If *pad* is true, the input is padded with "\\0" so its length is a - multiple of 4 characters before encoding. + If *pad* is true, the input is padded with ``b'\0'`` so its length is a + multiple of 4 bytes before encoding. .. versionadded:: 3.4 .. function:: b85decode(b) - Decode base85-encoded byte string. Padding is implicitly removed, if + Decode the base85-encoded :term:`bytes-like object` or ASCII string *b* and + return the decoded :class:`bytes`. Padding is implicitly removed, if necessary. .. versionadded:: 3.4 @@ -214,15 +232,15 @@ The legacy interface: Decode the contents of the binary *input* file and write the resulting binary data to the *output* file. *input* and *output* must be :term:`file objects - <file object>`. *input* will be read until ``input.read()`` returns an empty - bytes object. + <file object>`. *input* will be read until ``input.readline()`` returns an + empty bytes object. .. function:: decodebytes(s) decodestring(s) - Decode the byte string *s*, which must contain one or more lines of base64 - encoded data, and return a byte string containing the resulting binary data. + Decode the :term:`bytes-like object` *s*, which must contain one or more + lines of base64 encoded data, and return the decoded :class:`bytes`. ``decodestring`` is a deprecated alias. .. versionadded:: 3.1 @@ -233,17 +251,19 @@ The legacy interface: Encode the contents of the binary *input* file and write the resulting base64 encoded data to the *output* file. *input* and *output* must be :term:`file objects <file object>`. *input* will be read until ``input.read()`` returns - an empty bytes object. :func:`encode` returns the encoded data plus a trailing - newline character (``b'\n'``). + an empty bytes object. :func:`encode` inserts a newline character (``b'\n'``) + after every 76 bytes of the output, as well as ensuring that the output + always ends with a newline, as per :rfc:`2045` (MIME). .. function:: encodebytes(s) encodestring(s) - Encode the byte string *s*, which can contain arbitrary binary data, and - return a byte string containing one or more lines of base64-encoded data. - :func:`encodebytes` returns a string containing one or more lines of - base64-encoded data always including an extra trailing newline (``b'\n'``). + Encode the :term:`bytes-like object` *s*, which can contain arbitrary binary + data, and return :class:`bytes` containing the base64-encoded data, with newlines + (``b'\n'``) inserted after every 76 bytes of output, and ensuring that + there is a trailing newline, as per :rfc:`2045` (MIME). + ``encodestring`` is a deprecated alias. diff --git a/Doc/library/binascii.rst b/Doc/library/binascii.rst index dbe535d94d..878d8db53c 100644 --- a/Doc/library/binascii.rst +++ b/Doc/library/binascii.rst @@ -5,12 +5,13 @@ :synopsis: Tools for converting between binary and various ASCII-encoded binary representations. - .. index:: module: uu module: base64 module: binhex +-------------- + The :mod:`binascii` module contains a number of methods to convert between binary and various ASCII-encoded binary representations. Normally, you will not use these functions directly but use wrapper modules like :mod:`uu`, @@ -21,7 +22,7 @@ higher-level modules. .. note:: ``a2b_*`` functions accept Unicode strings containing only ASCII characters. - Other functions only accept :term:`bytes-like object`\ s (such as + Other functions only accept :term:`bytes-like objects <bytes-like object>` (such as :class:`bytes`, :class:`bytearray` and other objects that support the buffer protocol). @@ -112,15 +113,16 @@ The :mod:`binascii` module defines the following functions: possibly the last fragment). -.. function:: crc_hqx(data, crc) +.. function:: crc_hqx(data, value) - Compute the binhex4 crc value of *data*, starting with an initial *crc* and - returning the result. + Compute the binhex4 crc value of *data*, starting with *value* as the + initial crc, and return the result. -.. function:: crc32(data[, crc]) +.. function:: crc32(data[, value]) - Compute CRC-32, the 32-bit checksum of data, starting with an initial crc. This + Compute CRC-32, the 32-bit checksum of *data*, starting with an + initial CRC of *value*. The default initial CRC is zero. The algorithm is consistent with the ZIP file checksum. Since the algorithm is designed for use as a checksum algorithm, it is not suitable for use as a general hash algorithm. Use as follows:: @@ -128,15 +130,13 @@ The :mod:`binascii` module defines the following functions: print(binascii.crc32(b"hello world")) # Or, in two pieces: crc = binascii.crc32(b"hello") - crc = binascii.crc32(b" world", crc) & 0xffffffff + crc = binascii.crc32(b" world", crc) print('crc32 = {:#010x}'.format(crc)) -.. note:: - To generate the same numeric value across all Python versions and - platforms use crc32(data) & 0xffffffff. If you are only using - the checksum in packed binary format this is not necessary as the - return value is the correct 32bit binary representation - regardless of sign. + .. versionchanged:: 3.0 + The result is always unsigned. + To generate the same numeric value across all Python versions and + platforms, use ``crc32(data) & 0xffffffff``. .. function:: b2a_hex(data) @@ -152,8 +152,8 @@ The :mod:`binascii` module defines the following functions: Return the binary data represented by the hexadecimal string *hexstr*. This function is the inverse of :func:`b2a_hex`. *hexstr* must contain an even number - of hexadecimal digits (which can be upper or lower case), otherwise a - :exc:`TypeError` is raised. + of hexadecimal digits (which can be upper or lower case), otherwise an + :exc:`Error` exception is raised. .. exception:: Error diff --git a/Doc/library/binhex.rst b/Doc/library/binhex.rst index 43c78237d2..359ab23b2f 100644 --- a/Doc/library/binhex.rst +++ b/Doc/library/binhex.rst @@ -4,6 +4,9 @@ .. module:: binhex :synopsis: Encode and decode files in binhex4 format. +**Source code:** :source:`Lib/binhex.py` + +-------------- This module encodes and decodes files in binhex4 format, a format allowing representation of Macintosh files in ASCII. Only the data fork is handled. diff --git a/Doc/library/bisect.rst b/Doc/library/bisect.rst index 13b0147190..6bf7814b25 100644 --- a/Doc/library/bisect.rst +++ b/Doc/library/bisect.rst @@ -60,7 +60,7 @@ The following functions are provided: .. seealso:: `SortedCollection recipe - <http://code.activestate.com/recipes/577197-sortedcollection/>`_ that uses + <https://code.activestate.com/recipes/577197-sortedcollection/>`_ that uses bisect to build a full-featured collection class with straight-forward search methods and support for a key-function. The keys are precomputed to save unnecessary calls to the key function during searches. diff --git a/Doc/library/builtins.rst b/Doc/library/builtins.rst index 2cca1d05df..8fb1fef6da 100644 --- a/Doc/library/builtins.rst +++ b/Doc/library/builtins.rst @@ -4,6 +4,7 @@ .. module:: builtins :synopsis: The module that provides the built-in namespace. +-------------- This module provides direct access to all 'built-in' identifiers of Python; for example, ``builtins.open`` is the full name for the built-in function @@ -36,6 +37,6 @@ that wants to implement an :func:`open` function that wraps the built-in As an implementation detail, most modules have the name ``__builtins__`` made available as part of their globals. The value of ``__builtins__`` is normally -either this module or the value of this module's :attr:`__dict__` attribute. +either this module or the value of this module's :attr:`~object.__dict__` attribute. Since this is an implementation detail, it may not be used by alternate implementations of Python. diff --git a/Doc/library/bz2.rst b/Doc/library/bz2.rst index 488cda5968..6c49d9fffe 100644 --- a/Doc/library/bz2.rst +++ b/Doc/library/bz2.rst @@ -3,11 +3,15 @@ .. module:: bz2 :synopsis: Interfaces for bzip2 compression and decompression. + .. moduleauthor:: Gustavo Niemeyer <niemeyer@conectiva.com> .. moduleauthor:: Nadeem Vawda <nadeem.vawda@gmail.com> .. sectionauthor:: Gustavo Niemeyer <niemeyer@conectiva.com> .. sectionauthor:: Nadeem Vawda <nadeem.vawda@gmail.com> +**Source code:** :source:`Lib/bz2.py` + +-------------- This module provides a comprehensive interface for compressing and decompressing data using the bzip2 compression algorithm. @@ -120,6 +124,10 @@ All of the classes in this module may safely be accessed from multiple threads. .. versionchanged:: 3.4 The ``'x'`` (exclusive creation) mode was added. + .. versionchanged:: 3.5 + The :meth:`~io.BufferedIOBase.read` method now accepts an argument of + ``None``. + Incremental (de)compression --------------------------- @@ -162,15 +170,32 @@ Incremental (de)compression you need to decompress a multi-stream input with :class:`BZ2Decompressor`, you must use a new decompressor for each stream. - .. method:: decompress(data) + .. method:: decompress(data, max_length=-1) + + Decompress *data* (a :term:`bytes-like object`), returning + uncompressed data as bytes. Some of *data* may be buffered + internally, for use in later calls to :meth:`decompress`. The + returned data should be concatenated with the output of any + previous calls to :meth:`decompress`. - Provide data to the decompressor object. Returns a chunk of decompressed - data if possible, or an empty byte string otherwise. + If *max_length* is nonnegative, returns at most *max_length* + bytes of decompressed data. If this limit is reached and further + output can be produced, the :attr:`~.needs_input` attribute will + be set to ``False``. In this case, the next call to + :meth:`~.decompress` may provide *data* as ``b''`` to obtain + more of the output. - Attempting to decompress data after the end of the current stream is - reached raises an :exc:`EOFError`. If any data is found after the end of - the stream, it is ignored and saved in the :attr:`unused_data` attribute. + If all of the input data was decompressed and returned (either + because this was less than *max_length* bytes, or because + *max_length* was negative), the :attr:`~.needs_input` attribute + will be set to ``True``. + Attempting to decompress data after the end of stream is reached + raises an `EOFError`. Any data found after the end of the + stream is ignored and saved in the :attr:`~.unused_data` attribute. + + .. versionchanged:: 3.5 + Added the *max_length* parameter. .. attribute:: eof @@ -186,6 +211,13 @@ Incremental (de)compression If this attribute is accessed before the end of the stream has been reached, its value will be ``b''``. + .. attribute:: needs_input + + ``False`` if the :meth:`.decompress` method can provide more + decompressed data before requiring new uncompressed input. + + .. versionadded:: 3.5 + One-shot (de)compression ------------------------ diff --git a/Doc/library/calendar.rst b/Doc/library/calendar.rst index 3187c43a84..df76e33668 100644 --- a/Doc/library/calendar.rst +++ b/Doc/library/calendar.rst @@ -4,6 +4,7 @@ .. module:: calendar :synopsis: Functions for working with calendars, including some emulation of the Unix cal program. + .. sectionauthor:: Drew Csillag <drew_csillag@geocities.com> **Source code:** :source:`Lib/calendar.py` diff --git a/Doc/library/cgi.rst b/Doc/library/cgi.rst index 74abed5c55..41219eeaab 100644 --- a/Doc/library/cgi.rst +++ b/Doc/library/cgi.rst @@ -4,6 +4,7 @@ .. module:: cgi :synopsis: Helpers for running Python scripts via the Common Gateway Interface. +**Source code:** :source:`Lib/cgi.py` .. index:: pair: WWW; server @@ -13,8 +14,6 @@ single: URL single: Common Gateway Interface -**Source code:** :source:`Lib/cgi.py` - -------------- Support module for Common Gateway Interface (CGI) scripts. @@ -157,6 +156,9 @@ return bytes):: if not line: break linecount = linecount + 1 +:class:`FieldStorage` objects also support being used in a :keyword:`with` +statement, which will automatically close them when done. + If an error is encountered when obtaining the contents of an uploaded file (for example, when the user interrupts the form submission by clicking on a Back or Cancel button) the :attr:`~FieldStorage.done` attribute of the @@ -182,6 +184,10 @@ A form submitted via POST that also has a query string will contain both The :attr:`~FieldStorage.file` attribute is automatically closed upon the garbage collection of the creating :class:`FieldStorage` instance. +.. versionchanged:: 3.5 + Added support for the context management protocol to the + :class:`FieldStorage` class. + Higher Level Interface ---------------------- @@ -436,7 +442,9 @@ installing a copy of this module file (:file:`cgi.py`) as a CGI script. When invoked as a script, the file will dump its environment and the contents of the form in HTML form. Give it the right mode etc, and send it a request. If it's installed in the standard :file:`cgi-bin` directory, it should be possible to -send it a request by entering a URL into your browser of the form:: +send it a request by entering a URL into your browser of the form: + +.. code-block:: none http://yourhostname/cgi-bin/cgi.py?name=Joe+Blow&addr=At+Home @@ -528,4 +536,3 @@ Common problems and solutions order the field values should be supplied in, but knowing whether a request was received from a conforming browser, or even from a browser at all, is tedious and error-prone. - diff --git a/Doc/library/cgitb.rst b/Doc/library/cgitb.rst index 6827c8eb5b..b65a6355fe 100644 --- a/Doc/library/cgitb.rst +++ b/Doc/library/cgitb.rst @@ -3,9 +3,11 @@ .. module:: cgitb :synopsis: Configurable traceback handler for CGI scripts. + .. moduleauthor:: Ka-Ping Yee <ping@lfw.org> .. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org> +**Source code:** :source:`Lib/cgitb.py` .. index:: single: CGI; exceptions @@ -13,6 +15,8 @@ single: exceptions; in CGI scripts single: tracebacks; in CGI scripts +-------------- + The :mod:`cgitb` module provides a special exception handler for Python scripts. (Its name is a bit misleading. It was originally designed to display extensive traceback information in HTML for CGI scripts. It was later generalized to also diff --git a/Doc/library/chunk.rst b/Doc/library/chunk.rst index a90e9f808b..5e24df923e 100644 --- a/Doc/library/chunk.rst +++ b/Doc/library/chunk.rst @@ -3,9 +3,11 @@ .. module:: chunk :synopsis: Module to read IFF chunks. + .. moduleauthor:: Sjoerd Mullender <sjoerd@acm.org> .. sectionauthor:: Sjoerd Mullender <sjoerd@acm.org> +**Source code:** :source:`Lib/chunk.py` .. index:: single: Audio Interchange File Format @@ -14,6 +16,8 @@ single: Real Media File Format single: RMFF +-------------- + This module provides an interface for reading files that use EA IFF 85 chunks. [#]_ This format is used in at least the Audio Interchange File Format (AIFF/AIFF-C) and the Real Media File Format (RMFF). The WAVE audio file format diff --git a/Doc/library/cmath.rst b/Doc/library/cmath.rst index a981d94a11..62ddb6bc48 100644 --- a/Doc/library/cmath.rst +++ b/Doc/library/cmath.rst @@ -4,6 +4,7 @@ .. module:: cmath :synopsis: Mathematical functions for complex numbers. +-------------- This module is always available. It provides access to mathematical functions for complex numbers. The functions in this module accept integers, @@ -207,6 +208,38 @@ Classification functions and ``False`` otherwise. +.. function:: isclose(a, b, *, rel_tol=1e-09, abs_tol=0.0) + + Return ``True`` if the values *a* and *b* are close to each other and + ``False`` otherwise. + + Whether or not two values are considered close is determined according to + given absolute and relative tolerances. + + *rel_tol* is the relative tolerance -- it is the maximum allowed difference + between *a* and *b*, relative to the larger absolute value of *a* or *b*. + For example, to set a tolerance of 5%, pass ``rel_tol=0.05``. The default + tolerance is ``1e-09``, which assures that the two values are the same + within about 9 decimal digits. *rel_tol* must be greater than zero. + + *abs_tol* is the minimum absolute tolerance -- useful for comparisons near + zero. *abs_tol* must be at least zero. + + If no errors occur, the result will be: + ``abs(a-b) <= max(rel_tol * max(abs(a), abs(b)), abs_tol)``. + + The IEEE 754 special values of ``NaN``, ``inf``, and ``-inf`` will be + handled according to IEEE rules. Specifically, ``NaN`` is not considered + close to any other value, including ``NaN``. ``inf`` and ``-inf`` are only + considered close to themselves. + + .. versionadded:: 3.5 + + .. seealso:: + + :pep:`485` -- A function for testing approximate equality + + Constants --------- diff --git a/Doc/library/cmd.rst b/Doc/library/cmd.rst index 1ab2d7423f..f40cfdfd59 100644 --- a/Doc/library/cmd.rst +++ b/Doc/library/cmd.rst @@ -3,6 +3,7 @@ .. module:: cmd :synopsis: Build line-oriented command interpreters. + .. sectionauthor:: Eric S. Raymond <esr@snark.thyrsus.com> **Source code:** :source:`Lib/cmd.py` @@ -313,7 +314,9 @@ immediate playback:: Here is a sample session with the turtle shell showing the help functions, using -blank lines to repeat commands, and the simple record and playback facility:: +blank lines to repeat commands, and the simple record and playback facility: + +.. code-block:: none Welcome to the turtle shell. Type help or ? to list commands. @@ -372,4 +375,3 @@ blank lines to repeat commands, and the simple record and playback facility:: (turtle) bye Thank you for using Turtle - diff --git a/Doc/library/code.rst b/Doc/library/code.rst index 5b5d7cc8c1..443af699f2 100644 --- a/Doc/library/code.rst +++ b/Doc/library/code.rst @@ -4,6 +4,9 @@ .. module:: code :synopsis: Facilities to implement read-eval-print loops. +**Source code:** :source:`Lib/code.py` + +-------------- The ``code`` module provides facilities to implement read-eval-print loops in Python. Two classes and convenience functions are included which can be used to @@ -113,6 +116,9 @@ Interactive Interpreter Objects because it is within the interpreter object implementation. The output is written by the :meth:`write` method. + .. versionchanged:: 3.5 The full chained traceback is displayed instead + of just the primary traceback. + .. method:: InteractiveInterpreter.write(data) @@ -165,4 +171,3 @@ interpreter objects as well as the following additions. newline. When the user enters the EOF key sequence, :exc:`EOFError` is raised. The base implementation reads from ``sys.stdin``; a subclass may replace this with a different implementation. - diff --git a/Doc/library/codecs.rst b/Doc/library/codecs.rst index 628969c65d..1add30072f 100644 --- a/Doc/library/codecs.rst +++ b/Doc/library/codecs.rst @@ -3,10 +3,12 @@ .. module:: codecs :synopsis: Encode and decode data and streams. + .. moduleauthor:: Marc-André Lemburg <mal@lemburg.com> .. sectionauthor:: Marc-André Lemburg <mal@lemburg.com> .. sectionauthor:: Martin v. Löwis <martin@v.loewis.de> +**Source code:** :source:`Lib/codecs.py` .. index:: single: Unicode @@ -16,6 +18,8 @@ single: streams pair: stackable; streams +-------------- + This module defines base classes for standard Python codecs (encoders and decoders) and provides access to the internal Python codec registry, which manages the codec and error handling lookup process. Most standard codecs @@ -29,10 +33,9 @@ module features are restricted to use specifically with The module defines the following functions for encoding and decoding with any codec: -.. function:: encode(obj, [encoding[, errors]]) +.. function:: encode(obj, encoding='utf-8', errors='strict') - Encodes *obj* using the codec registered for *encoding*. The default - encoding is ``utf-8``. + Encodes *obj* using the codec registered for *encoding*. *Errors* may be given to set the desired error handling scheme. The default error handler is ``'strict'`` meaning that encoding errors raise @@ -40,10 +43,9 @@ any codec: :exc:`UnicodeEncodeError`). Refer to :ref:`codec-base-classes` for more information on codec error handling. -.. function:: decode(obj, [encoding[, errors]]) +.. function:: decode(obj, encoding='utf-8', errors='strict') - Decodes *obj* using the codec registered for *encoding*. The default - encoding is ``utf-8``. + Decodes *obj* using the codec registered for *encoding*. *Errors* may be given to set the desired error handling scheme. The default error handler is ``'strict'`` meaning that decoding errors raise @@ -104,7 +106,6 @@ The full details for each codec can also be looked up directly: To simplify access to the various codec components, the module provides these additional functions which use :func:`lookup` for the codec lookup: - .. function:: getencoder(encoding) Look up the codec for the given encoding and return its encoder function. @@ -139,16 +140,16 @@ these additional functions which use :func:`lookup` for the codec lookup: .. function:: getreader(encoding) - Look up the codec for the given encoding and return its StreamReader class or - factory function. + Look up the codec for the given encoding and return its :class:`StreamReader` + class or factory function. Raises a :exc:`LookupError` in case the encoding cannot be found. .. function:: getwriter(encoding) - Look up the codec for the given encoding and return its StreamWriter class or - factory function. + Look up the codec for the given encoding and return its :class:`StreamWriter` + class or factory function. Raises a :exc:`LookupError` in case the encoding cannot be found. @@ -274,6 +275,7 @@ implement the file protocols. Codec authors also need to define how the codec will handle encoding and decoding errors. +.. _surrogateescape: .. _error-handlers: Error Handlers @@ -315,10 +317,14 @@ The following error handlers are only applicable to | | reference (only for encoding). Implemented | | | in :func:`xmlcharrefreplace_errors`. | +-------------------------+-----------------------------------------------+ -| ``'backslashreplace'`` | Replace with backslashed escape sequences | -| | (only for encoding). Implemented in | +| ``'backslashreplace'`` | Replace with backslashed escape sequences. | +| | Implemented in | | | :func:`backslashreplace_errors`. | +-------------------------+-----------------------------------------------+ +| ``'namereplace'`` | Replace with ``\N{...}`` escape sequences | +| | (only for encoding). Implemented in | +| | :func:`namereplace_errors`. | ++-------------------------+-----------------------------------------------+ | ``'surrogateescape'`` | On decoding, replace byte with individual | | | surrogate code ranging from ``U+DC80`` to | | | ``U+DCFF``. This code will then be turned | @@ -344,6 +350,13 @@ In addition, the following error handler is specific to the given codecs: .. versionchanged:: 3.4 The ``'surrogatepass'`` error handlers now works with utf-16\* and utf-32\* codecs. +.. versionadded:: 3.5 + The ``'namereplace'`` error handler. + +.. versionchanged:: 3.5 + The ``'backslashreplace'`` error handlers now works with decoding and + translating. + The set of allowed values can be extended by registering a new named error handler: @@ -411,9 +424,17 @@ functions: .. function:: backslashreplace_errors(exception) - Implements the ``'backslashreplace'`` error handling (for encoding with + Implements the ``'backslashreplace'`` error handling (for + :term:`text encodings <text encoding>` only): malformed data is + replaced by a backslashed escape sequence. + +.. function:: namereplace_errors(exception) + + Implements the ``'namereplace'`` error handling (for encoding with :term:`text encodings <text encoding>` only): the - unencodable character is replaced by a backslashed escape sequence. + unencodable character is replaced by a ``\N{...}`` escape sequence. + + .. versionadded:: 3.5 .. _codec-objects: @@ -959,7 +980,7 @@ particular, the following variants typically exist: * an ISO 8859 codeset -* a Microsoft Windows code page, which is typically derived from a 8859 codeset, +* a Microsoft Windows code page, which is typically derived from an 8859 codeset, but replaces control characters with additional graphic characters * an IBM EBCDIC code page @@ -1144,8 +1165,16 @@ particular, the following variants typically exist: +-----------------+--------------------------------+--------------------------------+ | koi8_r | | Russian | +-----------------+--------------------------------+--------------------------------+ +| koi8_t | | Tajik | +| | | | +| | | .. versionadded:: 3.5 | ++-----------------+--------------------------------+--------------------------------+ | koi8_u | | Ukrainian | +-----------------+--------------------------------+--------------------------------+ +| kz1048 | kz_1048, strk1048_2002, rk1048 | Kazakh | +| | | | +| | | .. versionadded:: 3.5 | ++-----------------+--------------------------------+--------------------------------+ | mac_cyrillic | maccyrillic | Bulgarian, Byelorussian, | | | | Macedonian, Russian, Serbian | +-----------------+--------------------------------+--------------------------------+ @@ -1388,7 +1417,7 @@ parameters, such as :mod:`http.client` and :mod:`ftplib`, accept Unicode host names (:mod:`http.client` then also transparently sends an IDNA hostname in the :mailheader:`Host` field if it sends that field at all). -.. _section 3.1: http://tools.ietf.org/html/rfc3490#section-3.1 +.. _section 3.1: https://tools.ietf.org/html/rfc3490#section-3.1 When receiving host names from the wire (such as in reverse name lookup), no automatic conversion to Unicode is performed: Applications wishing to present @@ -1446,4 +1475,3 @@ This module implements a variant of the UTF-8 codec: On encoding a UTF-8 encoded BOM will be prepended to the UTF-8 encoded bytes. For the stateful encoder this is only done once (on the first write to the byte stream). For decoding an optional UTF-8 encoded BOM at the start of the data will be skipped. - diff --git a/Doc/library/codeop.rst b/Doc/library/codeop.rst index 9ae4176485..a52d2c62c4 100644 --- a/Doc/library/codeop.rst +++ b/Doc/library/codeop.rst @@ -3,9 +3,14 @@ .. module:: codeop :synopsis: Compile (possibly incomplete) Python code. + .. sectionauthor:: Moshe Zadka <moshez@zadka.site.co.il> .. sectionauthor:: Michael Hudson <mwh@python.net> +**Source code:** :source:`Lib/codeop.py` + +-------------- + The :mod:`codeop` module provides utilities upon which the Python read-eval-print loop can be emulated, as is done in the :mod:`code` module. As a result, you probably don't want to use the module directly; if you want to diff --git a/Doc/library/collections.abc.rst b/Doc/library/collections.abc.rst index d73f05ae56..aeb6a73f2e 100644 --- a/Doc/library/collections.abc.rst +++ b/Doc/library/collections.abc.rst @@ -3,20 +3,21 @@ .. module:: collections.abc :synopsis: Abstract base classes for containers + .. moduleauthor:: Raymond Hettinger <python at rcn.com> .. sectionauthor:: Raymond Hettinger <python at rcn.com> .. versionadded:: 3.3 Formerly, this module was part of the :mod:`collections` module. +**Source code:** :source:`Lib/_collections_abc.py` + .. testsetup:: * from collections import * import itertools __name__ = '<doctest>' -**Source code:** :source:`Lib/_collections_abc.py` - -------------- This module provides :term:`abstract base classes <abstract base class>` that @@ -33,13 +34,14 @@ The collections module offers the following :term:`ABCs <abstract base class>`: .. tabularcolumns:: |l|L|L|L| -========================= ===================== ====================== ==================================================== +========================== ====================== ======================= ==================================================== ABC Inherits from Abstract Methods Mixin Methods -========================= ===================== ====================== ==================================================== +========================== ====================== ======================= ==================================================== :class:`Container` ``__contains__`` :class:`Hashable` ``__hash__`` :class:`Iterable` ``__iter__`` :class:`Iterator` :class:`Iterable` ``__next__`` ``__iter__`` +:class:`Generator` :class:`Iterator` ``send``, ``throw`` ``close``, ``__iter__``, ``__next__`` :class:`Sized` ``__len__`` :class:`Callable` ``__call__`` @@ -53,6 +55,9 @@ ABC Inherits from Abstract Methods Mixin ``__len__``, ``insert`` +:class:`ByteString` :class:`Sequence` ``__getitem__``, Inherited :class:`Sequence` methods + ``__len__`` + :class:`Set` :class:`Sized`, ``__contains__``, ``__le__``, ``__lt__``, ``__eq__``, ``__ne__``, :class:`Iterable`, ``__iter__``, ``__gt__``, ``__ge__``, ``__and__``, ``__or__``, :class:`Container` ``__len__`` ``__sub__``, ``__xor__``, and ``isdisjoint`` @@ -80,7 +85,11 @@ ABC Inherits from Abstract Methods Mixin :class:`KeysView` :class:`MappingView`, ``__contains__``, :class:`Set` ``__iter__`` :class:`ValuesView` :class:`MappingView` ``__contains__``, ``__iter__`` -========================= ===================== ====================== ==================================================== +:class:`Awaitable` ``__await__`` +:class:`Coroutine` :class:`Awaitable` ``send``, ``throw`` ``close`` +:class:`AsyncIterable` ``__aiter__`` +:class:`AsyncIterator` :class:`AsyncIterable` ``__anext__`` ``__aiter__`` +========================== ====================== ======================= ==================================================== .. class:: Container @@ -102,11 +111,34 @@ ABC Inherits from Abstract Methods Mixin :meth:`~iterator.__next__` methods. See also the definition of :term:`iterator`. +.. class:: Generator + + ABC for generator classes that implement the protocol defined in + :pep:`342` that extends iterators with the :meth:`~generator.send`, + :meth:`~generator.throw` and :meth:`~generator.close` methods. + See also the definition of :term:`generator`. + + .. versionadded:: 3.5 + .. class:: Sequence MutableSequence + ByteString ABCs for read-only and mutable :term:`sequences <sequence>`. + Implementation note: Some of the mixin methods, such as + :meth:`__iter__`, :meth:`__reversed__` and :meth:`index`, make + repeated calls to the underlying :meth:`__getitem__` method. + Consequently, if :meth:`__getitem__` is implemented with constant + access speed, the mixin methods will have linear performance; + however, if the underlying method is linear (as it would be with a + linked list), the mixins will have quadratic performance and will + likely need to be overridden. + + .. versionchanged:: 3.5 + The index() method added support for *stop* and *start* + arguments. + .. class:: Set MutableSet @@ -124,6 +156,56 @@ ABC Inherits from Abstract Methods Mixin ABCs for mapping, items, keys, and values :term:`views <dictionary view>`. +.. class:: Awaitable + + ABC for :term:`awaitable` objects, which can be used in :keyword:`await` + expressions. Custom implementations must provide the :meth:`__await__` + method. + + :term:`Coroutine` objects and instances of the + :class:`~collections.abc.Coroutine` ABC are all instances of this ABC. + + .. note:: + In CPython, generator-based coroutines (generators decorated with + :func:`types.coroutine` or :func:`asyncio.coroutine`) are + *awaitables*, even though they do not have an :meth:`__await__` method. + Using ``isinstance(gencoro, Awaitable)`` for them will return ``False``. + Use :func:`inspect.isawaitable` to detect them. + + .. versionadded:: 3.5 + +.. class:: Coroutine + + ABC for coroutine compatible classes. These implement the + following methods, defined in :ref:`coroutine-objects`: + :meth:`~coroutine.send`, :meth:`~coroutine.throw`, and + :meth:`~coroutine.close`. Custom implementations must also implement + :meth:`__await__`. All :class:`Coroutine` instances are also instances of + :class:`Awaitable`. See also the definition of :term:`coroutine`. + + .. note:: + In CPython, generator-based coroutines (generators decorated with + :func:`types.coroutine` or :func:`asyncio.coroutine`) are + *awaitables*, even though they do not have an :meth:`__await__` method. + Using ``isinstance(gencoro, Coroutine)`` for them will return ``False``. + Use :func:`inspect.isawaitable` to detect them. + + .. versionadded:: 3.5 + +.. class:: AsyncIterable + + ABC for classes that provide ``__aiter__`` method. See also the + definition of :term:`asynchronous iterable`. + + .. versionadded:: 3.5 + +.. class:: AsyncIterator + + ABC for classes that provide ``__aiter__`` and ``__anext__`` + methods. See also the definition of :term:`asynchronous iterator`. + + .. versionadded:: 3.5 + These ABCs allow us to ask classes or instances if they provide particular functionality, for example:: @@ -140,19 +222,22 @@ The ABC supplies the remaining methods such as :meth:`__and__` and :meth:`isdisjoint`:: class ListBasedSet(collections.abc.Set): - ''' Alternate set implementation favoring space over speed - and not requiring the set elements to be hashable. ''' - def __init__(self, iterable): - self.elements = lst = [] - for value in iterable: - if value not in lst: - lst.append(value) - def __iter__(self): - return iter(self.elements) - def __contains__(self, value): - return value in self.elements - def __len__(self): - return len(self.elements) + ''' Alternate set implementation favoring space over speed + and not requiring the set elements to be hashable. ''' + def __init__(self, iterable): + self.elements = lst = [] + for value in iterable: + if value not in lst: + lst.append(value) + + def __iter__(self): + return iter(self.elements) + + def __contains__(self, value): + return value in self.elements + + def __len__(self): + return len(self.elements) s1 = ListBasedSet('abcdef') s2 = ListBasedSet('defghi') @@ -185,7 +270,7 @@ Notes on using :class:`Set` and :class:`MutableSet` as a mixin: .. seealso:: - * `OrderedSet recipe <http://code.activestate.com/recipes/576694/>`_ for an + * `OrderedSet recipe <https://code.activestate.com/recipes/576694/>`_ for an example built on :class:`MutableSet`. * For more about ABCs, see the :mod:`abc` module and :pep:`3119`. diff --git a/Doc/library/collections.rst b/Doc/library/collections.rst index 09718254eb..4936b3a0f8 100644 --- a/Doc/library/collections.rst +++ b/Doc/library/collections.rst @@ -3,17 +3,18 @@ .. module:: collections :synopsis: Container datatypes + .. moduleauthor:: Raymond Hettinger <python@rcn.com> .. sectionauthor:: Raymond Hettinger <python@rcn.com> +**Source code:** :source:`Lib/collections/__init__.py` + .. testsetup:: * from collections import * import itertools __name__ = '<doctest>' -**Source code:** :source:`Lib/collections/__init__.py` - -------------- This module implements specialized container datatypes providing alternatives to @@ -56,7 +57,7 @@ The class can be used to simulate nested scopes and is useful in templating. dictionary is provided so that a new chain always has at least one mapping. The underlying mappings are stored in a list. That list is public and can - accessed or updated using the *maps* attribute. There is no other state. + be accessed or updated using the *maps* attribute. There is no other state. Lookups search the underlying mappings successively until a key is found. In contrast, writes, updates, and deletions only operate on the first mapping. @@ -116,12 +117,12 @@ The class can be used to simulate nested scopes and is useful in templating. :meth:`~collections.ChainMap.parents` property. * The `Nested Contexts recipe - <http://code.activestate.com/recipes/577434/>`_ has options to control + <https://code.activestate.com/recipes/577434/>`_ has options to control whether writes and other mutations apply only to the first mapping or to any mapping in the chain. * A `greatly simplified read-only version of Chainmap - <http://code.activestate.com/recipes/305268/>`_. + <https://code.activestate.com/recipes/305268/>`_. :class:`ChainMap` Examples and Recipes @@ -374,12 +375,12 @@ or subtracting from an empty counter. .. seealso:: - * `Bag class <http://www.gnu.org/software/smalltalk/manual-base/html_node/Bag.html>`_ + * `Bag class <https://www.gnu.org/software/smalltalk/manual-base/html_node/Bag.html>`_ in Smalltalk. - * Wikipedia entry for `Multisets <http://en.wikipedia.org/wiki/Multiset>`_. + * Wikipedia entry for `Multisets <https://en.wikipedia.org/wiki/Multiset>`_. - * `C++ multisets <http://www.demo2s.com/Tutorial/Cpp/0380__set-multiset/Catalog0380__set-multiset.htm>`_ + * `C++ multisets <http://www.java2s.com/Tutorial/Cpp/0380__set-multiset/Catalog0380__set-multiset.htm>`_ tutorial with examples. * For mathematical operations on multisets and their use cases, see @@ -387,7 +388,7 @@ or subtracting from an empty counter. Section 4.6.3, Exercise 19*. * To enumerate all distinct multisets of a given size over a given set of - elements, see :func:`itertools.combinations_with_replacement`. + elements, see :func:`itertools.combinations_with_replacement`: map(Counter, combinations_with_replacement('ABC', 2)) --> AA AB AC BB BC CC @@ -437,6 +438,13 @@ or subtracting from an empty counter. Remove all elements from the deque leaving it with length 0. + .. method:: copy() + + Create a shallow copy of the deque. + + .. versionadded:: 3.5 + + .. method:: count(x) Count the number of deque elements equal to *x*. @@ -457,6 +465,25 @@ or subtracting from an empty counter. elements in the iterable argument. + .. method:: index(x[, start[, stop]]) + + Return the position of *x* in the deque (at or after index *start* + and before index *stop*). Returns the first match or raises + :exc:`ValueError` if not found. + + .. versionadded:: 3.5 + + + .. method:: insert(i, x) + + Insert *x* into the deque at position *i*. + + If the insertion would cause a bounded deque to grow beyond *maxlen*, + an :exc:`IndexError` is raised. + + .. versionadded:: 3.5 + + .. method:: pop() Remove and return an element from the right side of the deque. If no @@ -471,7 +498,7 @@ or subtracting from an empty counter. .. method:: remove(value) - Removed the first occurrence of *value*. If not found, raises a + Remove the first occurrence of *value*. If not found, raises a :exc:`ValueError`. @@ -504,6 +531,9 @@ the :keyword:`in` operator, and subscript references such as ``d[-1]``. Indexed access is O(1) at both ends but slows to O(n) in the middle. For fast random access, use lists instead. +Starting in version 3.5, deques support ``__add__()``, ``__mul__()``, +and ``__imul__()``. + Example: .. doctest:: @@ -763,6 +793,11 @@ they add the ability to access fields by name instead of position index. Named tuple instances do not have per-instance dictionaries, so they are lightweight and require no more memory than regular tuples. + For simple uses, where the only requirement is to be able to refer to a set + of values by name using attribute-style access, the + :class:`types.SimpleNamespace` type can be a suitable alternative to using + a namedtuple. + .. versionchanged:: 3.1 Added support for *rename*. @@ -879,15 +914,15 @@ functionality with a subclass. Here is how to add a calculated field and a fixed-width print format: >>> class Point(namedtuple('Point', 'x y')): - __slots__ = () - @property - def hypot(self): - return (self.x ** 2 + self.y ** 2) ** 0.5 - def __str__(self): - return 'Point: x=%6.3f y=%6.3f hypot=%6.3f' % (self.x, self.y, self.hypot) + __slots__ = () + @property + def hypot(self): + return (self.x ** 2 + self.y ** 2) ** 0.5 + def __str__(self): + return 'Point: x=%6.3f y=%6.3f hypot=%6.3f' % (self.x, self.y, self.hypot) >>> for p in Point(3, 4), Point(14, 5/7): - print(p) + print(p) Point: x= 3.000 y= 4.000 hypot= 5.000 Point: x=14.000 y= 0.714 hypot=14.018 @@ -899,6 +934,18 @@ create a new named tuple type from the :attr:`_fields` attribute: >>> Point3D = namedtuple('Point3D', Point._fields + ('z',)) +Docstrings can be customized by making direct assignments to the ``__doc__`` +fields: + + >>> Book = namedtuple('Book', ['id', 'title', 'authors']) + >>> Book.__doc__ += ': Hardcover book in active collection' + >>> Book.id.__doc__ = '13-digit ISBN' + >>> Book.title.__doc__ = 'Title of first printing' + >>> Book.authors.__doc__ = 'List of authors sorted by last name' + +.. versionchanged:: 3.5 + Property docstrings became writeable. + Default values can be implemented by using :meth:`_replace` to customize a prototype instance: @@ -907,21 +954,11 @@ customize a prototype instance: >>> johns_account = default_account._replace(owner='John') >>> janes_account = default_account._replace(owner='Jane') -Enumerated constants can be implemented with named tuples, but it is simpler -and more efficient to use a simple :class:`~enum.Enum`: - - >>> Status = namedtuple('Status', 'open pending closed')._make(range(3)) - >>> Status.open, Status.pending, Status.closed - (0, 1, 2) - >>> from enum import Enum - >>> class Status(Enum): - ... open, pending, closed = range(3) - .. seealso:: * `Recipe for named tuple abstract base class with a metaclass mix-in - <http://code.activestate.com/recipes/577629-namedtupleabc-abstract-base-class-mix-in-for-named/>`_ + <https://code.activestate.com/recipes/577629-namedtupleabc-abstract-base-class-mix-in-for-named/>`_ by Jan Kaliszewski. Besides providing an :term:`abstract base class` for named tuples, it also supports an alternate :term:`metaclass`-based constructor that is convenient for use cases where named tuples are being @@ -982,6 +1019,9 @@ The :class:`OrderedDict` constructor and :meth:`update` method both accept keyword arguments, but their order is lost because Python's function call semantics pass in keyword arguments using a regular unordered dictionary. +.. versionchanged:: 3.5 + The items, keys, and values :term:`views <dictionary view>` + of :class:`OrderedDict` now support reverse iteration using :func:`reversed`. :class:`OrderedDict` Examples and Recipes ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -990,7 +1030,7 @@ Since an ordered dictionary remembers its insertion order, it can be used in conjunction with sorting to make a sorted dictionary:: >>> # regular unsorted dictionary - >>> d = {'banana': 3, 'apple':4, 'pear': 1, 'orange': 2} + >>> d = {'banana': 3, 'apple': 4, 'pear': 1, 'orange': 2} >>> # dictionary sorted by key >>> OrderedDict(sorted(d.items(), key=lambda t: t[0])) @@ -1119,3 +1159,7 @@ attribute. be an instance of :class:`bytes`, :class:`str`, :class:`UserString` (or a subclass) or an arbitrary sequence which can be converted into a string using the built-in :func:`str` function. + + .. versionchanged:: 3.5 + New methods ``__getnewargs__``, ``__rmod__``, ``casefold``, + ``format_map``, ``isprintable``, and ``maketrans``. diff --git a/Doc/library/colorsys.rst b/Doc/library/colorsys.rst index 225306c2d3..c33f531cc1 100644 --- a/Doc/library/colorsys.rst +++ b/Doc/library/colorsys.rst @@ -3,6 +3,7 @@ .. module:: colorsys :synopsis: Conversion functions between RGB and other color systems. + .. sectionauthor:: David Ascher <da@python.net> **Source code:** :source:`Lib/colorsys.py` @@ -21,7 +22,7 @@ spaces, the coordinates are all between 0 and 1. More information about color spaces can be found at http://www.poynton.com/ColorFAQ.html and - http://www.cambridgeincolour.com/tutorials/color-spaces.htm. + https://www.cambridgeincolour.com/tutorials/color-spaces.htm. The :mod:`colorsys` module defines the following functions: diff --git a/Doc/library/compileall.rst b/Doc/library/compileall.rst index 9712de2fef..511c581ad0 100644 --- a/Doc/library/compileall.rst +++ b/Doc/library/compileall.rst @@ -8,7 +8,6 @@ -------------- - This module provides some utility functions to support installing Python libraries. These functions compile Python source files in a directory tree. This module can be used to create the cached byte-code files at library @@ -42,7 +41,8 @@ compile Python sources. .. cmdoption:: -q - Do not print the list of files compiled, print only error messages. + Do not print the list of files compiled. If passed once, error messages will + still be printed. If passed twice (``-qq``), all output is suppressed. .. cmdoption:: -d destdir @@ -70,9 +70,28 @@ compile Python sources. is to write files to their :pep:`3147` locations and names, which allows byte-code files from multiple versions of Python to coexist. +.. cmdoption:: -r + + Control the maximum recursion level for subdirectories. + If this is given, then ``-l`` option will not be taken into account. + :program:`python -m compileall <directory> -r 0` is equivalent to + :program:`python -m compileall <directory> -l`. + +.. cmdoption:: -j N + + Use *N* workers to compile the files within the given directory. + If ``0`` is used, then the result of :func:`os.cpu_count()` + will be used. + .. versionchanged:: 3.2 Added the ``-i``, ``-b`` and ``-h`` options. +.. versionchanged:: 3.5 + Added the ``-j``, ``-r``, and ``-qq`` options. ``-q`` option + was changed to a multilevel value. ``-b`` will always produce a + byte-code file ending in ``.pyc``, never ``.pyo``. + + There is no command-line option to control the optimization level used by the :func:`compile` function, because the Python interpreter itself already provides the option: :program:`python -O -m compileall`. @@ -80,7 +99,7 @@ provides the option: :program:`python -O -m compileall`. Public functions ---------------- -.. function:: compile_dir(dir, maxlevels=10, ddir=None, force=False, rx=None, quiet=False, legacy=False, optimize=-1) +.. function:: compile_dir(dir, maxlevels=10, ddir=None, force=False, rx=None, quiet=0, legacy=False, optimize=-1, workers=1) Recursively descend the directory tree named by *dir*, compiling all :file:`.py` files along the way. @@ -101,8 +120,9 @@ Public functions file considered for compilation, and if it returns a true value, the file is skipped. - If *quiet* is true, nothing is printed to the standard output unless errors - occur. + If *quiet* is ``False`` or ``0`` (the default), the filenames and other + information are printed to standard out. Set to ``1``, only errors are + printed. Set to ``2``, all output is suppressed. If *legacy* is true, byte-code files are written to their legacy locations and names, which may overwrite byte-code files created by another version of @@ -113,11 +133,26 @@ Public functions *optimize* specifies the optimization level for the compiler. It is passed to the built-in :func:`compile` function. + The argument *workers* specifies how many workers are used to + compile files in parallel. The default is to not use multiple workers. + If the platform can't use multiple workers and *workers* argument is given, + then sequential compilation will be used as a fallback. If *workers* is + lower than ``0``, a :exc:`ValueError` will be raised. + .. versionchanged:: 3.2 Added the *legacy* and *optimize* parameter. + .. versionchanged:: 3.5 + Added the *workers* parameter. + + .. versionchanged:: 3.5 + *quiet* parameter was changed to a multilevel value. -.. function:: compile_file(fullname, ddir=None, force=False, rx=None, quiet=False, legacy=False, optimize=-1) + .. versionchanged:: 3.5 + The *legacy* parameter only writes out ``.pyc`` files, not ``.pyo`` files + no matter what the value of *optimize* is. + +.. function:: compile_file(fullname, ddir=None, force=False, rx=None, quiet=0, legacy=False, optimize=-1) Compile the file with path *fullname*. @@ -131,8 +166,9 @@ Public functions file being compiled, and if it returns a true value, the file is not compiled and ``True`` is returned. - If *quiet* is true, nothing is printed to the standard output unless errors - occur. + If *quiet* is ``False`` or ``0`` (the default), the filenames and other + information are printed to standard out. Set to ``1``, only errors are + printed. Set to ``2``, all output is suppressed. If *legacy* is true, byte-code files are written to their legacy locations and names, which may overwrite byte-code files created by another version of @@ -145,8 +181,14 @@ Public functions .. versionadded:: 3.2 + .. versionchanged:: 3.5 + *quiet* parameter was changed to a multilevel value. + + .. versionchanged:: 3.5 + The *legacy* parameter only writes out ``.pyc`` files, not ``.pyo`` files + no matter what the value of *optimize* is. -.. function:: compile_path(skip_curdir=True, maxlevels=0, force=False, legacy=False, optimize=-1) +.. function:: compile_path(skip_curdir=True, maxlevels=0, force=False, quiet=0, legacy=False, optimize=-1) Byte-compile all the :file:`.py` files found along ``sys.path``. If *skip_curdir* is true (the default), the current directory is not included @@ -157,6 +199,12 @@ Public functions .. versionchanged:: 3.2 Added the *legacy* and *optimize* parameter. + .. versionchanged:: 3.5 + *quiet* parameter was changed to a multilevel value. + + .. versionchanged:: 3.5 + The *legacy* parameter only writes out ``.pyc`` files, not ``.pyo`` files + no matter what the value of *optimize* is. To force a recompile of all the :file:`.py` files in the :file:`Lib/` subdirectory and all its subdirectories:: diff --git a/Doc/library/concurrent.futures.rst b/Doc/library/concurrent.futures.rst index e63e741b2e..ae03f4b8f4 100644 --- a/Doc/library/concurrent.futures.rst +++ b/Doc/library/concurrent.futures.rst @@ -38,17 +38,26 @@ Executor Objects future = executor.submit(pow, 323, 1235) print(future.result()) - .. method:: map(func, *iterables, timeout=None) + .. method:: map(func, *iterables, timeout=None, chunksize=1) Equivalent to :func:`map(func, *iterables) <map>` except *func* is executed asynchronously and several calls to *func* may be made concurrently. The - returned iterator raises a :exc:`TimeoutError` if + returned iterator raises a :exc:`concurrent.futures.TimeoutError` if :meth:`~iterator.__next__` is called and the result isn't available after *timeout* seconds from the original call to :meth:`Executor.map`. *timeout* can be an int or a float. If *timeout* is not specified or ``None``, there is no limit to the wait time. If a call raises an exception, then that exception will be raised when its value is - retrieved from the iterator. + retrieved from the iterator. When using :class:`ProcessPoolExecutor`, this + method chops *iterables* into a number of chunks which it submits to the + pool as separate tasks. The (approximate) size of these chunks can be + specified by setting *chunksize* to a positive integer. For very long + iterables, using a large value for *chunksize* can significantly improve + performance compared to the default size of 1. With :class:`ThreadPoolExecutor`, + *chunksize* has no effect. + + .. versionchanged:: 3.5 + Added the *chunksize* argument. .. method:: shutdown(wait=True) @@ -90,12 +99,12 @@ the results of another :class:`Future`. For example:: import time def wait_on_b(): time.sleep(5) - print(b.result()) # b will never complete because it is waiting on a. + print(b.result()) # b will never complete because it is waiting on a. return 5 def wait_on_a(): time.sleep(5) - print(a.result()) # a will never complete because it is waiting on b. + print(a.result()) # a will never complete because it is waiting on b. return 6 @@ -115,11 +124,19 @@ And:: executor.submit(wait_on_future) -.. class:: ThreadPoolExecutor(max_workers) +.. class:: ThreadPoolExecutor(max_workers=None) An :class:`Executor` subclass that uses a pool of at most *max_workers* threads to execute calls asynchronously. + .. versionchanged:: 3.5 + If *max_workers* is ``None`` or + not given, it will default to the number of processors on the machine, + multiplied by ``5``, assuming that :class:`ThreadPoolExecutor` is often + used to overlap I/O instead of CPU work and the number of workers + should be higher than the number of workers + for :class:`ProcessPoolExecutor`. + .. _threadpoolexecutor-example: @@ -136,7 +153,7 @@ ThreadPoolExecutor Example 'http://www.bbc.co.uk/', 'http://some-made-up-domain.com/'] - # Retrieve a single page and report the url and contents + # Retrieve a single page and report the URL and contents def load_url(url, timeout): with urllib.request.urlopen(url, timeout=timeout) as conn: return conn.read() @@ -175,6 +192,8 @@ to a :class:`ProcessPoolExecutor` will result in deadlock. An :class:`Executor` subclass that executes calls asynchronously using a pool of at most *max_workers* processes. If *max_workers* is ``None`` or not given, it will default to the number of processors on the machine. + If *max_workers* is lower or equal to ``0``, then a :exc:`ValueError` + will be raised. .. versionchanged:: 3.3 When one of the worker processes terminates abruptly, a @@ -255,11 +274,12 @@ The :class:`Future` class encapsulates the asynchronous execution of a callable. Return the value returned by the call. If the call hasn't yet completed then this method will wait up to *timeout* seconds. If the call hasn't - completed in *timeout* seconds, then a :exc:`TimeoutError` will be - raised. *timeout* can be an int or float. If *timeout* is not specified - or ``None``, there is no limit to the wait time. + completed in *timeout* seconds, then a + :exc:`concurrent.futures.TimeoutError` will be raised. *timeout* can be + an int or float. If *timeout* is not specified or ``None``, there is no + limit to the wait time. - If the future is cancelled before completing then :exc:`CancelledError` + If the future is cancelled before completing then :exc:`.CancelledError` will be raised. If the call raised, this method will raise the same exception. @@ -268,11 +288,12 @@ The :class:`Future` class encapsulates the asynchronous execution of a callable. Return the exception raised by the call. If the call hasn't yet completed then this method will wait up to *timeout* seconds. If the - call hasn't completed in *timeout* seconds, then a :exc:`TimeoutError` - will be raised. *timeout* can be an int or float. If *timeout* is not - specified or ``None``, there is no limit to the wait time. + call hasn't completed in *timeout* seconds, then a + :exc:`concurrent.futures.TimeoutError` will be raised. *timeout* can be + an int or float. If *timeout* is not specified or ``None``, there is no + limit to the wait time. - If the future is cancelled before completing then :exc:`CancelledError` + If the future is cancelled before completing then :exc:`.CancelledError` will be raised. If the call completed without raising, ``None`` is returned. @@ -372,13 +393,12 @@ Module Functions Returns an iterator over the :class:`Future` instances (possibly created by different :class:`Executor` instances) given by *fs* that yields futures as they complete (finished or were cancelled). Any futures given by *fs* that - are duplicated will be returned once. Any futures that completed - before :func:`as_completed` is called will be yielded first. The returned - iterator raises a :exc:`TimeoutError` if :meth:`~iterator.__next__` is - called and the result isn't available after *timeout* seconds from the - original call to :func:`as_completed`. *timeout* can be an int or float. - If *timeout* is not specified or ``None``, there is no limit to the wait - time. + are duplicated will be returned once. Any futures that completed before + :func:`as_completed` is called will be yielded first. The returned iterator + raises a :exc:`concurrent.futures.TimeoutError` if :meth:`~iterator.__next__` + is called and the result isn't available after *timeout* seconds from the + original call to :func:`as_completed`. *timeout* can be an int or float. If + *timeout* is not specified or ``None``, there is no limit to the wait time. .. seealso:: @@ -391,6 +411,16 @@ Module Functions Exception classes ----------------- +.. currentmodule:: concurrent.futures + +.. exception:: CancelledError + + Raised when a future is cancelled. + +.. exception:: TimeoutError + + Raised when a future operation exceeds the given timeout. + .. currentmodule:: concurrent.futures.process .. exception:: BrokenProcessPool diff --git a/Doc/library/configparser.rst b/Doc/library/configparser.rst index 92551bc3bd..a9a5f5a2e5 100644 --- a/Doc/library/configparser.rst +++ b/Doc/library/configparser.rst @@ -11,12 +11,16 @@ .. sectionauthor:: Christopher G. Petrilli <petrilli@amber.org> .. sectionauthor:: Łukasz Langa <lukasz@langa.pl> +**Source code:** :source:`Lib/configparser.py` + .. index:: pair: .ini; file pair: configuration; file single: ini file single: Windows ini file +-------------- + This module provides the :class:`ConfigParser` class which implements a basic configuration language which provides a structure similar to what's found in Microsoft Windows INI files. You can use this to write Python programs which @@ -62,7 +66,7 @@ The structure of INI files is described `in the following section <#supported-ini-file-structure>`_. Essentially, the file consists of sections, each of which contains keys with values. :mod:`configparser` classes can read and write such files. Let's start by -creating the above configuration file programatically. +creating the above configuration file programmatically. .. doctest:: @@ -142,12 +146,13 @@ datatypes, you should convert on your own: >>> float(topsecret['CompressionLevel']) 9.0 -Extracting Boolean values is not that simple, though. Passing the value -to ``bool()`` would do no good since ``bool('False')`` is still -``True``. This is why config parsers also provide :meth:`getboolean`. -This method is case-insensitive and recognizes Boolean values from -``'yes'``/``'no'``, ``'on'``/``'off'`` and ``'1'``/``'0'`` [1]_. -For example: +Since this task is so common, config parsers provide a range of handy getter +methods to handle integers, floats and booleans. The last one is the most +interesting because simply passing the value to ``bool()`` would do no good +since ``bool('False')`` is still ``True``. This is why config parsers also +provide :meth:`getboolean`. This method is case-insensitive and recognizes +Boolean values from ``'yes'``/``'no'``, ``'on'``/``'off'``, +``'true'``/``'false'`` and ``'1'``/``'0'`` [1]_. For example: .. doctest:: @@ -159,10 +164,8 @@ For example: True Apart from :meth:`getboolean`, config parsers also provide equivalent -:meth:`getint` and :meth:`getfloat` methods, but these are far less -useful since conversion using :func:`int` and :func:`float` is -sufficient for these types. - +:meth:`getint` and :meth:`getfloat` methods. You can register your own +converters and customize the provided ones. [1]_ Fallback Values --------------- @@ -317,11 +320,11 @@ from ``get()`` calls. .. class:: ExtendedInterpolation() An alternative handler for interpolation which implements a more advanced - syntax, used for instance in ``zc.buildout``. Extended interpolation is + syntax, used for instance in ``zc.buildout``. Extended interpolation is using ``${section:option}`` to denote a value from a foreign section. - Interpolation can span multiple levels. For convenience, if the ``section:`` - part is omitted, interpolation defaults to the current section (and possibly - the default values from the special section). + Interpolation can span multiple levels. For convenience, if the + ``section:`` part is omitted, interpolation defaults to the current section + (and possibly the default values from the special section). For example, the configuration specified above with basic interpolation, would look like this with extended interpolation: @@ -399,13 +402,13 @@ However, there are a few differences that should be taken into account: * ``parser.popitem()`` never returns it. * ``parser.get(section, option, **kwargs)`` - the second argument is **not** - a fallback value. Note however that the section-level ``get()`` methods are + a fallback value. Note however that the section-level ``get()`` methods are compatible both with the mapping protocol and the classic configparser API. * ``parser.items()`` is compatible with the mapping protocol (returns a list of *section_name*, *section_proxy* pairs including the DEFAULTSECT). However, this method can also be invoked with arguments: ``parser.items(section, raw, - vars)``. The latter call returns a list of *option*, *value* pairs for + vars)``. The latter call returns a list of *option*, *value* pairs for a specified ``section``, with all interpolations expanded (unless ``raw=True`` is provided). @@ -539,9 +542,9 @@ the :meth:`__init__` options: * *delimiters*, default value: ``('=', ':')`` - Delimiters are substrings that delimit keys from values within a section. The - first occurrence of a delimiting substring on a line is considered a delimiter. - This means values (but not keys) can contain the delimiters. + Delimiters are substrings that delimit keys from values within a section. + The first occurrence of a delimiting substring on a line is considered + a delimiter. This means values (but not keys) can contain the delimiters. See also the *space_around_delimiters* argument to :meth:`ConfigParser.write`. @@ -553,7 +556,7 @@ the :meth:`__init__` options: Comment prefixes are strings that indicate the start of a valid comment within a config file. *comment_prefixes* are used only on otherwise empty lines (optionally indented) whereas *inline_comment_prefixes* can be used after - every valid value (e.g. section names, options and empty lines as well). By + every valid value (e.g. section names, options and empty lines as well). By default inline comments are disabled and ``'#'`` and ``';'`` are used as prefixes for whole line comments. @@ -563,10 +566,10 @@ the :meth:`__init__` options: Please note that config parsers don't support escaping of comment prefixes so using *inline_comment_prefixes* may prevent users from specifying option - values with characters used as comment prefixes. When in doubt, avoid setting - *inline_comment_prefixes*. In any circumstances, the only way of storing - comment prefix characters at the beginning of a line in multiline values is to - interpolate the prefix, for example:: + values with characters used as comment prefixes. When in doubt, avoid + setting *inline_comment_prefixes*. In any circumstances, the only way of + storing comment prefix characters at the beginning of a line in multiline + values is to interpolate the prefix, for example:: >>> from configparser import ConfigParser, ExtendedInterpolation >>> parser = ConfigParser(interpolation=ExtendedInterpolation()) @@ -611,7 +614,7 @@ the :meth:`__init__` options: When set to ``True``, the parser will not allow for any section or option duplicates while reading from a single source (using :meth:`read_file`, - :meth:`read_string` or :meth:`read_dict`). It is recommended to use strict + :meth:`read_string` or :meth:`read_dict`). It is recommended to use strict parsers in new applications. .. versionchanged:: 3.2 @@ -646,12 +649,12 @@ the :meth:`__init__` options: The convention of allowing a special section of default values for other sections or interpolation purposes is a powerful concept of this library, - letting users create complex declarative configurations. This section is + letting users create complex declarative configurations. This section is normally called ``"DEFAULT"`` but this can be customized to point to any - other valid section name. Some typical values include: ``"general"`` or - ``"common"``. The name provided is used for recognizing default sections when - reading from any source and is used when writing configuration back to - a file. Its current value can be retrieved using the + other valid section name. Some typical values include: ``"general"`` or + ``"common"``. The name provided is used for recognizing default sections + when reading from any source and is used when writing configuration back to + a file. Its current value can be retrieved using the ``parser_instance.default_section`` attribute and may be modified at runtime (i.e. to convert files from one format to another). @@ -660,14 +663,30 @@ the :meth:`__init__` options: Interpolation behaviour may be customized by providing a custom handler through the *interpolation* argument. ``None`` can be used to turn off interpolation completely, ``ExtendedInterpolation()`` provides a more - advanced variant inspired by ``zc.buildout``. More on the subject in the + advanced variant inspired by ``zc.buildout``. More on the subject in the `dedicated documentation section <#interpolation-of-values>`_. :class:`RawConfigParser` has a default value of ``None``. +* *converters*, default value: not set + + Config parsers provide option value getters that perform type conversion. By + default :meth:`getint`, :meth:`getfloat`, and :meth:`getboolean` are + implemented. Should other getters be desirable, users may define them in + a subclass or pass a dictionary where each key is a name of the converter and + each value is a callable implementing said conversion. For instance, passing + ``{'decimal': decimal.Decimal}`` would add :meth:`getdecimal` on both the + parser object and all section proxies. In other words, it will be possible + to write both ``parser_instance.getdecimal('section', 'key', fallback=0)`` + and ``parser_instance['section'].getdecimal('key', 0)``. + + If the converter needs to access the state of the parser, it can be + implemented as a method on a config parser subclass. If the name of this + method starts with ``get``, it will be available on all section proxies, in + the dict-compatible form (see the ``getdecimal()`` example above). More advanced customization may be achieved by overriding default values of -these parser attributes. The defaults are defined on the classes, so they -may be overridden by subclasses or by attribute assignment. +these parser attributes. The defaults are defined on the classes, so they may +be overridden by subclasses or by attribute assignment. .. attribute:: BOOLEAN_STATES @@ -725,10 +744,11 @@ may be overridden by subclasses or by attribute assignment. .. attribute:: SECTCRE - A compiled regular expression used to parse section headers. The default - matches ``[section]`` to the name ``"section"``. Whitespace is considered part - of the section name, thus ``[ larch ]`` will be read as a section of name - ``" larch "``. Override this attribute if that's unsuitable. For example: + A compiled regular expression used to parse section headers. The default + matches ``[section]`` to the name ``"section"``. Whitespace is considered + part of the section name, thus ``[ larch ]`` will be read as a section of + name ``" larch "``. Override this attribute if that's unsuitable. For + example: .. doctest:: @@ -815,13 +835,13 @@ To get interpolation, use :class:`ConfigParser`:: # Set the optional *raw* argument of get() to True if you wish to disable # interpolation in a single get operation. - print(cfg.get('Section1', 'foo', raw=False)) # -> "Python is fun!" - print(cfg.get('Section1', 'foo', raw=True)) # -> "%(bar)s is %(baz)s!" + print(cfg.get('Section1', 'foo', raw=False)) # -> "Python is fun!" + print(cfg.get('Section1', 'foo', raw=True)) # -> "%(bar)s is %(baz)s!" # The optional *vars* argument is a dict with members that will take # precedence in interpolation. print(cfg.get('Section1', 'foo', vars={'bar': 'Documentation', - 'baz': 'evil'})) + 'baz': 'evil'})) # The optional *fallback* argument can be used to provide a fallback value print(cfg.get('Section1', 'foo')) @@ -848,10 +868,10 @@ interpolation if an option used is not defined elsewhere. :: config = configparser.ConfigParser({'bar': 'Life', 'baz': 'hard'}) config.read('example.cfg') - print(config.get('Section1', 'foo')) # -> "Python is fun!" + print(config.get('Section1', 'foo')) # -> "Python is fun!" config.remove_option('Section1', 'bar') config.remove_option('Section1', 'baz') - print(config.get('Section1', 'foo')) # -> "Life is hard!" + print(config.get('Section1', 'foo')) # -> "Life is hard!" .. _configparser-objects: @@ -859,7 +879,7 @@ interpolation if an option used is not defined elsewhere. :: ConfigParser Objects -------------------- -.. class:: ConfigParser(defaults=None, dict_type=collections.OrderedDict, allow_no_value=False, delimiters=('=', ':'), comment_prefixes=('#', ';'), inline_comment_prefixes=None, strict=True, empty_lines_in_values=True, default_section=configparser.DEFAULTSECT, interpolation=BasicInterpolation()) +.. class:: ConfigParser(defaults=None, dict_type=collections.OrderedDict, allow_no_value=False, delimiters=('=', ':'), comment_prefixes=('#', ';'), inline_comment_prefixes=None, strict=True, empty_lines_in_values=True, default_section=configparser.DEFAULTSECT, interpolation=BasicInterpolation(), converters={}) The main configuration parser. When *defaults* is given, it is initialized into the dictionary of intrinsic defaults. When *dict_type* is given, it @@ -869,8 +889,8 @@ ConfigParser Objects When *delimiters* is given, it is used as the set of substrings that divide keys from values. When *comment_prefixes* is given, it will be used as the set of substrings that prefix comments in otherwise empty lines. - Comments can be indented. When *inline_comment_prefixes* is given, it will be - used as the set of substrings that prefix comments in non-empty lines. + Comments can be indented. When *inline_comment_prefixes* is given, it will + be used as the set of substrings that prefix comments in non-empty lines. When *strict* is ``True`` (the default), the parser won't allow for any section or option duplicates while reading from a single source (file, @@ -884,13 +904,13 @@ ConfigParser Objects When *default_section* is given, it specifies the name for the special section holding default values for other sections and interpolation purposes - (normally named ``"DEFAULT"``). This value can be retrieved and changed on + (normally named ``"DEFAULT"``). This value can be retrieved and changed on runtime using the ``default_section`` instance attribute. Interpolation behaviour may be customized by providing a custom handler through the *interpolation* argument. ``None`` can be used to turn off interpolation completely, ``ExtendedInterpolation()`` provides a more - advanced variant inspired by ``zc.buildout``. More on the subject in the + advanced variant inspired by ``zc.buildout``. More on the subject in the `dedicated documentation section <#interpolation-of-values>`_. All option names used in interpolation will be passed through the @@ -899,6 +919,12 @@ ConfigParser Objects converts option names to lower case), the values ``foo %(bar)s`` and ``foo %(BAR)s`` are equivalent. + When *converters* is given, it should be a dictionary where each key + represents the name of a type converter and each value is a callable + implementing the conversion from string to the desired datatype. Every + converter gets its own corresponding :meth:`get*()` method on the parser + object and section proxies. + .. versionchanged:: 3.1 The default *dict_type* is :class:`collections.OrderedDict`. @@ -907,6 +933,9 @@ ConfigParser Objects *empty_lines_in_values*, *default_section* and *interpolation* were added. + .. versionchanged:: 3.5 + The *converters* argument was added. + .. method:: defaults() @@ -944,7 +973,7 @@ ConfigParser Objects .. method:: has_option(section, option) If the given *section* exists, and contains the given *option*, return - :const:`True`; otherwise return :const:`False`. If the specified + :const:`True`; otherwise return :const:`False`. If the specified *section* is :const:`None` or an empty string, DEFAULT is assumed. @@ -1069,7 +1098,7 @@ ConfigParser Objects :meth:`get` method. .. versionchanged:: 3.2 - Items present in *vars* no longer appear in the result. The previous + Items present in *vars* no longer appear in the result. The previous behaviour mixed actual parser options with variables provided for interpolation. @@ -1170,7 +1199,7 @@ RawConfigParser Objects .. note:: Consider using :class:`ConfigParser` instead which checks types of - the values to be stored internally. If you don't want interpolation, you + the values to be stored internally. If you don't want interpolation, you can use ``ConfigParser(interpolation=None)``. @@ -1181,7 +1210,7 @@ RawConfigParser Objects *default section* name is passed, :exc:`ValueError` is raised. Type of *section* is not checked which lets users create non-string named - sections. This behaviour is unsupported and may cause internal errors. + sections. This behaviour is unsupported and may cause internal errors. .. method:: set(section, option, value) @@ -1282,3 +1311,4 @@ Exceptions .. [1] Config parsers allow for heavy customization. If you are interested in changing the behaviour outlined by the footnote reference, consult the `Customizing Parser Behaviour`_ section. + diff --git a/Doc/library/constants.rst b/Doc/library/constants.rst index 42b5af23a5..d5a0f09173 100644 --- a/Doc/library/constants.rst +++ b/Doc/library/constants.rst @@ -45,7 +45,6 @@ A small number of constants live in the built-in namespace. They are: for more details. - .. data:: Ellipsis The same as ``...``. Special value used mostly in conjunction with extended diff --git a/Doc/library/contextlib.rst b/Doc/library/contextlib.rst index 2b2ceceb2e..cf85fcd4b2 100644 --- a/Doc/library/contextlib.rst +++ b/Doc/library/contextlib.rst @@ -172,6 +172,16 @@ Functions and classes provided: .. versionadded:: 3.4 +.. function:: redirect_stderr(new_target) + + Similar to :func:`~contextlib.redirect_stdout` but redirecting + :data:`sys.stderr` to another file or file-like object. + + This context manager is :ref:`reentrant <reentrant-cms>`. + + .. versionadded:: 3.5 + + .. class:: ContextDecorator() A base class that enables a context manager to also be used as a decorator. @@ -593,7 +603,7 @@ an explicit ``with`` statement. .. seealso:: - :pep:`0343` - The "with" statement + :pep:`343` - The "with" statement The specification, background, and examples for the Python :keyword:`with` statement. @@ -634,7 +644,7 @@ to yield if an attempt is made to use them a second time:: Before After >>> with cm: - ... pass + ... pass ... Traceback (most recent call last): ... diff --git a/Doc/library/copy.rst b/Doc/library/copy.rst index 02ef0e5ee7..d0b861d469 100644 --- a/Doc/library/copy.rst +++ b/Doc/library/copy.rst @@ -4,6 +4,10 @@ .. module:: copy :synopsis: Shallow and deep copy operations. +**Source code:** :source:`Lib/copy.py` + +-------------- + Assignment statements in Python do not copy objects, they create bindings between a target and an object. For collections that are mutable or contain mutable items, a copy is sometimes needed so one can change one copy without @@ -44,7 +48,7 @@ copy operations: reference to themselves) may cause a recursive loop. * Because deep copy copies *everything* it may copy too much, e.g., - administrative data structures that should be shared even between copies. + even administrative data structures that should be shared even between copies. The :func:`deepcopy` function avoids these problems by: diff --git a/Doc/library/copyreg.rst b/Doc/library/copyreg.rst index 18306c7f99..eeabb4c658 100644 --- a/Doc/library/copyreg.rst +++ b/Doc/library/copyreg.rst @@ -4,11 +4,14 @@ .. module:: copyreg :synopsis: Register pickle support functions. +**Source code:** :source:`Lib/copyreg.py` .. index:: module: pickle module: copy +-------------- + The :mod:`copyreg` module offers a way to define functions used while pickling specific objects. The :mod:`pickle` and :mod:`copy` modules use those functions when pickling/copying those objects. The module provides configuration diff --git a/Doc/library/crypt.rst b/Doc/library/crypt.rst index b4c90cd592..a21c1e7a75 100644 --- a/Doc/library/crypt.rst +++ b/Doc/library/crypt.rst @@ -4,15 +4,19 @@ .. module:: crypt :platform: Unix :synopsis: The crypt() function used to check Unix passwords. + .. moduleauthor:: Steven D. Majewski <sdm7g@virginia.edu> .. sectionauthor:: Steven D. Majewski <sdm7g@virginia.edu> .. sectionauthor:: Peter Funk <pf@artcom-gmbh.de> +**Source code:** :source:`Lib/crypt.py` .. index:: single: crypt(3) pair: cipher; DES +-------------- + This module implements an interface to the :manpage:`crypt(3)` routine, which is a one-way hash function based upon a modified DES algorithm; see the Unix man page for further details. Possible uses include storing hashed passwords @@ -149,4 +153,4 @@ check it against the original:: hashed = crypt.crypt(plaintext) if not compare_hash(hashed, crypt.crypt(plaintext, hashed)): - raise ValueError("hashed version doesn't validate against original") + raise ValueError("hashed version doesn't validate against original") diff --git a/Doc/library/csv.rst b/Doc/library/csv.rst index e9a9cb5ddb..7fb4fc8256 100644 --- a/Doc/library/csv.rst +++ b/Doc/library/csv.rst @@ -3,13 +3,17 @@ .. module:: csv :synopsis: Write and read tabular data to and from delimited files. + .. sectionauthor:: Skip Montanaro <skip@pobox.com> +**Source code:** :source:`Lib/csv.py` .. index:: single: csv pair: data; tabular +-------------- + The so-called CSV (Comma Separated Values) format is the most common import and export format for spreadsheets and databases. CSV format was used for many years prior to attempts to describe the format in a standardized way in @@ -418,7 +422,7 @@ Writer Objects :class:`Writer` objects (:class:`DictWriter` instances and objects returned by the :func:`writer` function) have the following public methods. A *row* must be -a sequence of strings or numbers for :class:`Writer` objects and a dictionary +an iterable of strings or numbers for :class:`Writer` objects and a dictionary mapping fieldnames to strings or numbers (by passing them through :func:`str` first) for :class:`DictWriter` objects. Note that complex numbers are written out surrounded by parens. This may cause some problems for other programs which @@ -430,6 +434,8 @@ read CSV files (assuming they support complex numbers at all). Write the *row* parameter to the writer's file object, formatted according to the current dialect. + .. versionchanged:: 3.5 + Added support of arbitrary iterables. .. method:: csvwriter.writerows(rows) diff --git a/Doc/library/ctypes.rst b/Doc/library/ctypes.rst index 630d279174..a675790737 100644 --- a/Doc/library/ctypes.rst +++ b/Doc/library/ctypes.rst @@ -3,8 +3,10 @@ .. module:: ctypes :synopsis: A foreign function library for Python. + .. moduleauthor:: Thomas Heller <theller@python.net> +-------------- :mod:`ctypes` is a foreign function library for Python. It provides C compatible data types, and allows calling functions in DLLs or shared libraries. It can be @@ -20,11 +22,10 @@ Note: The code samples in this tutorial use :mod:`doctest` to make sure that they actually work. Since some code samples behave differently under Linux, Windows, or Mac OS X, they contain doctest directives in comments. -Note: Some code samples reference the ctypes :class:`c_int` type. This type is -an alias for the :class:`c_long` type on 32-bit systems. So, you should not be -confused if :class:`c_long` is printed if you would expect :class:`c_int` --- -they are actually the same type. - +Note: Some code samples reference the ctypes :class:`c_int` type. On platforms +where ``sizeof(long) == sizeof(int)`` it is an alias to :class:`c_long`. +So, you should not be confused if :class:`c_long` is printed if you would expect +:class:`c_int` --- they are actually the same type. .. _ctypes-loading-dynamic-link-libraries: @@ -52,24 +53,30 @@ library containing most standard C functions, and uses the cdecl calling convention:: >>> from ctypes import * - >>> print(windll.kernel32) # doctest: +WINDOWS + >>> print(windll.kernel32) # doctest: +WINDOWS <WinDLL 'kernel32', handle ... at ...> - >>> print(cdll.msvcrt) # doctest: +WINDOWS + >>> print(cdll.msvcrt) # doctest: +WINDOWS <CDLL 'msvcrt', handle ... at ...> - >>> libc = cdll.msvcrt # doctest: +WINDOWS + >>> libc = cdll.msvcrt # doctest: +WINDOWS >>> Windows appends the usual ``.dll`` file suffix automatically. +.. note:: + Accessing the standard C library through ``cdll.msvcrt`` will use an + outdated version of the library that may be incompatible with the one + being used by Python. Where possible, use native Python functionality, + or else import and use the ``msvcrt`` module. + On Linux, it is required to specify the filename *including* the extension to load a library, so attribute access can not be used to load libraries. Either the :meth:`LoadLibrary` method of the dll loaders should be used, or you should load the library by creating an instance of CDLL by calling the constructor:: - >>> cdll.LoadLibrary("libc.so.6") # doctest: +LINUX + >>> cdll.LoadLibrary("libc.so.6") # doctest: +LINUX <CDLL 'libc.so.6', handle ... at ...> - >>> libc = CDLL("libc.so.6") # doctest: +LINUX - >>> libc # doctest: +LINUX + >>> libc = CDLL("libc.so.6") # doctest: +LINUX + >>> libc # doctest: +LINUX <CDLL 'libc.so.6', handle ... at ...> >>> @@ -86,9 +93,9 @@ Functions are accessed as attributes of dll objects:: >>> from ctypes import * >>> libc.printf <_FuncPtr object at 0x...> - >>> print(windll.kernel32.GetModuleHandleA) # doctest: +WINDOWS + >>> print(windll.kernel32.GetModuleHandleA) # doctest: +WINDOWS <_FuncPtr object at 0x...> - >>> print(windll.kernel32.MyOwnFunction) # doctest: +WINDOWS + >>> print(windll.kernel32.MyOwnFunction) # doctest: +WINDOWS Traceback (most recent call last): File "<stdin>", line 1, in ? File "ctypes.py", line 239, in __getattr__ @@ -117,16 +124,16 @@ Sometimes, dlls export functions with names which aren't valid Python identifiers, like ``"??2@YAPAXI@Z"``. In this case you have to use :func:`getattr` to retrieve the function:: - >>> getattr(cdll.msvcrt, "??2@YAPAXI@Z") # doctest: +WINDOWS + >>> getattr(cdll.msvcrt, "??2@YAPAXI@Z") # doctest: +WINDOWS <_FuncPtr object at 0x...> >>> On Windows, some dlls export functions not by name but by ordinal. These functions can be accessed by indexing the dll object with the ordinal number:: - >>> cdll.kernel32[1] # doctest: +WINDOWS + >>> cdll.kernel32[1] # doctest: +WINDOWS <_FuncPtr object at 0x...> - >>> cdll.kernel32[0] # doctest: +WINDOWS + >>> cdll.kernel32[0] # doctest: +WINDOWS Traceback (most recent call last): File "<stdin>", line 1, in ? File "ctypes.py", line 310, in __getitem__ @@ -148,9 +155,9 @@ handle. This example calls both functions with a NULL pointer (``None`` should be used as the NULL pointer):: - >>> print(libc.time(None)) # doctest: +SKIP + >>> print(libc.time(None)) # doctest: +SKIP 1150640792 - >>> print(hex(windll.kernel32.GetModuleHandleA(None))) # doctest: +WINDOWS + >>> print(hex(windll.kernel32.GetModuleHandleA(None))) # doctest: +WINDOWS 0x1d000000 >>> @@ -159,11 +166,11 @@ of arguments or the wrong calling convention. Unfortunately this only works on Windows. It does this by examining the stack after the function returns, so although an error is raised the function *has* been called:: - >>> windll.kernel32.GetModuleHandleA() # doctest: +WINDOWS + >>> windll.kernel32.GetModuleHandleA() # doctest: +WINDOWS Traceback (most recent call last): File "<stdin>", line 1, in ? ValueError: Procedure probably called with not enough arguments (4 bytes missing) - >>> windll.kernel32.GetModuleHandleA(0, 0) # doctest: +WINDOWS + >>> windll.kernel32.GetModuleHandleA(0, 0) # doctest: +WINDOWS Traceback (most recent call last): File "<stdin>", line 1, in ? ValueError: Procedure probably called with too many arguments (4 bytes in excess) @@ -172,13 +179,13 @@ although an error is raised the function *has* been called:: The same exception is raised when you call an ``stdcall`` function with the ``cdecl`` calling convention, or vice versa:: - >>> cdll.kernel32.GetModuleHandleA(None) # doctest: +WINDOWS + >>> cdll.kernel32.GetModuleHandleA(None) # doctest: +WINDOWS Traceback (most recent call last): File "<stdin>", line 1, in ? ValueError: Procedure probably called with not enough arguments (4 bytes missing) >>> - >>> windll.msvcrt.printf(b"spam") # doctest: +WINDOWS + >>> windll.msvcrt.printf(b"spam") # doctest: +WINDOWS Traceback (most recent call last): File "<stdin>", line 1, in ? ValueError: Procedure probably called with too many arguments (4 bytes in excess) @@ -191,7 +198,7 @@ On Windows, :mod:`ctypes` uses win32 structured exception handling to prevent crashes from general protection faults when functions are called with invalid argument values:: - >>> windll.kernel32.GetModuleHandleA(32) # doctest: +WINDOWS + >>> windll.kernel32.GetModuleHandleA(32) # doctest: +WINDOWS Traceback (most recent call last): File "<stdin>", line 1, in ? OSError: exception: access violation reading 0x00000020 @@ -456,9 +463,9 @@ Here is a more advanced example, it uses the ``strchr`` function, which expects a string pointer and a char, and returns a pointer to a string:: >>> strchr = libc.strchr - >>> strchr(b"abcdef", ord("d")) # doctest: +SKIP + >>> strchr(b"abcdef", ord("d")) # doctest: +SKIP 8059983 - >>> strchr.restype = c_char_p # c_char_p is a pointer to a string + >>> strchr.restype = c_char_p # c_char_p is a pointer to a string >>> strchr(b"abcdef", ord("d")) b'def' >>> print(strchr(b"abcdef", ord("x"))) @@ -489,17 +496,17 @@ callable will be called with the *integer* the C function returns, and the result of this call will be used as the result of your function call. This is useful to check for error return values and automatically raise an exception:: - >>> GetModuleHandle = windll.kernel32.GetModuleHandleA # doctest: +WINDOWS + >>> GetModuleHandle = windll.kernel32.GetModuleHandleA # doctest: +WINDOWS >>> def ValidHandle(value): ... if value == 0: ... raise WinError() ... return value ... >>> - >>> GetModuleHandle.restype = ValidHandle # doctest: +WINDOWS - >>> GetModuleHandle(None) # doctest: +WINDOWS + >>> GetModuleHandle.restype = ValidHandle # doctest: +WINDOWS + >>> GetModuleHandle(None) # doctest: +WINDOWS 486539264 - >>> GetModuleHandle("something silly") # doctest: +WINDOWS + >>> GetModuleHandle("something silly") # doctest: +WINDOWS Traceback (most recent call last): File "<stdin>", line 1, in ? File "<stdin>", line 3, in ValidHandle @@ -665,17 +672,17 @@ positive integer:: TenPointsArrayType = POINT * 10 -Here is an example of an somewhat artificial data type, a structure containing 4 +Here is an example of a somewhat artificial data type, a structure containing 4 POINTs among other stuff:: >>> from ctypes import * >>> class POINT(Structure): - ... _fields_ = ("x", c_int), ("y", c_int) + ... _fields_ = ("x", c_int), ("y", c_int) ... >>> class MyStruct(Structure): - ... _fields_ = [("a", c_int), - ... ("b", c_float), - ... ("point_array", POINT * 4)] + ... _fields_ = [("a", c_int), + ... ("b", c_float), + ... ("point_array", POINT * 4)] >>> >>> print(len(MyStruct().point_array)) 4 @@ -716,8 +723,8 @@ Pointer instances are created by calling the :func:`pointer` function on a >>> pi = pointer(i) >>> -Pointer instances have a :attr:`contents` attribute which returns the object to -which the pointer points, the ``i`` object above:: +Pointer instances have a :attr:`~_Pointer.contents` attribute which +returns the object to which the pointer points, the ``i`` object above:: >>> pi.contents c_long(42) @@ -942,7 +949,7 @@ other, and finally follow the pointer chain a few times:: Callback functions ^^^^^^^^^^^^^^^^^^ -:mod:`ctypes` allows to create C callable function pointers from Python callables. +:mod:`ctypes` allows creating C callable function pointers from Python callables. These are sometimes called *callback functions*. First, you must create a class for the callback function. The class knows the @@ -992,7 +999,7 @@ passed:: The result:: - >>> qsort(ia, len(ia), sizeof(c_int), cmp_func) # doctest: +LINUX + >>> qsort(ia, len(ia), sizeof(c_int), cmp_func) # doctest: +LINUX py_cmp_func 5 1 py_cmp_func 33 99 py_cmp_func 7 33 @@ -1094,14 +1101,15 @@ access violation or whatever, so it's better to break out of the loop when we hit the NULL entry:: >>> for item in table: - ... print(item.name, item.size) - ... if item.name is None: - ... break + ... if item.name is None: + ... break + ... print(item.name.decode("ascii"), item.size) ... - __hello__ 104 - __phello__ -104 - __phello__.spam 104 - None 0 + _frozen_importlib 31764 + _frozen_importlib_external 41499 + __hello__ 161 + __phello__ -161 + __phello__.spam 161 >>> The fact that standard Python has a frozen module and a frozen package @@ -1291,7 +1299,7 @@ module instead of using :func:`find_library` to locate the library at runtime. Loading shared libraries ^^^^^^^^^^^^^^^^^^^^^^^^ -There are several ways to loaded shared libraries into the Python process. One +There are several ways to load shared libraries into the Python process. One way is to instantiate one of the following classes: @@ -1350,7 +1358,7 @@ details, consult the :manpage:`dlopen(3)` manpage, on Windows, *mode* is ignored. The *use_errno* parameter, when set to True, enables a ctypes mechanism that -allows to access the system :data:`errno` error number in a safe way. +allows accessing the system :data:`errno` error number in a safe way. :mod:`ctypes` maintains a thread-local copy of the systems :data:`errno` variable; if you call foreign functions created with ``use_errno=True`` then the :data:`errno` value before the function call is swapped with the ctypes private @@ -1421,7 +1429,7 @@ loader instance. Class which loads shared libraries. *dlltype* should be one of the :class:`CDLL`, :class:`PyDLL`, :class:`WinDLL`, or :class:`OleDLL` types. - :meth:`__getattr__` has special behavior: It allows to load a shared library by + :meth:`__getattr__` has special behavior: It allows loading a shared library by accessing it as attribute of a library loader instance. The result is cached, so repeated attribute accesses return the same library each time. @@ -1498,7 +1506,7 @@ They are instances of a private class: It is possible to assign a callable Python object that is not a ctypes type, in this case the function is assumed to return a C :c:type:`int`, and - the callable will be called with this integer, allowing to do further + the callable will be called with this integer, allowing further processing or error checking. Using this is deprecated, for more flexible post processing or error checking use a ctypes data type as :attr:`restype` and assign a callable to the :attr:`errcheck` attribute. @@ -1513,7 +1521,7 @@ They are instances of a private class: When a foreign function is called, each actual argument is passed to the :meth:`from_param` class method of the items in the :attr:`argtypes` - tuple, this method allows to adapt the actual argument to an object that + tuple, this method allows adapting the actual argument to an object that the foreign function accepts. For example, a :class:`c_char_p` item in the :attr:`argtypes` tuple will convert a string passed as argument into a bytes object using ctypes conversion rules. @@ -1521,7 +1529,7 @@ They are instances of a private class: New: It is now possible to put items in argtypes which are not ctypes types, but each item must have a :meth:`from_param` method which returns a value usable as argument (integer, string, ctypes instance). This allows - to define adapters that can adapt custom objects as function parameters. + defining adapters that can adapt custom objects as function parameters. .. attribute:: errcheck @@ -1535,12 +1543,12 @@ They are instances of a private class: *result* is what the foreign function returns, as specified by the :attr:`restype` attribute. - *func* is the foreign function object itself, this allows to reuse the + *func* is the foreign function object itself, this allows reusing the same callable object to check or post process the results of several functions. *arguments* is a tuple containing the parameters originally passed to - the function call, this allows to specialize the behavior on the + the function call, this allows specializing the behavior on the arguments used. The object that this function returns will be returned from the @@ -1785,7 +1793,7 @@ Utility functions If a bytes object is specified as first argument, the buffer is made one item larger than its length so that the last element in the array is a NUL termination character. An integer can be passed as second argument which allows - to specify the size of the array if the length of the bytes should not be used. + specifying the size of the array if the length of the bytes should not be used. @@ -1800,21 +1808,21 @@ Utility functions If a string is specified as first argument, the buffer is made one item larger than the length of the string so that the last element in the array is a NUL termination character. An integer can be passed as second argument which - allows to specify the size of the array if the length of the string should not + allows specifying the size of the array if the length of the string should not be used. .. function:: DllCanUnloadNow() - Windows only: This function is a hook which allows to implement in-process + Windows only: This function is a hook which allows implementing in-process COM servers with ctypes. It is called from the DllCanUnloadNow function that the _ctypes extension dll exports. .. function:: DllGetClassObject() - Windows only: This function is a hook which allows to implement in-process + Windows only: This function is a hook which allows implementing in-process COM servers with ctypes. It is called from the DllGetClassObject function that the ``_ctypes`` extension dll exports. @@ -1882,7 +1890,7 @@ Utility functions .. function:: POINTER(type) This factory function creates and returns a new ctypes pointer type. Pointer - types are cached an reused internally, so calling this function repeatedly is + types are cached and reused internally, so calling this function repeatedly is cheap. *type* must be a ctypes type. @@ -2321,7 +2329,7 @@ other data types containing pointer type fields. checked, only one field can be accessed when names are repeated. It is possible to define the :attr:`_fields_` class variable *after* the - class statement that defines the Structure subclass, this allows to create + class statement that defines the Structure subclass, this allows creating data types that directly or indirectly reference themselves:: class List(Structure): @@ -2342,7 +2350,7 @@ other data types containing pointer type fields. .. attribute:: _pack_ - An optional small integer that allows to override the alignment of + An optional small integer that allows overriding the alignment of structure fields in the instance. :attr:`_pack_` must already be defined when :attr:`_fields_` is assigned, otherwise it will have no effect. @@ -2354,8 +2362,8 @@ other data types containing pointer type fields. assigned, otherwise it will have no effect. The fields listed in this variable must be structure or union type fields. - :mod:`ctypes` will create descriptors in the structure type that allows to - access the nested fields directly, without the need to create the + :mod:`ctypes` will create descriptors in the structure type that allows + accessing the nested fields directly, without the need to create the structure or union field. Here is an example type (Windows):: @@ -2401,6 +2409,56 @@ other data types containing pointer type fields. Arrays and pointers ^^^^^^^^^^^^^^^^^^^ -Not yet written - please see the sections :ref:`ctypes-pointers` and section -:ref:`ctypes-arrays` in the tutorial. +.. class:: Array(\*args) + + Abstract base class for arrays. + + The recommended way to create concrete array types is by multiplying any + :mod:`ctypes` data type with a positive integer. Alternatively, you can subclass + this type and define :attr:`_length_` and :attr:`_type_` class variables. + Array elements can be read and written using standard + subscript and slice accesses; for slice reads, the resulting object is + *not* itself an :class:`Array`. + + + .. attribute:: _length_ + + A positive integer specifying the number of elements in the array. + Out-of-range subscripts result in an :exc:`IndexError`. Will be + returned by :func:`len`. + + + .. attribute:: _type_ + + Specifies the type of each element in the array. + + + Array subclass constructors accept positional arguments, used to + initialize the elements in order. + + +.. class:: _Pointer + + Private, abstract base class for pointers. + + Concrete pointer types are created by calling :func:`POINTER` with the + type that will be pointed to; this is done automatically by + :func:`pointer`. + + If a pointer points to an array, its elements can be read and + written using standard subscript and slice accesses. Pointer objects + have no size, so :func:`len` will raise :exc:`TypeError`. Negative + subscripts will read from the memory *before* the pointer (as in C), and + out-of-range subscripts will probably crash with an access violation (if + you're lucky). + + + .. attribute:: _type_ + + Specifies the type pointed to. + + .. attribute:: contents + + Returns the object to which to pointer points. Assigning to this + attribute changes the pointer to point to the assigned object. diff --git a/Doc/library/curses.ascii.rst b/Doc/library/curses.ascii.rst index 6bc1fb9ae3..db3c827406 100644 --- a/Doc/library/curses.ascii.rst +++ b/Doc/library/curses.ascii.rst @@ -3,9 +3,11 @@ .. module:: curses.ascii :synopsis: Constants and set-membership functions for ASCII characters. + .. moduleauthor:: Eric S. Raymond <esr@thyrsus.com> .. sectionauthor:: Eric S. Raymond <esr@thyrsus.com> +-------------- The :mod:`curses.ascii` module supplies name constants for ASCII characters and functions to test membership in various ASCII character classes. The constants @@ -113,12 +115,12 @@ C library: .. function:: isblank(c) - Checks for an ASCII whitespace character. + Checks for an ASCII whitespace character; space or horizontal tab. .. function:: iscntrl(c) - Checks for an ASCII control character (in the range 0x00 to 0x1f). + Checks for an ASCII control character (in the range 0x00 to 0x1f or 0x7f). .. function:: isdigit(c) diff --git a/Doc/library/curses.panel.rst b/Doc/library/curses.panel.rst index a3c8bda63c..c99c37a565 100644 --- a/Doc/library/curses.panel.rst +++ b/Doc/library/curses.panel.rst @@ -3,8 +3,10 @@ .. module:: curses.panel :synopsis: A panel stack extension that adds depth to curses windows. + .. sectionauthor:: A.M. Kuchling <amk@amk.ca> +-------------- Panels are windows with the added feature of depth, so they can be stacked on top of each other, and only the visible portions of each window will be diff --git a/Doc/library/curses.rst b/Doc/library/curses.rst index f3e60b4be9..b12a325f37 100644 --- a/Doc/library/curses.rst +++ b/Doc/library/curses.rst @@ -5,9 +5,12 @@ :synopsis: An interface to the curses library, providing portable terminal handling. :platform: Unix + .. sectionauthor:: Moshe Zadka <moshez@zadka.site.co.il> .. sectionauthor:: Eric Raymond <esr@thyrsus.com> +-------------- + The :mod:`curses` module provides an interface to the curses library, the de-facto standard for portable advanced terminal handling. @@ -599,6 +602,13 @@ The module :mod:`curses` defines the following functions: Only one *ch* can be pushed before :meth:`getch` is called. +.. function:: update_lines_cols() + + Update :envvar:`LINES` and :envvar:`COLS`. Useful for detecting manual screen resize. + + .. versionadded:: 3.5 + + .. function:: unget_wch(ch) Push *ch* so the next :meth:`get_wch` will return it. @@ -1501,9 +1511,9 @@ keys); also, the following keypad mappings are standard: +------------------+-----------+ | :kbd:`End` | KEY_END | +------------------+-----------+ -| :kbd:`Page Up` | KEY_NPAGE | +| :kbd:`Page Up` | KEY_PPAGE | +------------------+-----------+ -| :kbd:`Page Down` | KEY_PPAGE | +| :kbd:`Page Down` | KEY_NPAGE | +------------------+-----------+ The following table lists characters from the alternate character set. These are diff --git a/Doc/library/datetime.rst b/Doc/library/datetime.rst index 88f3f6eea7..9254ae8cfa 100644 --- a/Doc/library/datetime.rst +++ b/Doc/library/datetime.rst @@ -3,10 +3,15 @@ .. module:: datetime :synopsis: Basic date and time types. + .. moduleauthor:: Tim Peters <tim@zope.com> .. sectionauthor:: Tim Peters <tim@zope.com> .. sectionauthor:: A.M. Kuchling <amk@amk.ca> +**Source code:** :source:`Lib/datetime.py` + +-------------- + .. XXX what order should the types be discussed in? The :mod:`datetime` module supplies classes for manipulating dates and times in @@ -31,7 +36,7 @@ particular number represents metres, miles, or mass. Naive objects are easy to understand and to work with, at the cost of ignoring some aspects of reality. For applications requiring aware objects, :class:`.datetime` and :class:`.time` -objects have an optional time zone information attribute, :attr:`tzinfo`, that +objects have an optional time zone information attribute, :attr:`!tzinfo`, that can be set to an instance of a subclass of the abstract :class:`tzinfo` class. These :class:`tzinfo` objects capture information about the offset from UTC time, the time zone name, and whether Daylight Saving Time is in effect. Note @@ -83,7 +88,7 @@ Available Types An idealized time, independent of any particular day, assuming that every day has exactly 24\*60\*60 seconds (there is no notion of "leap seconds" here). Attributes: :attr:`hour`, :attr:`minute`, :attr:`second`, :attr:`microsecond`, - and :attr:`tzinfo`. + and :attr:`.tzinfo`. .. class:: datetime @@ -91,7 +96,7 @@ Available Types A combination of a date and a time. Attributes: :attr:`year`, :attr:`month`, :attr:`day`, :attr:`hour`, :attr:`minute`, :attr:`second`, :attr:`microsecond`, - and :attr:`tzinfo`. + and :attr:`.tzinfo`. .. class:: timedelta @@ -102,6 +107,7 @@ Available Types .. class:: tzinfo + :noindex: An abstract base class for time zone information objects. These are used by the :class:`.datetime` and :class:`.time` classes to provide a customizable notion of @@ -109,6 +115,7 @@ Available Types time). .. class:: timezone + :noindex: A class that implements the :class:`tzinfo` abstract base class as a fixed offset from the UTC. @@ -558,7 +565,7 @@ Instance methods: Return a 3-tuple, (ISO year, ISO week number, ISO weekday). The ISO calendar is a widely used variant of the Gregorian calendar. See - http://www.staff.science.uu.nl/~gent0113/calendar/isocalendar.htm for a good + https://www.staff.science.uu.nl/~gent0113/calendar/isocalendar.htm for a good explanation. The ISO year consists of 52 or 53 full weeks, and where a week starts on a @@ -602,7 +609,7 @@ Instance methods: .. method:: date.__format__(format) - Same as :meth:`.date.strftime`. This makes it possible to specify format + Same as :meth:`.date.strftime`. This makes it possible to specify a format string for a :class:`.date` object when using :meth:`str.format`. For a complete list of formatting directives, see :ref:`strftime-strptime-behavior`. @@ -695,7 +702,7 @@ Other constructors, all class methods: .. classmethod:: datetime.today() - Return the current local datetime, with :attr:`tzinfo` ``None``. This is + Return the current local datetime, with :attr:`.tzinfo` ``None``. This is equivalent to ``datetime.fromtimestamp(time.time())``. See also :meth:`now`, :meth:`fromtimestamp`. @@ -708,15 +715,15 @@ Other constructors, all class methods: (for example, this may be possible on platforms supplying the C :c:func:`gettimeofday` function). - Else *tz* must be an instance of a class :class:`tzinfo` subclass, and the - current date and time are converted to *tz*'s time zone. In this case the + If *tz* is not ``None``, it must be an instance of a :class:`tzinfo` subclass, and the + current date and time are converted to *tz*’s time zone. In this case the result is equivalent to ``tz.fromutc(datetime.utcnow().replace(tzinfo=tz))``. See also :meth:`today`, :meth:`utcnow`. .. classmethod:: datetime.utcnow() - Return the current UTC date and time, with :attr:`tzinfo` ``None``. This is like + Return the current UTC date and time, with :attr:`.tzinfo` ``None``. This is like :meth:`now`, but returns the current UTC date and time, as a naive :class:`.datetime` object. An aware current UTC datetime can be obtained by calling ``datetime.now(timezone.utc)``. See also :meth:`now`. @@ -728,8 +735,8 @@ Other constructors, all class methods: specified, the timestamp is converted to the platform's local date and time, and the returned :class:`.datetime` object is naive. - Else *tz* must be an instance of a class :class:`tzinfo` subclass, and the - timestamp is converted to *tz*'s time zone. In this case the result is + If *tz* is not ``None``, it must be an instance of a :class:`tzinfo` subclass, and the + timestamp is converted to *tz*’s time zone. In this case the result is equivalent to ``tz.fromutc(datetime.utcfromtimestamp(timestamp).replace(tzinfo=tz))``. @@ -754,16 +761,22 @@ Other constructors, all class methods: .. classmethod:: datetime.utcfromtimestamp(timestamp) Return the UTC :class:`.datetime` corresponding to the POSIX timestamp, with - :attr:`tzinfo` ``None``. This may raise :exc:`OverflowError`, if the timestamp is + :attr:`.tzinfo` ``None``. This may raise :exc:`OverflowError`, if the timestamp is out of the range of values supported by the platform C :c:func:`gmtime` function, and :exc:`OSError` on :c:func:`gmtime` failure. - It's common for this to be restricted to years in 1970 through 2038. See also - :meth:`fromtimestamp`. + It's common for this to be restricted to years in 1970 through 2038. + + To get an aware :class:`.datetime` object, call :meth:`fromtimestamp`:: + + datetime.fromtimestamp(timestamp, timezone.utc) - On the POSIX compliant platforms, ``utcfromtimestamp(timestamp)`` - is equivalent to the following expression:: + On the POSIX compliant platforms, it is equivalent to the following + expression:: - datetime(1970, 1, 1) + timedelta(seconds=timestamp) + datetime(1970, 1, 1, tzinfo=timezone.utc) + timedelta(seconds=timestamp) + + except the latter formula always supports the full years range: between + :const:`MINYEAR` and :const:`MAXYEAR` inclusive. .. versionchanged:: 3.3 Raise :exc:`OverflowError` instead of :exc:`ValueError` if the timestamp @@ -777,17 +790,17 @@ Other constructors, all class methods: Return the :class:`.datetime` corresponding to the proleptic Gregorian ordinal, where January 1 of year 1 has ordinal 1. :exc:`ValueError` is raised unless ``1 <= ordinal <= datetime.max.toordinal()``. The hour, minute, second and - microsecond of the result are all 0, and :attr:`tzinfo` is ``None``. + microsecond of the result are all 0, and :attr:`.tzinfo` is ``None``. .. classmethod:: datetime.combine(date, time) Return a new :class:`.datetime` object whose date components are equal to the - given :class:`date` object's, and whose time components and :attr:`tzinfo` + given :class:`date` object's, and whose time components and :attr:`.tzinfo` attributes are equal to the given :class:`.time` object's. For any :class:`.datetime` object *d*, ``d == datetime.combine(d.date(), d.timetz())``. If date is a - :class:`.datetime` object, its time components and :attr:`tzinfo` attributes + :class:`.datetime` object, its time components and :attr:`.tzinfo` attributes are ignored. @@ -883,7 +896,7 @@ Supported operations: (1) datetime2 is a duration of timedelta removed from datetime1, moving forward in time if ``timedelta.days`` > 0, or backward if ``timedelta.days`` < 0. The - result has the same :attr:`tzinfo` attribute as the input datetime, and + result has the same :attr:`~.datetime.tzinfo` attribute as the input datetime, and datetime2 - datetime1 == timedelta after. :exc:`OverflowError` is raised if datetime2.year would be smaller than :const:`MINYEAR` or larger than :const:`MAXYEAR`. Note that no time zone adjustments are done even if the @@ -891,7 +904,7 @@ Supported operations: (2) Computes the datetime2 such that datetime2 + timedelta == datetime1. As for - addition, the result has the same :attr:`tzinfo` attribute as the input + addition, the result has the same :attr:`~.datetime.tzinfo` attribute as the input datetime, and no time zone adjustments are done even if the input is aware. This isn't quite equivalent to datetime1 + (-timedelta), because -timedelta in isolation can overflow in cases where datetime1 - timedelta does not. @@ -901,12 +914,12 @@ Supported operations: both operands are naive, or if both are aware. If one is aware and the other is naive, :exc:`TypeError` is raised. - If both are naive, or both are aware and have the same :attr:`tzinfo` attribute, - the :attr:`tzinfo` attributes are ignored, and the result is a :class:`timedelta` + If both are naive, or both are aware and have the same :attr:`~.datetime.tzinfo` attribute, + the :attr:`~.datetime.tzinfo` attributes are ignored, and the result is a :class:`timedelta` object *t* such that ``datetime2 + t == datetime1``. No time zone adjustments are done in this case. - If both are aware and have different :attr:`tzinfo` attributes, ``a-b`` acts + If both are aware and have different :attr:`~.datetime.tzinfo` attributes, ``a-b`` acts as if *a* and *b* were first converted to naive UTC datetimes first. The result is ``(a.replace(tzinfo=None) - a.utcoffset()) - (b.replace(tzinfo=None) - b.utcoffset())`` except that the implementation never overflows. @@ -919,14 +932,14 @@ Supported operations: is raised if an order comparison is attempted. For equality comparisons, naive instances are never equal to aware instances. - If both comparands are aware, and have the same :attr:`tzinfo` attribute, the - common :attr:`tzinfo` attribute is ignored and the base datetimes are - compared. If both comparands are aware and have different :attr:`tzinfo` + If both comparands are aware, and have the same :attr:`~.datetime.tzinfo` attribute, the + common :attr:`~.datetime.tzinfo` attribute is ignored and the base datetimes are + compared. If both comparands are aware and have different :attr:`~.datetime.tzinfo` attributes, the comparands are first adjusted by subtracting their UTC offsets (obtained from ``self.utcoffset()``). .. versionchanged:: 3.3 - Equality comparisons between naive and aware :class:`datetime` + Equality comparisons between naive and aware :class:`.datetime` instances don't raise :exc:`TypeError`. .. note:: @@ -954,7 +967,7 @@ Instance methods: .. method:: datetime.time() Return :class:`.time` object with same hour, minute, second and microsecond. - :attr:`tzinfo` is ``None``. See also method :meth:`timetz`. + :attr:`.tzinfo` is ``None``. See also method :meth:`timetz`. .. method:: datetime.timetz() @@ -973,7 +986,7 @@ Instance methods: .. method:: datetime.astimezone(tz=None) - Return a :class:`datetime` object with new :attr:`tzinfo` attribute *tz*, + Return a :class:`.datetime` object with new :attr:`.tzinfo` attribute *tz*, adjusting the date and time data so the result is the same UTC time as *self*, but in *tz*'s local time. @@ -983,7 +996,7 @@ Instance methods: not return ``None``). If called without arguments (or with ``tz=None``) the system local - timezone is assumed. The ``tzinfo`` attribute of the converted + timezone is assumed. The ``.tzinfo`` attribute of the converted datetime instance will be set to an instance of :class:`timezone` with the zone name and offset obtained from the OS. @@ -1019,7 +1032,7 @@ Instance methods: .. method:: datetime.utcoffset() - If :attr:`tzinfo` is ``None``, returns ``None``, else returns + If :attr:`.tzinfo` is ``None``, returns ``None``, else returns ``self.tzinfo.utcoffset(self)``, and raises an exception if the latter doesn't return ``None``, or a :class:`timedelta` object representing a whole number of minutes with magnitude less than one day. @@ -1027,7 +1040,7 @@ Instance methods: .. method:: datetime.dst() - If :attr:`tzinfo` is ``None``, returns ``None``, else returns + If :attr:`.tzinfo` is ``None``, returns ``None``, else returns ``self.tzinfo.dst(self)``, and raises an exception if the latter doesn't return ``None``, or a :class:`timedelta` object representing a whole number of minutes with magnitude less than one day. @@ -1035,7 +1048,7 @@ Instance methods: .. method:: datetime.tzname() - If :attr:`tzinfo` is ``None``, returns ``None``, else returns + If :attr:`.tzinfo` is ``None``, returns ``None``, else returns ``self.tzinfo.tzname(self)``, raises an exception if the latter doesn't return ``None`` or a string object, @@ -1047,7 +1060,7 @@ Instance methods: d.hour, d.minute, d.second, d.weekday(), yday, dst))``, where ``yday = d.toordinal() - date(d.year, 1, 1).toordinal() + 1`` is the day number within the current year starting with ``1`` for January 1st. The :attr:`tm_isdst` flag - of the result is set according to the :meth:`dst` method: :attr:`tzinfo` is + of the result is set according to the :meth:`dst` method: :attr:`.tzinfo` is ``None`` or :meth:`dst` returns ``None``, :attr:`tm_isdst` is set to ``-1``; else if :meth:`dst` returns a non-zero value, :attr:`tm_isdst` is set to ``1``; else :attr:`tm_isdst` is set to ``0``. @@ -1074,18 +1087,18 @@ Instance methods: .. method:: datetime.timestamp() - Return POSIX timestamp corresponding to the :class:`datetime` + Return POSIX timestamp corresponding to the :class:`.datetime` instance. The return value is a :class:`float` similar to that returned by :func:`time.time`. - Naive :class:`datetime` instances are assumed to represent local + Naive :class:`.datetime` instances are assumed to represent local time and this method relies on the platform C :c:func:`mktime` - function to perform the conversion. Since :class:`datetime` + function to perform the conversion. Since :class:`.datetime` supports wider range of values than :c:func:`mktime` on many platforms, this method may raise :exc:`OverflowError` for times far in the past or far in the future. - For aware :class:`datetime` instances, the return value is computed + For aware :class:`.datetime` instances, the return value is computed as:: (dt - datetime(1970, 1, 1, tzinfo=timezone.utc)).total_seconds() @@ -1095,7 +1108,7 @@ Instance methods: .. note:: There is no method to obtain the POSIX timestamp directly from a - naive :class:`datetime` instance representing UTC time. If your + naive :class:`.datetime` instance representing UTC time. If your application uses this convention and your system timezone is not set to UTC, you can obtain the POSIX timestamp by supplying ``tzinfo=timezone.utc``:: @@ -1171,7 +1184,7 @@ Instance methods: .. method:: datetime.__format__(format) - Same as :meth:`.datetime.strftime`. This makes it possible to specify format + Same as :meth:`.datetime.strftime`. This makes it possible to specify a format string for a :class:`.datetime` object when using :meth:`str.format`. For a complete list of formatting directives, see :ref:`strftime-strptime-behavior`. @@ -1359,9 +1372,9 @@ Supported operations: comparisons, naive instances are never equal to aware instances. If both comparands are aware, and have - the same :attr:`tzinfo` attribute, the common :attr:`tzinfo` attribute is + the same :attr:`~time.tzinfo` attribute, the common :attr:`~time.tzinfo` attribute is ignored and the base times are compared. If both comparands are aware and - have different :attr:`tzinfo` attributes, the comparands are first adjusted by + have different :attr:`~time.tzinfo` attributes, the comparands are first adjusted by subtracting their UTC offsets (obtained from ``self.utcoffset()``). In order to stop mixed-type comparisons from falling back to the default comparison by object address, when a :class:`.time` object is compared to an object of a @@ -1376,10 +1389,13 @@ Supported operations: * efficient pickling -* in Boolean contexts, a :class:`.time` object is considered to be true if and - only if, after converting it to minutes and subtracting :meth:`utcoffset` (or - ``0`` if that's ``None``), the result is non-zero. +In boolean contexts, a :class:`.time` object is always considered to be true. +.. versionchanged:: 3.5 + Before Python 3.5, a :class:`.time` object was considered to be false if it + represented midnight in UTC. This behavior was considered obscure and + error-prone and has been removed in Python 3.5. See :issue:`13936` for full + details. Instance methods: @@ -1413,7 +1429,7 @@ Instance methods: .. method:: time.__format__(format) - Same as :meth:`.time.strftime`. This makes it possible to specify format string + Same as :meth:`.time.strftime`. This makes it possible to specify a format string for a :class:`.time` object when using :meth:`str.format`. For a complete list of formatting directives, see :ref:`strftime-strptime-behavior`. @@ -1421,7 +1437,7 @@ Instance methods: .. method:: time.utcoffset() - If :attr:`tzinfo` is ``None``, returns ``None``, else returns + If :attr:`.tzinfo` is ``None``, returns ``None``, else returns ``self.tzinfo.utcoffset(None)``, and raises an exception if the latter doesn't return ``None`` or a :class:`timedelta` object representing a whole number of minutes with magnitude less than one day. @@ -1429,7 +1445,7 @@ Instance methods: .. method:: time.dst() - If :attr:`tzinfo` is ``None``, returns ``None``, else returns + If :attr:`.tzinfo` is ``None``, returns ``None``, else returns ``self.tzinfo.dst(None)``, and raises an exception if the latter doesn't return ``None``, or a :class:`timedelta` object representing a whole number of minutes with magnitude less than one day. @@ -1437,7 +1453,7 @@ Instance methods: .. method:: time.tzname() - If :attr:`tzinfo` is ``None``, returns ``None``, else returns + If :attr:`.tzinfo` is ``None``, returns ``None``, else returns ``self.tzinfo.tzname(None)``, or raises an exception if the latter doesn't return ``None`` or a string object. @@ -1474,28 +1490,30 @@ Example: :class:`tzinfo` Objects ----------------------- -:class:`tzinfo` is an abstract base class, meaning that this class should not be -instantiated directly. You need to derive a concrete subclass, and (at least) -supply implementations of the standard :class:`tzinfo` methods needed by the -:class:`.datetime` methods you use. The :mod:`datetime` module supplies -a simple concrete subclass of :class:`tzinfo` :class:`timezone` which can represent -timezones with fixed offset from UTC such as UTC itself or North American EST and -EDT. +.. class:: tzinfo() + + This is an abstract base class, meaning that this class should not be + instantiated directly. You need to derive a concrete subclass, and (at least) + supply implementations of the standard :class:`tzinfo` methods needed by the + :class:`.datetime` methods you use. The :mod:`datetime` module supplies + a simple concrete subclass of :class:`tzinfo`, :class:`timezone`, which can represent + timezones with fixed offset from UTC such as UTC itself or North American EST and + EDT. -An instance of (a concrete subclass of) :class:`tzinfo` can be passed to the -constructors for :class:`.datetime` and :class:`.time` objects. The latter objects -view their attributes as being in local time, and the :class:`tzinfo` object -supports methods revealing offset of local time from UTC, the name of the time -zone, and DST offset, all relative to a date or time object passed to them. + An instance of (a concrete subclass of) :class:`tzinfo` can be passed to the + constructors for :class:`.datetime` and :class:`.time` objects. The latter objects + view their attributes as being in local time, and the :class:`tzinfo` object + supports methods revealing offset of local time from UTC, the name of the time + zone, and DST offset, all relative to a date or time object passed to them. -Special requirement for pickling: A :class:`tzinfo` subclass must have an -:meth:`__init__` method that can be called with no arguments, else it can be -pickled but possibly not unpickled again. This is a technical requirement that -may be relaxed in the future. + Special requirement for pickling: A :class:`tzinfo` subclass must have an + :meth:`__init__` method that can be called with no arguments, else it can be + pickled but possibly not unpickled again. This is a technical requirement that + may be relaxed in the future. -A concrete subclass of :class:`tzinfo` may need to implement the following -methods. Exactly which methods are needed depends on the uses made of aware -:mod:`datetime` objects. If in doubt, simply implement all of them. + A concrete subclass of :class:`tzinfo` may need to implement the following + methods. Exactly which methods are needed depends on the uses made of aware + :mod:`datetime` objects. If in doubt, simply implement all of them. .. method:: tzinfo.utcoffset(dt) @@ -1528,7 +1546,7 @@ methods. Exactly which methods are needed depends on the uses made of aware (see :meth:`utcoffset` for details). Note that DST offset, if applicable, has already been added to the UTC offset returned by :meth:`utcoffset`, so there's no need to consult :meth:`dst` unless you're interested in obtaining DST info - separately. For example, :meth:`datetime.timetuple` calls its :attr:`tzinfo` + separately. For example, :meth:`datetime.timetuple` calls its :attr:`~.datetime.tzinfo` attribute's :meth:`dst` method to determine how the :attr:`tm_isdst` flag should be set, and :meth:`tzinfo.fromutc` calls :meth:`dst` to account for DST changes when crossing time zones. @@ -1693,7 +1711,7 @@ only EST (fixed offset -5 hours), or only EDT (fixed offset -4 hours)). *pytz* library brings the *IANA timezone database* (also known as the Olson database) to Python and its usage is recommended. - `IANA timezone database <http://www.iana.org/time-zones>`_ + `IANA timezone database <https://www.iana.org/time-zones>`_ The Time Zone Database (often called tz or zoneinfo) contains code and data that represent the history of local time for many representative locations around the globe. It is updated periodically to reflect changes @@ -1854,7 +1872,7 @@ format codes. +-----------+--------------------------------+------------------------+-------+ | ``%z`` | UTC offset in the form +HHMM | (empty), +0000, -0400, | \(6) | | | or -HHMM (empty string if the | +1030 | | -| | the object is naive). | | | +| | object is naive). | | | +-----------+--------------------------------+------------------------+-------+ | ``%Z`` | Time zone name (empty string | (empty), UTC, EST, CST | | | | if the object is naive). | | | diff --git a/Doc/library/dbm.rst b/Doc/library/dbm.rst index e6a82d67ce..2a1db91427 100644 --- a/Doc/library/dbm.rst +++ b/Doc/library/dbm.rst @@ -4,10 +4,14 @@ .. module:: dbm :synopsis: Interfaces to various Unix "database" formats. +**Source code:** :source:`Lib/dbm/__init__.py` + +-------------- + :mod:`dbm` is a generic interface to variants of the DBM database --- :mod:`dbm.gnu` or :mod:`dbm.ndbm`. If none of these modules is installed, the slow-but-simple implementation in module :mod:`dbm.dumb` will be used. There -is a `third party interface <http://www.jcea.es/programacion/pybsddb.htm>`_ to +is a `third party interface <https://www.jcea.es/programacion/pybsddb.htm>`_ to the Oracle Berkeley DB. @@ -124,6 +128,9 @@ The individual submodules are described in the following sections. :platform: Unix :synopsis: GNU's reinterpretation of dbm. +**Source code:** :source:`Lib/dbm/gnu.py` + +-------------- This module is quite similar to the :mod:`dbm` module, but uses the GNU library ``gdbm`` instead to provide some additional functionality. Please note that the @@ -233,6 +240,9 @@ supported. :platform: Unix :synopsis: The standard "database" interface, based on ndbm. +**Source code:** :source:`Lib/dbm/ndbm.py` + +-------------- The :mod:`dbm.ndbm` module provides an interface to the Unix "(n)dbm" library. Dbm objects behave like mappings (dictionaries), except that keys and values are @@ -295,6 +305,8 @@ to locate the appropriate header file to simplify building this module. .. module:: dbm.dumb :synopsis: Portable implementation of the simple DBM interface. +**Source code:** :source:`Lib/dbm/dumb.py` + .. index:: single: databases .. note:: @@ -304,6 +316,8 @@ to locate the appropriate header file to simplify building this module. module is not written for speed and is not nearly as heavily used as the other database modules. +-------------- + The :mod:`dbm.dumb` module provides a persistent dictionary-like interface which is written entirely in Python. Unlike other modules such as :mod:`dbm.gnu` no external library is required. As with other persistent mappings, the keys and @@ -325,13 +339,18 @@ The module defines the following: dumbdbm database is created, files with :file:`.dat` and :file:`.dir` extensions are created. - The optional *flag* argument is currently ignored; the database is always opened - for update, and will be created if it does not exist. + The optional *flag* argument supports only the semantics of ``'c'`` + and ``'n'`` values. Other values will default to database being always + opened for update, and will be created if it does not exist. The optional *mode* argument is the Unix mode of the file, used only when the database has to be created. It defaults to octal ``0o666`` (and will be modified by the prevailing umask). + .. versionchanged:: 3.5 + :func:`.open` always creates a new database when the flag has the value + ``'n'``. + In addition to the methods provided by the :class:`collections.abc.MutableMapping` class, :class:`dumbdbm` objects provide the following methods: diff --git a/Doc/library/decimal.rst b/Doc/library/decimal.rst index ca41c3a824..971600c35b 100644 --- a/Doc/library/decimal.rst +++ b/Doc/library/decimal.rst @@ -12,6 +12,8 @@ .. moduleauthor:: Stefan Krah <skrah at bytereef.org> .. sectionauthor:: Raymond D. Hettinger <python at rcn.com> +**Source code:** :source:`Lib/decimal.py` + .. import modules for testing inline doctests with the Sphinx doctest builder .. testsetup:: * @@ -21,6 +23,8 @@ # make sure each group gets a fresh context setcontext(Context()) +-------------- + The :mod:`decimal` module provides support for fast correctly-rounded decimal floating point arithmetic. It offers several advantages over the :class:`float` datatype: @@ -107,9 +111,6 @@ reset them before monitoring a calculation. * IBM's General Decimal Arithmetic Specification, `The General Decimal Arithmetic Specification <http://speleotrove.com/decimal/decarith.html>`_. - * IEEE standard 854-1987, `Unofficial IEEE 854 Text - <http://754r.ucbtest.org/standards/854.pdf>`_. - .. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -162,7 +163,7 @@ an exception:: >>> c.traps[FloatOperation] = True >>> Decimal(3.14) Traceback (most recent call last): - File "<stdin>", line 1, in <module> + File "<stdin>", line 1, in <module> decimal.FloatOperation: [<class 'decimal.FloatOperation'>] >>> Decimal('3.5') < 3.7 Traceback (most recent call last): @@ -742,7 +743,7 @@ Decimal objects * ``"NaN"``, indicating that the operand is a quiet NaN (Not a Number). * ``"sNaN"``, indicating that the operand is a signaling NaN. - .. method:: quantize(exp, rounding=None, context=None, watchexp=True) + .. method:: quantize(exp, rounding=None, context=None) Return a value equal to the first operand after rounding and having the exponent of the second operand. @@ -765,14 +766,8 @@ Decimal objects ``context`` argument; if neither argument is given the rounding mode of the current thread's context is used. - If *watchexp* is set (default), then an error is returned whenever the - resulting exponent is greater than :attr:`Emax` or less than - :attr:`Etiny`. - - .. deprecated:: 3.3 - *watchexp* is an implementation detail from the pure Python version - and is not present in the C version. It will be removed in version - 3.4, where it defaults to ``True``. + An error is returned whenever the resulting exponent is greater than + :attr:`Emax` or less than :attr:`Etiny`. .. method:: radix() @@ -2092,4 +2087,3 @@ Alternatively, inputs can be rounded upon creation using the >>> Context(prec=5, rounding=ROUND_DOWN).create_decimal('1.2345678') Decimal('1.2345') - diff --git a/Doc/library/development.rst b/Doc/library/development.rst index 06e7048a04..d2b5fa2aa4 100644 --- a/Doc/library/development.rst +++ b/Doc/library/development.rst @@ -16,6 +16,7 @@ The list of modules described in this chapter is: .. toctree:: + typing.rst pydoc.rst doctest.rst unittest.rst diff --git a/Doc/library/difflib.rst b/Doc/library/difflib.rst index cead818adf..59a6478d8d 100644 --- a/Doc/library/difflib.rst +++ b/Doc/library/difflib.rst @@ -3,15 +3,20 @@ .. module:: difflib :synopsis: Helpers for computing differences between objects. + .. moduleauthor:: Tim Peters <tim_one@users.sourceforge.net> .. sectionauthor:: Tim Peters <tim_one@users.sourceforge.net> .. Markup by Fred L. Drake, Jr. <fdrake@acm.org> +**Source code:** :source:`Lib/difflib.py` + .. testsetup:: import sys from difflib import * +-------------- + This module provides classes and functions for comparing sequences. It can be used for example, for comparing files, and can produce difference information in various formats, including HTML and context and unified @@ -25,7 +30,9 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module. little fancier than, an algorithm published in the late 1980's by Ratcliff and Obershelp under the hyperbolic name "gestalt pattern matching." The idea is to find the longest contiguous matching subsequence that contains no "junk" - elements (the Ratcliff and Obershelp algorithm doesn't address junk). The same + elements; these "junk" elements are ones that are uninteresting in some + sense, such as blank lines or whitespace. (Handling junk is an + extension to the Ratcliff and Obershelp algorithm.) The same idea is then applied recursively to the pieces of the sequences to the left and to the right of the matching subsequence. This does not yield minimal edit sequences, but does tend to yield matches that "look right" to people. @@ -100,7 +107,8 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module. The following methods are public: - .. method:: make_file(fromlines, tolines, fromdesc='', todesc='', context=False, numlines=5) + .. method:: make_file(fromlines, tolines, fromdesc='', todesc='', context=False, \ + numlines=5, *, charset='utf-8') Compares *fromlines* and *tolines* (lists of strings) and returns a string which is a complete HTML file containing a table showing line by line differences with @@ -119,6 +127,10 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module. the next difference highlight at the top of the browser without any leading context). + .. versionchanged:: 3.5 + *charset* keyword-only argument was added. The default charset of + HTML document changed from ``'ISO-8859-1'`` to ``'utf-8'``. + .. method:: make_table(fromlines, tolines, fromdesc='', todesc='', context=False, numlines=5) Compares *fromlines* and *tolines* (lists of strings) and returns a string which @@ -208,7 +220,7 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module. Compare *a* and *b* (lists of strings); return a :class:`Differ`\ -style delta (a :term:`generator` generating the delta lines). - Optional keyword parameters *linejunk* and *charjunk* are for filter functions + Optional keyword parameters *linejunk* and *charjunk* are filtering functions (or ``None``): *linejunk*: A function that accepts a single string argument, and returns @@ -222,7 +234,7 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module. *charjunk*: A function that accepts a character (a string of length 1), and returns if the character is junk, or false if not. The default is module-level function :func:`IS_CHARACTER_JUNK`, which filters out whitespace characters (a - blank or tab; note: bad idea to include newline in this!). + blank or tab; it's a bad idea to include newline in this!). :file:`Tools/scripts/ndiff.py` is a command-line front-end to this function. @@ -306,6 +318,21 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module. See :ref:`difflib-interface` for a more detailed example. +.. function:: diff_bytes(dfunc, a, b, fromfile=b'', tofile=b'', fromfiledate=b'', tofiledate=b'', n=3, lineterm=b'\\n') + + Compare *a* and *b* (lists of bytes objects) using *dfunc*; yield a + sequence of delta lines (also bytes) in the format returned by *dfunc*. + *dfunc* must be a callable, typically either :func:`unified_diff` or + :func:`context_diff`. + + Allows you to compare data with unknown or inconsistent encoding. All + inputs except *n* must be bytes objects, not str. Works by losslessly + converting all inputs (except *n*) to str, and calling ``dfunc(a, b, + fromfile, tofile, fromfiledate, tofiledate, n, lineterm)``. The output of + *dfunc* is then converted back to bytes, so the delta lines that you + receive have the same unknown/inconsistent encodings as *a* and *b*. + + .. versionadded:: 3.5 .. function:: IS_LINE_JUNK(line) @@ -477,16 +504,14 @@ The :class:`SequenceMatcher` class has this constructor: | | are equal). | +---------------+---------------------------------------------+ - For example: + For example:: >>> a = "qabxcd" >>> b = "abycdf" >>> s = SequenceMatcher(None, a, b) >>> for tag, i1, i2, j1, j2 in s.get_opcodes(): - print('{:7} a[{}:{}] --> b[{}:{}] {!r:>8} --> {!r}'.format( - tag, i1, i2, j1, j2, a[i1:i2], b[j1:j2])) - - + ... print('{:7} a[{}:{}] --> b[{}:{}] {!r:>8} --> {!r}'.format( + ... tag, i1, i2, j1, j2, a[i1:i2], b[j1:j2])) delete a[0:1] --> b[0:0] 'q' --> '' equal a[1:3] --> b[0:2] 'ab' --> 'ab' replace a[3:4] --> b[2:3] 'x' --> 'y' @@ -591,7 +616,7 @@ If you want to know how to change the first sequence into the second, use work. * `Simple version control recipe - <http://code.activestate.com/recipes/576729/>`_ for a small application + <https://code.activestate.com/recipes/576729/>`_ for a small application built with :class:`SequenceMatcher`. @@ -622,6 +647,12 @@ The :class:`Differ` class has this constructor: length 1), and returns true if the character is junk. The default is ``None``, meaning that no character is considered junk. + These junk-filtering functions speed up matching to find + differences and do not cause any differing lines or characters to + be ignored. Read the description of the + :meth:`~SequenceMatcher.find_longest_match` method's *isjunk* + parameter for an explanation. + :class:`Differ` objects are used (deltas generated) via a single method: @@ -713,65 +744,4 @@ This example shows how to use difflib to create a ``diff``-like utility. It is also contained in the Python source distribution, as :file:`Tools/scripts/diff.py`. -.. testcode:: - - """ Command line interface to difflib.py providing diffs in four formats: - - * ndiff: lists every line and highlights interline changes. - * context: highlights clusters of changes in a before/after format. - * unified: highlights clusters of changes in an inline format. - * html: generates side by side comparison with change highlights. - - """ - - import sys, os, time, difflib, optparse - - def main(): - # Configure the option parser - usage = "usage: %prog [options] fromfile tofile" - parser = optparse.OptionParser(usage) - parser.add_option("-c", action="store_true", default=False, - help='Produce a context format diff (default)') - parser.add_option("-u", action="store_true", default=False, - help='Produce a unified format diff') - hlp = 'Produce HTML side by side diff (can use -c and -l in conjunction)' - parser.add_option("-m", action="store_true", default=False, help=hlp) - parser.add_option("-n", action="store_true", default=False, - help='Produce a ndiff format diff') - parser.add_option("-l", "--lines", type="int", default=3, - help='Set number of context lines (default 3)') - (options, args) = parser.parse_args() - - if len(args) == 0: - parser.print_help() - sys.exit(1) - if len(args) != 2: - parser.error("need to specify both a fromfile and tofile") - - n = options.lines - fromfile, tofile = args # as specified in the usage string - - # we're passing these as arguments to the diff function - fromdate = time.ctime(os.stat(fromfile).st_mtime) - todate = time.ctime(os.stat(tofile).st_mtime) - with open(fromfile) as fromf, open(tofile) as tof: - fromlines, tolines = list(fromf), list(tof) - - if options.u: - diff = difflib.unified_diff(fromlines, tolines, fromfile, tofile, - fromdate, todate, n=n) - elif options.n: - diff = difflib.ndiff(fromlines, tolines) - elif options.m: - diff = difflib.HtmlDiff().make_file(fromlines, tolines, fromfile, - tofile, context=options.c, - numlines=n) - else: - diff = difflib.context_diff(fromlines, tolines, fromfile, tofile, - fromdate, todate, n=n) - - # we're using writelines because diff is a generator - sys.stdout.writelines(diff) - - if __name__ == '__main__': - main() +.. literalinclude:: ../../Tools/scripts/diff.py diff --git a/Doc/library/dis.rst b/Doc/library/dis.rst index 273fb20634..d2d8ac7839 100644 --- a/Doc/library/dis.rst +++ b/Doc/library/dis.rst @@ -48,8 +48,9 @@ code. .. class:: Bytecode(x, *, first_line=None, current_offset=None) - Analyse the bytecode corresponding to a function, method, string of source - code, or a code object (as returned by :func:`compile`). + + Analyse the bytecode corresponding to a function, generator, method, string + of source code, or a code object (as returned by :func:`compile`). This is a convenience wrapper around many of the functions listed below, most notably :func:`get_instructions`, as iterating over a :class:`Bytecode` @@ -109,7 +110,7 @@ operation is being performed, so the intermediate analysis object isn't useful: .. function:: code_info(x) Return a formatted multi-line string with detailed code object information - for the supplied function, method, source code string or code object. + for the supplied function, generator, method, source code string or code object. Note that the exact contents of code info strings are highly implementation dependent and they may change arbitrarily across Python VMs or Python @@ -136,13 +137,13 @@ operation is being performed, so the intermediate analysis object isn't useful: .. function:: dis(x=None, *, file=None) Disassemble the *x* object. *x* can denote either a module, a class, a - method, a function, a code object, a string of source code or a byte sequence - of raw bytecode. For a module, it disassembles all functions. For a class, - it disassembles all methods. For a code object or sequence of raw bytecode, - it prints one line per bytecode instruction. Strings are first compiled to - code objects with the :func:`compile` built-in function before being - disassembled. If no object is provided, this function disassembles the last - traceback. + method, a function, a generator, a code object, a string of source code or + a byte sequence of raw bytecode. For a module, it disassembles all functions. + For a class, it disassembles all methods (including class and static methods). + For a code object or sequence of raw bytecode, it prints one line per bytecode + instruction. Strings are first compiled to code objects with the :func:`compile` + built-in function before being disassembled. If no object is provided, this + function disassembles the last traceback. The disassembly is written as text to the supplied *file* argument if provided and to ``sys.stdout`` otherwise. @@ -345,6 +346,14 @@ result back on the stack. Implements ``TOS = iter(TOS)``. +.. opcode:: GET_YIELD_FROM_ITER + + If ``TOS`` is a :term:`generator iterator` or :term:`coroutine` object + it is left as is. Otherwise, implements ``TOS = iter(TOS)``. + + .. versionadded:: 3.5 + + **Binary operations** Binary operations remove the top of the stack (TOS) and the second top-most @@ -361,6 +370,13 @@ result back on the stack. Implements ``TOS = TOS1 * TOS``. +.. opcode:: BINARY_MATRIX_MULTIPLY + + Implements ``TOS = TOS1 @ TOS``. + + .. versionadded:: 3.5 + + .. opcode:: BINARY_FLOOR_DIVIDE Implements ``TOS = TOS1 // TOS``. @@ -433,6 +449,13 @@ the original TOS1. Implements in-place ``TOS = TOS1 * TOS``. +.. opcode:: INPLACE_MATRIX_MULTIPLY + + Implements in-place ``TOS = TOS1 @ TOS``. + + .. versionadded:: 3.5 + + .. opcode:: INPLACE_FLOOR_DIVIDE Implements in-place ``TOS = TOS1 // TOS``. @@ -493,6 +516,40 @@ the original TOS1. Implements ``del TOS1[TOS]``. +**Coroutine opcodes** + +.. opcode:: GET_AWAITABLE + + Implements ``TOS = get_awaitable(TOS)``, where ``get_awaitable(o)`` + returns ``o`` if ``o`` is a coroutine object or a generator object with + the CO_ITERABLE_COROUTINE flag, or resolves + ``o.__await__``. + + +.. opcode:: GET_AITER + + Implements ``TOS = get_awaitable(TOS.__aiter__())``. See ``GET_AWAITABLE`` + for details about ``get_awaitable`` + + +.. opcode:: GET_ANEXT + + Implements ``PUSH(get_awaitable(TOS.__anext__()))``. See ``GET_AWAITABLE`` + for details about ``get_awaitable`` + + +.. opcode:: BEFORE_ASYNC_WITH + + Resolves ``__aenter__`` and ``__aexit__`` from the object on top of the + stack. Pushes ``__aexit__`` and result of ``__aenter__()`` to the stack. + + +.. opcode:: SETUP_ASYNC_WITH + + Creates a new frame object. + + + **Miscellaneous opcodes** .. opcode:: PRINT_EXPR @@ -597,7 +654,7 @@ iterations of the loop. :opcode:`UNPACK_SEQUENCE`). -.. opcode:: WITH_CLEANUP +.. opcode:: WITH_CLEANUP_START Cleans up the stack when a :keyword:`with` statement block exits. TOS is the context manager's :meth:`__exit__` bound method. Below TOS are 1--3 values @@ -609,7 +666,13 @@ iterations of the loop. * (SECOND, THIRD, FOURTH) = exc_info() In the last case, ``TOS(SECOND, THIRD, FOURTH)`` is called, otherwise - ``TOS(None, None, None)``. In addition, TOS is removed from the stack. + ``TOS(None, None, None)``. Pushes SECOND and result of the call + to the stack. + + +.. opcode:: WITH_CLEANUP_FINISH + + Pops exception type and result of 'exit' function call from the stack. If the stack represents an exception, *and* the function call returns a 'true' value, this information is "zapped" and replaced with a single @@ -645,7 +708,7 @@ the more significant byte last. Implements assignment with a starred target: Unpacks an iterable in TOS into individual values, where the total number of values can be smaller than the - number of items in the iterable: one the new values will be a list of all + number of items in the iterable: one of the new values will be a list of all leftover items. The low byte of *counts* is the number of values before the list value, the @@ -795,10 +858,6 @@ the more significant byte last. Pushes a try block from a try-except clause onto the block stack. *delta* points to the finally block. -.. opcode:: STORE_MAP - - Store a key and value pair in a dictionary. Pops the key and value while - leaving the dictionary on the stack. .. opcode:: LOAD_FAST (var_num) diff --git a/Doc/library/distribution.rst b/Doc/library/distribution.rst index c4954d1b4a..3e6e84b42a 100644 --- a/Doc/library/distribution.rst +++ b/Doc/library/distribution.rst @@ -12,3 +12,4 @@ with a local index server, or without any index server at all. distutils.rst ensurepip.rst venv.rst + zipapp.rst diff --git a/Doc/library/distutils.rst b/Doc/library/distutils.rst index e3d1314572..62abc85ac3 100644 --- a/Doc/library/distutils.rst +++ b/Doc/library/distutils.rst @@ -4,8 +4,10 @@ .. module:: distutils :synopsis: Support for building and installing Python modules into an existing Python installation. + .. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org> +-------------- The :mod:`distutils` package provides support for building and installing additional modules into a Python installation. The new modules may be either @@ -15,7 +17,7 @@ collections of Python packages which include modules coded in both Python and C. Most Python users will *not* want to use this module directly, but instead use the cross-version tools maintained by the Python Packaging Authority. In particular, -`setuptools <https://setuptools.pypa.io/en/latest/setuptools.html>`__ is an +`setuptools <https://setuptools.readthedocs.io/en/latest/>`__ is an enhanced alternative to :mod:`distutils` that provides: * support for declaring project dependencies diff --git a/Doc/library/doctest.rst b/Doc/library/doctest.rst index ea6e1b5481..66a521ebe7 100644 --- a/Doc/library/doctest.rst +++ b/Doc/library/doctest.rst @@ -5,11 +5,15 @@ .. module:: doctest :synopsis: Test pieces of code within docstrings. + .. moduleauthor:: Tim Peters <tim@python.org> .. sectionauthor:: Tim Peters <tim@python.org> .. sectionauthor:: Moshe Zadka <moshez@debian.org> .. sectionauthor:: Edward Loper <edloper@users.sourceforge.net> +**Source code:** :source:`Lib/doctest.py` + +-------------- The :mod:`doctest` module searches for pieces of text that look like interactive Python sessions, and then executes those sessions to verify that they work @@ -84,14 +88,18 @@ Here's a complete but small example module:: doctest.testmod() If you run :file:`example.py` directly from the command line, :mod:`doctest` -works its magic:: +works its magic: + +.. code-block:: shell-session $ python example.py $ There's no output! That's normal, and it means all the examples worked. Pass ``-v`` to the script, and :mod:`doctest` prints a detailed log of what -it's trying, and prints a summary at the end:: +it's trying, and prints a summary at the end: + +.. code-block:: shell-session $ python example.py -v Trying: @@ -105,7 +113,9 @@ it's trying, and prints a summary at the end:: [1, 1, 2, 6, 24, 120] ok -And so on, eventually ending with:: +And so on, eventually ending with: + +.. code-block:: none Trying: factorial(1e100) @@ -192,7 +202,9 @@ file. This can be done with the :func:`testfile` function:: That short script executes and verifies any interactive Python examples contained in the file :file:`example.txt`. The file content is treated as if it were a single giant docstring; the file doesn't need to contain a Python -program! For example, perhaps :file:`example.txt` contains this:: +program! For example, perhaps :file:`example.txt` contains this: + +.. code-block:: none The ``example`` module ====================== @@ -1053,15 +1065,9 @@ from text files and modules with doctests: This function uses the same search technique as :func:`testmod`. - .. note:: - Unlike :func:`testmod` and :class:`DocTestFinder`, this function raises - a :exc:`ValueError` if *module* contains no docstrings. You can prevent - this error by passing a :class:`DocTestFinder` instance as the - *test_finder* argument with its *exclude_empty* keyword argument set - to ``False``:: - - >>> finder = doctest.DocTestFinder(exclude_empty=False) - >>> suite = doctest.DocTestSuite(test_finder=finder) + .. versionchanged:: 3.5 + :func:`DocTestSuite` returns an empty :class:`unittest.TestSuite` if *module* + contains no docstrings instead of raising :exc:`ValueError`. Under the covers, :func:`DocTestSuite` creates a :class:`unittest.TestSuite` out diff --git a/Doc/library/email-examples.rst b/Doc/library/email-examples.rst index cbbcb78e24..ad93b5ccb8 100644 --- a/Doc/library/email-examples.rst +++ b/Doc/library/email-examples.rst @@ -59,9 +59,11 @@ way we could process it: .. literalinclude:: ../includes/email-read-alternative-new-api.py -Up to the prompt, the output from the above is:: +Up to the prompt, the output from the above is: - To: Penelope Pussycat <"penelope@example.com">, Fabrette Pussycat <"fabrette@example.com"> +.. code-block:: none + + To: Penelope Pussycat <penelope@example.com>, Fabrette Pussycat <fabrette@example.com> From: Pepé Le Pew <pepe@example.com> Subject: Ayons asperges pour le déjeuner diff --git a/Doc/library/email.charset.rst b/Doc/library/email.charset.rst index 80ef3d62cc..161d86a3b7 100644 --- a/Doc/library/email.charset.rst +++ b/Doc/library/email.charset.rst @@ -4,6 +4,9 @@ .. module:: email.charset :synopsis: Character Sets +**Source code:** :source:`Lib/email/charset.py` + +-------------- This module provides a class :class:`Charset` for representing character sets and character set conversions in email messages, as well as a character set @@ -101,9 +104,10 @@ Import this class from the :mod:`email.charset` module. returns the string ``base64`` if *body_encoding* is ``BASE64``, and returns the string ``7bit`` otherwise. + .. XXX to_splittable and from_splittable are not there anymore! - .. method to_splittable(s) + .. to_splittable(s) Convert a possibly multibyte string to a safely splittable format. *s* is the string to split. @@ -118,7 +122,7 @@ Import this class from the :mod:`email.charset` module. the Unicode replacement character ``'U+FFFD'``. - .. method from_splittable(ustr[, to_output]) + .. from_splittable(ustr[, to_output]) Convert a splittable string back into an encoded string. *ustr* is a Unicode string to "unsplit". diff --git a/Doc/library/email.contentmanager.rst b/Doc/library/email.contentmanager.rst index f53d34b34c..cd63728372 100644 --- a/Doc/library/email.contentmanager.rst +++ b/Doc/library/email.contentmanager.rst @@ -7,6 +7,10 @@ .. moduleauthor:: R. David Murray <rdmurray@bitdance.com> .. sectionauthor:: R. David Murray <rdmurray@bitdance.com> +.. versionadded:: 3.4 + as a :term:`provisional module <provisional package>`. + +**Source code:** :source:`Lib/email/contentmanager.py` .. note:: @@ -15,8 +19,7 @@ changes (up to and including removal of the module) may occur if deemed necessary by the core developers. -.. versionadded:: 3.4 - as a :term:`provisional module <provisional package>`. +-------------- The :mod:`~email.message` module provides a class that can represent an arbitrary email message. That basic message model has a useful and flexible diff --git a/Doc/library/email.encoders.rst b/Doc/library/email.encoders.rst index 916ba5d18a..9d7f9bf5e9 100644 --- a/Doc/library/email.encoders.rst +++ b/Doc/library/email.encoders.rst @@ -4,6 +4,9 @@ .. module:: email.encoders :synopsis: Encoders for email message payloads. +**Source code:** :source:`Lib/email/encoders.py` + +-------------- When creating :class:`~email.message.Message` objects from scratch, you often need to encode the payloads for transport through compliant mail servers. This diff --git a/Doc/library/email.errors.rst b/Doc/library/email.errors.rst index 294e3edbdc..8470783e81 100644 --- a/Doc/library/email.errors.rst +++ b/Doc/library/email.errors.rst @@ -4,6 +4,9 @@ .. module:: email.errors :synopsis: The exception classes used by the email package. +**Source code:** :source:`Lib/email/errors.py` + +-------------- The following exception classes are defined in the :mod:`email.errors` module: diff --git a/Doc/library/email.generator.rst b/Doc/library/email.generator.rst index 48d41e1dc7..d596ed8d85 100644 --- a/Doc/library/email.generator.rst +++ b/Doc/library/email.generator.rst @@ -4,6 +4,9 @@ .. module:: email.generator :synopsis: Generate flat text email messages from a message structure. +**Source code:** :source:`Lib/email/generator.py` + +-------------- One of the most common tasks is to generate the flat text of the email message represented by a message object structure. You will need to do this if you want @@ -43,7 +46,7 @@ Here are the public methods of the :class:`Generator` class, imported from the followed by a space at the beginning of the line. This is the only guaranteed portable way to avoid having such lines be mistaken for a Unix mailbox format envelope header separator (see `WHY THE CONTENT-LENGTH FORMAT IS BAD - <http://www.jwz.org/doc/content-length.html>`_ for details). *mangle_from_* + <https://www.jwz.org/doc/content-length.html>`_ for details). *mangle_from_* defaults to ``True``, but you might want to set this to ``False`` if you are not writing Unix mailbox format files. @@ -123,7 +126,7 @@ formatted string representation of a message object. For more detail, see i.e. ``From`` followed by a space at the beginning of the line. This is the only guaranteed portable way to avoid having such lines be mistaken for a Unix mailbox format envelope header separator (see `WHY THE CONTENT-LENGTH - FORMAT IS BAD <http://www.jwz.org/doc/content-length.html>`_ for details). + FORMAT IS BAD <https://www.jwz.org/doc/content-length.html>`_ for details). *mangle_from_* defaults to ``True``, but you might want to set this to ``False`` if you are not writing Unix mailbox format files. diff --git a/Doc/library/email.header.rst b/Doc/library/email.header.rst index 346d23fc8c..e94837c1c9 100644 --- a/Doc/library/email.header.rst +++ b/Doc/library/email.header.rst @@ -4,6 +4,9 @@ .. module:: email.header :synopsis: Representing non-ASCII headers +**Source code:** :source:`Lib/email/header.py` + +-------------- :rfc:`2822` is the base standard that describes the format of email messages. It derives from the older :rfc:`822` standard which came into widespread use at diff --git a/Doc/library/email.headerregistry.rst b/Doc/library/email.headerregistry.rst index db3aade416..0707bd858a 100644 --- a/Doc/library/email.headerregistry.rst +++ b/Doc/library/email.headerregistry.rst @@ -7,6 +7,10 @@ .. moduleauthor:: R. David Murray <rdmurray@bitdance.com> .. sectionauthor:: R. David Murray <rdmurray@bitdance.com> +.. versionadded:: 3.3 + as a :term:`provisional module <provisional package>`. + +**Source code:** :source:`Lib/email/headerregistry.py` .. note:: @@ -15,8 +19,7 @@ changes (up to and including removal of the module) may occur if deemed necessary by the core developers. -.. versionadded:: 3.3 - as a :term:`provisional module <provisional package>`. +-------------- Headers are represented by customized subclasses of :class:`str`. The particular class used to represent a given header is determined by the @@ -171,7 +174,7 @@ headers. :class:`~datetime.datetime` instance. This means, for example, that the following code is valid and does what one would expect:: - msg['Date'] = datetime(2011, 7, 15, 21) + msg['Date'] = datetime(2011, 7, 15, 21) Because this is a naive ``datetime`` it will be interpreted as a UTC timestamp, and the resulting value will have a timezone of ``-0000``. Much diff --git a/Doc/library/email.iterators.rst b/Doc/library/email.iterators.rst index f92f460249..f3e9e184a5 100644 --- a/Doc/library/email.iterators.rst +++ b/Doc/library/email.iterators.rst @@ -4,6 +4,9 @@ .. module:: email.iterators :synopsis: Iterate over a message object tree. +**Source code:** :source:`Lib/email/iterators.py` + +-------------- Iterating over a message object tree is fairly easy with the :meth:`Message.walk <email.message.Message.walk>` method. The diff --git a/Doc/library/email.message.rst b/Doc/library/email.message.rst index aeea94221d..91a694f3ec 100644 --- a/Doc/library/email.message.rst +++ b/Doc/library/email.message.rst @@ -4,6 +4,9 @@ .. module:: email.message :synopsis: The base class representing email messages. +**Source code:** :source:`Lib/email/message.py` + +-------------- The central class in the :mod:`email` package is the :class:`Message` class, imported from the :mod:`email.message` module. It is the base class for the @@ -578,6 +581,15 @@ Here are the methods of the :class:`Message` class: will be *failobj*. + .. method:: get_content_disposition() + + Return the lowercased value (without parameters) of the message's + :mailheader:`Content-Disposition` header if it has one, or ``None``. The + possible values for this method are *inline*, *attachment* or ``None`` + if the message follows :rfc:`2183`. + + .. versionadded:: 3.5 + .. method:: walk() The :meth:`walk` method is an all-purpose generator which can be used to diff --git a/Doc/library/email.mime.rst b/Doc/library/email.mime.rst index 1d70225fc8..8297deaf93 100644 --- a/Doc/library/email.mime.rst +++ b/Doc/library/email.mime.rst @@ -4,6 +4,9 @@ .. module:: email.mime :synopsis: Build MIME messages. +**Source code:** :source:`Lib/email/mime/` + +-------------- Ordinarily, you get a message object structure by passing a file or some text to a parser, which parses the text and returns the root message object. However @@ -195,7 +198,8 @@ Here are the classes: set of the text and is passed as an argument to the :class:`~email.mime.nonmultipart.MIMENonMultipart` constructor; it defaults to ``us-ascii`` if the string contains only ``ascii`` code points, and - ``utf-8`` otherwise. + ``utf-8`` otherwise. The *_charset* parameter accepts either a string or a + :class:`~email.charset.Charset` instance. Unless the *_charset* argument is explicitly set to ``None``, the MIMEText object created will have both a :mailheader:`Content-Type` header @@ -206,3 +210,6 @@ Here are the classes: ``Content-Transfer-Encoding`` header, after which a ``set_payload`` call will automatically encode the new payload (and add a new :mailheader:`Content-Transfer-Encoding` header). + + .. versionchanged:: 3.5 + *_charset* also accepts :class:`~email.charset.Charset` instances. diff --git a/Doc/library/email.parser.rst b/Doc/library/email.parser.rst index 177adc62e7..71b339a15e 100644 --- a/Doc/library/email.parser.rst +++ b/Doc/library/email.parser.rst @@ -4,6 +4,9 @@ .. module:: email.parser :synopsis: Parse flat text email messages to produce a message object structure. +**Source code:** :source:`Lib/email/parser.py` + +-------------- Message object structures can be created in one of two ways: they can be created from whole cloth by instantiating :class:`~email.message.Message` objects and @@ -247,7 +250,7 @@ in the top-level :mod:`email` package namespace. and *policy* are interpreted as with the :class:`~email.parser.Parser` class constructor. - .. versionchanged:: + .. versionchanged:: 3.3 Removed the *strict* argument. Added the *policy* keyword. .. function:: message_from_binary_file(fp, _class=email.message.Message, *, \ diff --git a/Doc/library/email.policy.rst b/Doc/library/email.policy.rst index d4e3fc186a..47f321227a 100644 --- a/Doc/library/email.policy.rst +++ b/Doc/library/email.policy.rst @@ -9,6 +9,9 @@ .. versionadded:: 3.3 +**Source code:** :source:`Lib/email/policy.py` + +-------------- The :mod:`email` package's prime focus is the handling of email messages as described by the various email and MIME RFCs. However, the general format of @@ -187,6 +190,18 @@ added matters. To illustrate:: :const:`False` (the default), defects will be passed to the :meth:`register_defect` method. + + + .. attribute:: mangle_from\_ + + If :const:`True`, lines starting with *"From "* in the body are + escaped by putting a ``>`` in front of them. This parameter is used when + the message is being serialized by a generator. + Default: :const:`False`. + + .. versionadded:: 3.5 + The *mangle_from_* parameter. + The following :class:`Policy` method is intended to be called by code using the email library to create policy instances with custom settings: @@ -319,6 +334,13 @@ added matters. To illustrate:: :const:`compat32`, that is used as the default policy. Thus the default behavior of the email package is to maintain compatibility with Python 3.2. + The following attributes have values that are different from the + :class:`Policy` default: + + .. attribute:: mangle_from_ + + The default is ``True``. + The class provides the following concrete implementations of the abstract methods of :class:`Policy`: @@ -356,6 +378,14 @@ added matters. To illustrate:: line breaks and any (RFC invalid) binary data it may contain. +An instance of :class:`Compat32` is provided as a module constant: + +.. data:: compat32 + + An instance of :class:`Compat32`, providing backward compatibility with the + behavior of the email package in Python 3.2. + + .. note:: The documentation below describes new policies that are included in the @@ -378,6 +408,14 @@ added matters. To illustrate:: In addition to the settable attributes listed above that apply to all policies, this policy adds the following additional attributes: + .. attribute:: utf8 + + If ``False``, follow :rfc:`5322`, supporting non-ASCII characters in + headers by encoding them as "encoded words". If ``True``, follow + :rfc:`6532` and use ``utf-8`` encoding for headers. Messages + formatted in this way may be passed to SMTP servers that support + the ``SMTPUTF8`` extension (:rfc:`6531`). + .. attribute:: refold_source If the value for a header in the ``Message`` object originated from a @@ -499,6 +537,14 @@ more closely to the RFCs relevant to their domains. Like ``default``, but with ``linesep`` set to ``\r\n``, which is RFC compliant. +.. data:: SMTPUTF8 + + The same as ``SMTP`` except that :attr:`~EmailPolicy.utf8` is ``True``. + Useful for serializing messages to a message store without using encoded + words in the headers. Should only be used for SMTP trasmission if the + sender or recipient addresses have non-ASCII characters (the + :meth:`smtplib.SMTP.send_message` method handles this automatically). + .. data:: HTTP Suitable for serializing headers with for use in HTTP traffic. Like diff --git a/Doc/library/email.rst b/Doc/library/email.rst index 95c0a2f4a8..e8bb02bff5 100644 --- a/Doc/library/email.rst +++ b/Doc/library/email.rst @@ -4,10 +4,14 @@ .. module:: email :synopsis: Package supporting the parsing, manipulating, and generating email messages, including MIME documents. + .. moduleauthor:: Barry A. Warsaw <barry@python.org> .. sectionauthor:: Barry A. Warsaw <barry@python.org> .. Copyright (C) 2001-2010 Python Software Foundation +**Source code:** :source:`Lib/email/__init__.py` + +-------------- The :mod:`email` package is a library for managing email messages, including MIME and other :rfc:`2822`\ -based message documents. It is specifically *not* diff --git a/Doc/library/email.util.rst b/Doc/library/email.util.rst index 219e2847ea..5cff7465df 100644 --- a/Doc/library/email.util.rst +++ b/Doc/library/email.util.rst @@ -4,6 +4,9 @@ .. module:: email.utils :synopsis: Miscellaneous email package utilities. +**Source code:** :source:`Lib/email/utils.py` + +-------------- There are several useful utilities provided in the :mod:`email.utils` module: diff --git a/Doc/library/ensurepip.rst b/Doc/library/ensurepip.rst index d589f1cf12..6aeeabc306 100644 --- a/Doc/library/ensurepip.rst +++ b/Doc/library/ensurepip.rst @@ -7,6 +7,8 @@ .. versionadded:: 3.4 +-------------- + The :mod:`ensurepip` package provides support for bootstrapping the ``pip`` installer into an existing Python installation or virtual environment. This bootstrapping approach reflects the fact that ``pip`` is an independent diff --git a/Doc/library/enum.rst b/Doc/library/enum.rst index 1c76e8728d..a3d5afcb9e 100644 --- a/Doc/library/enum.rst +++ b/Doc/library/enum.rst @@ -314,11 +314,11 @@ Then:: >>> str(Mood.funky) 'my custom str! 1' -The rules for what is allowed are as follows: _sunder_ names (starting and -ending with a single underscore) are reserved by enum and cannot be used; -all other attributes defined within an enumeration will become members of this -enumeration, with the exception of *__dunder__* names and descriptors (methods -are also descriptors). +The rules for what is allowed are as follows: names that start and end with +a single underscore are reserved by enum and cannot be used; all other +attributes defined within an enumeration will become members of this +enumeration, with the exception of special methods (:meth:`__str__`, +:meth:`__add__`, etc.) and descriptors (methods are also descriptors). Note: if your enumeration defines :meth:`__new__` and/or :meth:`__init__` then whatever value(s) were given to the enum member will be passed into those @@ -400,7 +400,8 @@ The second argument is the *source* of enumeration member names. It can be a whitespace-separated string of names, a sequence of names, a sequence of 2-tuples with key/value pairs, or a mapping (e.g. dictionary) of names to values. The last two options enable assigning arbitrary values to -enumerations; the others auto-assign increasing integers starting with 1. A +enumerations; the others auto-assign increasing integers starting with 1 (use +the ``start`` parameter to specify a different starting value). A new class derived from :class:`Enum` is returned. In other words, the above assignment to :class:`Animal` is equivalent to:: @@ -430,7 +431,7 @@ The solution is to specify the module name explicitly as follows:: the source, pickling will be disabled. The new pickle protocol 4 also, in some circumstances, relies on -:attr:`__qualname__` being set to the location where pickle will be able +:attr:`~definition.__qualname__` being set to the location where pickle will be able to find the class. For example, if the class was made available in class SomeData in the global scope:: @@ -438,12 +439,12 @@ SomeData in the global scope:: The complete signature is:: - Enum(value='NewEnumName', names=<...>, *, module='...', qualname='...', type=<mixed-in class>) + Enum(value='NewEnumName', names=<...>, *, module='...', qualname='...', type=<mixed-in class>, start=1) :value: What the new Enum class will record as its name. :names: The Enum members. This can be a whitespace or comma separated string - (values will start at 1):: + (values will start at 1 unless otherwise specified):: 'red green blue' | 'red,green,blue' | 'red, green, blue' @@ -465,6 +466,11 @@ The complete signature is:: :type: type to mix in to new Enum class. +:start: number to start counting at if only names are passed in. + +.. versionchanged:: 3.5 + The *start* parameter was added. + Derived Enumerations -------------------- @@ -549,12 +555,12 @@ Some rules: 3. When another data type is mixed in, the :attr:`value` attribute is *not the same* as the enum member itself, although it is equivalent and will compare equal. -4. %-style formatting: `%s` and `%r` call :class:`Enum`'s :meth:`__str__` and - :meth:`__repr__` respectively; other codes (such as `%i` or `%h` for - IntEnum) treat the enum member as its mixed-in type. -5. :meth:`str.__format__` (or :func:`format`) will use the mixed-in - type's :meth:`__format__`. If the :class:`Enum`'s :func:`str` or - :func:`repr` is desired use the `!s` or `!r` :class:`str` format codes. +4. %-style formatting: `%s` and `%r` call the :class:`Enum` class's + :meth:`__str__` and :meth:`__repr__` respectively; other codes (such as + `%i` or `%h` for IntEnum) treat the enum member as its mixed-in type. +5. :meth:`str.format` (or :func:`format`) will use the mixed-in + type's :meth:`__format__`. If the :class:`Enum` class's :func:`str` or + :func:`repr` is desired, use the `!s` or `!r` format codes. Interesting examples @@ -724,18 +730,24 @@ member instances. Finer Points ^^^^^^^^^^^^ -Enum members are instances of an Enum class, and even though they are -accessible as `EnumClass.member`, they are not accessible directly from -the member:: +:class:`Enum` members are instances of an :class:`Enum` class, and even +though they are accessible as `EnumClass.member`, they should not be accessed +directly from the member as that lookup may fail or, worse, return something +besides the :class:`Enum` member you looking for:: - >>> Color.red - <Color.red: 1> - >>> Color.red.blue - Traceback (most recent call last): + >>> class FieldTypes(Enum): + ... name = 0 + ... value = 1 + ... size = 2 ... - AttributeError: 'Color' object has no attribute 'blue' + >>> FieldTypes.value.size + <FieldTypes.size: 2> + >>> FieldTypes.size.value + 2 + +.. versionchanged:: 3.5 -Likewise, the :attr:`__members__` is only available on the class. +The :attr:`__members__` attribute is only available on the class. If you give your :class:`Enum` subclass extra methods, like the `Planet`_ class above, those methods will show up in a :func:`dir` of the member, diff --git a/Doc/library/errno.rst b/Doc/library/errno.rst index d2163b6258..1cbd51c582 100644 --- a/Doc/library/errno.rst +++ b/Doc/library/errno.rst @@ -4,6 +4,7 @@ .. module:: errno :synopsis: Standard errno system symbols. +---------------- This module makes available standard ``errno`` system symbols. The value of each symbol is the corresponding integer value. The names and descriptions are @@ -41,7 +42,10 @@ defined by the module. The specific list of defined symbols is available as .. data:: EINTR - Interrupted system call + Interrupted system call. + + .. seealso:: + This error is mapped to the exception :exc:`InterruptedError`. .. data:: EIO diff --git a/Doc/library/exceptions.rst b/Doc/library/exceptions.rst index ef39e6c6f0..5a7193393c 100644 --- a/Doc/library/exceptions.rst +++ b/Doc/library/exceptions.rst @@ -162,7 +162,8 @@ The following exceptions are the exceptions that are usually raised. .. exception:: GeneratorExit - Raised when a :term:`generator`\'s :meth:`close` method is called. It + Raised when a :term:`generator` or :term:`coroutine` is closed; + see :meth:`generator.close` and :meth:`coroutine.close`. It directly inherits from :exc:`BaseException` instead of :exc:`Exception` since it is technically not an error. @@ -287,7 +288,7 @@ The following exceptions are the exceptions that are usually raised. .. versionchanged:: 3.3 :exc:`EnvironmentError`, :exc:`IOError`, :exc:`WindowsError`, - :exc:`VMSError`, :exc:`socket.error`, :exc:`select.error` and + :exc:`socket.error`, :exc:`select.error` and :exc:`mmap.error` have been merged into :exc:`OSError`, and the constructor may return a subclass. @@ -308,6 +309,16 @@ The following exceptions are the exceptions that are usually raised. handling in C, most floating point operations are not checked. +.. exception:: RecursionError + + This exception is derived from :exc:`RuntimeError`. It is raised when the + interpreter detects that the maximum recursion depth (see + :func:`sys.getrecursionlimit`) is exceeded. + + .. versionadded:: 3.5 + Previously, a plain :exc:`RuntimeError` was raised. + + .. exception:: ReferenceError This exception is raised when a weak reference proxy, created by the @@ -333,14 +344,30 @@ The following exceptions are the exceptions that are usually raised. given as an argument when constructing the exception, and defaults to :const:`None`. - When a generator function returns, a new :exc:`StopIteration` instance is + When a :term:`generator` or :term:`coroutine` function + returns, a new :exc:`StopIteration` instance is raised, and the value returned by the function is used as the :attr:`value` parameter to the constructor of the exception. + If a generator function defined in the presence of a ``from __future__ + import generator_stop`` directive raises :exc:`StopIteration`, it will be + converted into a :exc:`RuntimeError` (retaining the :exc:`StopIteration` + as the new exception's cause). + .. versionchanged:: 3.3 Added ``value`` attribute and the ability for generator functions to use it to return a value. + .. versionchanged:: 3.5 + Introduced the RuntimeError transformation. + +.. exception:: StopAsyncIteration + + Must be raised by :meth:`__anext__` method of an + :term:`asynchronous iterator` object to stop the iteration. + + .. versionadded:: 3.5 + .. exception:: SyntaxError Raised when the parser encounters a syntax error. This may occur in an @@ -563,7 +590,12 @@ depending on the system error code. .. exception:: InterruptedError Raised when a system call is interrupted by an incoming signal. - Corresponds to :c:data:`errno` ``EINTR``. + Corresponds to :c:data:`errno` :py:data:`~errno.EINTR`. + + .. versionchanged:: 3.5 + Python now retries system calls when a syscall is interrupted by a + signal, except if the signal handler raises an exception (see :pep:`475` + for the rationale), instead of raising :exc:`InterruptedError`. .. exception:: IsADirectoryError diff --git a/Doc/library/faulthandler.rst b/Doc/library/faulthandler.rst index eb2016a7b9..deedea1f26 100644 --- a/Doc/library/faulthandler.rst +++ b/Doc/library/faulthandler.rst @@ -6,6 +6,8 @@ .. versionadded:: 3.3 +---------------- + This module contains functions to dump Python tracebacks explicitly, on a fault, after a timeout, or on a user signal. Call :func:`faulthandler.enable` to install fault handlers for the :const:`SIGSEGV`, :const:`SIGFPE`, @@ -47,6 +49,9 @@ Dumping the traceback Dump the tracebacks of all threads into *file*. If *all_threads* is ``False``, dump only the current thread. + .. versionchanged:: 3.5 + Added support for passing file descriptor to this function. + Fault handler state ------------------- @@ -59,6 +64,12 @@ Fault handler state produce tracebacks for every running thread. Otherwise, dump only the current thread. + The *file* must be kept open until the fault handler is disabled: see + :ref:`issue with file descriptors <faulthandler-fd>`. + + .. versionchanged:: 3.5 + Added support for passing file descriptor to this function. + .. function:: disable() Disable the fault handler: uninstall the signal handlers installed by @@ -82,9 +93,16 @@ Dumping the tracebacks after a timeout call replaces previous parameters and resets the timeout. The timer has a sub-second resolution. + The *file* must be kept open until the traceback is dumped or + :func:`cancel_dump_traceback_later` is called: see :ref:`issue with file + descriptors <faulthandler-fd>`. + This function is implemented using a watchdog thread and therefore is not available if Python is compiled with threads disabled. + .. versionchanged:: 3.5 + Added support for passing file descriptor to this function. + .. function:: cancel_dump_traceback_later() Cancel the last call to :func:`dump_traceback_later`. @@ -99,8 +117,14 @@ Dumping the traceback on a user signal the traceback of all threads, or of the current thread if *all_threads* is ``False``, into *file*. Call the previous handler if chain is ``True``. + The *file* must be kept open until the signal is unregistered by + :func:`unregister`: see :ref:`issue with file descriptors <faulthandler-fd>`. + Not available on Windows. + .. versionchanged:: 3.5 + Added support for passing file descriptor to this function. + .. function:: unregister(signum) Unregister a user signal: uninstall the handler of the *signum* signal @@ -110,6 +134,8 @@ Dumping the traceback on a user signal Not available on Windows. +.. _faulthandler-fd: + Issue with file descriptors --------------------------- diff --git a/Doc/library/fcntl.rst b/Doc/library/fcntl.rst index b798f22d7f..88112f6b7e 100644 --- a/Doc/library/fcntl.rst +++ b/Doc/library/fcntl.rst @@ -4,15 +4,19 @@ .. module:: fcntl :platform: Unix :synopsis: The fcntl() and ioctl() system calls. -.. sectionauthor:: Jaap Vermeulen +.. sectionauthor:: Jaap Vermeulen .. index:: pair: UNIX; file control pair: UNIX; I/O control +---------------- + This module performs file control and I/O control on file descriptors. It is an -interface to the :c:func:`fcntl` and :c:func:`ioctl` Unix routines. +interface to the :c:func:`fcntl` and :c:func:`ioctl` Unix routines. For a +complete description of these calls, see :manpage:`fcntl(2)` and +:manpage:`ioctl(2)` Unix manual pages. All functions in this module take a file descriptor *fd* as their first argument. This can be an integer file descriptor, such as returned by @@ -28,41 +32,41 @@ descriptor. The module defines the following functions: -.. function:: fcntl(fd, op[, arg]) +.. function:: fcntl(fd, cmd, arg=0) - Perform the operation *op* on file descriptor *fd* (file objects providing + Perform the operation *cmd* on file descriptor *fd* (file objects providing a :meth:`~io.IOBase.fileno` method are accepted as well). The values used - for *op* are operating system dependent, and are available as constants + for *cmd* are operating system dependent, and are available as constants in the :mod:`fcntl` module, using the same names as used in the relevant C - header files. The argument *arg* is optional, and defaults to the integer - value ``0``. When present, it can either be an integer value, or a string. - With the argument missing or an integer value, the return value of this function - is the integer return value of the C :c:func:`fcntl` call. When the argument is - a string it represents a binary structure, e.g. created by :func:`struct.pack`. - The binary data is copied to a buffer whose address is passed to the C - :c:func:`fcntl` call. The return value after a successful call is the contents - of the buffer, converted to a string object. The length of the returned string - will be the same as the length of the *arg* argument. This is limited to 1024 - bytes. If the information returned in the buffer by the operating system is - larger than 1024 bytes, this is most likely to result in a segmentation - violation or a more subtle data corruption. + header files. The argument *arg* can either be an integer value, or a + :class:`bytes` object. With an integer value, the return value of this + function is the integer return value of the C :c:func:`fcntl` call. When + the argument is bytes it represents a binary structure, e.g. created by + :func:`struct.pack`. The binary data is copied to a buffer whose address is + passed to the C :c:func:`fcntl` call. The return value after a successful + call is the contents of the buffer, converted to a :class:`bytes` object. + The length of the returned object will be the same as the length of the + *arg* argument. This is limited to 1024 bytes. If the information returned + in the buffer by the operating system is larger than 1024 bytes, this is + most likely to result in a segmentation violation or a more subtle data + corruption. If the :c:func:`fcntl` fails, an :exc:`OSError` is raised. -.. function:: ioctl(fd, op[, arg[, mutate_flag]]) +.. function:: ioctl(fd, request, arg=0, mutate_flag=True) This function is identical to the :func:`~fcntl.fcntl` function, except that the argument handling is even more complicated. - The op parameter is limited to values that can fit in 32-bits. - Additional constants of interest for use as the *op* argument can be + The *request* parameter is limited to values that can fit in 32-bits. + Additional constants of interest for use as the *request* argument can be found in the :mod:`termios` module, under the same names as used in the relevant C header files. - The parameter *arg* can be one of an integer, absent (treated identically to the - integer ``0``), an object supporting the read-only buffer interface (most likely - a plain Python string) or an object supporting the read-write buffer interface. + The parameter *arg* can be one of an integer, an object supporting the + read-only buffer interface (like :class:`bytes`) or an object supporting + the read-write buffer interface (like :class:`bytearray`). In all but the last case, behaviour is as for the :func:`~fcntl.fcntl` function. @@ -72,7 +76,7 @@ The module defines the following functions: If it is false, the buffer's mutability is ignored and behaviour is as for a read-only buffer, except that the 1024 byte limit mentioned above is avoided -- - so long as the buffer you pass is as least as long as what the operating system + so long as the buffer you pass is at least as long as what the operating system wants to put there, things should work. If *mutate_flag* is true (the default), then the buffer is (in effect) passed @@ -83,7 +87,7 @@ The module defines the following functions: buffer 1024 bytes long which is then passed to :func:`ioctl` and copied back into the supplied buffer. - If the :c:func:`ioctl` fails, an :exc:`IOError` exception is raised. + If the :c:func:`ioctl` fails, an :exc:`OSError` exception is raised. An example:: @@ -99,27 +103,27 @@ The module defines the following functions: array('h', [13341]) -.. function:: flock(fd, op) +.. function:: flock(fd, operation) - Perform the lock operation *op* on file descriptor *fd* (file objects providing + Perform the lock operation *operation* on file descriptor *fd* (file objects providing a :meth:`~io.IOBase.fileno` method are accepted as well). See the Unix manual :manpage:`flock(2)` for details. (On some systems, this function is emulated using :c:func:`fcntl`.) - If the :c:func:`flock` fails, an :exc:`IOError` exception is raised. + If the :c:func:`flock` fails, an :exc:`OSError` exception is raised. -.. function:: lockf(fd, operation, [length, [start, [whence]]]) +.. function:: lockf(fd, cmd, len=0, start=0, whence=0) This is essentially a wrapper around the :func:`~fcntl.fcntl` locking calls. - *fd* is the file descriptor of the file to lock or unlock, and *operation* + *fd* is the file descriptor of the file to lock or unlock, and *cmd* is one of the following values: * :const:`LOCK_UN` -- unlock * :const:`LOCK_SH` -- acquire a shared lock * :const:`LOCK_EX` -- acquire an exclusive lock - When *operation* is :const:`LOCK_SH` or :const:`LOCK_EX`, it can also be + When *cmd* is :const:`LOCK_SH` or :const:`LOCK_EX`, it can also be bitwise ORed with :const:`LOCK_NB` to avoid blocking on lock acquisition. If :const:`LOCK_NB` is used and the lock cannot be acquired, an :exc:`OSError` will be raised and the exception will have an *errno* @@ -128,7 +132,7 @@ The module defines the following functions: systems, :const:`LOCK_EX` can only be used if the file descriptor refers to a file opened for writing. - *length* is the number of bytes to lock, *start* is the byte offset at + *len* is the number of bytes to lock, *start* is the byte offset at which the lock starts, relative to *whence*, and *whence* is as with :func:`io.IOBase.seek`, specifically: @@ -137,7 +141,7 @@ The module defines the following functions: * :const:`2` -- relative to the end of the file (:data:`os.SEEK_END`) The default for *start* is 0, which means to start at the beginning of the file. - The default for *length* is 0 which means to lock to the end of the file. The + The default for *len* is 0 which means to lock to the end of the file. The default for *whence* is also 0. Examples (all on a SVR4 compliant system):: @@ -151,9 +155,9 @@ Examples (all on a SVR4 compliant system):: rv = fcntl.fcntl(f, fcntl.F_SETLKW, lockdata) Note that in the first example the return value variable *rv* will hold an -integer value; in the second example it will hold a string value. The structure -lay-out for the *lockdata* variable is system dependent --- therefore using the -:func:`flock` call may be better. +integer value; in the second example it will hold a :class:`bytes` object. The +structure lay-out for the *lockdata* variable is system dependent --- therefore +using the :func:`flock` call may be better. .. seealso:: diff --git a/Doc/library/filecmp.rst b/Doc/library/filecmp.rst index 06d3f21300..31b9b4afab 100644 --- a/Doc/library/filecmp.rst +++ b/Doc/library/filecmp.rst @@ -3,6 +3,7 @@ .. module:: filecmp :synopsis: Compare files efficiently. + .. sectionauthor:: Moshe Zadka <moshez@zadka.site.co.il> **Source code:** :source:`Lib/filecmp.py` diff --git a/Doc/library/fileinput.rst b/Doc/library/fileinput.rst index ee06830ad8..aa4c529b2a 100644 --- a/Doc/library/fileinput.rst +++ b/Doc/library/fileinput.rst @@ -3,6 +3,7 @@ .. module:: fileinput :synopsis: Loop over standard input or a list of files. + .. moduleauthor:: Guido van Rossum <guido@python.org> .. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org> @@ -71,6 +72,9 @@ The following function is the primary interface of this module: .. versionchanged:: 3.2 Can be used as a context manager. + .. versionchanged:: 3.5.2 + The *bufsize* parameter is no longer used. + The following functions use the global state created by :func:`fileinput.input`; if there is no active state, :exc:`RuntimeError` is raised. @@ -161,7 +165,10 @@ available for subclassing as well: Can be used as a context manager. .. deprecated:: 3.4 - The ``'rU'`` and ``'U'`` modes. + The ``'rU'`` and ``'U'`` modes. + + .. versionchanged:: 3.5.2 + The *bufsize* parameter is no longer used. **Optional in-place filtering:** if the keyword argument ``inplace=True`` is @@ -190,7 +197,7 @@ The two following opening hooks are provided by this module: .. function:: hook_encoded(encoding) - Returns a hook which opens each file with :func:`codecs.open`, using the given + Returns a hook which opens each file with :func:`open`, using the given *encoding* to read the file. Usage example: ``fi = diff --git a/Doc/library/fnmatch.rst b/Doc/library/fnmatch.rst index 68b437f2c0..85ac484965 100644 --- a/Doc/library/fnmatch.rst +++ b/Doc/library/fnmatch.rst @@ -4,13 +4,12 @@ .. module:: fnmatch :synopsis: Unix shell style filename pattern matching. +**Source code:** :source:`Lib/fnmatch.py` .. index:: single: filenames; wildcard expansion .. index:: module: re -**Source code:** :source:`Lib/fnmatch.py` - -------------- This module provides support for Unix shell-style wildcards, which are *not* the diff --git a/Doc/library/formatter.rst b/Doc/library/formatter.rst index 1847a8094b..6c10ac6fab 100644 --- a/Doc/library/formatter.rst +++ b/Doc/library/formatter.rst @@ -6,9 +6,9 @@ :deprecated: .. deprecated:: 3.4 - Due to lack of usage, the formatter module has been deprecated and is slated - for removal in Python 3.6. + Due to lack of usage, the formatter module has been deprecated. +-------------- This module supports two interface definitions, each with multiple implementations: The *formatter* interface, and the *writer* interface which is diff --git a/Doc/library/fpectl.rst b/Doc/library/fpectl.rst index fb15f69445..e4b528cf0b 100644 --- a/Doc/library/fpectl.rst +++ b/Doc/library/fpectl.rst @@ -4,10 +4,10 @@ .. module:: fpectl :platform: Unix :synopsis: Provide control for floating point exception handling. + .. moduleauthor:: Lee Busby <busby1@llnl.gov> .. sectionauthor:: Lee Busby <busby1@llnl.gov> - .. note:: The :mod:`fpectl` module is not built by default, and its usage is discouraged @@ -16,6 +16,8 @@ .. index:: single: IEEE-754 +-------------- + Most computers carry out floating point operations in conformance with the so-called IEEE-754 standard. On any real computer, some floating point operations produce results that cannot be expressed as a normal floating point diff --git a/Doc/library/fractions.rst b/Doc/library/fractions.rst index 3d2529d1e5..b5a818e1ca 100644 --- a/Doc/library/fractions.rst +++ b/Doc/library/fractions.rst @@ -3,6 +3,7 @@ .. module:: fractions :synopsis: Rational numbers. + .. moduleauthor:: Jeffrey Yasskin <jyasskin at gmail.com> .. sectionauthor:: Jeffrey Yasskin <jyasskin at gmail.com> @@ -172,6 +173,9 @@ another rational number, or from a string. sign as *b* if *b* is nonzero; otherwise it takes the sign of *a*. ``gcd(0, 0)`` returns ``0``. + .. deprecated:: 3.5 + Use :func:`math.gcd` instead. + .. seealso:: diff --git a/Doc/library/ftplib.rst b/Doc/library/ftplib.rst index f06e678ecd..233e11fb5a 100644 --- a/Doc/library/ftplib.rst +++ b/Doc/library/ftplib.rst @@ -4,19 +4,18 @@ .. module:: ftplib :synopsis: FTP protocol client (requires sockets). +**Source code:** :source:`Lib/ftplib.py` .. index:: pair: FTP; protocol single: FTP; ftplib (standard module) -**Source code:** :source:`Lib/ftplib.py` - -------------- This module defines the class :class:`FTP` and a few related items. The :class:`FTP` class implements the client side of the FTP protocol. You can use this to write Python programs that perform a variety of automated FTP jobs, such -as mirroring other ftp servers. It is also used by the module +as mirroring other FTP servers. It is also used by the module :mod:`urllib.request` to handle URLs that use FTP. For more information on FTP (File Transfer Protocol), see Internet :rfc:`959`. @@ -148,12 +147,6 @@ The module defines the following items: typically used by FTP clients to load user authentication information before prompting the user. - .. index:: single: ftpmirror.py - - The file :file:`Tools/scripts/ftpmirror.py` in the Python source distribution is - a script that can mirror FTP sites, or portions thereof, using the :mod:`ftplib` - module. It can be used as an extended example that applies this module. - .. _ftp-objects: @@ -314,7 +307,7 @@ followed by ``lines`` for the text version or ``binary`` for the binary version. .. method:: FTP.mlsd(path="", facts=[]) - List a directory in a standardized format by using MLSD command + List a directory in a standardized format by using ``MLSD`` command (:rfc:`3659`). If *path* is omitted the current directory is assumed. *facts* is a list of strings representing the type of information desired (e.g. ``["type", "size", "perm"]``). Return a generator object yielding a @@ -333,7 +326,7 @@ followed by ``lines`` for the text version or ``binary`` for the binary version. directory). Multiple arguments can be used to pass non-standard options to the ``NLST`` command. - .. deprecated:: 3.3 use :meth:`mlsd` instead. + .. note:: If your server supports the command, :meth:`mlsd` offers a better API. .. method:: FTP.dir(argument[, ...]) @@ -345,7 +338,7 @@ followed by ``lines`` for the text version or ``binary`` for the binary version. as a *callback* function as for :meth:`retrlines`; the default prints to ``sys.stdout``. This method returns ``None``. - .. deprecated:: 3.3 use :meth:`mlsd` instead. + .. note:: If your server supports the command, :meth:`mlsd` offers a better API. .. method:: FTP.rename(fromname, toname) diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst index 21aeaf9f5c..efa5bd38a2 100644 --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -156,11 +156,12 @@ are always available. They are listed here in alphabetical order. .. function:: chr(i) - Return the string representing a character whose Unicode code point is the integer - *i*. For example, ``chr(97)`` returns the string ``'a'``. This is the - inverse of :func:`ord`. The valid range for the argument is from 0 through - 1,114,111 (0x10FFFF in base 16). :exc:`ValueError` will be raised if *i* is - outside that range. + Return the string representing a character whose Unicode code point is the + integer *i*. For example, ``chr(97)`` returns the string ``'a'``, while + ``chr(8364)`` returns the string ``'€'``. This is the inverse of :func:`ord`. + + The valid range for the argument is from 0 through 1,114,111 (0x10FFFF in + base 16). :exc:`ValueError` will be raised if *i* is outside that range. .. function:: classmethod(function) @@ -229,7 +230,7 @@ are always available. They are listed here in alphabetical order. or ``2`` (docstrings are removed too). This function raises :exc:`SyntaxError` if the compiled source is invalid, - and :exc:`TypeError` if the source contains null bytes. + and :exc:`ValueError` if the source contains null bytes. If you want to parse Python code into its AST representation, see :func:`ast.parse`. @@ -245,6 +246,10 @@ are always available. They are listed here in alphabetical order. Allowed use of Windows and Mac newlines. Also input in ``'exec'`` mode does not have to end in a newline anymore. Added the *optimize* parameter. + .. versionchanged:: 3.5 + Previously, :exc:`TypeError` was raised when null bytes were encountered + in *source*. + .. class:: complex([real[, imag]]) @@ -299,7 +304,7 @@ are always available. They are listed here in alphabetical order. :func:`dir` reports their attributes. If the object does not provide :meth:`__dir__`, the function tries its best to - gather information from the object's :attr:`__dict__` attribute, if defined, and + gather information from the object's :attr:`~object.__dict__` attribute, if defined, and from its type object. The resulting list is not necessarily complete, and may be inaccurate when the object has a custom :func:`__getattr__`. @@ -972,9 +977,11 @@ are always available. They are listed here in alphabetical order. Characters not supported by the encoding are replaced with the appropriate XML character reference ``&#nnn;``. - * ``'backslashreplace'`` (also only supported when writing) - replaces unsupported characters with Python's backslashed escape - sequences. + * ``'backslashreplace'`` replaces malformed data by Python's backslashed + escape sequences. + + * ``'namereplace'`` (also only supported when writing) + replaces unsupported characters with ``\N{...}`` escape sequences. .. index:: single: universal newlines; open() built-in function @@ -999,8 +1006,8 @@ are always available. They are listed here in alphabetical order. If *closefd* is ``False`` and a file descriptor rather than a filename was given, the underlying file descriptor will be kept open when the file is - closed. If a filename is given *closefd* has no effect and must be ``True`` - (the default). + closed. If a filename is given *closefd* must be ``True`` (the default) + otherwise an error will be raised. A custom opener can be used by passing a callable as *opener*. The underlying file descriptor for the file object is then obtained by calling *opener* with @@ -1062,14 +1069,20 @@ are always available. They are listed here in alphabetical order. The ``'U'`` mode. + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise an + exception, the function now retries the system call instead of raising an + :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + + .. versionchanged:: 3.5 + The ``'namereplace'`` error handler was added. -.. XXX works for bytes too, but should it? .. function:: ord(c) Given a string representing one Unicode character, return an integer - representing the Unicode code - point of that character. For example, ``ord('a')`` returns the integer ``97`` - and ``ord('\u2020')`` returns ``8224``. This is the inverse of :func:`chr`. + representing the Unicode code point of that character. For example, + ``ord('a')`` returns the integer ``97`` and ``ord('€')`` (Euro sign) + returns ``8364``. This is the inverse of :func:`chr`. .. function:: pow(x, y[, z]) @@ -1186,6 +1199,9 @@ are always available. They are listed here in alphabetical order. The returned property object also has the attributes ``fget``, ``fset``, and ``fdel`` corresponding to the constructor arguments. + .. versionchanged:: 3.5 + The docstrings of property objects are now writeable. + .. _func-range: .. function:: range(stop) @@ -1218,8 +1234,8 @@ are always available. They are listed here in alphabetical order. .. function:: round(number[, ndigits]) Return the floating point value *number* rounded to *ndigits* digits after - the decimal point. If *ndigits* is omitted, it defaults to zero. Delegates - to ``number.__round__(ndigits)``. + the decimal point. If *ndigits* is omitted, it returns the nearest integer + to its input. Delegates to ``number.__round__(ndigits)``. For the built-in types supporting :func:`round`, values are rounded to the closest multiple of 10 to the power minus *ndigits*; if two multiples are @@ -1404,7 +1420,7 @@ are always available. They are listed here in alphabetical order. For practical suggestions on how to design cooperative classes using :func:`super`, see `guide to using super() - <http://rhettinger.wordpress.com/2011/05/26/super-considered-super/>`_. + <https://rhettinger.wordpress.com/2011/05/26/super-considered-super/>`_. .. _func-tuple: @@ -1430,11 +1446,12 @@ are always available. They are listed here in alphabetical order. With three arguments, return a new type object. This is essentially a dynamic form of the :keyword:`class` statement. The *name* string is the - class name and becomes the :attr:`~class.__name__` attribute; the *bases* + class name and becomes the :attr:`~definition.__name__` attribute; the *bases* tuple itemizes the base classes and becomes the :attr:`~class.__bases__` attribute; and the *dict* dictionary is the namespace containing definitions - for class body and becomes the :attr:`~object.__dict__` attribute. For - example, the following two statements create identical :class:`type` objects: + for class body and is copied to a standard dictionary to become the + :attr:`~object.__dict__` attribute. For example, the following two + statements create identical :class:`type` objects: >>> class X: ... a = 1 @@ -1447,12 +1464,12 @@ are always available. They are listed here in alphabetical order. .. function:: vars([object]) Return the :attr:`~object.__dict__` attribute for a module, class, instance, - or any other object with a :attr:`__dict__` attribute. + or any other object with a :attr:`~object.__dict__` attribute. - Objects such as modules and instances have an updateable :attr:`__dict__` + Objects such as modules and instances have an updateable :attr:`~object.__dict__` attribute; however, other objects may have write restrictions on their - :attr:`__dict__` attributes (for example, classes use a - dictproxy to prevent direct dictionary updates). + :attr:`~object.__dict__` attributes (for example, classes use a + :class:`types.MappingProxyType` to prevent direct dictionary updates). Without an argument, :func:`vars` acts like :func:`locals`. Note, the locals dictionary is only useful for reads since updates to the locals @@ -1484,7 +1501,9 @@ are always available. They are listed here in alphabetical order. The left-to-right evaluation order of the iterables is guaranteed. This makes possible an idiom for clustering a data series into n-length groups - using ``zip(*[iter(s)]*n)``. + using ``zip(*[iter(s)]*n)``. This repeats the *same* iterator ``n`` times + so that each output tuple has the result of ``n`` calls to the iterator. + This has the effect of dividing the input into n-length chunks. :func:`zip` should only be used with unequal length inputs when you don't care about trailing, unmatched values from the longer iterables. If those diff --git a/Doc/library/functools.rst b/Doc/library/functools.rst index 46aa88767e..127e3fa64d 100644 --- a/Doc/library/functools.rst +++ b/Doc/library/functools.rst @@ -3,6 +3,7 @@ .. module:: functools :synopsis: Higher-order functions and operations on callable objects. + .. moduleauthor:: Peter Harris <scav@blueyonder.co.uk> .. moduleauthor:: Raymond Hettinger <python@rcn.com> .. moduleauthor:: Nick Coghlan <ncoghlan@gmail.com> @@ -73,7 +74,7 @@ The :mod:`functools` module defines the following functions: bypassing the cache, or for rewrapping the function with a different cache. An `LRU (least recently used) cache - <http://en.wikipedia.org/wiki/Cache_algorithms#Examples>`_ works + <https://en.wikipedia.org/wiki/Cache_algorithms#Examples>`_ works best when the most recent calls are the best predictors of upcoming calls (for example, the most popular articles on a news server tend to change each day). The cache's size limit assures that the cache does not grow without bound on @@ -99,9 +100,9 @@ The :mod:`functools` module defines the following functions: CacheInfo(hits=3, misses=8, maxsize=32, currsize=8) Example of efficiently computing - `Fibonacci numbers <http://en.wikipedia.org/wiki/Fibonacci_number>`_ + `Fibonacci numbers <https://en.wikipedia.org/wiki/Fibonacci_number>`_ using a cache to implement a - `dynamic programming <http://en.wikipedia.org/wiki/Dynamic_programming>`_ + `dynamic programming <https://en.wikipedia.org/wiki/Dynamic_programming>`_ technique:: @lru_cache(maxsize=None) @@ -176,7 +177,7 @@ The :mod:`functools` module defines the following functions: def newfunc(*fargs, **fkeywords): newkeywords = keywords.copy() newkeywords.update(fkeywords) - return func(*(args + fargs), **newkeywords) + return func(*args, *fargs, **newkeywords) newfunc.func = func newfunc.args = args newfunc.keywords = keywords @@ -375,10 +376,10 @@ The :mod:`functools` module defines the following functions: assigned directly to the matching attributes on the wrapper function and which attributes of the wrapper function are updated with the corresponding attributes from the original function. The default values for these arguments are the - module level constants *WRAPPER_ASSIGNMENTS* (which assigns to the wrapper - function's *__name__*, *__module__*, *__annotations__* and *__doc__*, the - documentation string) and *WRAPPER_UPDATES* (which updates the wrapper - function's *__dict__*, i.e. the instance dictionary). + module level constants ``WRAPPER_ASSIGNMENTS`` (which assigns to the wrapper + function's ``__module__``, ``__name__``, ``__qualname__``, ``__annotations__`` + and ``__doc__``, the documentation string) and ``WRAPPER_UPDATES`` (which + updates the wrapper function's ``__dict__``, i.e. the instance dictionary). To allow access to the original function for introspection and other purposes (e.g. bypassing a caching decorator such as :func:`lru_cache`), this function @@ -473,7 +474,7 @@ have three read-only attributes: :class:`partial` objects are like :class:`function` objects in that they are callable, weak referencable, and can have attributes. There are some important -differences. For instance, the :attr:`__name__` and :attr:`__doc__` attributes +differences. For instance, the :attr:`~definition.__name__` and :attr:`__doc__` attributes are not created automatically. Also, :class:`partial` objects defined in classes behave like static methods and do not transform into bound methods during instance attribute look-up. diff --git a/Doc/library/gc.rst b/Doc/library/gc.rst index 813554206a..4af16a9dfd 100644 --- a/Doc/library/gc.rst +++ b/Doc/library/gc.rst @@ -3,9 +3,11 @@ .. module:: gc :synopsis: Interface to the cycle-detecting garbage collector. + .. moduleauthor:: Neil Schemenauer <nas@arctrix.com> .. sectionauthor:: Neil Schemenauer <nas@arctrix.com> +-------------- This module provides an interface to the optional garbage collector. It provides the ability to disable the collector, tune the collection frequency, @@ -186,7 +188,7 @@ values but should not rebind them): added to this list rather than freed. .. versionchanged:: 3.2 - If this list is non-empty at interpreter shutdown, a + If this list is non-empty at :term:`interpreter shutdown`, a :exc:`ResourceWarning` is emitted, which is silent by default. If :const:`DEBUG_UNCOLLECTABLE` is set, in addition all uncollectable objects are printed. @@ -252,8 +254,8 @@ The following constants are provided for use with :func:`set_debug`: to the ``garbage`` list. .. versionchanged:: 3.2 - Also print the contents of the :data:`garbage` list at interpreter - shutdown, if it isn't empty. + Also print the contents of the :data:`garbage` list at + :term:`interpreter shutdown`, if it isn't empty. .. data:: DEBUG_SAVEALL diff --git a/Doc/library/getopt.rst b/Doc/library/getopt.rst index f9a1e53e38..336deab28c 100644 --- a/Doc/library/getopt.rst +++ b/Doc/library/getopt.rst @@ -7,8 +7,6 @@ **Source code:** :source:`Lib/getopt.py` --------------- - .. note:: The :mod:`getopt` module is a parser for command line options whose API is @@ -17,6 +15,8 @@ less code and get better help and error messages should consider using the :mod:`argparse` module instead. +-------------- + This module helps scripts to parse the command line arguments in ``sys.argv``. It supports the same conventions as the Unix :c:func:`getopt` function (including the special meanings of arguments of the form '``-``' and '``--``'). Long @@ -124,7 +124,7 @@ In a script, typical usage is something like this:: opts, args = getopt.getopt(sys.argv[1:], "ho:v", ["help", "output="]) except getopt.GetoptError as err: # print help information and exit: - print(err) # will print something like "option -a not recognized" + print(err) # will print something like "option -a not recognized" usage() sys.exit(2) output = None diff --git a/Doc/library/getpass.rst b/Doc/library/getpass.rst index 211563e23e..5eb9f04a8d 100644 --- a/Doc/library/getpass.rst +++ b/Doc/library/getpass.rst @@ -3,10 +3,15 @@ .. module:: getpass :synopsis: Portable reading of passwords and retrieval of the userid. + .. moduleauthor:: Piers Lauder <piers@cs.su.oz.au> .. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org> .. Windows (& Mac?) support by Guido van Rossum. +**Source code:** :source:`Lib/getpass.py` + +-------------- + The :mod:`getpass` module provides two functions: @@ -23,8 +28,6 @@ The :mod:`getpass` module provides two functions: a warning message to *stream* and reading from ``sys.stdin`` and issuing a :exc:`GetPassWarning`. - Availability: Macintosh, Unix, Windows. - .. note:: If you call getpass from within IDLE, the input may be done in the terminal you launched IDLE from rather than the idle window itself. @@ -36,7 +39,7 @@ The :mod:`getpass` module provides two functions: .. function:: getuser() - Return the "login name" of the user. Availability: Unix, Windows. + Return the "login name" of the user. This function checks the environment variables :envvar:`LOGNAME`, :envvar:`USER`, :envvar:`LNAME` and :envvar:`USERNAME`, in order, and returns diff --git a/Doc/library/gettext.rst b/Doc/library/gettext.rst index ff23b59156..ea439b527f 100644 --- a/Doc/library/gettext.rst +++ b/Doc/library/gettext.rst @@ -3,6 +3,7 @@ .. module:: gettext :synopsis: Multilingual internationalization services. + .. moduleauthor:: Barry A. Warsaw <barry@python.org> .. sectionauthor:: Barry A. Warsaw <barry@python.org> @@ -344,9 +345,9 @@ will assume message ids as Unicode strings, not byte strings. The entire set of key/value pairs are placed into a dictionary and set as the "protected" :attr:`_info` instance variable. -If the :file:`.mo` file's magic number is invalid, or if other problems occur -while reading the file, instantiating a :class:`GNUTranslations` class can raise -:exc:`OSError`. +If the :file:`.mo` file's magic number is invalid, the major version number is +unexpected, or if other problems occur while reading the file, instantiating a +:class:`GNUTranslations` class can raise :exc:`OSError`. The following methods are overridden from the base class implementation: diff --git a/Doc/library/glob.rst b/Doc/library/glob.rst index 8e9af376a8..328eef30f1 100644 --- a/Doc/library/glob.rst +++ b/Doc/library/glob.rst @@ -4,11 +4,10 @@ .. module:: glob :synopsis: Unix shell style pathname pattern expansion. +**Source code:** :source:`Lib/glob.py` .. index:: single: filenames; pathname expansion -**Source code:** :source:`Lib/glob.py` - -------------- The :mod:`glob` module finds all the pathnames matching a specified pattern @@ -29,7 +28,7 @@ For example, ``'[?]'`` matches the character ``'?'``. The :mod:`pathlib` module offers high-level path objects. -.. function:: glob(pathname) +.. function:: glob(pathname, *, recursive=False) Return a possibly-empty list of path names that match *pathname*, which must be a string containing a path specification. *pathname* can be either absolute @@ -37,8 +36,19 @@ For example, ``'[?]'`` matches the character ``'?'``. :file:`../../Tools/\*/\*.gif`), and can contain shell-style wildcards. Broken symlinks are included in the results (as in the shell). + If *recursive* is true, the pattern "``**``" will match any files and zero or + more directories and subdirectories. If the pattern is followed by an + ``os.sep``, only directories and subdirectories match. + + .. note:: + Using the "``**``" pattern in large directory trees may consume + an inordinate amount of time. + + .. versionchanged:: 3.5 + Support for recursive globs using "``**``". + -.. function:: iglob(pathname) +.. function:: iglob(pathname, recursive=False) Return an :term:`iterator` which yields the same values as :func:`glob` without actually storing them all simultaneously. @@ -55,8 +65,9 @@ For example, ``'[?]'`` matches the character ``'?'``. .. versionadded:: 3.4 -For example, consider a directory containing only the following files: -:file:`1.gif`, :file:`2.txt`, and :file:`card.gif`. :func:`glob` will produce +For example, consider a directory containing the following files: +:file:`1.gif`, :file:`2.txt`, :file:`card.gif` and a subdirectory :file:`sub` +which contains only the file :file:`3.txt`. :func:`glob` will produce the following results. Notice how any leading components of the path are preserved. :: @@ -67,6 +78,10 @@ preserved. :: ['1.gif', 'card.gif'] >>> glob.glob('?.gif') ['1.gif'] + >>> glob.glob('**/*.txt', recursive=True) + ['2.txt', 'sub/3.txt'] + >>> glob.glob('./**/', recursive=True) + ['./', './sub/'] If the directory contains files starting with ``.`` they won't be matched by default. For example, consider a directory containing :file:`card.gif` and diff --git a/Doc/library/grp.rst b/Doc/library/grp.rst index 88821406a3..a30e6229db 100644 --- a/Doc/library/grp.rst +++ b/Doc/library/grp.rst @@ -5,6 +5,7 @@ :platform: Unix :synopsis: The group database (getgrnam() and friends). +-------------- This module provides access to the Unix group database. It is available on all Unix versions. diff --git a/Doc/library/gzip.rst b/Doc/library/gzip.rst index 355cf9ccfe..792d57a8a3 100644 --- a/Doc/library/gzip.rst +++ b/Doc/library/gzip.rst @@ -90,13 +90,9 @@ The module defines the following items: is no compression. The default is ``9``. The *mtime* argument is an optional numeric timestamp to be written to - the stream when compressing. All :program:`gzip` compressed streams are - required to contain a timestamp. If omitted or ``None``, the current - time is used. This module ignores the timestamp when decompressing; - however, some programs, such as :program:`gunzip`\ , make use of it. - The format of the timestamp is the same as that of the return value of - ``time.time()`` and of the ``st_mtime`` attribute of the object returned - by ``os.stat()``. + the last modification time field in the stream when compressing. It + should only be provided in compression mode. If omitted or ``None``, the + current time is used. See the :attr:`mtime` attribute for more details. Calling a :class:`GzipFile` object's :meth:`close` method does not close *fileobj*, since you might wish to append more material after the compressed @@ -108,9 +104,9 @@ The module defines the following items: including iteration and the :keyword:`with` statement. Only the :meth:`truncate` method isn't implemented. - :class:`GzipFile` also provides the following method: + :class:`GzipFile` also provides the following method and attribute: - .. method:: peek([n]) + .. method:: peek(n) Read *n* uncompressed bytes without advancing the file position. At most one single read on the compressed stream is done to satisfy @@ -124,9 +120,21 @@ The module defines the following items: .. versionadded:: 3.2 + .. attribute:: mtime + + When decompressing, the value of the last modification time field in + the most recently read header may be read from this attribute, as an + integer. The initial value before reading any headers is ``None``. + + All :program:`gzip` compressed streams are required to contain this + timestamp field. Some programs, such as :program:`gunzip`\ , make use + of the timestamp. The format is the same as the return value of + :func:`time.time` and the :attr:`~os.stat_result.st_mtime` attribute of + the object returned by :func:`os.stat`. + .. versionchanged:: 3.1 Support for the :keyword:`with` statement was added, along with the - *mtime* argument. + *mtime* constructor argument and :attr:`mtime` attribute. .. versionchanged:: 3.2 Support for zero-padded and unseekable files was added. @@ -137,6 +145,12 @@ The module defines the following items: .. versionchanged:: 3.4 Added support for the ``'x'`` and ``'xb'`` modes. + .. versionchanged:: 3.5 + Added support for writing arbitrary + :term:`bytes-like objects <bytes-like object>`. + The :meth:`~io.BufferedIOBase.read` method now accepts an argument of + ``None``. + .. function:: compress(data, compresslevel=9) @@ -175,9 +189,10 @@ Example of how to create a compressed GZIP file:: Example of how to GZIP compress an existing file:: import gzip + import shutil with open('/home/joe/file.txt', 'rb') as f_in: with gzip.open('/home/joe/file.txt.gz', 'wb') as f_out: - f_out.writelines(f_in) + shutil.copyfileobj(f_in, f_out) Example of how to GZIP compress a binary string:: diff --git a/Doc/library/hashlib.rst b/Doc/library/hashlib.rst index 769f96f017..30be3354af 100644 --- a/Doc/library/hashlib.rst +++ b/Doc/library/hashlib.rst @@ -3,16 +3,16 @@ .. module:: hashlib :synopsis: Secure hash and message digest algorithms. + .. moduleauthor:: Gregory P. Smith <greg@krypto.org> .. sectionauthor:: Gregory P. Smith <greg@krypto.org> +**Source code:** :source:`Lib/hashlib.py` .. index:: single: message digest, MD5 single: secure hash algorithm, SHA1, SHA224, SHA256, SHA384, SHA512 -**Source code:** :source:`Lib/hashlib.py` - -------------- This module implements a common interface to many different secure hash and @@ -41,7 +41,7 @@ Hash algorithms There is one constructor method named for each type of :dfn:`hash`. All return a hash object with the same simple interface. For example: use :func:`sha1` to create a SHA1 hash object. You can now feed this object with :term:`bytes-like -object`\ s (normally :class:`bytes`) using the :meth:`update` method. +objects <bytes-like object>` (normally :class:`bytes`) using the :meth:`update` method. At any point you can ask it for the :dfn:`digest` of the concatenation of the data fed to it so far using the :meth:`digest` or :meth:`hexdigest` methods. @@ -185,22 +185,23 @@ brute-force attacks. A good password hashing function must be tunable, slow, and include a `salt <https://en.wikipedia.org/wiki/Salt_%28cryptography%29>`_. -.. function:: pbkdf2_hmac(name, password, salt, rounds, dklen=None) +.. function:: pbkdf2_hmac(hash_name, password, salt, iterations, dklen=None) The function provides PKCS#5 password-based key derivation function 2. It uses HMAC as pseudorandom function. - The string *name* is the desired name of the hash digest algorithm for + The string *hash_name* is the desired name of the hash digest algorithm for HMAC, e.g. 'sha1' or 'sha256'. *password* and *salt* are interpreted as buffers of bytes. Applications and libraries should limit *password* to - a sensible value (e.g. 1024). *salt* should be about 16 or more bytes from + a sensible length (e.g. 1024). *salt* should be about 16 or more bytes from a proper source, e.g. :func:`os.urandom`. - The number of *rounds* should be chosen based on the hash algorithm and - computing power. As of 2013, at least 100,000 rounds of SHA-256 is suggested. + The number of *iterations* should be chosen based on the hash algorithm and + computing power. As of 2013, at least 100,000 iterations of SHA-256 are + suggested. *dklen* is the length of the derived key. If *dklen* is ``None`` then the - digest size of the hash algorithm *name* is used, e.g. 64 for SHA-512. + digest size of the hash algorithm *hash_name* is used, e.g. 64 for SHA-512. >>> import hashlib, binascii >>> dk = hashlib.pbkdf2_hmac('sha256', b'password', b'salt', 100000) @@ -231,5 +232,5 @@ include a `salt <https://en.wikipedia.org/wiki/Salt_%28cryptography%29>`_. Wikipedia article with information on which algorithms have known issues and what that means regarding their use. - http://www.ietf.org/rfc/rfc2898.txt + https://www.ietf.org/rfc/rfc2898.txt PKCS #5: Password-Based Cryptography Specification Version 2.0 diff --git a/Doc/library/heapq.rst b/Doc/library/heapq.rst index f8970bed73..7e33e74814 100644 --- a/Doc/library/heapq.rst +++ b/Doc/library/heapq.rst @@ -3,6 +3,7 @@ .. module:: heapq :synopsis: Heap queue algorithm (a.k.a. priority queue). + .. moduleauthor:: Kevin O'Connor .. sectionauthor:: Guido van Rossum <guido@python.org> .. sectionauthor:: François Pinard @@ -82,7 +83,7 @@ The following functions are provided: The module also offers three general purpose functions based on heaps. -.. function:: merge(*iterables) +.. function:: merge(*iterables, key=None, reverse=False) Merge multiple sorted inputs into a single sorted output (for example, merge timestamped entries from multiple log files). Returns an :term:`iterator` @@ -92,6 +93,18 @@ The module also offers three general purpose functions based on heaps. not pull the data into memory all at once, and assumes that each of the input streams is already sorted (smallest to largest). + Has two optional arguments which must be specified as keyword arguments. + + *key* specifies a :term:`key function` of one argument that is used to + extract a comparison key from each input element. The default value is + ``None`` (compare the elements directly). + + *reverse* is a boolean value. If set to ``True``, then the input elements + are merged as if each comparison were reversed. + + .. versionchanged:: 3.5 + Added the optional *key* and *reverse* parameters. + .. function:: nlargest(n, iterable, key=None) @@ -120,7 +133,7 @@ the iterable into an actual heap. Basic Examples -------------- -A `heapsort <http://en.wikipedia.org/wiki/Heapsort>`_ can be implemented by +A `heapsort <https://en.wikipedia.org/wiki/Heapsort>`_ can be implemented by pushing all values onto a heap and then popping off the smallest values one at a time:: @@ -151,7 +164,7 @@ Heap elements can be tuples. This is useful for assigning comparison values Priority Queue Implementation Notes ----------------------------------- -A `priority queue <http://en.wikipedia.org/wiki/Priority_queue>`_ is common use +A `priority queue <https://en.wikipedia.org/wiki/Priority_queue>`_ is common use for a heap, and it presents several implementation challenges: * Sort stability: how do you get two tasks with equal priorities to be returned @@ -230,7 +243,7 @@ for a tournament. The numbers below are *k*, not ``a[k]``:: 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 -In the tree above, each cell *k* is topping ``2*k+1`` and ``2*k+2``. In an usual +In the tree above, each cell *k* is topping ``2*k+1`` and ``2*k+2``. In a usual binary tournament we see in sports, each cell is the winner over the two cells it tops, and we can trace the winner down the tree to see all opponents s/he had. However, in many computer applications of such tournaments, we do not need diff --git a/Doc/library/hmac.rst b/Doc/library/hmac.rst index 1446da6ee6..bb44866879 100644 --- a/Doc/library/hmac.rst +++ b/Doc/library/hmac.rst @@ -3,6 +3,7 @@ .. module:: hmac :synopsis: Keyed-Hashing for Message Authentication (HMAC) implementation + .. moduleauthor:: Gerhard Häring <ghaering@users.sourceforge.net> .. sectionauthor:: Gerhard Häring <ghaering@users.sourceforge.net> diff --git a/Doc/library/html.entities.rst b/Doc/library/html.entities.rst index e10e46e2b8..067e1b1e5a 100644 --- a/Doc/library/html.entities.rst +++ b/Doc/library/html.entities.rst @@ -3,6 +3,7 @@ .. module:: html.entities :synopsis: Definitions of HTML general entities. + .. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org> **Source code:** :source:`Lib/html/entities.py` @@ -43,4 +44,4 @@ This module defines four dictionaries, :data:`html5`, .. rubric:: Footnotes -.. [#] See http://www.w3.org/TR/html5/syntax.html#named-character-references +.. [#] See https://www.w3.org/TR/html5/syntax.html#named-character-references diff --git a/Doc/library/html.parser.rst b/Doc/library/html.parser.rst index fef9c38411..ac844a683b 100644 --- a/Doc/library/html.parser.rst +++ b/Doc/library/html.parser.rst @@ -4,33 +4,24 @@ .. module:: html.parser :synopsis: A simple parser that can handle HTML and XHTML. +**Source code:** :source:`Lib/html/parser.py` .. index:: single: HTML single: XHTML -**Source code:** :source:`Lib/html/parser.py` - -------------- This module defines a class :class:`HTMLParser` which serves as the basis for parsing text files formatted in HTML (HyperText Mark-up Language) and XHTML. -.. class:: HTMLParser(strict=False, *, convert_charrefs=False) +.. class:: HTMLParser(*, convert_charrefs=True) - Create a parser instance. + Create a parser instance able to parse invalid markup. - If *convert_charrefs* is ``True`` (default: ``False``), all character + If *convert_charrefs* is ``True`` (the default), all character references (except the ones in ``script``/``style`` elements) are automatically converted to the corresponding Unicode characters. - The use of ``convert_charrefs=True`` is encouraged and will become - the default in Python 3.5. - - If *strict* is ``False`` (the default), the parser will accept and parse - invalid markup. If *strict* is ``True`` the parser will raise an - :exc:`~html.parser.HTMLParseError` exception instead [#]_ when it's not - able to parse the markup. The use of ``strict=True`` is discouraged and - the *strict* argument is deprecated. An :class:`.HTMLParser` instance is fed HTML data and calls handler methods when start tags, end tags, text, comments, and other markup elements are @@ -40,31 +31,11 @@ parsing text files formatted in HTML (HyperText Mark-up Language) and XHTML. This parser does not check that end tags match start tags or call the end-tag handler for elements which are closed implicitly by closing an outer element. - .. versionchanged:: 3.2 - *strict* argument added. - - .. deprecated-removed:: 3.3 3.5 - The *strict* argument and the strict mode have been deprecated. - The parser is now able to accept and parse invalid markup too. - .. versionchanged:: 3.4 *convert_charrefs* keyword argument added. -An exception is defined as well: - - -.. exception:: HTMLParseError - - Exception raised by the :class:`HTMLParser` class when it encounters an error - while parsing and *strict* is ``True``. This exception provides three - attributes: :attr:`msg` is a brief message explaining the error, - :attr:`lineno` is the number of the line on which the broken construct was - detected, and :attr:`offset` is the number of characters into the line at - which the construct starts. - - .. deprecated-removed:: 3.3 3.5 - This exception has been deprecated because it's never raised by the parser - (when the default non-strict mode is used). + .. versionchanged:: 3.5 + The default value for argument *convert_charrefs* is now ``True``. Example HTML Parser Application @@ -79,8 +50,10 @@ as they are encountered:: class MyHTMLParser(HTMLParser): def handle_starttag(self, tag, attrs): print("Encountered a start tag:", tag) + def handle_endtag(self, tag): print("Encountered an end tag :", tag) + def handle_data(self, data): print("Encountered some data :", data) @@ -88,7 +61,9 @@ as they are encountered:: parser.feed('<html><head><title>Test</title></head>' '<body><h1>Parse me!</h1></body></html>') -The output will then be:: +The output will then be: + +.. code-block:: none Encountered a start tag: html Encountered a start tag: head @@ -159,8 +134,8 @@ implementations do nothing (except for :meth:`~HTMLParser.handle_startendtag`): and quotes in the *value* have been removed, and character and entity references have been replaced. - For instance, for the tag ``<A HREF="http://www.cwi.nl/">``, this method - would be called as ``handle_starttag('a', [('href', 'http://www.cwi.nl/')])``. + For instance, for the tag ``<A HREF="https://www.cwi.nl/">``, this method + would be called as ``handle_starttag('a', [('href', 'https://www.cwi.nl/')])``. All entity references from :mod:`html.entities` are replaced in the attribute values. @@ -246,8 +221,7 @@ implementations do nothing (except for :meth:`~HTMLParser.handle_startendtag`): The *data* parameter will be the entire contents of the declaration inside the ``<![...]>`` markup. It is sometimes useful to be overridden by a - derived class. The base class implementation raises an :exc:`HTMLParseError` - when *strict* is ``True``. + derived class. The base class implementation does nothing. .. _htmlparser-examples: @@ -266,21 +240,27 @@ examples:: print("Start tag:", tag) for attr in attrs: print(" attr:", attr) + def handle_endtag(self, tag): print("End tag :", tag) + def handle_data(self, data): print("Data :", data) + def handle_comment(self, data): print("Comment :", data) + def handle_entityref(self, name): c = chr(name2codepoint[name]) print("Named ent:", c) + def handle_charref(self, name): if name.startswith('x'): c = chr(int(name[1:], 16)) else: c = chr(int(name)) print("Num ent :", c) + def handle_decl(self, data): print("Decl :", data) @@ -312,7 +292,7 @@ further parsing:: attr: ('type', 'text/css') Data : #python { color: green } End tag : style - >>> + >>> parser.feed('<script type="text/javascript">' ... 'alert("<strong>hello!</strong>");</script>') Start tag: script @@ -358,9 +338,3 @@ Parsing invalid HTML (e.g. unquoted attributes) also works:: Data : tag soup End tag : p End tag : a - -.. rubric:: Footnotes - -.. [#] For backward compatibility reasons *strict* mode does not raise - exceptions for all non-compliant HTML. That is, some invalid HTML - is tolerated even in *strict* mode. diff --git a/Doc/library/http.client.rst b/Doc/library/http.client.rst index 807f685248..8954648976 100644 --- a/Doc/library/http.client.rst +++ b/Doc/library/http.client.rst @@ -4,6 +4,7 @@ .. module:: http.client :synopsis: HTTP and HTTPS protocol client (requires sockets). +**Source code:** :source:`Lib/http/client.py` .. index:: pair: HTTP; protocol @@ -11,8 +12,6 @@ .. index:: module: urllib.request -**Source code:** :source:`Lib/http/client.py` - -------------- This module defines classes which implement the client side of the HTTP and @@ -21,8 +20,8 @@ HTTPS protocols. It is normally not used directly --- the module .. seealso:: - The `Requests package <http://requests.readthedocs.org/>`_ - is recommended for a higher-level http client interface. + The `Requests package <https://requests.readthedocs.org/>`_ + is recommended for a higher-level HTTP client interface. .. note:: @@ -180,6 +179,17 @@ The following exceptions are raised as appropriate: is received in the HTTP protocol from the server. +.. exception:: RemoteDisconnected + + A subclass of :exc:`ConnectionResetError` and :exc:`BadStatusLine`. Raised + by :meth:`HTTPConnection.getresponse` when the attempt to read the response + results in no data read from the connection, indicating that the remote end + has closed the connection. + + .. versionadded:: 3.5 + Previously, :exc:`BadStatusLine`\ ``('')`` was raised. + + The constants defined in this module are: .. data:: HTTP_PORT @@ -191,221 +201,15 @@ The constants defined in this module are: The default port for the HTTPS protocol (always ``443``). -and also the following constants for integer status codes: - -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| Constant | Value | Definition | -+==========================================+=========+=======================================================================+ -| :const:`CONTINUE` | ``100`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.1.1 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.1.1>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`SWITCHING_PROTOCOLS` | ``101`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.1.2 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.1.2>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`PROCESSING` | ``102`` | WEBDAV, `RFC 2518, Section 10.1 | -| | | <http://www.webdav.org/specs/rfc2518.html#STATUS_102>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`OK` | ``200`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.2.1 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.2.1>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`CREATED` | ``201`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.2.2 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.2.2>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`ACCEPTED` | ``202`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.2.3 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.2.3>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`NON_AUTHORITATIVE_INFORMATION` | ``203`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.2.4 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.2.4>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`NO_CONTENT` | ``204`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.2.5 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.2.5>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`RESET_CONTENT` | ``205`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.2.6 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.2.6>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`PARTIAL_CONTENT` | ``206`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.2.7 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.2.7>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`MULTI_STATUS` | ``207`` | WEBDAV `RFC 2518, Section 10.2 | -| | | <http://www.webdav.org/specs/rfc2518.html#STATUS_207>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`IM_USED` | ``226`` | Delta encoding in HTTP, | -| | | :rfc:`3229`, Section 10.4.1 | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`MULTIPLE_CHOICES` | ``300`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.3.1 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3.1>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`MOVED_PERMANENTLY` | ``301`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.3.2 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3.2>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`FOUND` | ``302`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.3.3 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3.3>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`SEE_OTHER` | ``303`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.3.4 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3.4>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`NOT_MODIFIED` | ``304`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.3.5 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3.5>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`USE_PROXY` | ``305`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.3.6 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3.6>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`TEMPORARY_REDIRECT` | ``307`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.3.8 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3.8>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`BAD_REQUEST` | ``400`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.1 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.1>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`UNAUTHORIZED` | ``401`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.2 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.2>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`PAYMENT_REQUIRED` | ``402`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.3 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.3>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`FORBIDDEN` | ``403`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.4 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.4>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`NOT_FOUND` | ``404`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.5 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.5>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`METHOD_NOT_ALLOWED` | ``405`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.6 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.6>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`NOT_ACCEPTABLE` | ``406`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.7 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.7>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`PROXY_AUTHENTICATION_REQUIRED` | ``407`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.8 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.8>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`REQUEST_TIMEOUT` | ``408`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.9 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.9>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`CONFLICT` | ``409`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.10 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.10>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`GONE` | ``410`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.11 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.11>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`LENGTH_REQUIRED` | ``411`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.12 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.12>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`PRECONDITION_FAILED` | ``412`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.13 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.13>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`REQUEST_ENTITY_TOO_LARGE` | ``413`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.14 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.14>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`REQUEST_URI_TOO_LONG` | ``414`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.15 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.15>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`UNSUPPORTED_MEDIA_TYPE` | ``415`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.16 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.16>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`REQUESTED_RANGE_NOT_SATISFIABLE` | ``416`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.17 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.17>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`EXPECTATION_FAILED` | ``417`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.4.18 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.18>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`UNPROCESSABLE_ENTITY` | ``422`` | WEBDAV, `RFC 2518, Section 10.3 | -| | | <http://www.webdav.org/specs/rfc2518.html#STATUS_422>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`LOCKED` | ``423`` | WEBDAV `RFC 2518, Section 10.4 | -| | | <http://www.webdav.org/specs/rfc2518.html#STATUS_423>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`FAILED_DEPENDENCY` | ``424`` | WEBDAV, `RFC 2518, Section 10.5 | -| | | <http://www.webdav.org/specs/rfc2518.html#STATUS_424>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`UPGRADE_REQUIRED` | ``426`` | HTTP Upgrade to TLS, | -| | | :rfc:`2817`, Section 6 | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`PRECONDITION_REQUIRED` | ``428`` | Additional HTTP Status Codes, | -| | | :rfc:`6585`, Section 3 | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`TOO_MANY_REQUESTS` | ``429`` | Additional HTTP Status Codes, | -| | | :rfc:`6585`, Section 4 | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`REQUEST_HEADER_FIELDS_TOO_LARGE` | ``431`` | Additional HTTP Status Codes, | -| | | :rfc:`6585`, Section 5 | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`INTERNAL_SERVER_ERROR` | ``500`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.5.1 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.5.1>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`NOT_IMPLEMENTED` | ``501`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.5.2 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.5.2>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`BAD_GATEWAY` | ``502`` | HTTP/1.1 `RFC 2616, Section | -| | | 10.5.3 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.5.3>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`SERVICE_UNAVAILABLE` | ``503`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.5.4 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.5.4>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`GATEWAY_TIMEOUT` | ``504`` | HTTP/1.1 `RFC 2616, Section | -| | | 10.5.5 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.5.5>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`HTTP_VERSION_NOT_SUPPORTED` | ``505`` | HTTP/1.1, `RFC 2616, Section | -| | | 10.5.6 | -| | | <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.5.6>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`INSUFFICIENT_STORAGE` | ``507`` | WEBDAV, `RFC 2518, Section 10.6 | -| | | <http://www.webdav.org/specs/rfc2518.html#STATUS_507>`_ | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`NOT_EXTENDED` | ``510`` | An HTTP Extension Framework, | -| | | :rfc:`2774`, Section 7 | -+------------------------------------------+---------+-----------------------------------------------------------------------+ -| :const:`NETWORK_AUTHENTICATION_REQUIRED` | ``511`` | Additional HTTP Status Codes, | -| | | :rfc:`6585`, Section 6 | -+------------------------------------------+---------+-----------------------------------------------------------------------+ - -.. versionchanged:: 3.3 - Added codes ``428``, ``429``, ``431`` and ``511`` from :rfc:`6585`. - - .. data:: responses This dictionary maps the HTTP 1.1 status codes to the W3C names. Example: ``http.client.responses[http.client.NOT_FOUND]`` is ``'Not Found'``. +See :ref:`http-status-codes` for a list of HTTP status codes that are +available in this module as constants. + .. _httpconnection-objects: @@ -458,6 +262,11 @@ HTTPConnection Objects Note that you must have read the whole response before you can send a new request to the server. + .. versionchanged:: 3.5 + If a :exc:`ConnectionError` or subclass is raised, the + :class:`HTTPConnection` object will be ready to reconnect when + a new request is sent. + .. method:: HTTPConnection.set_debuglevel(level) @@ -496,7 +305,9 @@ HTTPConnection Objects .. method:: HTTPConnection.connect() - Connect to the server specified when the object was created. + Connect to the server specified when the object was created. By default, + this is called automatically when making a request if the client does not + already have a connection. .. method:: HTTPConnection.close() @@ -550,6 +361,10 @@ server. It provides access to the request headers and the entity body. The response is an iterable object and can be used in a with statement. +.. versionchanged:: 3.5 + The :class:`io.BufferedIOBase` interface is now implemented and + all of its reader operations are supported. + .. method:: HTTPResponse.read([amt]) @@ -625,7 +440,7 @@ Here is an example session that uses the ``GET`` method:: >>> conn.request("GET", "/") >>> r1 = conn.getresponse() >>> while not r1.closed: - ... print(r1.read(200)) # 200 bytes + ... print(r1.read(200)) # 200 bytes b'<!doctype html>\n<!--[if"... ... >>> # Example of an invalid request diff --git a/Doc/library/http.cookiejar.rst b/Doc/library/http.cookiejar.rst index ca68aacc92..5370601544 100644 --- a/Doc/library/http.cookiejar.rst +++ b/Doc/library/http.cookiejar.rst @@ -3,6 +3,7 @@ .. module:: http.cookiejar :synopsis: Classes for automatic handling of HTTP cookies. + .. moduleauthor:: John J. Lee <jjl@pobox.com> .. sectionauthor:: John J. Lee <jjl@pobox.com> @@ -115,7 +116,7 @@ The following classes are provided: :mod:`http.cookiejar` and :mod:`http.cookies` modules do not depend on each other. - http://curl.haxx.se/rfc/cookie_spec.html + https://curl.haxx.se/rfc/cookie_spec.html The specification of the original Netscape cookie protocol. Though this is still the dominant protocol, the 'Netscape cookie protocol' implemented by all the major browsers (and :mod:`http.cookiejar`) only bears a passing resemblance to diff --git a/Doc/library/http.cookies.rst b/Doc/library/http.cookies.rst index d0c1e54dbe..4b45d4bc38 100644 --- a/Doc/library/http.cookies.rst +++ b/Doc/library/http.cookies.rst @@ -3,6 +3,7 @@ .. module:: http.cookies :synopsis: Support for HTTP state management (cookies). + .. moduleauthor:: Timothy O'Malley <timo@alum.mit.edu> .. sectionauthor:: Moshe Zadka <moshez@zadka.site.co.il> @@ -141,28 +142,45 @@ Morsel Objects in HTTP requests, and is not accessible through JavaScript. This is intended to mitigate some forms of cross-site scripting. - The keys are case-insensitive. + The keys are case-insensitive and their default value is ``''``. + + .. versionchanged:: 3.5 + :meth:`~Morsel.__eq__` now takes :attr:`~Morsel.key` and :attr:`~Morsel.value` + into account. .. attribute:: Morsel.value The value of the cookie. + .. deprecated:: 3.5 + assigning to ``value``; use :meth:`~Morsel.set` instead. + .. attribute:: Morsel.coded_value The encoded value of the cookie --- this is what should be sent. + .. deprecated:: 3.5 + assigning to ``coded_value``; use :meth:`~Morsel.set` instead. + .. attribute:: Morsel.key The name of the cookie. + .. deprecated:: 3.5 + assigning to ``key``; use :meth:`~Morsel.set` instead. + .. method:: Morsel.set(key, value, coded_value) Set the *key*, *value* and *coded_value* attributes. + .. deprecated:: 3.5 + The undocumented *LegalChars* parameter is ignored and will be removed in + a future version. + .. method:: Morsel.isReservedKey(K) @@ -193,6 +211,30 @@ Morsel Objects The meaning for *attrs* is the same as in :meth:`output`. +.. method:: Morsel.update(values) + + Update the values in the Morsel dictionary with the values in the dictionary + *values*. Raise an error if any of the keys in the *values* dict is not a + valid :rfc:`2109` attribute. + + .. versionchanged:: 3.5 + an error is raised for invalid keys. + + +.. method:: Morsel.copy(value) + + Return a shallow copy of the Morsel object. + + .. versionchanged:: 3.5 + return a Morsel object instead of a dict. + + +.. method:: Morsel.setdefault(key, value=None) + + Raise an error if key is not a valid :rfc:`2109` attribute, otherwise + behave the same as :meth:`dict.setdefault`. + + .. _cookie-example: Example diff --git a/Doc/library/http.rst b/Doc/library/http.rst index a387a37ddd..be661c5e8c 100644 --- a/Doc/library/http.rst +++ b/Doc/library/http.rst @@ -1,7 +1,18 @@ :mod:`http` --- HTTP modules ============================ -``http`` is a package that collects several modules for working with the +.. module:: http + :synopsis: HTTP status codes and messages + +**Source code:** :source:`Lib/http/__init__.py` + +.. index:: + pair: HTTP; protocol + single: HTTP; http (standard module) + +-------------- + +:mod:`http` is a package that collects several modules for working with the HyperText Transfer Protocol: * :mod:`http.client` is a low-level HTTP protocol client; for high-level URL @@ -9,3 +20,105 @@ HyperText Transfer Protocol: * :mod:`http.server` contains basic HTTP server classes based on :mod:`socketserver` * :mod:`http.cookies` has utilities for implementing state management with cookies * :mod:`http.cookiejar` provides persistence of cookies + +:mod:`http` is also a module that defines a number of HTTP status codes and +associated messages through the :class:`http.HTTPStatus` enum: + +.. class:: HTTPStatus + + .. versionadded:: 3.5 + + A subclass of :class:`enum.IntEnum` that defines a set of HTTP status codes, + reason phrases and long descriptions written in English. + + Usage:: + + >>> from http import HTTPStatus + >>> HTTPStatus.OK + <HTTPStatus.OK: 200> + >>> HTTPStatus.OK == 200 + True + >>> http.HTTPStatus.OK.value + 200 + >>> HTTPStatus.OK.phrase + 'OK' + >>> HTTPStatus.OK.description + 'Request fulfilled, document follows' + >>> list(HTTPStatus) + [<HTTPStatus.CONTINUE: 100>, <HTTPStatus.SWITCHING_PROTOCOLS: 101>, ...] + +.. _http-status-codes: + +HTTP status codes +----------------- + +Supported, +`IANA-registered <https://www.iana.org/assignments/http-status-codes/http-status-codes.xhtml>`_ +status codes available in :class:`http.HTTPStatus` are: + +======= =================================== ================================================================== +Code Enum Name Details +======= =================================== ================================================================== +``100`` ``CONTINUE`` HTTP/1.1 :rfc:`7231`, Section 6.2.1 +``101`` ``SWITCHING_PROTOCOLS`` HTTP/1.1 :rfc:`7231`, Section 6.2.2 +``102`` ``PROCESSING`` WebDAV :rfc:`2518`, Section 10.1 +``200`` ``OK`` HTTP/1.1 :rfc:`7231`, Section 6.3.1 +``201`` ``CREATED`` HTTP/1.1 :rfc:`7231`, Section 6.3.2 +``202`` ``ACCEPTED`` HTTP/1.1 :rfc:`7231`, Section 6.3.3 +``203`` ``NON_AUTHORITATIVE_INFORMATION`` HTTP/1.1 :rfc:`7231`, Section 6.3.4 +``204`` ``NO_CONTENT`` HTTP/1.1 :rfc:`7231`, Section 6.3.5 +``205`` ``RESET_CONTENT`` HTTP/1.1 :rfc:`7231`, Section 6.3.6 +``206`` ``PARTIAL_CONTENT`` HTTP/1.1 :rfc:`7233`, Section 4.1 +``207`` ``MULTI_STATUS`` WebDAV :rfc:`4918`, Section 11.1 +``208`` ``ALREADY_REPORTED`` WebDAV Binding Extensions :rfc:`5842`, Section 7.1 (Experimental) +``226`` ``IM_USED`` Delta Encoding in HTTP :rfc:`3229`, Section 10.4.1 +``300`` ``MULTIPLE_CHOICES`` HTTP/1.1 :rfc:`7231`, Section 6.4.1 +``301`` ``MOVED_PERMANENTLY`` HTTP/1.1 :rfc:`7231`, Section 6.4.2 +``302`` ``FOUND`` HTTP/1.1 :rfc:`7231`, Section 6.4.3 +``303`` ``SEE_OTHER`` HTTP/1.1 :rfc:`7231`, Section 6.4.4 +``304`` ``NOT_MODIFIED`` HTTP/1.1 :rfc:`7232`, Section 4.1 +``305`` ``USE_PROXY`` HTTP/1.1 :rfc:`7231`, Section 6.4.5 +``307`` ``TEMPORARY_REDIRECT`` HTTP/1.1 :rfc:`7231`, Section 6.4.7 +``308`` ``PERMANENT_REDIRECT`` Permanent Redirect :rfc:`7238`, Section 3 (Experimental) +``400`` ``BAD_REQUEST`` HTTP/1.1 :rfc:`7231`, Section 6.5.1 +``401`` ``UNAUTHORIZED`` HTTP/1.1 Authentication :rfc:`7235`, Section 3.1 +``402`` ``PAYMENT_REQUIRED`` HTTP/1.1 :rfc:`7231`, Section 6.5.2 +``403`` ``FORBIDDEN`` HTTP/1.1 :rfc:`7231`, Section 6.5.3 +``404`` ``NOT_FOUND`` HTTP/1.1 :rfc:`7231`, Section 6.5.4 +``405`` ``METHOD_NOT_ALLOWED`` HTTP/1.1 :rfc:`7231`, Section 6.5.5 +``406`` ``NOT_ACCEPTABLE`` HTTP/1.1 :rfc:`7231`, Section 6.5.6 +``407`` ``PROXY_AUTHENTICATION_REQUIRED`` HTTP/1.1 Authentication :rfc:`7235`, Section 3.2 +``408`` ``REQUEST_TIMEOUT`` HTTP/1.1 :rfc:`7231`, Section 6.5.7 +``409`` ``CONFLICT`` HTTP/1.1 :rfc:`7231`, Section 6.5.8 +``410`` ``GONE`` HTTP/1.1 :rfc:`7231`, Section 6.5.9 +``411`` ``LENGTH_REQUIRED`` HTTP/1.1 :rfc:`7231`, Section 6.5.10 +``412`` ``PRECONDITION_FAILED`` HTTP/1.1 :rfc:`7232`, Section 4.2 +``413`` ``REQUEST_ENTITY_TOO_LARGE`` HTTP/1.1 :rfc:`7231`, Section 6.5.11 +``414`` ``REQUEST_URI_TOO_LONG`` HTTP/1.1 :rfc:`7231`, Section 6.5.12 +``415`` ``UNSUPPORTED_MEDIA_TYPE`` HTTP/1.1 :rfc:`7231`, Section 6.5.13 +``416`` ``REQUEST_RANGE_NOT_SATISFIABLE`` HTTP/1.1 Range Requests :rfc:`7233`, Section 4.4 +``417`` ``EXPECTATION_FAILED`` HTTP/1.1 :rfc:`7231`, Section 6.5.14 +``422`` ``UNPROCESSABLE_ENTITY`` WebDAV :rfc:`4918`, Section 11.2 +``423`` ``LOCKED`` WebDAV :rfc:`4918`, Section 11.3 +``424`` ``FAILED_DEPENDENCY`` WebDAV :rfc:`4918`, Section 11.4 +``426`` ``UPGRADE_REQUIRED`` HTTP/1.1 :rfc:`7231`, Section 6.5.15 +``428`` ``PRECONDITION_REQUIRED`` Additional HTTP Status Codes :rfc:`6585` +``429`` ``TOO_MANY_REQUESTS`` Additional HTTP Status Codes :rfc:`6585` +``431`` ``REQUEST_HEADER_FIELDS_TOO_LARGE`` Additional HTTP Status Codes :rfc:`6585` +``500`` ``INTERNAL_SERVER_ERROR`` HTTP/1.1 :rfc:`7231`, Section 6.6.1 +``501`` ``NOT_IMPLEMENTED`` HTTP/1.1 :rfc:`7231`, Section 6.6.2 +``502`` ``BAD_GATEWAY`` HTTP/1.1 :rfc:`7231`, Section 6.6.3 +``503`` ``SERVICE_UNAVAILABLE`` HTTP/1.1 :rfc:`7231`, Section 6.6.4 +``504`` ``GATEWAY_TIMEOUT`` HTTP/1.1 :rfc:`7231`, Section 6.6.5 +``505`` ``HTTP_VERSION_NOT_SUPPORTED`` HTTP/1.1 :rfc:`7231`, Section 6.6.6 +``506`` ``VARIANT_ALSO_NEGOTIATES`` Transparent Content Negotiation in HTTP :rfc:`2295`, Section 8.1 (Experimental) +``507`` ``INSUFFICIENT_STORAGE`` WebDAV :rfc:`4918`, Section 11.5 +``508`` ``LOOP_DETECTED`` WebDAV Binding Extensions :rfc:`5842`, Section 7.2 (Experimental) +``510`` ``NOT_EXTENDED`` An HTTP Extension Framework :rfc:`2774`, Section 7 (Experimental) +``511`` ``NETWORK_AUTHENTICATION_REQUIRED`` Additional HTTP Status Codes :rfc:`6585`, Section 6 +======= =================================== ================================================================== + +In order to preserve backwards compatibility, enum values are also present +in the :mod:`http.client` module in the form of constants. The enum name is +equal to the constant name (i.e. ``http.HTTPStatus.OK`` is also available as +``http.client.OK``). diff --git a/Doc/library/http.server.rst b/Doc/library/http.server.rst index 0bde35b02a..16c4fac947 100644 --- a/Doc/library/http.server.rst +++ b/Doc/library/http.server.rst @@ -4,6 +4,7 @@ .. module:: http.server :synopsis: HTTP server and request handlers. +**Source code:** :source:`Lib/http/server.py` .. index:: pair: WWW; server @@ -11,8 +12,6 @@ single: URL single: httpd -**Source code:** :source:`Lib/http/server.py` - -------------- This module defines classes for implementing HTTP servers (Web servers). @@ -97,7 +96,6 @@ of which this module provides three different variants: :mod:`http.client` is used to parse the headers and it requires that the HTTP request provide a valid :rfc:`2822` style header. - .. attribute:: rfile Contains an input stream, positioned at the start of the optional input @@ -109,7 +107,7 @@ of which this module provides three different variants: client. Proper adherence to the HTTP protocol must be used when writing to this stream. - :class:`BaseHTTPRequestHandler` has the following class variables: + :class:`BaseHTTPRequestHandler` has the following attributes: .. attribute:: server_version @@ -125,13 +123,10 @@ of which this module provides three different variants: .. attribute:: error_message_format - Specifies a format string for building an error response to the client. It - uses parenthesized, keyed format specifiers, so the format operand must be - a dictionary. The *code* key should be an integer, specifying the numeric - HTTP error code value. *message* should be a string containing a - (detailed) error message of what occurred, and *explain* should be an - explanation of the error code number. Default *message* and *explain* - values can found in the :attr:`responses` class variable. + Specifies a format string that should be used by :meth:`send_error` method + for building an error response to the client. The string is filled by + default with variables from :attr:`responses` based on the status code + that passed to :meth:`send_error`. .. attribute:: error_content_type @@ -154,11 +149,11 @@ of which this module provides three different variants: .. attribute:: responses - This variable contains a mapping of error code integers to two-element tuples + This attribute contains a mapping of error code integers to two-element tuples containing a short and long message. For example, ``{code: (shortmessage, longmessage)}``. The *shortmessage* is usually used as the *message* key in an - error response, and *longmessage* as the *explain* key (see the - :attr:`error_message_format` class variable). + error response, and *longmessage* as the *explain* key. It is used by + :meth:`send_response_only` and :meth:`send_error` methods. A :class:`BaseHTTPRequestHandler` instance has the following methods: @@ -191,17 +186,18 @@ of which this module provides three different variants: specifies the HTTP error code, with *message* as an optional, short, human readable description of the error. The *explain* argument can be used to provide more detailed information about the error; it will be formatted - using the :attr:`error_message_format` class variable and emitted, after + using the :attr:`error_message_format` attribute and emitted, after a complete set of headers, as the response body. The :attr:`responses` - class variable holds the default values for *message* and *explain* that + attribute holds the default values for *message* and *explain* that will be used if no value is provided; for unknown codes the default value - for both is the string ``???``. + for both is the string ``???``. The body will be empty if the method is + HEAD or the response code is one of the following: ``1xx``, + ``204 No Content``, ``205 Reset Content``, ``304 Not Modified``. .. versionchanged:: 3.4 The error response includes a Content-Length header. Added the *explain* argument. - .. method:: send_response(code, message=None) Adds a response header to the headers buffer and logs the accepted @@ -217,7 +213,6 @@ of which this module provides three different variants: Headers are stored to an internal buffer and :meth:`end_headers` needs to be called explicitly. - .. method:: send_header(keyword, value) Adds the HTTP header to an internal buffer which will be written to the @@ -229,7 +224,6 @@ of which this module provides three different variants: .. versionchanged:: 3.2 Headers are stored in an internal buffer. - .. method:: send_response_only(code, message=None) Sends the response header only, used for the purposes when ``100 @@ -279,7 +273,7 @@ of which this module provides three different variants: .. method:: version_string() Returns the server software's version string. This is a combination of the - :attr:`server_version` and :attr:`sys_version` class variables. + :attr:`server_version` and :attr:`sys_version` attributes. .. method:: date_time_string(timestamp=None) diff --git a/Doc/library/idle.rst b/Doc/library/idle.rst index 4384d56814..4a8c257ea2 100644 --- a/Doc/library/idle.rst +++ b/Doc/library/idle.rst @@ -3,12 +3,16 @@ IDLE ==== +.. moduleauthor:: Guido van Rossum <guido@python.org> + +**Source code:** :source:`Lib/idlelib/` + .. index:: single: IDLE single: Python Editor single: Integrated Development Environment -.. moduleauthor:: Guido van Rossum <guido@python.org> +-------------- IDLE is Python's Integrated Development and Learning Environment. @@ -128,7 +132,7 @@ Find Selection Search for the currently selected string, if there is one. Find in Files... - Open a file search dialog. Put results in an new output window. + Open a file search dialog. Put results in a new output window. Replace... Open a search-and-replace dialog. @@ -520,7 +524,7 @@ functions to be used from IDLE's Python shell. Command line usage ^^^^^^^^^^^^^^^^^^ -:: +.. code-block:: none idle.py [-c command] [-d] [-e] [-h] [-i] [-r file] [-s] [-t title] [-] [arg] ... @@ -550,14 +554,16 @@ IDLE-console differences As much as possible, the result of executing Python code with IDLE is the same as executing the same code in a console window. However, the different -interface and operation occasionally affects results. - -For instance, IDLE normally executes user code in a separate process from -the IDLE GUI itself. The IDLE versions of sys.stdin, .stdout, and .stderr in the -execution process get input from and send output to the GUI process, -which keeps control of the keyboard and screen. This is normally transparent, -but code that access these object will see different attribute values. -Also, functions that directly access the keyboard and screen will not work. +interface and operation occasionally affects visible results. For instance, +``sys.modules`` starts with more entries. + +IDLE also replaces ``sys.stdin``, ``sys.stdout``, and ``sys.stderr`` with +objects that get input from and send output to the Shell window. +When this window has the focus, it controls the keyboard and screen. +This is normally transparent, but functions that directly access the keyboard +and screen will not work. If ``sys`` is reset with ``importlib.reload(sys)``, +IDLE's changes are lost and things li ke ``input``, ``raw_input``, and +``print`` will not work correctly. With IDLE's Shell, one enters, edits, and recalls complete statements. Some consoles only work with a single physical line at a time. diff --git a/Doc/library/imaplib.rst b/Doc/library/imaplib.rst index fa736fe3af..c25e7d8611 100644 --- a/Doc/library/imaplib.rst +++ b/Doc/library/imaplib.rst @@ -3,6 +3,7 @@ .. module:: imaplib :synopsis: IMAP4 protocol client (requires sockets). + .. moduleauthor:: Piers Lauder <piers@communitysolutions.com.au> .. sectionauthor:: Piers Lauder <piers@communitysolutions.com.au> .. revised by ESR, January 2000 @@ -10,14 +11,13 @@ .. changes for IMAP4_stream by Piers Lauder <piers@communitysolutions.com.au>, November 2002 +**Source code:** :source:`Lib/imaplib.py` .. index:: pair: IMAP4; protocol pair: IMAP4_SSL; protocol pair: IMAP4_stream; protocol -**Source code:** :source:`Lib/imaplib.py` - -------------- This module defines three classes, :class:`IMAP4`, :class:`IMAP4_SSL` and @@ -37,6 +37,19 @@ base class: initialized. If *host* is not specified, ``''`` (the local host) is used. If *port* is omitted, the standard IMAP4 port (143) is used. + The :class:`IMAP4` class supports the :keyword:`with` statement. When used + like this, the IMAP4 ``LOGOUT`` command is issued automatically when the + :keyword:`with` statement exits. E.g.:: + + >>> from imaplib import IMAP4 + >>> with IMAP4("domain.org") as M: + ... M.noop() + ... + ('OK', [b'Nothing Accomplished. d25if65hy903weo.87']) + + .. versionchanged:: 3.5 + Support for the :keyword:`with` statement was added. + Three exceptions are defined as attributes of the :class:`IMAP4` class: @@ -64,7 +77,8 @@ Three exceptions are defined as attributes of the :class:`IMAP4` class: There's also a subclass for secure connections: -.. class:: IMAP4_SSL(host='', port=IMAP4_SSL_PORT, keyfile=None, certfile=None, ssl_context=None) +.. class:: IMAP4_SSL(host='', port=IMAP4_SSL_PORT, keyfile=None, \ + certfile=None, ssl_context=None) This is a subclass derived from :class:`IMAP4` that connects over an SSL encrypted socket (to use this class you need a socket module that was compiled @@ -143,7 +157,7 @@ example of usage. Documents describing the protocol, and sources and binaries for servers implementing it, can all be found at the University of Washington's *IMAP - Information Center* (http://www.washington.edu/imap/). + Information Center* (https://www.washington.edu/imap/). .. _imap4-objects: @@ -198,6 +212,10 @@ An :class:`IMAP4` instance has the following methods: that will be base64 encoded and sent to the server. It should return ``None`` if the client abort response ``*`` should be sent instead. + .. versionchanged:: 3.5 + string usernames and passwords are now encoded to ``utf-8`` instead of + being limited to ASCII. + .. method:: IMAP4.check() @@ -230,6 +248,16 @@ An :class:`IMAP4` instance has the following methods: Delete the ACLs (remove any rights) set for who on mailbox. +.. method:: IMAP4.enable(capability) + + Enable *capability* (see :rfc:`5161`). Most capabilities do not need to be + enabled. Currently only the ``UTF8=ACCEPT`` capability is supported + (see :RFC:`6855`). + + .. versionadded:: 3.5 + The :meth:`enable` method itself, and :RFC:`6855` support. + + .. method:: IMAP4.expunge() Permanently remove deleted items from selected mailbox. Generates an ``EXPUNGE`` @@ -367,7 +395,9 @@ An :class:`IMAP4` instance has the following methods: Search mailbox for matching messages. *charset* may be ``None``, in which case no ``CHARSET`` will be specified in the request to the server. The IMAP protocol requires that at least one criterion be specified; an exception will be - raised when the server returns an error. + raised when the server returns an error. *charset* must be ``None`` if + the ``UTF8=ACCEPT`` capability was enabled using the :meth:`enable` + command. Example:: @@ -529,6 +559,15 @@ The following attributes are defined on instances of :class:`IMAP4`: the module variable ``Debug``. Values greater than three trace each command. +.. attribute:: IMAP4.utf8_enabled + + Boolean value that is normally ``False``, but is set to ``True`` if an + :meth:`enable` command is successfully issued for the ``UTF8=ACCEPT`` + capability. + + .. versionadded:: 3.5 + + .. _imap4-example: IMAP4 Example diff --git a/Doc/library/imghdr.rst b/Doc/library/imghdr.rst index 9e8952339c..f11f6dcf8e 100644 --- a/Doc/library/imghdr.rst +++ b/Doc/library/imghdr.rst @@ -48,6 +48,14 @@ from :func:`what`: +------------+-----------------------------------+ | ``'png'`` | Portable Network Graphics | +------------+-----------------------------------+ +| ``'webp'`` | WebP files | ++------------+-----------------------------------+ +| ``'exr'`` | OpenEXR Files | ++------------+-----------------------------------+ + +.. versionadded:: 3.5 + The *exr* and *webp* formats were added. + You can extend the list of file types :mod:`imghdr` can recognize by appending to this variable: diff --git a/Doc/library/imp.rst b/Doc/library/imp.rst index 83a52e4e14..9828ba6fe2 100644 --- a/Doc/library/imp.rst +++ b/Doc/library/imp.rst @@ -5,11 +5,15 @@ :synopsis: Access the implementation of the import statement. :deprecated: +**Source code:** :source:`Lib/imp.py` + .. deprecated:: 3.4 The :mod:`imp` package is pending deprecation in favor of :mod:`importlib`. .. index:: statement: import +-------------- + This module provides an interface to the mechanisms used to implement the :keyword:`import` statement. It defines the following constants and functions: @@ -197,11 +201,9 @@ file paths. value would be ``/foo/bar/__pycache__/baz.cpython-32.pyc`` for Python 3.2. The ``cpython-32`` string comes from the current magic tag (see :func:`get_tag`; if :attr:`sys.implementation.cache_tag` is not defined then - :exc:`NotImplementedError` will be raised). The returned path will end in - ``.pyc`` when ``__debug__`` is ``True`` or ``.pyo`` for an optimized Python - (i.e. ``__debug__`` is ``False``). By passing in ``True`` or ``False`` for - *debug_override* you can override the system's value for ``__debug__`` for - extension selection. + :exc:`NotImplementedError` will be raised). By passing in ``True`` or + ``False`` for *debug_override* you can override the system's value for + ``__debug__``, leading to optimized bytecode. *path* need not exist. @@ -212,6 +214,9 @@ file paths. .. deprecated:: 3.4 Use :func:`importlib.util.cache_from_source` instead. + .. versionchanged:: 3.5 + The *debug_override* parameter no longer creates a ``.pyo`` file. + .. function:: source_from_cache(path) diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst index 42812f67a0..06e9ea3a55 100644 --- a/Doc/library/importlib.rst +++ b/Doc/library/importlib.rst @@ -9,6 +9,9 @@ .. versionadded:: 3.1 +**Source code:** :source:`Lib/importlib/__init__.py` + +-------------- Introduction ------------ @@ -52,9 +55,18 @@ generically as an :term:`importer`) to participate in the import process. :pep:`366` Main module explicit relative imports + :pep:`420` + Implicit namespace packages + :pep:`451` A ModuleSpec Type for the Import System + :pep:`488` + Elimination of PYO files + + :pep:`489` + Multi-phase extension module initialization + :pep:`3120` Using UTF-8 as the Default Source Encoding @@ -69,6 +81,10 @@ Functions An implementation of the built-in :func:`__import__` function. + .. note:: + Programmatic importing of modules should use :func:`import_module` + instead of this function. + .. function:: import_module(name, package=None) Import a module. The *name* argument specifies what module to @@ -81,12 +97,15 @@ Functions The :func:`import_module` function acts as a simplifying wrapper around :func:`importlib.__import__`. This means all semantics of the function are - derived from :func:`importlib.__import__`, including requiring the package - from which an import is occurring to have been previously imported - (i.e., *package* must already be imported). The most important difference - is that :func:`import_module` returns the specified package or module - (e.g. ``pkg.mod``), while :func:`__import__` returns the - top-level package or module (e.g. ``pkg``). + derived from :func:`importlib.__import__`. The most important difference + between these two functions is that :func:`import_module` returns the + specified package or module (e.g. ``pkg.mod``), while :func:`__import__` + returns the top-level package or module (e.g. ``pkg``). + + If you are dynamically importing a module that was created since the + interpreter began execution (e.g., created a Python source file), you may + need to call :func:`invalidate_caches` in order for the new module to be + noticed by the import system. .. versionchanged:: 3.3 Parent packages are automatically imported. @@ -192,6 +211,11 @@ Functions .. module:: importlib.abc :synopsis: Abstract base classes related to import +**Source code:** :source:`Lib/importlib/abc.py` + +-------------- + + The :mod:`importlib.abc` module contains all of the core abstract base classes used by :keyword:`import`. Some subclasses of the core abstract base classes are also provided to help in implementing the core ABCs. @@ -217,7 +241,7 @@ ABC hierarchy:: .. deprecated:: 3.3 Use :class:`MetaPathFinder` or :class:`PathEntryFinder` instead. - .. method:: find_module(fullname, path=None) + .. abstractmethod:: find_module(fullname, path=None) An abstact method for finding a :term:`loader` for the specified module. Originally specified in :pep:`302`, this method was meant @@ -341,13 +365,16 @@ ABC hierarchy:: .. method:: create_module(spec) - An optional method that returns the module object to use when - importing a module. create_module() may also return ``None``, - indicating that the default module creation should take place - instead. + A method that returns the module object to use when + importing a module. This method may return ``None``, + indicating that default module creation semantics should take place. .. versionadded:: 3.4 + .. versionchanged:: 3.5 + Starting in Python 3.6, this method will not be optional when + :meth:`exec_module` is defined. + .. method:: exec_module(module) An abstract method that executes the module in its own namespace @@ -411,7 +438,7 @@ ABC hierarchy:: .. deprecated:: 3.4 The recommended API for loading a module is :meth:`exec_module` - (and optionally :meth:`create_module`). Loaders should implement + (and :meth:`create_module`). Loaders should implement it instead of load_module(). The import machinery takes care of all the other responsibilities of load_module() when exec_module() is implemented. @@ -437,7 +464,7 @@ ABC hierarchy:: :pep:`302` protocol for loading arbitrary resources from the storage back-end. - .. method:: get_data(path) + .. abstractmethod:: get_data(path) An abstract method to return the bytes for the data located at *path*. Loaders that have a file-like storage back-end @@ -473,7 +500,7 @@ ABC hierarchy:: .. versionchanged:: 3.4 No longer abstract and a concrete implementation is provided. - .. method:: get_source(fullname) + .. abstractmethod:: get_source(fullname) An abstract method to return the source of a module. It is returned as a text string using :term:`universal newlines`, translating all @@ -493,7 +520,7 @@ ABC hierarchy:: .. versionchanged:: 3.4 Raises :exc:`ImportError` instead of :exc:`NotImplementedError`. - .. method:: source_to_code(data, path='<string>') + .. staticmethod:: source_to_code(data, path='<string>') Create a code object from Python source. @@ -502,8 +529,14 @@ ABC hierarchy:: the "path" to where the source code originated from, which can be an abstract concept (e.g. location in a zip file). + With the subsequent code object one can execute it in a module by + running ``exec(code, module.__dict__)``. + .. versionadded:: 3.4 + .. versionchanged:: 3.5 + Made the method static. + .. method:: exec_module(module) Implementation of :meth:`Loader.exec_module`. @@ -524,7 +557,7 @@ ABC hierarchy:: when implemented, helps a module to be executed as a script. The ABC represents an optional :pep:`302` protocol. - .. method:: get_filename(fullname) + .. abstractmethod:: get_filename(fullname) An abstract method that is to return the value of :attr:`__file__` for the specified module. If no path is available, :exc:`ImportError` is @@ -564,11 +597,11 @@ ABC hierarchy:: .. deprecated:: 3.4 Use :meth:`Loader.exec_module` instead. - .. method:: get_filename(fullname) + .. abstractmethod:: get_filename(fullname) Returns :attr:`path`. - .. method:: get_data(path) + .. abstractmethod:: get_data(path) Reads *path* as a binary file and returns the bytes from it. @@ -596,7 +629,7 @@ ABC hierarchy:: .. method:: path_stats(path) Optional abstract method which returns a :class:`dict` containing - metadata about the specifed path. Supported dictionary keys are: + metadata about the specified path. Supported dictionary keys are: - ``'mtime'`` (mandatory): an integer or floating-point number representing the modification time of the source code; @@ -672,6 +705,10 @@ ABC hierarchy:: .. module:: importlib.machinery :synopsis: Importers and path hooks +**Source code:** :source:`Lib/importlib/machinery.py` + +-------------- + This module contains the various objects that help :keyword:`import` find and load modules. @@ -689,6 +726,9 @@ find and load modules. .. versionadded:: 3.3 + .. deprecated:: 3.5 + Use :attr:`BYTECODE_SUFFIXES` instead. + .. attribute:: OPTIMIZED_BYTECODE_SUFFIXES A list of strings representing the file suffixes for optimized bytecode @@ -696,14 +736,19 @@ find and load modules. .. versionadded:: 3.3 + .. deprecated:: 3.5 + Use :attr:`BYTECODE_SUFFIXES` instead. + .. attribute:: BYTECODE_SUFFIXES A list of strings representing the recognized file suffixes for bytecode - modules. Set to either :attr:`DEBUG_BYTECODE_SUFFIXES` or - :attr:`OPTIMIZED_BYTECODE_SUFFIXES` based on whether ``__debug__`` is true. + modules (including the leading dot). .. versionadded:: 3.3 + .. versionchanged:: 3.5 + The value is no longer dependent on ``__debug__``. + .. attribute:: EXTENSION_SUFFIXES A list of strings representing the recognized file suffixes for @@ -732,9 +777,9 @@ find and load modules. Only class methods are defined by this class to alleviate the need for instantiation. - .. note:: - Due to limitations in the extension module C-API, for now - BuiltinImporter does not implement :meth:`Loader.exec_module`. + .. versionchanged:: 3.5 + As part of :pep:`489`, the builtin importer now implements + :meth:`Loader.create_module` and :meth:`Loader.exec_module` .. class:: FrozenImporter @@ -782,6 +827,11 @@ find and load modules. .. versionadded:: 3.4 + .. versionchanged:: 3.5 + If the current working directory -- represented by an empty string -- + is no longer valid then ``None`` is returned but no value is cached + in :data:`sys.path_importer_cache`. + .. classmethod:: find_module(fullname, path=None) A legacy wrapper around :meth:`find_spec`. @@ -944,14 +994,18 @@ find and load modules. Path to the extension module. - .. method:: load_module(name=None) + .. method:: create_module(spec) + + Creates the module object from the given specification in accordance + with :pep:`489`. - Loads the extension module if and only if *fullname* is the same as - :attr:`name` or is ``None``. + .. versionadded:: 3.5 - .. note:: - Due to limitations in the extension module C-API, for now - ExtensionFileLoader does not implement :meth:`Loader.exec_module`. + .. method:: exec_module(module) + + Initializes the given module object in accordance with :pep:`489`. + + .. versionadded:: 3.5 .. method:: is_package(fullname) @@ -1037,6 +1091,11 @@ find and load modules. .. module:: importlib.util :synopsis: Utility code for importers + +**Source code:** :source:`Lib/importlib/util.py` + +-------------- + This module contains the various objects that help in the construction of an :term:`importer`. @@ -1047,23 +1106,37 @@ an :term:`importer`. .. versionadded:: 3.4 -.. function:: cache_from_source(path, debug_override=None) +.. function:: cache_from_source(path, debug_override=None, *, optimization=None) - Return the :pep:`3147` path to the byte-compiled file associated with the - source *path*. For example, if *path* is ``/foo/bar/baz.py`` the return + Return the :pep:`3147`/:pep:`488` path to the byte-compiled file associated + with the source *path*. For example, if *path* is ``/foo/bar/baz.py`` the return value would be ``/foo/bar/__pycache__/baz.cpython-32.pyc`` for Python 3.2. The ``cpython-32`` string comes from the current magic tag (see :func:`get_tag`; if :attr:`sys.implementation.cache_tag` is not defined then - :exc:`NotImplementedError` will be raised). The returned path will end in - ``.pyc`` when ``__debug__`` is ``True`` or ``.pyo`` for an optimized Python - (i.e. ``__debug__`` is ``False``). By passing in ``True`` or ``False`` for - *debug_override* you can override the system's value for ``__debug__`` for - extension selection. - - *path* need not exist. + :exc:`NotImplementedError` will be raised). + + The *optimization* parameter is used to specify the optimization level of the + bytecode file. An empty string represents no optimization, so + ``/foo/bar/baz.py`` with an *optimization* of ``''`` will result in a + bytecode path of ``/foo/bar/__pycache__/baz.cpython-32.pyc``. ``None`` causes + the interpter's optimization level to be used. Any other value's string + representation being used, so ``/foo/bar/baz.py`` with an *optimization* of + ``2`` will lead to the bytecode path of + ``/foo/bar/__pycache__/baz.cpython-32.opt-2.pyc``. The string representation + of *optimization* can only be alphanumeric, else :exc:`ValueError` is raised. + + The *debug_override* parameter is deprecated and can be used to override + the system's value for ``__debug__``. A ``True`` value is the equivalent of + setting *optimization* to the empty string. A ``False`` value is the same as + setting *optimization* to ``1``. If both *debug_override* an *optimization* + are not ``None`` then :exc:`TypeError` is raised. .. versionadded:: 3.4 + .. versionchanged:: 3.5 + The *optimization* parameter was added and the *debug_override* parameter + was deprecated. + .. function:: source_from_cache(path) @@ -1071,7 +1144,7 @@ an :term:`importer`. file path. For example, if *path* is ``/foo/bar/__pycache__/baz.cpython-32.pyc`` the returned path would be ``/foo/bar/baz.py``. *path* need not exist, however if it does not conform - to :pep:`3147` format, a ``ValueError`` is raised. If + to :pep:`3147` or :pep:`488` format, a ``ValueError`` is raised. If :attr:`sys.implementation.cache_tag` is not defined, :exc:`NotImplementedError` is raised. @@ -1117,6 +1190,21 @@ an :term:`importer`. .. versionadded:: 3.4 +.. function:: module_from_spec(spec) + + Create a new module based on **spec** and ``spec.loader.create_module()``. + + If ``spec.loader.create_module()`` does not return ``None``, then any + pre-existing attributes will not be reset. Also, no :exc:`AttributeError` + will be raised if triggered while accessing **spec** or setting an attribute + on the module. + + This function is preferred over using :class:`types.ModuleType` to create a + new module as **spec** is used to set as many import-controlled attributes on + the module as possible. + + .. versionadded:: 3.5 + .. decorator:: module_for_loader A :term:`decorator` for :meth:`importlib.abc.Loader.load_module` @@ -1195,3 +1283,40 @@ an :term:`importer`. module will be file-based. .. versionadded:: 3.4 + +.. class:: LazyLoader(loader) + + A class which postpones the execution of the loader of a module until the + module has an attribute accessed. + + This class **only** works with loaders that define + :meth:`~importlib.abc.Loader.exec_module` as control over what module type + is used for the module is required. For those same reasons, the loader's + :meth:`~importlib.abc.Loader.create_module` method will be ignored (i.e., the + loader's method should only return ``None``; this excludes + :class:`BuiltinImporter` and :class:`ExtensionFileLoader`). Finally, + modules which substitute the object placed into :attr:`sys.modules` will + not work as there is no way to properly replace the module references + throughout the interpreter safely; :exc:`ValueError` is raised if such a + substitution is detected. + + .. note:: + For projects where startup time is critical, this class allows for + potentially minimizing the cost of loading a module if it is never used. + For projects where startup time is not essential then use of this class is + **heavily** discouraged due to error messages created during loading being + postponed and thus occurring out of context. + + .. versionadded:: 3.5 + + .. classmethod:: factory(loader) + + A static method which returns a callable that creates a lazy loader. This + is meant to be used in situations where the loader is passed by class + instead of by instance. + :: + + suffixes = importlib.machinery.SOURCE_SUFFIXES + loader = importlib.machinery.SourceFileLoader + lazy_loader = importlib.util.LazyLoader.factory(loader) + finder = importlib.machinery.FileFinder(path, (lazy_loader, suffixes)) diff --git a/Doc/library/inspect.rst b/Doc/library/inspect.rst index 57eb4fff82..8e7ed19f29 100644 --- a/Doc/library/inspect.rst +++ b/Doc/library/inspect.rst @@ -3,6 +3,7 @@ .. module:: inspect :synopsis: Extract information and source code from live objects. + .. moduleauthor:: Ka-Ping Yee <ping@lfw.org> .. sectionauthor:: Ka-Ping Yee <ping@lfw.org> @@ -28,7 +29,7 @@ Types and members ----------------- The :func:`getmembers` function retrieves the members of an object such as a -class or module. The sixteen functions whose names begin with "is" are mainly +class or module. The functions whose names begin with "is" are mainly provided as convenient choices for the second argument to :func:`getmembers`. They also help you determine when you can expect to find the following special attributes: @@ -88,6 +89,12 @@ attributes: | | __globals__ | global namespace in which | | | | this function was defined | +-----------+-----------------+---------------------------+ +| | __annotations__ | mapping of parameters | +| | | names to annotations; | +| | | ``"return"`` key is | +| | | reserved for return | +| | | annotations. | ++-----------+-----------------+---------------------------+ | traceback | tb_frame | frame object at this | | | | level | +-----------+-----------------+---------------------------+ @@ -168,6 +175,33 @@ attributes: | | | arguments and local | | | | variables | +-----------+-----------------+---------------------------+ +| generator | __name__ | name | ++-----------+-----------------+---------------------------+ +| | __qualname__ | qualified name | ++-----------+-----------------+---------------------------+ +| | gi_frame | frame | ++-----------+-----------------+---------------------------+ +| | gi_running | is the generator running? | ++-----------+-----------------+---------------------------+ +| | gi_code | code | ++-----------+-----------------+---------------------------+ +| | gi_yieldfrom | object being iterated by | +| | | ``yield from``, or | +| | | ``None`` | ++-----------+-----------------+---------------------------+ +| coroutine | __name__ | name | ++-----------+-----------------+---------------------------+ +| | __qualname__ | qualified name | ++-----------+-----------------+---------------------------+ +| | cr_await | object being awaited on, | +| | | or ``None`` | ++-----------+-----------------+---------------------------+ +| | cr_frame | frame | ++-----------+-----------------+---------------------------+ +| | cr_running | is the coroutine running? | ++-----------+-----------------+---------------------------+ +| | cr_code | code | ++-----------+-----------------+---------------------------+ | builtin | __doc__ | documentation string | +-----------+-----------------+---------------------------+ | | __name__ | original name of this | @@ -180,6 +214,13 @@ attributes: | | | ``None`` | +-----------+-----------------+---------------------------+ +.. versionchanged:: 3.5 + + Add ``__qualname__`` and ``gi_yieldfrom`` attributes to generators. + + The ``__name__`` attribute of generators is now set from the function + name, instead of the code name, and it can now be modified. + .. function:: getmembers(object[, predicate]) @@ -261,6 +302,41 @@ attributes: Return true if the object is a generator. +.. function:: iscoroutinefunction(object) + + Return true if the object is a :term:`coroutine function` + (a function defined with an :keyword:`async def` syntax). + + .. versionadded:: 3.5 + + +.. function:: iscoroutine(object) + + Return true if the object is a :term:`coroutine` created by an + :keyword:`async def` function. + + .. versionadded:: 3.5 + + +.. function:: isawaitable(object) + + Return true if the object can be used in :keyword:`await` expression. + + Can also be used to distinguish generator-based coroutines from regular + generators:: + + def gen(): + yield + @types.coroutine + def gen_coro(): + yield + + assert not isawaitable(gen()) + assert isawaitable(gen_coro()) + + .. versionadded:: 3.5 + + .. function:: istraceback(object) Return true if the object is a traceback. @@ -298,8 +374,9 @@ attributes: are true. This, for example, is true of ``int.__add__``. An object passing this test - has a :attr:`__get__` attribute but not a :attr:`__set__` attribute, but - beyond that the set of attributes varies. :attr:`__name__` is usually + has a :meth:`~object.__get__` method but not a :meth:`~object.__set__` + method, but beyond that the set of attributes varies. A + :attr:`~definition.__name__` attribute is usually sensible, and :attr:`__doc__` often is. Methods implemented via descriptors that also pass one of the other tests @@ -312,11 +389,11 @@ attributes: Return true if the object is a data descriptor. - Data descriptors have both a :attr:`__get__` and a :attr:`__set__` attribute. + Data descriptors have both a :attr:`~object.__get__` and a :attr:`~object.__set__` method. Examples are properties (defined in Python), getsets, and members. The latter two are defined in C and there are more specific tests available for those types, which is robust across Python implementations. Typically, data - descriptors will also have :attr:`__name__` and :attr:`__doc__` attributes + descriptors will also have :attr:`~definition.__name__` and :attr:`__doc__` attributes (properties, getsets, and members have both of these attributes), but this is not guaranteed. @@ -351,6 +428,12 @@ Retrieving source code .. function:: getdoc(object) Get the documentation string for an object, cleaned up with :func:`cleandoc`. + If the documentation string for an object is not provided and the object is + a class, a method, a property or a descriptor, retrieve the documentation + string from the inheritance hierarchy. + + .. versionchanged:: 3.5 + Documentation strings are now inherited if not overridden. .. function:: getcomments(object) @@ -408,8 +491,12 @@ Retrieving source code .. function:: cleandoc(doc) Clean up indentation from docstrings that are indented to line up with blocks - of code. Any whitespace that can be uniformly removed from the second line - onwards is removed. Also, all tabs are expanded to spaces. + of code. + + All leading whitespace is removed from the first line. Any leading whitespace + that can be uniformly removed from the second line onwards is removed. Empty + lines at the beginning and end are subsequently removed. Also, all tabs are + expanded to spaces. .. _inspect-signature-object: @@ -423,7 +510,7 @@ The Signature object represents the call signature of a callable object and its return annotation. To retrieve a Signature object, use the :func:`signature` function. -.. function:: signature(callable) +.. function:: signature(callable, \*, follow_wrapped=True) Return a :class:`Signature` object for the given ``callable``:: @@ -448,6 +535,11 @@ function. Raises :exc:`ValueError` if no signature can be provided, and :exc:`TypeError` if that type of object is not supported. + .. versionadded:: 3.5 + ``follow_wrapped`` parameter. Pass ``False`` to get a signature of + ``callable`` specifically (``callable.__wrapped__`` will not be used to + unwrap decorated callables.) + .. note:: Some callables may not be introspectable in certain implementations of @@ -473,6 +565,9 @@ function. Signature objects are *immutable*. Use :meth:`Signature.replace` to make a modified copy. + .. versionchanged:: 3.5 + Signature objects are picklable and hashable. + .. attribute:: Signature.empty A special class-level marker to specify absence of a return annotation. @@ -517,12 +612,30 @@ function. >>> str(new_sig) "(a, b) -> 'new return anno'" + .. classmethod:: Signature.from_callable(obj, \*, follow_wrapped=True) + + Return a :class:`Signature` (or its subclass) object for a given callable + ``obj``. Pass ``follow_wrapped=False`` to get a signature of ``obj`` + without unwrapping its ``__wrapped__`` chain. + + This method simplifies subclassing of :class:`Signature`:: + + class MySignature(Signature): + pass + sig = MySignature.from_callable(min) + assert isinstance(sig, MySignature) + + .. versionadded:: 3.5 + .. class:: Parameter(name, kind, \*, default=Parameter.empty, annotation=Parameter.empty) Parameter objects are *immutable*. Instead of modifying a Parameter object, you can use :meth:`Parameter.replace` to create a modified copy. + .. versionchanged:: 3.5 + Parameter objects are picklable and hashable. + .. attribute:: Parameter.empty A special class-level marker to specify absence of default values and @@ -639,27 +752,8 @@ function. Arguments for which :meth:`Signature.bind` or :meth:`Signature.bind_partial` relied on a default value are skipped. - However, if needed, it is easy to include them. - - :: - - >>> def foo(a, b=10): - ... pass - - >>> sig = signature(foo) - >>> ba = sig.bind(5) - - >>> ba.args, ba.kwargs - ((5,), {}) - - >>> for param in sig.parameters.values(): - ... if (param.name not in ba.arguments - ... and param.default is not param.empty): - ... ba.arguments[param.name] = param.default - - >>> ba.args, ba.kwargs - ((5, 10), {}) - + However, if needed, use :meth:`BoundArguments.apply_defaults` to add + them. .. attribute:: BoundArguments.args @@ -675,11 +769,31 @@ function. A reference to the parent :class:`Signature` object. + .. method:: BoundArguments.apply_defaults() + + Set default values for missing arguments. + + For variable-positional arguments (``*args``) the default is an + empty tuple. + + For variable-keyword arguments (``**kwargs``) the default is an + empty dict. + + :: + + >>> def foo(a, b='ham', *args): pass + >>> ba = inspect.signature(foo).bind('spam') + >>> ba.apply_defaults() + >>> ba.arguments + OrderedDict([('a', 'spam'), ('b', 'ham'), ('args', ())]) + + .. versionadded:: 3.5 + The :attr:`args` and :attr:`kwargs` properties can be used to invoke functions:: def test(a, *, b): - ... + ... sig = signature(test) ba = sig.bind(10, b=20) @@ -719,8 +833,9 @@ Classes and functions *n* elements listed in *args*. .. deprecated:: 3.0 - Use :func:`getfullargspec` instead, which provides information about - keyword-only arguments and annotations. + Use :func:`signature` and + :ref:`Signature Object <inspect-signature-object>`, which provide a + better introspecting API for callables. .. function:: getfullargspec(func) @@ -741,15 +856,16 @@ Classes and functions The first four items in the tuple correspond to :func:`getargspec`. - .. note:: - Consider using the new :ref:`Signature Object <inspect-signature-object>` - interface, which provides a better way of introspecting functions. - .. versionchanged:: 3.4 This function is now based on :func:`signature`, but still ignores ``__wrapped__`` attributes and includes the already bound first parameter in the signature output for bound methods. + .. deprecated:: 3.5 + Use :func:`signature` and + :ref:`Signature Object <inspect-signature-object>`, which provide a + better introspecting API for callables. + .. function:: getargvalues(frame) @@ -759,6 +875,11 @@ Classes and functions are the names of the ``*`` and ``**`` arguments or ``None``. *locals* is the locals dictionary of the given frame. + .. deprecated:: 3.5 + Use :func:`signature` and + :ref:`Signature Object <inspect-signature-object>`, which provide a + better introspecting API for callables. + .. function:: formatargspec(args[, varargs, varkw, defaults, kwonlyargs, kwonlydefaults, annotations[, formatarg, formatvarargs, formatvarkw, formatvalue, formatreturns, formatannotations]]) @@ -781,6 +902,11 @@ Classes and functions >>> formatargspec(*getfullargspec(f)) '(a: int, b: float)' + .. deprecated:: 3.5 + Use :func:`signature` and + :ref:`Signature Object <inspect-signature-object>`, which provide a + better introspecting API for callables. + .. function:: formatargvalues(args[, varargs, varkw, locals, formatarg, formatvarargs, formatvarkw, formatvalue]) @@ -788,6 +914,11 @@ Classes and functions :func:`getargvalues`. The format\* arguments are the corresponding optional formatting functions that are called to turn names and values into strings. + .. deprecated:: 3.5 + Use :func:`signature` and + :ref:`Signature Object <inspect-signature-object>`, which provide a + better introspecting API for callables. + .. function:: getmro(cls) @@ -822,8 +953,8 @@ Classes and functions .. versionadded:: 3.2 - .. note:: - Consider using the new :meth:`Signature.bind` instead. + .. deprecated:: 3.5 + Use :meth:`Signature.bind` and :meth:`Signature.bind_partial` instead. .. function:: getclosurevars(func) @@ -864,11 +995,17 @@ Classes and functions The interpreter stack --------------------- -When the following functions return "frame records," each record is a tuple of -six items: the frame object, the filename, the line number of the current line, +When the following functions return "frame records," each record is a +:term:`named tuple` +``FrameInfo(frame, filename, lineno, function, code_context, index)``. +The tuple contains the frame object, the filename, the line number of the +current line, the function name, a list of lines of context from the source code, and the index of the current line within that list. +.. versionchanged:: 3.5 + Return a named tuple instead of a tuple. + .. note:: Keeping references to frame objects, as found in the first element of the frame @@ -913,6 +1050,11 @@ line. returned list represents *frame*; the last entry represents the outermost call on *frame*'s stack. + .. versionchanged:: 3.5 + A list of :term:`named tuples <named tuple>` + ``FrameInfo(frame, filename, lineno, function, code_context, index)`` + is returned. + .. function:: getinnerframes(traceback, context=1) @@ -921,6 +1063,11 @@ line. list represents *traceback*; the last entry represents where the exception was raised. + .. versionchanged:: 3.5 + A list of :term:`named tuples <named tuple>` + ``FrameInfo(frame, filename, lineno, function, code_context, index)`` + is returned. + .. function:: currentframe() @@ -940,6 +1087,11 @@ line. returned list represents the caller; the last entry represents the outermost call on the stack. + .. versionchanged:: 3.5 + A list of :term:`named tuples <named tuple>` + ``FrameInfo(frame, filename, lineno, function, code_context, index)`` + is returned. + .. function:: trace(context=1) @@ -948,6 +1100,11 @@ line. entry in the list represents the caller; the last entry represents where the exception was raised. + .. versionchanged:: 3.5 + A list of :term:`named tuples <named tuple>` + ``FrameInfo(frame, filename, lineno, function, code_context, index)`` + is returned. + Fetching attributes statically ------------------------------ @@ -1007,8 +1164,8 @@ code execution:: pass -Current State of a Generator ----------------------------- +Current State of Generators and Coroutines +------------------------------------------ When implementing coroutine schedulers and for other advanced uses of generators, it is useful to determine whether a generator is currently @@ -1028,6 +1185,21 @@ generator to be determined easily. .. versionadded:: 3.2 +.. function:: getcoroutinestate(coroutine) + + Get current state of a coroutine object. The function is intended to be + used with coroutine objects created by :keyword:`async def` functions, but + will accept any coroutine-like object that has ``cr_running`` and + ``cr_frame`` attributes. + + Possible states are: + * CORO_CREATED: Waiting to start execution. + * CORO_RUNNING: Currently being executed by the interpreter. + * CORO_SUSPENDED: Currently suspended at an await expression. + * CORO_CLOSED: Execution has completed. + + .. versionadded:: 3.5 + The current internal state of the generator can also be queried. This is mostly useful for testing purposes, to ensure that internal state is being updated as expected: @@ -1052,6 +1224,13 @@ updated as expected: .. versionadded:: 3.3 +.. function:: getcoroutinelocals(coroutine) + + This function is analogous to :func:`~inspect.getgeneratorlocals`, but + works for coroutine objects created by :keyword:`async def` functions. + + .. versionadded:: 3.5 + .. _inspect-module-cli: diff --git a/Doc/library/io.rst b/Doc/library/io.rst index 592bc484ad..23df18f8e7 100644 --- a/Doc/library/io.rst +++ b/Doc/library/io.rst @@ -3,6 +3,7 @@ .. module:: io :synopsis: Core tools for working with streams. + .. moduleauthor:: Guido van Rossum <guido@python.org> .. moduleauthor:: Mike Verdone <mike.verdone@gmail.com> .. moduleauthor:: Mark Russell <mark.russell@zen.co.uk> @@ -11,6 +12,10 @@ .. moduleauthor:: Benjamin Peterson <benjamin@python.org> .. sectionauthor:: Benjamin Peterson <benjamin@python.org> +**Source code:** :source:`Lib/io.py` + +-------------- + .. _io-overview: Overview @@ -66,7 +71,8 @@ The text stream API is described in detail in the documentation of Binary I/O ^^^^^^^^^^ -Binary I/O (also called *buffered I/O*) expects and produces :class:`bytes` +Binary I/O (also called *buffered I/O*) expects +:term:`bytes-like objects <bytes-like object>` and produces :class:`bytes` objects. No encoding, decoding, or newline translation is performed. This category of streams can be used for all kinds of non-text data, and also when manual control over the handling of text data is desired. @@ -130,7 +136,7 @@ High-level Module Interface In-memory streams ^^^^^^^^^^^^^^^^^ -It is also possible to use a :class:`str` or :class:`bytes`-like object as a +It is also possible to use a :class:`str` or :term:`bytes-like object` as a file for both reading and writing. For strings :class:`StringIO` can be used like a file opened in text mode. :class:`BytesIO` can be used like a file opened in binary mode. Both provide full read-write capabilities with random @@ -227,9 +233,10 @@ I/O Base Classes when operations they do not support are called. The basic type used for binary data read from or written to a file is - :class:`bytes`. :class:`bytearray`\s are accepted too, and in some cases - (such as :meth:`readinto`) required. Text I/O classes work with - :class:`str` data. + :class:`bytes`. Other :term:`bytes-like objects <bytes-like object>` are + accepted as method arguments too. In some cases, such as + :meth:`~RawIOBase.readinto`, a writable object such as :class:`bytearray` + is required. Text I/O classes work with :class:`str` data. Note that calling any method (even inquiries) on a closed stream is undefined. Implementations may raise :exc:`ValueError` in this case. @@ -339,8 +346,11 @@ I/O Base Classes if *size* is not specified). The current stream position isn't changed. This resizing can extend or reduce the current file size. In case of extension, the contents of the new file area depend on the platform - (on most systems, additional bytes are zero-filled, on Windows they're - undetermined). The new file size is returned. + (on most systems, additional bytes are zero-filled). The new file size + is returned. + + .. versionchanged:: 3.5 + Windows will now zero-fill files when extending. .. method:: writable() @@ -390,18 +400,22 @@ I/O Base Classes .. method:: readinto(b) - Read up to ``len(b)`` bytes into :class:`bytearray` *b* and return the - number of bytes read. If the object is in non-blocking mode and no - bytes are available, ``None`` is returned. + Read bytes into a pre-allocated, writable + :term:`bytes-like object` *b*, and return the + number of bytes read. If the object is in non-blocking mode and no bytes + are available, ``None`` is returned. .. method:: write(b) - Write the given :class:`bytes` or :class:`bytearray` object, *b*, to the - underlying raw stream and return the number of bytes written. This can - be less than ``len(b)``, depending on specifics of the underlying raw + Write the given :term:`bytes-like object`, *b*, to the + underlying raw stream, and return the number of + bytes written. This can be less than the length of *b* in + bytes, depending on specifics of the underlying raw stream, and especially if it is in non-blocking mode. ``None`` is returned if the raw stream is set not to block and no single byte could - be readily written to it. + be readily written to it. The caller may release or mutate *b* after + this method returns, so the implementation should only access *b* + during the method call. .. class:: BufferedIOBase @@ -465,26 +479,39 @@ I/O Base Classes .. method:: read1(size=-1) - Read and return up to *size* bytes, with at most one call to the underlying - raw stream's :meth:`~RawIOBase.read` method. This can be useful if you - are implementing your own buffering on top of a :class:`BufferedIOBase` + Read and return up to *size* bytes, with at most one call to the + underlying raw stream's :meth:`~RawIOBase.read` (or + :meth:`~RawIOBase.readinto`) method. This can be useful if you are + implementing your own buffering on top of a :class:`BufferedIOBase` object. .. method:: readinto(b) - Read up to ``len(b)`` bytes into bytearray *b* and return the number of - bytes read. + Read bytes into a pre-allocated, writable + :term:`bytes-like object` *b* and return the number of bytes read. Like :meth:`read`, multiple reads may be issued to the underlying raw stream, unless the latter is interactive. - A :exc:`BlockingIOError` is raised if the underlying raw stream is in - non blocking-mode, and has no data available at the moment. + A :exc:`BlockingIOError` is raised if the underlying raw stream is in non + blocking-mode, and has no data available at the moment. + + .. method:: readinto1(b) + + Read bytes into a pre-allocated, writable + :term:`bytes-like object` *b*, using at most one call to + the underlying raw stream's :meth:`~RawIOBase.read` (or + :meth:`~RawIOBase.readinto`) method. Return the number of bytes read. + + A :exc:`BlockingIOError` is raised if the underlying raw stream is in non + blocking-mode, and has no data available at the moment. + + .. versionadded:: 3.5 .. method:: write(b) - Write the given :class:`bytes` or :class:`bytearray` object, *b* and - return the number of bytes written (never less than ``len(b)``, since if + Write the given :term:`bytes-like object`, *b*, and return the number + of bytes written (always equal to the length of *b* in bytes, since if the write fails an :exc:`OSError` will be raised). Depending on the actual implementation, these bytes may be readily written to the underlying stream, or held in a buffer for performance and latency @@ -494,6 +521,9 @@ I/O Base Classes data needed to be written to the raw stream but it couldn't accept all the data without blocking. + The caller may release or mutate *b* after this method returns, + so the implementation should only access *b* during the method call. + Raw File I/O ^^^^^^^^^^^^ @@ -507,9 +537,12 @@ Raw File I/O The *name* can be one of two things: * a character string or :class:`bytes` object representing the path to the - file which will be opened; + file which will be opened. In this case closefd must be True (the default) + otherwise an error will be raised. * an integer representing the number of an existing OS-level file descriptor - to which the resulting :class:`FileIO` object will give access. + to which the resulting :class:`FileIO` object will give access. When the + FileIO object is closed this fd will be closed as well, unless *closefd* + is set to ``False``. The *mode* can be ``'r'``, ``'w'``, ``'x'`` or ``'a'`` for reading (default), writing, exclusive creation or appending. The file will be @@ -566,7 +599,8 @@ than raw I/O does. :class:`BufferedIOBase`. The buffer is discarded when the :meth:`~IOBase.close` method is called. - The argument *initial_bytes* contains optional initial :class:`bytes` data. + The optional argument *initial_bytes* is a :term:`bytes-like object` that + contains initial data. :class:`BytesIO` provides or overrides these methods in addition to those from :class:`BufferedIOBase` and :class:`IOBase`: @@ -598,6 +632,11 @@ than raw I/O does. In :class:`BytesIO`, this is the same as :meth:`read`. + .. method:: readinto1() + + In :class:`BytesIO`, this is the same as :meth:`readinto`. + + .. versionadded:: 3.5 .. class:: BufferedReader(raw, buffer_size=DEFAULT_BUFFER_SIZE) @@ -659,7 +698,7 @@ than raw I/O does. .. method:: write(b) - Write the :class:`bytes` or :class:`bytearray` object, *b* and return the + Write the :term:`bytes-like object`, *b*, and return the number of bytes written. When in non-blocking mode, a :exc:`BlockingIOError` is raised if the buffer needs to be written out but the raw stream blocks. @@ -808,11 +847,13 @@ Text I/O exception if there is an encoding error (the default of ``None`` has the same effect), or pass ``'ignore'`` to ignore errors. (Note that ignoring encoding errors can lead to data loss.) ``'replace'`` causes a replacement marker - (such as ``'?'``) to be inserted where there is malformed data. When - writing, ``'xmlcharrefreplace'`` (replace with the appropriate XML character - reference) or ``'backslashreplace'`` (replace with backslashed escape - sequences) can be used. Any other error handling name that has been - registered with :func:`codecs.register_error` is also valid. + (such as ``'?'``) to be inserted where there is malformed data. + ``'backslashreplace'`` causes malformed data to be replaced by a + backslashed escape sequence. When writing, ``'xmlcharrefreplace'`` + (replace with the appropriate XML character reference) or ``'namereplace'`` + (replace with ``\N{...}`` escape sequences) can be used. Any other error + handling name that has been registered with + :func:`codecs.register_error` is also valid. .. index:: single: universal newlines; io.TextIOWrapper class diff --git a/Doc/library/ipaddress.rst b/Doc/library/ipaddress.rst index 301048ef6a..23526b6c42 100644 --- a/Doc/library/ipaddress.rst +++ b/Doc/library/ipaddress.rst @@ -3,6 +3,7 @@ .. module:: ipaddress :synopsis: IPv4/IPv6 manipulation library. + .. moduleauthor:: Peter Moody **Source code:** :source:`Lib/ipaddress.py` @@ -146,6 +147,20 @@ write code that handles both IP versions correctly. the appropriate length (most significant octet first). This is 4 bytes for IPv4 and 16 bytes for IPv6. + .. attribute:: reverse_pointer + + The name of the reverse DNS PTR record for the IP address, e.g.:: + + >>> ipaddress.ip_address("127.0.0.1").reverse_pointer + '1.0.0.127.in-addr.arpa' + >>> ipaddress.ip_address("2001:db8::1").reverse_pointer + '1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.b.d.0.1.0.0.2.ip6.arpa' + + This is the name that could be used for performing a PTR lookup, not the + resolved hostname itself. + + .. versionadded:: 3.5 + .. attribute:: is_multicast ``True`` if the address is reserved for multicast use. See @@ -184,8 +199,8 @@ write code that handles both IP versions correctly. ``True`` if the address is reserved for link-local usage. See :RFC:`3927`. -.. _iana-ipv4-special-registry: http://www.iana.org/assignments/iana-ipv4-special-registry/iana-ipv4-special-registry.xhtml -.. _iana-ipv6-special-registry: http://www.iana.org/assignments/iana-ipv6-special-registry/iana-ipv6-special-registry.xhtml +.. _iana-ipv4-special-registry: https://www.iana.org/assignments/iana-ipv4-special-registry/iana-ipv4-special-registry.xhtml +.. _iana-ipv6-special-registry: https://www.iana.org/assignments/iana-ipv6-special-registry/iana-ipv6-special-registry.xhtml .. class:: IPv6Address(address) @@ -226,6 +241,7 @@ write code that handles both IP versions correctly. :class:`IPv4Address` class: .. attribute:: packed + .. attribute:: reverse_pointer .. attribute:: version .. attribute:: max_prefixlen .. attribute:: is_multicast @@ -377,6 +393,12 @@ so to avoid duplication they are only documented for :class:`IPv4Network`. 3. An integer packed into a :class:`bytes` object of length 4, big-endian. The interpretation is similar to an integer *address*. + 4. A two-tuple of an address description and a netmask, where the address + description is either a string, a 32-bits integer, a 4-bytes packed + integer, or an existing IPv4Address object; and the netmask is either + an integer representing the prefix length (e.g. ``24``) or a string + representing the prefix mask (e.g. ``255.255.255.0``). + An :exc:`AddressValueError` is raised if *address* is not a valid IPv4 address. A :exc:`NetmaskValueError` is raised if the mask is not valid for an IPv4 address. @@ -389,6 +411,10 @@ so to avoid duplication they are only documented for :class:`IPv4Network`. objects will raise :exc:`TypeError` if the argument's IP version is incompatible to ``self`` + .. versionchanged:: 3.5 + + Added the two-tuple form for the *address* constructor parameter. + .. attribute:: version .. attribute:: max_prefixlen @@ -553,6 +579,11 @@ so to avoid duplication they are only documented for :class:`IPv4Network`. 3. An integer packed into a :class:`bytes` object of length 16, big-endian. The interpretation is similar to an integer *address*. + 4. A two-tuple of an address description and a netmask, where the address + description is either a string, a 128-bits integer, a 16-bytes packed + integer, or an existing IPv6Address object; and the netmask is an + integer representing the prefix length. + An :exc:`AddressValueError` is raised if *address* is not a valid IPv6 address. A :exc:`NetmaskValueError` is raised if the mask is not valid for an IPv6 address. @@ -561,6 +592,10 @@ so to avoid duplication they are only documented for :class:`IPv4Network`. then :exc:`ValueError` is raised. Otherwise, the host bits are masked out to determine the appropriate network address. + .. versionchanged:: 3.5 + + Added the two-tuple form for the *address* constructor parameter. + .. attribute:: version .. attribute:: max_prefixlen .. attribute:: is_multicast @@ -619,7 +654,7 @@ network. For iteration, *all* hosts are returned, including unusable hosts example:: >>> for addr in IPv4Network('192.0.2.0/28'): - ... addr + ... addr ... IPv4Address('192.0.2.0') IPv4Address('192.0.2.1') diff --git a/Doc/library/itertools.rst b/Doc/library/itertools.rst index f489535c53..dfc1ddc2d7 100644 --- a/Doc/library/itertools.rst +++ b/Doc/library/itertools.rst @@ -3,14 +3,15 @@ .. module:: itertools :synopsis: Functions creating iterators for efficient looping. + .. moduleauthor:: Raymond Hettinger <python@rcn.com> .. sectionauthor:: Raymond Hettinger <python@rcn.com> - .. testsetup:: from itertools import * +-------------- This module implements a number of :term:`iterator` building blocks inspired by constructs from APL, Haskell, and SML. Each has been recast in a form @@ -87,19 +88,27 @@ loops that truncate the stream. .. function:: accumulate(iterable[, func]) - Make an iterator that returns accumulated sums. Elements may be any addable - type including :class:`~decimal.Decimal` or :class:`~fractions.Fraction`. - If the optional *func* argument is supplied, it should be a function of two - arguments and it will be used instead of addition. + Make an iterator that returns accumulated sums, or accumulated + results of other binary functions (specified via the optional + *func* argument). If *func* is supplied, it should be a function + of two arguments. Elements of the input *iterable* may be any type + that can be accepted as arguments to *func*. (For example, with + the default operation of addition, elements may be any addable + type including :class:`~decimal.Decimal` or + :class:`~fractions.Fraction`.) If the input iterable is empty, the + output iterable will also be empty. - Equivalent to:: + Roughly equivalent to:: def accumulate(iterable, func=operator.add): 'Return running totals' # accumulate([1,2,3,4,5]) --> 1 3 6 10 15 # accumulate([1,2,3,4,5], operator.mul) --> 1 2 6 24 120 it = iter(iterable) - total = next(it) + try: + total = next(it) + except StopIteration: + return yield total for element in it: total = func(total, element) @@ -109,7 +118,7 @@ loops that truncate the stream. :func:`min` for a running minimum, :func:`max` for a running maximum, or :func:`operator.mul` for a running product. Amortization tables can be built by accumulating interest and applying payments. First-order - `recurrence relations <http://en.wikipedia.org/wiki/Recurrence_relation>`_ + `recurrence relations <https://en.wikipedia.org/wiki/Recurrence_relation>`_ can be modeled by supplying the initial value in the iterable and using only the accumulated total in *func* argument:: @@ -124,7 +133,7 @@ loops that truncate the stream. >>> list(accumulate(cashflows, lambda bal, pmt: bal*1.05 + pmt)) [1000, 960.0, 918.0, 873.9000000000001, 827.5950000000001] - # Chaotic recurrence relation http://en.wikipedia.org/wiki/Logistic_map + # Chaotic recurrence relation https://en.wikipedia.org/wiki/Logistic_map >>> logistic_map = lambda x, _: r * x * (1 - x) >>> r = 3.8 >>> x0 = 0.4 @@ -148,7 +157,7 @@ loops that truncate the stream. Make an iterator that returns elements from the first iterable until it is exhausted, then proceeds to the next iterable, until all of the iterables are exhausted. Used for treating consecutive sequences as a single sequence. - Equivalent to:: + Roughly equivalent to:: def chain(*iterables): # chain('ABC', 'DEF') --> A B C D E F @@ -181,7 +190,7 @@ loops that truncate the stream. value. So if the input elements are unique, there will be no repeat values in each combination. - Equivalent to:: + Roughly equivalent to:: def combinations(iterable, r): # combinations('ABCD', 2) --> AB AC AD BC BD CD @@ -230,7 +239,7 @@ loops that truncate the stream. value. So if the input elements are unique, the generated combinations will also be unique. - Equivalent to:: + Roughly equivalent to:: def combinations_with_replacement(iterable, r): # combinations_with_replacement('ABC', 2) --> AA AB AC BB BC CC @@ -270,7 +279,7 @@ loops that truncate the stream. Make an iterator that filters elements from *data* returning only those that have a corresponding element in *selectors* that evaluates to ``True``. Stops when either the *data* or *selectors* iterables has been exhausted. - Equivalent to:: + Roughly equivalent to:: def compress(data, selectors): # compress('ABCDEF', [1,0,1,0,1,1]) --> A C E F @@ -283,7 +292,7 @@ loops that truncate the stream. Make an iterator that returns evenly spaced values starting with number *start*. Often used as an argument to :func:`map` to generate consecutive data points. - Also, used with :func:`zip` to add sequence numbers. Equivalent to:: + Also, used with :func:`zip` to add sequence numbers. Roughly equivalent to:: def count(start=0, step=1): # count(10) --> 10 11 12 13 14 ... @@ -304,7 +313,7 @@ loops that truncate the stream. Make an iterator returning elements from the iterable and saving a copy of each. When the iterable is exhausted, return elements from the saved copy. Repeats - indefinitely. Equivalent to:: + indefinitely. Roughly equivalent to:: def cycle(iterable): # cycle('ABCD') --> A B C D A B C D A B C D ... @@ -325,7 +334,7 @@ loops that truncate the stream. Make an iterator that drops elements from the iterable as long as the predicate is true; afterwards, returns every element. Note, the iterator does not produce *any* output until the predicate first becomes false, so it may have a lengthy - start-up time. Equivalent to:: + start-up time. Roughly equivalent to:: def dropwhile(predicate, iterable): # dropwhile(lambda x: x<5, [1,4,6,4,1]) --> 6 4 1 @@ -341,7 +350,7 @@ loops that truncate the stream. Make an iterator that filters elements from iterable returning only those for which the predicate is ``False``. If *predicate* is ``None``, return the items - that are false. Equivalent to:: + that are false. Roughly equivalent to:: def filterfalse(predicate, iterable): # filterfalse(lambda x: x%2, range(10)) --> 0 2 4 6 8 @@ -378,7 +387,7 @@ loops that truncate the stream. groups.append(list(g)) # Store group iterator as a list uniquekeys.append(k) - :func:`groupby` is equivalent to:: + :func:`groupby` is roughly equivalent to:: class groupby: # [k for k, g in groupby('AAAABBBCCDAABBB')] --> A B C D A B @@ -400,7 +409,10 @@ loops that truncate the stream. def _grouper(self, tgtkey): while self.currkey == tgtkey: yield self.currvalue - self.currvalue = next(self.it) # Exit on StopIteration + try: + self.currvalue = next(self.it) + except StopIteration: + return self.currkey = self.keyfunc(self.currvalue) @@ -415,7 +427,7 @@ loops that truncate the stream. specified position. Unlike regular slicing, :func:`islice` does not support negative values for *start*, *stop*, or *step*. Can be used to extract related fields from data where the internal structure has been flattened (for example, a - multi-line report may list a name field on every third line). Equivalent to:: + multi-line report may list a name field on every third line). Roughly equivalent to:: def islice(iterable, *args): # islice('ABCDEFG', 2) --> A B @@ -424,7 +436,10 @@ loops that truncate the stream. # islice('ABCDEFG', 0, None, 2) --> A C E G s = slice(*args) it = iter(range(s.start or 0, s.stop or sys.maxsize, s.step or 1)) - nexti = next(it) + try: + nexti = next(it) + except StopIteration: + return for i, element in enumerate(iterable): if i == nexti: yield element @@ -450,7 +465,7 @@ loops that truncate the stream. value. So if the input elements are unique, there will be no repeat values in each permutation. - Equivalent to:: + Roughly equivalent to:: def permutations(iterable, r=None): # permutations('ABCD', 2) --> AB AC AD BA BC BD CA CB CD DA DB DC @@ -496,7 +511,7 @@ loops that truncate the stream. Cartesian product of input iterables. - Equivalent to nested for-loops in a generator expression. For example, + Roughly equivalent to nested for-loops in a generator expression. For example, ``product(A, B)`` returns the same as ``((x,y) for x in A for y in B)``. The nested loops cycle like an odometer with the rightmost element advancing @@ -508,7 +523,7 @@ loops that truncate the stream. repetitions with the optional *repeat* keyword argument. For example, ``product(A, repeat=4)`` means the same as ``product(A, A, A, A)``. - This function is equivalent to the following code, except that the + This function is roughly equivalent to the following code, except that the actual implementation does not build up intermediate results in memory:: def product(*args, repeat=1): @@ -527,7 +542,9 @@ loops that truncate the stream. Make an iterator that returns *object* over and over again. Runs indefinitely unless the *times* argument is specified. Used as argument to :func:`map` for invariant parameters to the called function. Also used with :func:`zip` to - create an invariant part of a tuple record. Equivalent to:: + create an invariant part of a tuple record. + + Roughly equivalent to:: def repeat(object, times=None): # repeat(10, 3) --> 10 10 10 @@ -550,7 +567,7 @@ loops that truncate the stream. the iterable. Used instead of :func:`map` when argument parameters are already grouped in tuples from a single iterable (the data has been "pre-zipped"). The difference between :func:`map` and :func:`starmap` parallels the distinction - between ``function(a,b)`` and ``function(*c)``. Equivalent to:: + between ``function(a,b)`` and ``function(*c)``. Roughly equivalent to:: def starmap(function, iterable): # starmap(pow, [(2,5), (3,2), (10,3)]) --> 32 9 1000 @@ -561,7 +578,7 @@ loops that truncate the stream. .. function:: takewhile(predicate, iterable) Make an iterator that returns elements from the iterable as long as the - predicate is true. Equivalent to:: + predicate is true. Roughly equivalent to:: def takewhile(predicate, iterable): # takewhile(lambda x: x<5, [1,4,6,4,1]) --> 1 4 @@ -574,7 +591,7 @@ loops that truncate the stream. .. function:: tee(iterable, n=2) - Return *n* independent iterators from a single iterable. Equivalent to:: + Return *n* independent iterators from a single iterable. Roughly equivalent to:: def tee(iterable, n=2): it = iter(iterable) @@ -582,7 +599,10 @@ loops that truncate the stream. def gen(mydeque): while True: if not mydeque: # when the local deque is empty - newval = next(it) # fetch a new value and + try: + newval = next(it) # fetch a new value and + except StopIteration: + return for d in deques: # load it to all the deques d.append(newval) yield mydeque.popleft() @@ -602,7 +622,7 @@ loops that truncate the stream. Make an iterator that aggregates elements from each of the iterables. If the iterables are of uneven length, missing values are filled-in with *fillvalue*. - Iteration continues until the longest iterable is exhausted. Equivalent to:: + Iteration continues until the longest iterable is exhausted. Roughly equivalent to:: class ZipExhausted(Exception): pass @@ -657,6 +677,11 @@ which incur interpreter overhead. "Return function(0), function(1), ..." return map(function, count(start)) + def tail(n, iterable): + "Return an iterator over the last n items" + # tail(3, 'ABCDEFG') --> E F G + return iter(collections.deque(iterable, maxlen=n)) + def consume(iterator, n): "Advance the iterator n-steps ahead. If n is none, consume entirely." # Use functions that consume iterators at C speed. @@ -671,6 +696,11 @@ which incur interpreter overhead. "Returns the nth item or a default value" return next(islice(iterable, n, None), default) + def all_equal(iterable): + "Returns True if all the elements are equal to each other" + g = groupby(iterable) + return next(g, True) and not next(g, False) + def quantify(iterable, pred=bool): "Count how many times the predicate is true" return sum(map(pred, iterable)) @@ -779,7 +809,7 @@ which incur interpreter overhead. try: if first is not None: yield first() # For database APIs needing an initial cast to db.first() - while 1: + while True: yield func() except exception: pass diff --git a/Doc/library/json.rst b/Doc/library/json.rst index a01e636496..ee582667b6 100644 --- a/Doc/library/json.rst +++ b/Doc/library/json.rst @@ -3,14 +3,19 @@ .. module:: json :synopsis: Encode and decode the JSON format. + .. moduleauthor:: Bob Ippolito <bob@redivi.com> .. sectionauthor:: Bob Ippolito <bob@redivi.com> +**Source code:** :source:`Lib/json/__init__.py` + +-------------- + `JSON (JavaScript Object Notation) <http://json.org>`_, specified by :rfc:`7159` (which obsoletes :rfc:`4627`) and by `ECMA-404 <http://www.ecma-international.org/publications/standards/Ecma-404.htm>`_, is a lightweight data interchange format inspired by -`JavaScript <http://en.wikipedia.org/wiki/JavaScript>`_ object literal syntax +`JavaScript <https://en.wikipedia.org/wiki/JavaScript>`_ object literal syntax (although it is not a strict subset of JavaScript [#rfc-errata]_ ). :mod:`json` exposes an API familiar to users of the standard library @@ -97,7 +102,7 @@ Extending :class:`JSONEncoder`:: .. highlight:: bash -Using json.tool from the shell to validate and pretty-print:: +Using :mod:`json.tool` from the shell to validate and pretty-print:: $ echo '{"json":"obj"}' | python -m json.tool { @@ -106,6 +111,8 @@ Using json.tool from the shell to validate and pretty-print:: $ echo '{1.2:3.4}' | python -m json.tool Expecting property name enclosed in double quotes: line 1 column 2 (char 1) +See :ref:`json-commandline` for detailed documentation. + .. highlight:: python3 .. note:: @@ -128,7 +135,7 @@ Basic Usage :term:`file-like object`) using this :ref:`conversion table <py-to-json-table>`. - If *skipkeys* is ``True`` (default: ``False``), then dict keys that are not + If *skipkeys* is true (default: ``False``), then dict keys that are not of a basic type (:class:`str`, :class:`int`, :class:`float`, :class:`bool`, ``None``) will be skipped instead of raising a :exc:`TypeError`. @@ -136,18 +143,19 @@ Basic Usage :class:`bytes` objects. Therefore, ``fp.write()`` must support :class:`str` input. - If *ensure_ascii* is ``True`` (the default), the output is guaranteed to + If *ensure_ascii* is true (the default), the output is guaranteed to have all incoming non-ASCII characters escaped. If *ensure_ascii* is - ``False``, these characters will be output as-is. + false, these characters will be output as-is. - If *check_circular* is ``False`` (default: ``True``), then the circular + If *check_circular* is false (default: ``True``), then the circular reference check for container types will be skipped and a circular reference will result in an :exc:`OverflowError` (or worse). - If *allow_nan* is ``False`` (default: ``True``), then it will be a + If *allow_nan* is false (default: ``True``), then it will be a :exc:`ValueError` to serialize out of range :class:`float` values (``nan``, - ``inf``, ``-inf``) in strict compliance of the JSON specification, instead of - using the JavaScript equivalents (``NaN``, ``Infinity``, ``-Infinity``). + ``inf``, ``-inf``) in strict compliance of the JSON specification. + If *allow_nan* is true, their JavaScript equivalents (``NaN``, + ``Infinity``, ``-Infinity``) will be used. If *indent* is a non-negative integer or string, then JSON array elements and object members will be pretty-printed with that indent level. An indent level @@ -167,10 +175,12 @@ Basic Usage .. versionchanged:: 3.4 Use ``(',', ': ')`` as default if *indent* is not ``None``. - *default(obj)* is a function that should return a serializable version of - *obj* or raise :exc:`TypeError`. The default simply raises :exc:`TypeError`. + If specified, *default* should be a function that gets called for objects that + can't otherwise be serialized. It should return a JSON encodable version of + the object or raise a :exc:`TypeError`. If not specified, :exc:`TypeError` + is raised. - If *sort_keys* is ``True`` (default: ``False``), then the output of + If *sort_keys* is true (default: ``False``), then the output of dictionaries will be sorted by key. To use a custom :class:`JSONEncoder` subclass (e.g. one that overrides the @@ -248,7 +258,7 @@ Basic Usage will be passed to the constructor of the class. If the data being deserialized is not a valid JSON document, a - :exc:`ValueError` will be raised. + :exc:`JSONDecodeError` will be raised. .. function:: loads(s, encoding=None, cls=None, object_hook=None, parse_float=None, parse_int=None, parse_constant=None, object_pairs_hook=None, **kw) @@ -259,7 +269,7 @@ Basic Usage *encoding* which is ignored and deprecated. If the data being deserialized is not a valid JSON document, a - :exc:`ValueError` will be raised. + :exc:`JSONDecodeError` will be raised. Encoders and Decoders --------------------- @@ -326,19 +336,22 @@ Encoders and Decoders ``'false'``. This can be used to raise an exception if invalid JSON numbers are encountered. - If *strict* is ``False`` (``True`` is the default), then control characters + If *strict* is false (``True`` is the default), then control characters will be allowed inside strings. Control characters in this context are those with character codes in the 0-31 range, including ``'\t'`` (tab), ``'\n'``, ``'\r'`` and ``'\0'``. If the data being deserialized is not a valid JSON document, a - :exc:`ValueError` will be raised. + :exc:`JSONDecodeError` will be raised. .. method:: decode(s) Return the Python representation of *s* (a :class:`str` instance containing a JSON document). + :exc:`JSONDecodeError` will be raised if the given JSON document is not + valid. + .. method:: raw_decode(s) Decode a JSON document from *s* (a :class:`str` beginning with a @@ -383,26 +396,26 @@ Encoders and Decoders for ``o`` if possible, otherwise it should call the superclass implementation (to raise :exc:`TypeError`). - If *skipkeys* is ``False`` (the default), then it is a :exc:`TypeError` to + If *skipkeys* is false (the default), then it is a :exc:`TypeError` to attempt encoding of keys that are not str, int, float or None. If - *skipkeys* is ``True``, such items are simply skipped. + *skipkeys* is true, such items are simply skipped. - If *ensure_ascii* is ``True`` (the default), the output is guaranteed to + If *ensure_ascii* is true (the default), the output is guaranteed to have all incoming non-ASCII characters escaped. If *ensure_ascii* is - ``False``, these characters will be output as-is. + false, these characters will be output as-is. - If *check_circular* is ``True`` (the default), then lists, dicts, and custom + If *check_circular* is true (the default), then lists, dicts, and custom encoded objects will be checked for circular references during encoding to prevent an infinite recursion (which would cause an :exc:`OverflowError`). Otherwise, no such check takes place. - If *allow_nan* is ``True`` (the default), then ``NaN``, ``Infinity``, and + If *allow_nan* is true (the default), then ``NaN``, ``Infinity``, and ``-Infinity`` will be encoded as such. This behavior is not JSON specification compliant, but is consistent with most JavaScript based encoders and decoders. Otherwise, it will be a :exc:`ValueError` to encode such floats. - If *sort_keys* is ``True`` (default ``False``), then the output of dictionaries + If *sort_keys* is true (default: ``False``), then the output of dictionaries will be sorted by key; this is useful for regression tests to ensure that JSON serializations can be compared on a day-to-day basis. @@ -424,9 +437,10 @@ Encoders and Decoders .. versionchanged:: 3.4 Use ``(',', ': ')`` as default if *indent* is not ``None``. - If specified, *default* is a function that gets called for objects that can't - otherwise be serialized. It should return a JSON encodable version of the - object or raise a :exc:`TypeError`. + If specified, *default* should be a function that gets called for objects that + can't otherwise be serialized. It should return a JSON encodable version of + the object or raise a :exc:`TypeError`. If not specified, :exc:`TypeError` + is raised. .. method:: default(o) @@ -467,6 +481,36 @@ Encoders and Decoders mysocket.write(chunk) +Exceptions +---------- + +.. exception:: JSONDecodeError(msg, doc, pos, end=None) + + Subclass of :exc:`ValueError` with the following additional attributes: + + .. attribute:: msg + + The unformatted error message. + + .. attribute:: doc + + The JSON document being parsed. + + .. attribute:: pos + + The start index of *doc* where parsing failed. + + .. attribute:: lineno + + The line corresponding to *pos*. + + .. attribute:: colno + + The column corresponding to *pos*. + + .. versionadded:: 3.5 + + Standard Compliance and Interoperability ---------------------------------------- @@ -588,11 +632,79 @@ when serializing Python :class:`int` values of extremely large magnitude, or when serializing instances of "exotic" numerical types such as :class:`decimal.Decimal`. +.. highlight:: bash + +.. _json-commandline: + +Command Line Interface +---------------------- + +.. module:: json.tool + :synopsis: A command line to validate and pretty-print JSON. + +**Source code:** :source:`Lib/json/tool.py` + +-------------- + +The :mod:`json.tool` module provides a simple command line interface to validate +and pretty-print JSON objects. + +If the optional ``infile`` and ``outfile`` arguments are not +specified, :attr:`sys.stdin` and :attr:`sys.stdout` will be used respectively:: + + $ echo '{"json": "obj"}' | python -m json.tool + { + "json": "obj" + } + $ echo '{1.2:3.4}' | python -m json.tool + Expecting property name enclosed in double quotes: line 1 column 2 (char 1) + +.. versionchanged:: 3.5 + The output is now in the same order as the input. Use the + :option:`--sort-keys` option to sort the output of dictionaries + alphabetically by key. + +Command line options +^^^^^^^^^^^^^^^^^^^^ + +.. cmdoption:: infile + + The JSON file to be validated or pretty-printed:: + + $ python -m json.tool mp_films.json + [ + { + "title": "And Now for Something Completely Different", + "year": 1971 + }, + { + "title": "Monty Python and the Holy Grail", + "year": 1975 + } + ] + + If *infile* is not specified, read from :attr:`sys.stdin`. + +.. cmdoption:: outfile + + Write the output of the *infile* to the given *outfile*. Otherwise, write it + to :attr:`sys.stdout`. + +.. cmdoption:: --sort-keys + + Sort the output of dictionaries alphabetically by key. + + .. versionadded:: 3.5 + +.. cmdoption:: -h, --help + + Show the help message. + .. rubric:: Footnotes .. [#rfc-errata] As noted in `the errata for RFC 7159 - <http://www.rfc-editor.org/errata_search.php?rfc=7159>`_, + <https://www.rfc-editor.org/errata_search.php?rfc=7159>`_, JSON permits literal U+2028 (LINE SEPARATOR) and U+2029 (PARAGRAPH SEPARATOR) characters in strings, whereas JavaScript (as of ECMAScript Edition 5.1) does not. diff --git a/Doc/library/linecache.rst b/Doc/library/linecache.rst index f18b1cdac0..ae3de9fc0a 100644 --- a/Doc/library/linecache.rst +++ b/Doc/library/linecache.rst @@ -3,6 +3,7 @@ .. module:: linecache :synopsis: This module provides random access to individual lines from text files. + .. sectionauthor:: Moshe Zadka <moshez@zadka.site.co.il> **Source code:** :source:`Lib/linecache.py` @@ -47,6 +48,14 @@ The :mod:`linecache` module defines the following functions: changed on disk, and you require the updated version. If *filename* is omitted, it will check all the entries in the cache. +.. function:: lazycache(filename, module_globals) + + Capture enough detail about a non-file-based module to permit getting its + lines later via :func:`getline` even if *module_globals* is None in the later + call. This avoids doing I/O until a line is actually needed, without having + to carry the module globals around indefinitely. + + .. versionadded:: 3.5 Example:: diff --git a/Doc/library/locale.rst b/Doc/library/locale.rst index b14c551b6e..5aaf4a398f 100644 --- a/Doc/library/locale.rst +++ b/Doc/library/locale.rst @@ -3,9 +3,13 @@ .. module:: locale :synopsis: Internationalization services. + .. moduleauthor:: Martin von Löwis <martin@v.loewis.de> .. sectionauthor:: Martin von Löwis <martin@v.loewis.de> +**Source code:** :source:`Lib/locale.py` + +-------------- The :mod:`locale` module opens access to the POSIX locale database and functionality. The POSIX locale mechanism allows programmers to deal with @@ -387,6 +391,14 @@ The :mod:`locale` module defines the following exception and functions: ``str(float)``, but takes the decimal point into account. +.. function:: delocalize(string) + + Converts a string into a normalized number string, following the + :const:`LC_NUMERIC` settings. + + .. versionadded:: 3.5 + + .. function:: atof(string) Converts a string to a floating point number, following the :const:`LC_NUMERIC` @@ -459,13 +471,13 @@ The :mod:`locale` module defines the following exception and functions: Example:: >>> import locale - >>> loc = locale.getlocale() # get current locale + >>> loc = locale.getlocale() # get current locale # use German locale; name might vary with platform >>> locale.setlocale(locale.LC_ALL, 'de_DE') - >>> locale.strcoll('f\xe4n', 'foo') # compare a string containing an umlaut - >>> locale.setlocale(locale.LC_ALL, '') # use user's preferred locale - >>> locale.setlocale(locale.LC_ALL, 'C') # use default (C) locale - >>> locale.setlocale(locale.LC_ALL, loc) # restore saved locale + >>> locale.strcoll('f\xe4n', 'foo') # compare a string containing an umlaut + >>> locale.setlocale(locale.LC_ALL, '') # use user's preferred locale + >>> locale.setlocale(locale.LC_ALL, 'C') # use default (C) locale + >>> locale.setlocale(locale.LC_ALL, loc) # restore saved locale Background, details, hints, tips and caveats diff --git a/Doc/library/logging.config.rst b/Doc/library/logging.config.rst index fd6a47778f..b4c9bc3dc1 100644 --- a/Doc/library/logging.config.rst +++ b/Doc/library/logging.config.rst @@ -4,10 +4,11 @@ .. module:: logging.config :synopsis: Configuration of the logging module. - .. moduleauthor:: Vinay Sajip <vinay_sajip@red-dove.com> .. sectionauthor:: Vinay Sajip <vinay_sajip@red-dove.com> +**Source code:** :source:`Lib/logging/config.py` + .. sidebar:: Important This page contains only reference information. For tutorials, @@ -17,8 +18,6 @@ * :ref:`Advanced Tutorial <logging-advanced-tutorial>` * :ref:`Logging Cookbook <logging-cookbook>` -**Source code:** :source:`Lib/logging/config.py` - -------------- This section describes the API for configuring the logging module. @@ -244,7 +243,9 @@ otherwise, the context is used to determine what to instantiate. handler. All *other* keys are passed through as keyword arguments to the - handler's constructor. For example, given the snippet:: + handler's constructor. For example, given the snippet: + + .. code-block:: yaml handlers: console: @@ -353,7 +354,9 @@ it unambiguously, and then using the id in the source object's configuration to indicate that a connection exists between the source and the destination object with that id. -So, for example, consider the following YAML snippet:: +So, for example, consider the following YAML snippet: + +.. code-block:: yaml formatters: brief: @@ -410,7 +413,9 @@ to provide a 'factory' - a callable which is called with a configuration dictionary and which returns the instantiated object. This is signalled by an absolute import path to the factory being made available under the special key ``'()'``. Here's a concrete -example:: +example: + +.. code-block:: yaml formatters: brief: @@ -627,7 +632,9 @@ configuration must be specified in a section called ``[logger_root]``. :func:`dictConfig`, so it's worth considering transitioning to this newer API when it's convenient to do so. -Examples of these sections in the file are given below. :: +Examples of these sections in the file are given below. + +.. code-block:: ini [loggers] keys=root,log02,log03,log04,log05,log06,log07 @@ -639,7 +646,9 @@ Examples of these sections in the file are given below. :: keys=form01,form02,form03,form04,form05,form06,form07,form08,form09 The root logger must specify a level and a list of handlers. An example of a -root logger section is given below. :: +root logger section is given below. + +.. code-block:: ini [logger_root] level=NOTSET @@ -656,7 +665,9 @@ appear in the ``[handlers]`` section. These names must appear in the file. For loggers other than the root logger, some additional information is required. -This is illustrated by the following example. :: +This is illustrated by the following example. + +.. code-block:: ini [logger_parser] level=DEBUG @@ -674,7 +685,8 @@ indicate that messages are **not** propagated to handlers up the hierarchy. The say the name used by the application to get the logger. Sections which specify handler configuration are exemplified by the following. -:: + +.. code-block:: ini [handler_hand01] class=StreamHandler @@ -694,7 +706,9 @@ a corresponding section in the configuration file. The ``args`` entry, when :func:`eval`\ uated in the context of the ``logging`` package's namespace, is the list of arguments to the constructor for the handler class. Refer to the constructors for the relevant handlers, or to the examples -below, to see how typical entries are constructed. :: +below, to see how typical entries are constructed. + +.. code-block:: ini [handler_hand02] class=FileHandler @@ -745,7 +759,9 @@ below, to see how typical entries are constructed. :: formatter=form09 args=('localhost:9022', '/log', 'GET') -Sections which specify formatter configuration are typified by the following. :: +Sections which specify formatter configuration are typified by the following. + +.. code-block:: ini [formatter_form01] format=F1 %(asctime)s %(levelname)s %(message)s @@ -781,5 +797,3 @@ condensed format. Module :mod:`logging.handlers` Useful handlers included with the logging module. - - diff --git a/Doc/library/logging.handlers.rst b/Doc/library/logging.handlers.rst index 9e558e549d..855adabf86 100644 --- a/Doc/library/logging.handlers.rst +++ b/Doc/library/logging.handlers.rst @@ -4,10 +4,11 @@ .. module:: logging.handlers :synopsis: Handlers for the logging module. - .. moduleauthor:: Vinay Sajip <vinay_sajip@red-dove.com> .. sectionauthor:: Vinay Sajip <vinay_sajip@red-dove.com> +**Source code:** :source:`Lib/logging/handlers.py` + .. sidebar:: Important This page contains only reference information. For tutorials, @@ -17,8 +18,6 @@ * :ref:`Advanced Tutorial <logging-advanced-tutorial>` * :ref:`Logging Cookbook <logging-cookbook>` -**Source code:** :source:`Lib/logging/handlers.py` - -------------- .. currentmodule:: logging @@ -544,7 +543,7 @@ supports sending logging messages to a remote or local Unix syslog. (See: :issue:`12168`.) In earlier versions, the message sent to the syslog daemons was always terminated with a NUL byte, because early versions of these daemons expected a NUL terminated message - even - though it's not in the relevant specification (RF 5424). More recent + though it's not in the relevant specification (RFC 5424). More recent versions of these daemons don't expect the NUL byte but strip it off if it's there, and even more recent daemons (which adhere more closely to RFC 5424) pass the NUL byte on as part of the message. @@ -853,7 +852,7 @@ supports sending logging messages to a Web server, using either ``GET`` or credentials, you should also specify secure=True so that your userid and password are not passed in cleartext across the wire. - .. versionchanged:: 3.4.3 + .. versionchanged:: 3.5 The *context* parameter was added. .. method:: mapLogRecord(record) @@ -866,7 +865,7 @@ supports sending logging messages to a Web server, using either ``GET`` or .. method:: emit(record) - Sends the record to the Web server as an URL-encoded dictionary. The + Sends the record to the Web server as a URL-encoded dictionary. The :meth:`mapLogRecord` method is used to convert the record to the dictionary to be sent. @@ -953,13 +952,20 @@ applications where threads servicing clients need to respond as quickly as possible, while any potentially slow operations (such as sending an email via :class:`SMTPHandler`) are done on a separate thread. -.. class:: QueueListener(queue, *handlers) +.. class:: QueueListener(queue, *handlers, respect_handler_level=False) Returns a new instance of the :class:`QueueListener` class. The instance is initialized with the queue to send messages to and a list of handlers which will handle entries placed on the queue. The queue can be any queue- like object; it's passed as-is to the :meth:`dequeue` method, which needs - to know how to get messages from it. + to know how to get messages from it. If ``respect_handler_level`` is ``True``, + a handler's level is respected (compared with the level for the message) when + deciding whether to pass messages to that handler; otherwise, the behaviour + is as in previous Python versions - to always pass each message to each + handler. + + .. versionchanged:: 3.5 + The ``respect_handler_levels`` argument was added. .. method:: dequeue(block) diff --git a/Doc/library/logging.rst b/Doc/library/logging.rst index 8fd7e40599..72da385d72 100644 --- a/Doc/library/logging.rst +++ b/Doc/library/logging.rst @@ -4,10 +4,10 @@ .. module:: logging :synopsis: Flexible event logging system for applications. - .. moduleauthor:: Vinay Sajip <vinay_sajip@red-dove.com> .. sectionauthor:: Vinay Sajip <vinay_sajip@red-dove.com> +**Source code:** :source:`Lib/logging/__init__.py` .. index:: pair: Errors; logging @@ -20,9 +20,6 @@ * :ref:`Advanced Tutorial <logging-advanced-tutorial>` * :ref:`Logging Cookbook <logging-cookbook>` - -**Source code:** :source:`Lib/logging/__init__.py` - -------------- This module defines functions and classes which implement a flexible event @@ -159,11 +156,13 @@ is the module's name in the Python package namespace. *msg* using the string formatting operator. (Note that this means that you can use keywords in the format string, together with a single dictionary argument.) - There are three keyword arguments in *kwargs* which are inspected: *exc_info* - which, if it does not evaluate as false, causes exception information to be + There are three keyword arguments in *kwargs* which are inspected: + *exc_info*, *stack_info*, and *extra*. + + If *exc_info* does not evaluate as false, it causes exception information to be added to the logging message. If an exception tuple (in the format returned by - :func:`sys.exc_info`) is provided, it is used; otherwise, :func:`sys.exc_info` - is called to get the exception information. + :func:`sys.exc_info`) or an exception instance is provided, it is used; + otherwise, :func:`sys.exc_info` is called to get the exception information. The second optional keyword argument is *stack_info*, which defaults to ``False``. If true, stack information is added to the logging @@ -220,6 +219,9 @@ is the module's name in the Python package namespace. .. versionadded:: 3.2 The *stack_info* parameter was added. + .. versionchanged:: 3.5 + The *exc_info* parameter can now accept exception instances. + .. method:: Logger.info(msg, *args, **kwargs) @@ -1241,7 +1243,7 @@ with the :mod:`warnings` module. The proposal which described this feature for inclusion in the Python standard library. - `Original Python logging package <http://www.red-dove.com/python_logging.html>`_ + `Original Python logging package <https://www.red-dove.com/python_logging.html>`_ This is the original source for the :mod:`logging` package. The version of the package available from this site is suitable for use with Python 1.5.2, 2.1.x and 2.2.x, which do not include the :mod:`logging` package in the standard diff --git a/Doc/library/lzma.rst b/Doc/library/lzma.rst index b71051da85..f99c495ce9 100644 --- a/Doc/library/lzma.rst +++ b/Doc/library/lzma.rst @@ -3,11 +3,15 @@ .. module:: lzma :synopsis: A Python wrapper for the liblzma compression library. + .. moduleauthor:: Nadeem Vawda <nadeem.vawda@gmail.com> .. sectionauthor:: Nadeem Vawda <nadeem.vawda@gmail.com> .. versionadded:: 3.3 +**Source code:** :source:`Lib/lzma.py` + +-------------- This module provides classes and convenience functions for compressing and decompressing data using the LZMA compression algorithm. Also included is a file @@ -110,6 +114,10 @@ Reading and writing compressed files .. versionchanged:: 3.4 Added support for the ``"x"`` and ``"xb"`` modes. + .. versionchanged:: 3.5 + The :meth:`~io.BufferedIOBase.read` method now accepts an argument of + ``None``. + Compressing and decompressing data in memory -------------------------------------------- @@ -221,13 +229,32 @@ Compressing and decompressing data in memory decompress a multi-stream input with :class:`LZMADecompressor`, you must create a new decompressor for each stream. - .. method:: decompress(data) + .. method:: decompress(data, max_length=-1) + + Decompress *data* (a :term:`bytes-like object`), returning + uncompressed data as bytes. Some of *data* may be buffered + internally, for use in later calls to :meth:`decompress`. The + returned data should be concatenated with the output of any + previous calls to :meth:`decompress`. + + If *max_length* is nonnegative, returns at most *max_length* + bytes of decompressed data. If this limit is reached and further + output can be produced, the :attr:`~.needs_input` attribute will + be set to ``False``. In this case, the next call to + :meth:`~.decompress` may provide *data* as ``b''`` to obtain + more of the output. - Decompress *data* (a :class:`bytes` object), returning a :class:`bytes` - object containing the decompressed data for at least part of the input. - Some of *data* may be buffered internally, for use in later calls to - :meth:`decompress`. The returned data should be concatenated with the - output of any previous calls to :meth:`decompress`. + If all of the input data was decompressed and returned (either + because this was less than *max_length* bytes, or because + *max_length* was negative), the :attr:`~.needs_input` attribute + will be set to ``True``. + + Attempting to decompress data after the end of stream is reached + raises an `EOFError`. Any data found after the end of the + stream is ignored and saved in the :attr:`~.unused_data` attribute. + + .. versionchanged:: 3.5 + Added the *max_length* parameter. .. attribute:: check @@ -245,6 +272,12 @@ Compressing and decompressing data in memory Before the end of the stream is reached, this will be ``b""``. + .. attribute:: needs_input + + ``False`` if the :meth:`.decompress` method can provide more + decompressed data before requiring new uncompressed input. + + .. versionadded:: 3.5 .. function:: compress(data, format=FORMAT_XZ, check=-1, preset=None, filters=None) diff --git a/Doc/library/macpath.rst b/Doc/library/macpath.rst index b7a5d89239..b08bbe0809 100644 --- a/Doc/library/macpath.rst +++ b/Doc/library/macpath.rst @@ -4,6 +4,9 @@ .. module:: macpath :synopsis: Mac OS 9 path manipulation functions. +**Source code:** :source:`Lib/macpath.py` + +-------------- This module is the Mac OS 9 (and earlier) implementation of the :mod:`os.path` module. It can be used to manipulate old-style Macintosh pathnames on Mac OS X diff --git a/Doc/library/mailbox.rst b/Doc/library/mailbox.rst index d29902dc85..81244c2ed0 100644 --- a/Doc/library/mailbox.rst +++ b/Doc/library/mailbox.rst @@ -3,9 +3,13 @@ .. module:: mailbox :synopsis: Manipulate mailboxes in various formats + .. moduleauthor:: Gregory K. Johnson <gkj@gregorykjohnson.com> .. sectionauthor:: Gregory K. Johnson <gkj@gregorykjohnson.com> +**Source code:** :source:`Lib/mailbox.py` + +-------------- This module defines two classes, :class:`Mailbox` and :class:`Message`, for accessing and manipulating on-disk mailboxes and the messages they contain. @@ -422,7 +426,7 @@ Supported mailbox formats are Maildir, mbox, MH, Babyl, and MMDF. `maildir man page from qmail <http://www.qmail.org/man/man5/maildir.html>`_ The original specification of the format. - `Using maildir format <http://cr.yp.to/proto/maildir.html>`_ + `Using maildir format <https://cr.yp.to/proto/maildir.html>`_ Notes on Maildir by its inventor. Includes an updated name-creation scheme and details on "info" semantics. @@ -484,7 +488,7 @@ Supported mailbox formats are Maildir, mbox, MH, Babyl, and MMDF. `mbox man page from tin <http://www.tin.org/bin/man.cgi?section=5&topic=mbox>`_ Another specification of the format, with details on locking. - `Configuring Netscape Mail on Unix: Why The Content-Length Format is Bad <http://www.jwz.org/doc/content-length.html>`_ + `Configuring Netscape Mail on Unix: Why The Content-Length Format is Bad <https://www.jwz.org/doc/content-length.html>`_ An argument for using the original mbox format rather than a variation. `"mbox" is a family of several mutually incompatible mailbox formats <http://homepage.ntlworld.com/jonathan.deboynepollard/FGA/mail-mbox-formats.html>`_ @@ -690,10 +694,10 @@ Supported mailbox formats are Maildir, mbox, MH, Babyl, and MMDF. .. seealso:: - `Format of Version 5 Babyl Files <http://quimby.gnus.org/notes/BABYL>`_ + `Format of Version 5 Babyl Files <https://quimby.gnus.org/notes/BABYL>`_ A specification of the Babyl format. - `Reading Mail with Rmail <http://www.gnu.org/software/emacs/manual/html_node/emacs/Rmail.html>`_ + `Reading Mail with Rmail <https://www.gnu.org/software/emacs/manual/html_node/emacs/Rmail.html>`_ The Rmail manual, with some information on Babyl semantics. @@ -744,7 +748,7 @@ Supported mailbox formats are Maildir, mbox, MH, Babyl, and MMDF. `mmdf man page from tin <http://www.tin.org/bin/man.cgi?section=5&topic=mmdf>`_ A specification of MMDF format from the documentation of tin, a newsreader. - `MMDF <http://en.wikipedia.org/wiki/MMDF>`_ + `MMDF <https://en.wikipedia.org/wiki/MMDF>`_ A Wikipedia article describing the Multichannel Memorandum Distribution Facility. diff --git a/Doc/library/mailcap.rst b/Doc/library/mailcap.rst index 8115e42603..896afd1d73 100644 --- a/Doc/library/mailcap.rst +++ b/Doc/library/mailcap.rst @@ -70,7 +70,7 @@ standard. However, mailcap files are supported on most Unix systems. An example usage:: >>> import mailcap - >>> d=mailcap.getcaps() + >>> d = mailcap.getcaps() >>> mailcap.findmatch(d, 'video/mpeg', filename='tmp1223') ('xmpeg tmp1223', {'view': 'xmpeg %s'}) diff --git a/Doc/library/marshal.rst b/Doc/library/marshal.rst index af43944b2c..1ffc6effc7 100644 --- a/Doc/library/marshal.rst +++ b/Doc/library/marshal.rst @@ -5,6 +5,7 @@ :synopsis: Convert Python objects to streams of bytes and back (with different constraints). +-------------- This module contains functions that can read and write Python values in a binary format. The format is specific to Python, but independent of machine @@ -16,7 +17,6 @@ rarely does). [#]_ .. index:: module: pickle module: shelve - object: code This is not a general "persistence" module. For general persistence and transfer of Python objects through RPC calls, see the modules :mod:`pickle` and @@ -34,13 +34,15 @@ supports a substantially wider range of objects than marshal. maliciously constructed data. Never unmarshal data received from an untrusted or unauthenticated source. +.. index:: object; code, code object + Not all Python object types are supported; in general, only objects whose value is independent from a particular invocation of Python can be written and read by this module. The following types are supported: booleans, integers, floating point numbers, complex numbers, strings, bytes, bytearrays, tuples, lists, sets, frozensets, dictionaries, and code objects, where it should be understood that tuples, lists, sets, frozensets and dictionaries are only supported as long as -the values contained therein are themselves supported. +the values contained therein are themselves supported. The singletons :const:`None`, :const:`Ellipsis` and :exc:`StopIteration` can also be marshalled and unmarshalled. For format *version* lower than 3, recursive lists, sets and dictionaries cannot diff --git a/Doc/library/math.rst b/Doc/library/math.rst index 0fc7c7c9b9..3fdea18cfd 100644 --- a/Doc/library/math.rst +++ b/Doc/library/math.rst @@ -8,6 +8,8 @@ from math import fsum +-------------- + This module is always available. It provides access to the mathematical functions defined by the C standard. @@ -97,7 +99,49 @@ Number-theoretic and representation functions For further discussion and two alternative approaches, see the `ASPN cookbook recipes for accurate floating point summation - <http://code.activestate.com/recipes/393090/>`_\. + <https://code.activestate.com/recipes/393090/>`_\. + + +.. function:: gcd(a, b) + + Return the greatest common divisor of the integers *a* and *b*. If either + *a* or *b* is nonzero, then the value of ``gcd(a, b)`` is the largest + positive integer that divides both *a* and *b*. ``gcd(0, 0)`` returns + ``0``. + + .. versionadded:: 3.5 + + +.. function:: isclose(a, b, *, rel_tol=1e-09, abs_tol=0.0) + + Return ``True`` if the values *a* and *b* are close to each other and + ``False`` otherwise. + + Whether or not two values are considered close is determined according to + given absolute and relative tolerances. + + *rel_tol* is the relative tolerance -- it is the maximum allowed difference + between *a* and *b*, relative to the larger absolute value of *a* or *b*. + For example, to set a tolerance of 5%, pass ``rel_tol=0.05``. The default + tolerance is ``1e-09``, which assures that the two values are the same + within about 9 decimal digits. *rel_tol* must be greater than zero. + + *abs_tol* is the minimum absolute tolerance -- useful for comparisons near + zero. *abs_tol* must be at least zero. + + If no errors occur, the result will be: + ``abs(a-b) <= max(rel_tol * max(abs(a), abs(b)), abs_tol)``. + + The IEEE 754 special values of ``NaN``, ``inf``, and ``-inf`` will be + handled according to IEEE rules. Specifically, ``NaN`` is not considered + close to any other value, including ``NaN``. ``inf`` and ``-inf`` are only + considered close to themselves. + + .. versionadded:: 3.5 + + .. seealso:: + + :pep:`485` -- A function for testing approximate equality .. function:: isfinite(x) @@ -162,7 +206,7 @@ Power and logarithmic functions Return ``e**x - 1``. For small floats *x*, the subtraction in ``exp(x) - 1`` can result in a `significant loss of precision - <http://en.wikipedia.org/wiki/Loss_of_significance>`_\; the :func:`expm1` + <https://en.wikipedia.org/wiki/Loss_of_significance>`_\; the :func:`expm1` function provides a way to compute this quantity to full precision:: >>> from math import exp, expm1 @@ -290,7 +334,7 @@ Angular conversion Hyperbolic functions -------------------- -`Hyperbolic functions <http://en.wikipedia.org/wiki/Hyperbolic_function>`_ +`Hyperbolic functions <https://en.wikipedia.org/wiki/Hyperbolic_function>`_ are analogs of trigonometric functions that are based on hyperbolas instead of circles. @@ -329,12 +373,12 @@ Special functions .. function:: erf(x) - Return the `error function <http://en.wikipedia.org/wiki/Error_function>`_ at + Return the `error function <https://en.wikipedia.org/wiki/Error_function>`_ at *x*. The :func:`erf` function can be used to compute traditional statistical functions such as the `cumulative standard normal distribution - <http://en.wikipedia.org/wiki/Normal_distribution#Cumulative_distribution_function>`_:: + <https://en.wikipedia.org/wiki/Normal_distribution#Cumulative_distribution_function>`_:: def phi(x): 'Cumulative distribution function for the standard normal distribution' @@ -346,17 +390,17 @@ Special functions .. function:: erfc(x) Return the complementary error function at *x*. The `complementary error - function <http://en.wikipedia.org/wiki/Error_function>`_ is defined as + function <https://en.wikipedia.org/wiki/Error_function>`_ is defined as ``1.0 - erf(x)``. It is used for large values of *x* where a subtraction from one would cause a `loss of significance - <http://en.wikipedia.org/wiki/Loss_of_significance>`_\. + <https://en.wikipedia.org/wiki/Loss_of_significance>`_\. .. versionadded:: 3.2 .. function:: gamma(x) - Return the `Gamma function <http://en.wikipedia.org/wiki/Gamma_function>`_ at + Return the `Gamma function <https://en.wikipedia.org/wiki/Gamma_function>`_ at *x*. .. versionadded:: 3.2 @@ -383,6 +427,22 @@ Constants The mathematical constant e = 2.718281..., to available precision. +.. data:: inf + + A floating-point positive infinity. (For negative infinity, use + ``-math.inf``.) Equivalent to the output of ``float('inf')``. + + .. versionadded:: 3.5 + + +.. data:: nan + + A floating-point "not a number" (NaN) value. Equivalent to the output of + ``float('nan')``. + + .. versionadded:: 3.5 + + .. impl-detail:: The :mod:`math` module consists mostly of thin wrappers around the platform C diff --git a/Doc/library/mimetypes.rst b/Doc/library/mimetypes.rst index 8739ea3dcd..464248c3ea 100644 --- a/Doc/library/mimetypes.rst +++ b/Doc/library/mimetypes.rst @@ -3,13 +3,13 @@ .. module:: mimetypes :synopsis: Mapping of filename extensions to MIME types. + .. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org> +**Source code:** :source:`Lib/mimetypes.py` .. index:: pair: MIME; content type -**Source code:** :source:`Lib/mimetypes.py` - -------------- The :mod:`mimetypes` module converts between a filename or URL and the MIME type @@ -44,7 +44,7 @@ the information :func:`init` sets up. The optional *strict* argument is a flag specifying whether the list of known MIME types is limited to only the official types `registered with IANA - <http://www.iana.org/assignments/media-types/media-types.xhtml>`_. + <https://www.iana.org/assignments/media-types/media-types.xhtml>`_. When *strict* is ``True`` (the default), only the IANA types are supported; when *strict* is ``False``, some additional non-standard but commonly used MIME types are also recognized. diff --git a/Doc/library/mmap.rst b/Doc/library/mmap.rst index 18e05e31f7..8f538339aa 100644 --- a/Doc/library/mmap.rst +++ b/Doc/library/mmap.rst @@ -4,6 +4,7 @@ .. module:: mmap :synopsis: Interface to memory-mapped files for Unix and Windows. +-------------- Memory-mapped file objects behave like both :class:`bytearray` and like :term:`file objects <file object>`. You can use mmap objects in most places @@ -127,7 +128,7 @@ To map anonymous memory, -1 should be passed as the fileno along with the length import mmap with mmap.mmap(-1, 13) as mm: - mm.write("Hello world!") + mm.write(b"Hello world!") .. versionadded:: 3.2 Context manager support. @@ -144,7 +145,7 @@ To map anonymous memory, -1 should be passed as the fileno along with the length pid = os.fork() - if pid == 0: # In a child process + if pid == 0: # In a child process mm.seek(0) print(mm.readline()) @@ -174,6 +175,9 @@ To map anonymous memory, -1 should be passed as the fileno along with the length Optional arguments *start* and *end* are interpreted as in slice notation. Returns ``-1`` on failure. + .. versionchanged:: 3.5 + Writable :term:`bytes-like object` is now accepted. + .. method:: flush([offset[, size]]) @@ -234,6 +238,9 @@ To map anonymous memory, -1 should be passed as the fileno along with the length Optional arguments *start* and *end* are interpreted as in slice notation. Returns ``-1`` on failure. + .. versionchanged:: 3.5 + Writable :term:`bytes-like object` is now accepted. + .. method:: seek(pos[, whence]) @@ -261,6 +268,9 @@ To map anonymous memory, -1 should be passed as the fileno along with the length were written. If the mmap was created with :const:`ACCESS_READ`, then writing to it will raise a :exc:`TypeError` exception. + .. versionchanged:: 3.5 + Writable :term:`bytes-like object` is now accepted. + .. method:: write_byte(byte) diff --git a/Doc/library/modulefinder.rst b/Doc/library/modulefinder.rst index e84a4964a0..7b39ce7d1a 100644 --- a/Doc/library/modulefinder.rst +++ b/Doc/library/modulefinder.rst @@ -1,12 +1,11 @@ :mod:`modulefinder` --- Find modules used by a script ===================================================== -.. sectionauthor:: A.M. Kuchling <amk@amk.ca> - - .. module:: modulefinder :synopsis: Find modules used by a script. +.. sectionauthor:: A.M. Kuchling <amk@amk.ca> + **Source code:** :source:`Lib/modulefinder.py` -------------- diff --git a/Doc/library/msilib.rst b/Doc/library/msilib.rst index 4145c8e7cc..0a42032974 100644 --- a/Doc/library/msilib.rst +++ b/Doc/library/msilib.rst @@ -4,12 +4,16 @@ .. module:: msilib :platform: Windows :synopsis: Creation of Microsoft Installer files, and CAB files. + .. moduleauthor:: Martin v. Löwis <martin@v.loewis.de> .. sectionauthor:: Martin v. Löwis <martin@v.loewis.de> +**Source code:** :source:`Lib/msilib/__init__.py` .. index:: single: msi +-------------- + The :mod:`msilib` supports the creation of Microsoft Installer (``.msi``) files. Because these files often contain an embedded "cabinet" file (``.cab``), it also exposes an API to create CAB files. Support for reading ``.cab`` files is @@ -120,9 +124,9 @@ structures. .. seealso:: - `FCICreateFile <http://msdn.microsoft.com/library?url=/library/en-us/devnotes/winprog/fcicreate.asp>`_ - `UuidCreate <http://msdn.microsoft.com/library?url=/library/en-us/rpc/rpc/uuidcreate.asp>`_ - `UuidToString <http://msdn.microsoft.com/library?url=/library/en-us/rpc/rpc/uuidtostring.asp>`_ + `FCICreateFile <https://msdn.microsoft.com/library?url=/library/en-us/devnotes/winprog/fcicreate.asp>`_ + `UuidCreate <https://msdn.microsoft.com/library?url=/library/en-us/rpc/rpc/uuidcreate.asp>`_ + `UuidToString <https://msdn.microsoft.com/library?url=/library/en-us/rpc/rpc/uuidtostring.asp>`_ .. _database-objects: @@ -151,9 +155,9 @@ Database Objects .. seealso:: - `MSIDatabaseOpenView <http://msdn.microsoft.com/library?url=/library/en-us/msi/setup/msidatabaseopenview.asp>`_ - `MSIDatabaseCommit <http://msdn.microsoft.com/library?url=/library/en-us/msi/setup/msidatabasecommit.asp>`_ - `MSIGetSummaryInformation <http://msdn.microsoft.com/library?url=/library/en-us/msi/setup/msigetsummaryinformation.asp>`_ + `MSIDatabaseOpenView <https://msdn.microsoft.com/library?url=/library/en-us/msi/setup/msidatabaseopenview.asp>`_ + `MSIDatabaseCommit <https://msdn.microsoft.com/library?url=/library/en-us/msi/setup/msidatabasecommit.asp>`_ + `MSIGetSummaryInformation <https://msdn.microsoft.com/library?url=/library/en-us/msi/setup/msigetsummaryinformation.asp>`_ .. _view-objects: @@ -199,11 +203,11 @@ View Objects .. seealso:: - `MsiViewExecute <http://msdn.microsoft.com/library?url=/library/en-us/msi/setup/msiviewexecute.asp>`_ - `MSIViewGetColumnInfo <http://msdn.microsoft.com/library?url=/library/en-us/msi/setup/msiviewgetcolumninfo.asp>`_ - `MsiViewFetch <http://msdn.microsoft.com/library?url=/library/en-us/msi/setup/msiviewfetch.asp>`_ - `MsiViewModify <http://msdn.microsoft.com/library?url=/library/en-us/msi/setup/msiviewmodify.asp>`_ - `MsiViewClose <http://msdn.microsoft.com/library?url=/library/en-us/msi/setup/msiviewclose.asp>`_ + `MsiViewExecute <https://msdn.microsoft.com/library?url=/library/en-us/msi/setup/msiviewexecute.asp>`_ + `MSIViewGetColumnInfo <https://msdn.microsoft.com/library?url=/library/en-us/msi/setup/msiviewgetcolumninfo.asp>`_ + `MsiViewFetch <https://msdn.microsoft.com/library?url=/library/en-us/msi/setup/msiviewfetch.asp>`_ + `MsiViewModify <https://msdn.microsoft.com/library?url=/library/en-us/msi/setup/msiviewmodify.asp>`_ + `MsiViewClose <https://msdn.microsoft.com/library?url=/library/en-us/msi/setup/msiviewclose.asp>`_ .. _summary-objects: @@ -243,10 +247,10 @@ Summary Information Objects .. seealso:: - `MsiSummaryInfoGetProperty <http://msdn.microsoft.com/library?url=/library/en-us/msi/setup/msisummaryinfogetproperty.asp>`_ - `MsiSummaryInfoGetPropertyCount <http://msdn.microsoft.com/library?url=/library/en-us/msi/setup/msisummaryinfogetpropertycount.asp>`_ - `MsiSummaryInfoSetProperty <http://msdn.microsoft.com/library?url=/library/en-us/msi/setup/msisummaryinfosetproperty.asp>`_ - `MsiSummaryInfoPersist <http://msdn.microsoft.com/library?url=/library/en-us/msi/setup/msisummaryinfopersist.asp>`_ + `MsiSummaryInfoGetProperty <https://msdn.microsoft.com/library?url=/library/en-us/msi/setup/msisummaryinfogetproperty.asp>`_ + `MsiSummaryInfoGetPropertyCount <https://msdn.microsoft.com/library?url=/library/en-us/msi/setup/msisummaryinfogetpropertycount.asp>`_ + `MsiSummaryInfoSetProperty <https://msdn.microsoft.com/library?url=/library/en-us/msi/setup/msisummaryinfosetproperty.asp>`_ + `MsiSummaryInfoPersist <https://msdn.microsoft.com/library?url=/library/en-us/msi/setup/msisummaryinfopersist.asp>`_ .. _record-objects: @@ -297,11 +301,11 @@ Record Objects .. seealso:: - `MsiRecordGetFieldCount <http://msdn.microsoft.com/library?url=/library/en-us/msi/setup/msirecordgetfieldcount.asp>`_ - `MsiRecordSetString <http://msdn.microsoft.com/library?url=/library/en-us/msi/setup/msirecordsetstring.asp>`_ - `MsiRecordSetStream <http://msdn.microsoft.com/library?url=/library/en-us/msi/setup/msirecordsetstream.asp>`_ - `MsiRecordSetInteger <http://msdn.microsoft.com/library?url=/library/en-us/msi/setup/msirecordsetinteger.asp>`_ - `MsiRecordClear <http://msdn.microsoft.com/library?url=/library/en-us/msi/setup/msirecordclear.asp>`_ + `MsiRecordGetFieldCount <https://msdn.microsoft.com/library?url=/library/en-us/msi/setup/msirecordgetfieldcount.asp>`_ + `MsiRecordSetString <https://msdn.microsoft.com/library?url=/library/en-us/msi/setup/msirecordsetstring.asp>`_ + `MsiRecordSetStream <https://msdn.microsoft.com/library?url=/library/en-us/msi/setup/msirecordsetstream.asp>`_ + `MsiRecordSetInteger <https://msdn.microsoft.com/library?url=/library/en-us/msi/setup/msirecordsetinteger.asp>`_ + `MsiRecordClear <https://msdn.microsoft.com/library?url=/library/en-us/msi/setup/msirecordclear.asp>`_ .. _msi-errors: @@ -393,10 +397,10 @@ Directory Objects .. seealso:: - `Directory Table <http://msdn.microsoft.com/library?url=/library/en-us/msi/setup/directory_table.asp>`_ - `File Table <http://msdn.microsoft.com/library?url=/library/en-us/msi/setup/file_table.asp>`_ - `Component Table <http://msdn.microsoft.com/library?url=/library/en-us/msi/setup/component_table.asp>`_ - `FeatureComponents Table <http://msdn.microsoft.com/library?url=/library/en-us/msi/setup/featurecomponents_table.asp>`_ + `Directory Table <https://msdn.microsoft.com/library?url=/library/en-us/msi/setup/directory_table.asp>`_ + `File Table <https://msdn.microsoft.com/library?url=/library/en-us/msi/setup/file_table.asp>`_ + `Component Table <https://msdn.microsoft.com/library?url=/library/en-us/msi/setup/component_table.asp>`_ + `FeatureComponents Table <https://msdn.microsoft.com/library?url=/library/en-us/msi/setup/featurecomponents_table.asp>`_ .. _features: @@ -421,7 +425,7 @@ Features .. seealso:: - `Feature Table <http://msdn.microsoft.com/library?url=/library/en-us/msi/setup/feature_table.asp>`_ + `Feature Table <https://msdn.microsoft.com/library?url=/library/en-us/msi/setup/feature_table.asp>`_ .. _msi-gui: @@ -516,13 +520,13 @@ for installing Python packages. .. seealso:: - `Dialog Table <http://msdn.microsoft.com/library?url=/library/en-us/msi/setup/dialog_table.asp>`_ - `Control Table <http://msdn.microsoft.com/library?url=/library/en-us/msi/setup/control_table.asp>`_ - `Control Types <http://msdn.microsoft.com/library?url=/library/en-us/msi/setup/controls.asp>`_ - `ControlCondition Table <http://msdn.microsoft.com/library?url=/library/en-us/msi/setup/controlcondition_table.asp>`_ - `ControlEvent Table <http://msdn.microsoft.com/library?url=/library/en-us/msi/setup/controlevent_table.asp>`_ - `EventMapping Table <http://msdn.microsoft.com/library?url=/library/en-us/msi/setup/eventmapping_table.asp>`_ - `RadioButton Table <http://msdn.microsoft.com/library?url=/library/en-us/msi/setup/radiobutton_table.asp>`_ + `Dialog Table <https://msdn.microsoft.com/library?url=/library/en-us/msi/setup/dialog_table.asp>`_ + `Control Table <https://msdn.microsoft.com/library?url=/library/en-us/msi/setup/control_table.asp>`_ + `Control Types <https://msdn.microsoft.com/library?url=/library/en-us/msi/setup/controls.asp>`_ + `ControlCondition Table <https://msdn.microsoft.com/library?url=/library/en-us/msi/setup/controlcondition_table.asp>`_ + `ControlEvent Table <https://msdn.microsoft.com/library?url=/library/en-us/msi/setup/controlevent_table.asp>`_ + `EventMapping Table <https://msdn.microsoft.com/library?url=/library/en-us/msi/setup/eventmapping_table.asp>`_ + `RadioButton Table <https://msdn.microsoft.com/library?url=/library/en-us/msi/setup/radiobutton_table.asp>`_ .. _msi-tables: diff --git a/Doc/library/msvcrt.rst b/Doc/library/msvcrt.rst index fadaf05a09..b334eeb314 100644 --- a/Doc/library/msvcrt.rst +++ b/Doc/library/msvcrt.rst @@ -4,8 +4,10 @@ .. module:: msvcrt :platform: Windows :synopsis: Miscellaneous useful routines from the MS VC++ runtime. + .. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org> +-------------- These functions provide access to some useful capabilities on Windows platforms. Some higher-level modules use these functions to build the Windows diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst index f7dc11bf3f..d20098ffa7 100644 --- a/Doc/library/multiprocessing.rst +++ b/Doc/library/multiprocessing.rst @@ -4,6 +4,9 @@ .. module:: multiprocessing :synopsis: Process-based parallelism. +**Source code:** :source:`Lib/multiprocessing/` + +-------------- Introduction ------------ @@ -316,7 +319,7 @@ However, if you really do need to use some shared data then proxies. A manager returned by :func:`Manager` will support types - :class:`list`, :class:`dict`, :class:`Namespace`, :class:`Lock`, + :class:`list`, :class:`dict`, :class:`~managers.Namespace`, :class:`Lock`, :class:`RLock`, :class:`Semaphore`, :class:`BoundedSemaphore`, :class:`Condition`, :class:`Event`, :class:`Barrier`, :class:`Queue`, :class:`Value` and :class:`Array`. For example, :: @@ -361,8 +364,9 @@ processes in a few different ways. For example:: - from multiprocessing import Pool - from time import sleep + from multiprocessing import Pool, TimeoutError + import time + import os def f(x): return x*x @@ -378,15 +382,29 @@ For example:: for i in pool.imap_unordered(f, range(10)): print(i) - # evaluate "f(10)" asynchronously - res = pool.apply_async(f, [10]) - print(res.get(timeout=1)) # prints "100" + # evaluate "f(20)" asynchronously + res = pool.apply_async(f, (20,)) # runs in *only* one process + print(res.get(timeout=1)) # prints "400" + + # evaluate "os.getpid()" asynchronously + res = pool.apply_async(os.getpid, ()) # runs in *only* one process + print(res.get(timeout=1)) # prints the PID of that process + + # launching multiple evaluations asynchronously *may* use more processes + multiple_results = [pool.apply_async(os.getpid, ()) for i in range(4)] + print([res.get(timeout=1) for res in multiple_results]) + + # make a single worker sleep for 10 secs + res = pool.apply_async(time.sleep, (10,)) + try: + print(res.get(timeout=1)) + except TimeoutError: + print("We lacked patience and got a multiprocessing.TimeoutError") - # make worker sleep for 10 secs - res = pool.apply_async(sleep, [10]) - print(res.get(timeout=1)) # raises multiprocessing.TimeoutError + print("For the moment, the pool remains available for more work") # exiting the 'with'-block has stopped the pool + print("Now the pool is closed and no longer available") Note that the methods of a pool should only ever be used by the process which created it. @@ -901,8 +919,10 @@ Miscellaneous If the ``freeze_support()`` line is omitted then trying to run the frozen executable will raise :exc:`RuntimeError`. - If the module is being run normally by the Python interpreter then - :func:`freeze_support` has no effect. + Calling ``freeze_support()`` has no effect when invoked on any operating + system other than Windows. In addition, if the module is being run + normally by the Python interpreter on Windows (the program has not been + frozen), then ``freeze_support()`` has no effect. .. function:: get_all_start_methods() @@ -990,7 +1010,7 @@ Connection objects are usually created using :func:`Pipe` -- see also using :meth:`recv`. The object must be picklable. Very large pickles (approximately 32 MB+, - though it depends on the OS) may raise a ValueError exception. + though it depends on the OS) may raise a :exc:`ValueError` exception. .. method:: recv() @@ -1458,6 +1478,9 @@ processes. Note that accessing the ctypes object through the wrapper can be a lot slower than accessing the raw ctypes object. + .. versionchanged:: 3.5 + Synchronized objects support the :term:`context manager` protocol. + The table below compares the syntax for creating shared ctypes objects from shared memory with the normal ctypes syntax. (In the table ``MyStruct`` is some @@ -1742,24 +1765,26 @@ their parent process exits. The manager classes are defined in the lproxy[0] = d -Namespace objects ->>>>>>>>>>>>>>>>> +.. class:: Namespace -A namespace object has no public methods, but does have writable attributes. -Its representation shows the values of its attributes. + A type that can register with :class:`SyncManager`. -However, when using a proxy for a namespace object, an attribute beginning with -``'_'`` will be an attribute of the proxy and not an attribute of the referent: + A namespace object has no public methods, but does have writable attributes. + Its representation shows the values of its attributes. -.. doctest:: + However, when using a proxy for a namespace object, an attribute beginning + with ``'_'`` will be an attribute of the proxy and not an attribute of the + referent: - >>> manager = multiprocessing.Manager() - >>> Global = manager.Namespace() - >>> Global.x = 10 - >>> Global.y = 'hello' - >>> Global._z = 12.3 # this is an attribute of the proxy - >>> print(Global) - Namespace(x=10, y='hello') + .. doctest:: + + >>> manager = multiprocessing.Manager() + >>> Global = manager.Namespace() + >>> Global.x = 10 + >>> Global.y = 'hello' + >>> Global._z = 12.3 # this is an attribute of the proxy + >>> print(Global) + Namespace(x=10, y='hello') Customized managers @@ -1945,9 +1970,9 @@ itself. This means, for example, that one shared object can contain a second: >>> l = manager.list(range(10)) >>> l._callmethod('__len__') 10 - >>> l._callmethod('__getitem__', (slice(2, 7),)) # equiv to `l[2:7]` + >>> l._callmethod('__getitem__', (slice(2, 7),)) # equivalent to l[2:7] [2, 3, 4, 5, 6] - >>> l._callmethod('__getitem__', (20,)) # equiv to `l[20]` + >>> l._callmethod('__getitem__', (20,)) # equivalent to l[20] Traceback (most recent call last): ... IndexError: list index out of range @@ -2164,13 +2189,14 @@ with the :class:`Pool` class. The following example demonstrates the use of a pool:: from multiprocessing import Pool + import time def f(x): return x*x if __name__ == '__main__': with Pool(processes=4) as pool: # start 4 worker processes - result = pool.apply_async(f, (10,)) # evaluate "f(10)" asynchronously + result = pool.apply_async(f, (10,)) # evaluate "f(10)" asynchronously in a single process print(result.get(timeout=1)) # prints "100" unless your computer is *very* slow print(pool.map(f, range(10))) # prints "[0, 1, 4,..., 81]" @@ -2180,9 +2206,8 @@ The following example demonstrates the use of a pool:: print(next(it)) # prints "1" print(it.next(timeout=1)) # prints "4" unless your computer is *very* slow - import time result = pool.apply_async(time.sleep, (10,)) - print(result.get(timeout=1)) # raises TimeoutError + print(result.get(timeout=1)) # raises multiprocessing.TimeoutError .. _multiprocessing-listeners-clients: @@ -2456,7 +2481,7 @@ the connection.) If authentication is requested but no authentication key is specified then the return value of ``current_process().authkey`` is used (see -:class:`~multiprocessing.Process`). This value will automatically inherited by +:class:`~multiprocessing.Process`). This value will be automatically inherited by any :class:`~multiprocessing.Process` object that the current process creates. This means that (by default) all processes of a multi-process program will share a single authentication key which can be used when setting up connections @@ -2641,8 +2666,8 @@ Explicitly pass resources to child processes ... do something using "lock" ... if __name__ == '__main__': - lock = Lock() - for i in range(10): + lock = Lock() + for i in range(10): Process(target=f).start() should be rewritten as :: @@ -2653,8 +2678,8 @@ Explicitly pass resources to child processes ... do something using "l" ... if __name__ == '__main__': - lock = Lock() - for i in range(10): + lock = Lock() + for i in range(10): Process(target=f, args=(lock,)).start() Beware of replacing :data:`sys.stdin` with a "file like object" @@ -2667,7 +2692,7 @@ Beware of replacing :data:`sys.stdin` with a "file like object" in issues with processes-in-processes. This has been changed to:: sys.stdin.close() - sys.stdin = open(os.devnull) + sys.stdin = open(os.open(os.devnull, os.O_RDONLY), closefd=False) Which solves the fundamental issue of processes colliding with each other resulting in a bad file descriptor error, but introduces a potential danger @@ -2698,12 +2723,7 @@ start method. More picklability - Ensure that all arguments to :meth:`Process.__init__` are - picklable. This means, in particular, that bound or unbound - methods cannot be used directly as the ``target`` (unless you use - the *fork* start method) --- just define a function and use that - instead. - + Ensure that all arguments to :meth:`Process.__init__` are picklable. Also, if you subclass :class:`~multiprocessing.Process` then make sure that instances will be picklable when the :meth:`Process.start <multiprocessing.Process.start>` method is called. diff --git a/Doc/library/netrc.rst b/Doc/library/netrc.rst index 23ffed69eb..cdc2616368 100644 --- a/Doc/library/netrc.rst +++ b/Doc/library/netrc.rst @@ -4,6 +4,7 @@ .. module:: netrc :synopsis: Loading of .netrc files. + .. moduleauthor:: Eric S. Raymond <esr@snark.thyrsus.com> .. sectionauthor:: Eric S. Raymond <esr@snark.thyrsus.com> diff --git a/Doc/library/nis.rst b/Doc/library/nis.rst index ade2a7aad0..10c67cbb81 100644 --- a/Doc/library/nis.rst +++ b/Doc/library/nis.rst @@ -5,9 +5,11 @@ .. module:: nis :platform: Unix :synopsis: Interface to Sun's NIS (Yellow Pages) library. + .. moduleauthor:: Fred Gansevles <Fred.Gansevles@cs.utwente.nl> .. sectionauthor:: Moshe Zadka <moshez@zadka.site.co.il> +-------------- The :mod:`nis` module gives a thin wrapper around the NIS library, useful for central administration of several hosts. @@ -26,7 +28,7 @@ The :mod:`nis` module defines the following functions: Note that *mapname* is first checked if it is an alias to another name. - The *domain* argument allows to override the NIS domain used for the lookup. If + The *domain* argument allows overriding the NIS domain used for the lookup. If unspecified, lookup is in the default NIS domain. @@ -38,7 +40,7 @@ The :mod:`nis` module defines the following functions: Note that *mapname* is first checked if it is an alias to another name. - The *domain* argument allows to override the NIS domain used for the lookup. If + The *domain* argument allows overriding the NIS domain used for the lookup. If unspecified, lookup is in the default NIS domain. @@ -46,7 +48,7 @@ The :mod:`nis` module defines the following functions: Return a list of all valid maps. - The *domain* argument allows to override the NIS domain used for the lookup. If + The *domain* argument allows overriding the NIS domain used for the lookup. If unspecified, lookup is in the default NIS domain. diff --git a/Doc/library/nntplib.rst b/Doc/library/nntplib.rst index 9fb1b4594d..9790e3a0b2 100644 --- a/Doc/library/nntplib.rst +++ b/Doc/library/nntplib.rst @@ -4,13 +4,12 @@ .. module:: nntplib :synopsis: NNTP protocol client (requires sockets). +**Source code:** :source:`Lib/nntplib.py` .. index:: pair: NNTP; protocol single: Network News Transfer Protocol -**Source code:** :source:`Lib/nntplib.py` - -------------- This module defines the class :class:`NNTP` which implements the client side of diff --git a/Doc/library/numbers.rst b/Doc/library/numbers.rst index 8ab07d0b93..1b594952ea 100644 --- a/Doc/library/numbers.rst +++ b/Doc/library/numbers.rst @@ -4,6 +4,9 @@ .. module:: numbers :synopsis: Numeric abstract base classes (Complex, Real, Integral, etc.). +**Source code:** :source:`Lib/numbers.py` + +-------------- The :mod:`numbers` module (:pep:`3141`) defines a hierarchy of numeric :term:`abstract base classes <abstract base class>` which progressively define @@ -35,7 +38,7 @@ The numeric tower Abstract. Retrieves the imaginary component of this number. - .. method:: conjugate() + .. abstractmethod:: conjugate() Abstract. Returns the complex conjugate. For example, ``(1+3j).conjugate() == (1-3j)``. diff --git a/Doc/library/operator.rst b/Doc/library/operator.rst index f9e2a3d003..8121b480cb 100644 --- a/Doc/library/operator.rst +++ b/Doc/library/operator.rst @@ -3,16 +3,16 @@ .. module:: operator :synopsis: Functions corresponding to the standard operators. + .. sectionauthor:: Skip Montanaro <skip@automatrix.com> +**Source code:** :source:`Lib/operator.py` .. testsetup:: import operator from operator import itemgetter, iadd -**Source code:** :source:`Lib/operator.py` - -------------- The :mod:`operator` module exports a set of efficient functions corresponding to @@ -138,6 +138,14 @@ The mathematical and bitwise operations are the most numerous: Return ``a * b``, for *a* and *b* numbers. +.. function:: matmul(a, b) + __matmul__(a, b) + + Return ``a @ b``. + + .. versionadded:: 3.5 + + .. function:: neg(obj) __neg__(obj) @@ -391,6 +399,8 @@ Python syntax and the functions in the :mod:`operator` module. +-----------------------+-------------------------+---------------------------------------+ | Multiplication | ``a * b`` | ``mul(a, b)`` | +-----------------------+-------------------------+---------------------------------------+ +| Matrix Multiplication | ``a @ b`` | ``matmul(a, b)`` | ++-----------------------+-------------------------+---------------------------------------+ | Negation (Arithmetic) | ``- a`` | ``neg(a)`` | +-----------------------+-------------------------+---------------------------------------+ | Negation (Logical) | ``not a`` | ``not_(a)`` | @@ -499,6 +509,14 @@ will perform the update, so no subsequent assignment is necessary: ``a = imul(a, b)`` is equivalent to ``a *= b``. +.. function:: imatmul(a, b) + __imatmul__(a, b) + + ``a = imatmul(a, b)`` is equivalent to ``a @= b``. + + .. versionadded:: 3.5 + + .. function:: ior(a, b) __ior__(a, b) diff --git a/Doc/library/optparse.rst b/Doc/library/optparse.rst index 160c29d3b5..e5f40f492c 100644 --- a/Doc/library/optparse.rst +++ b/Doc/library/optparse.rst @@ -4,15 +4,16 @@ .. module:: optparse :synopsis: Command-line option parsing library. :deprecated: + .. moduleauthor:: Greg Ward <gward@python.net> .. sectionauthor:: Greg Ward <gward@python.net> +**Source code:** :source:`Lib/optparse.py` + .. deprecated:: 3.2 The :mod:`optparse` module is deprecated and will not be developed further; development will continue with the :mod:`argparse` module. -**Source code:** :source:`Lib/optparse.py` - -------------- :mod:`optparse` is a more convenient, flexible, and powerful library for parsing @@ -25,7 +26,7 @@ GNU/POSIX syntax, and additionally generates usage and help messages for you. Here's an example of using :mod:`optparse` in a simple script:: from optparse import OptionParser - [...] + ... parser = OptionParser() parser.add_option("-f", "--file", dest="filename", help="write report to FILE", metavar="FILE") @@ -252,7 +253,7 @@ First, you need to import the OptionParser class; then, early in the main program, create an OptionParser instance:: from optparse import OptionParser - [...] + ... parser = OptionParser() Then you can start defining options. The basic syntax is:: @@ -677,7 +678,9 @@ automatically adds a ``--version`` option to your parser. If it encounters this option on the command line, it expands your ``version`` string (by replacing ``%prog``), prints it to stdout, and exits. -For example, if your script is called ``/usr/bin/foo``:: +For example, if your script is called ``/usr/bin/foo``: + +.. code-block:: shell-session $ /usr/bin/foo --version foo 1.0 @@ -718,7 +721,7 @@ you can call :func:`OptionParser.error` to signal an application-defined error condition:: (options, args) = parser.parse_args() - [...] + ... if options.a and options.b: parser.error("options -a and -b are mutually exclusive") @@ -727,14 +730,18 @@ program's usage message and an error message to standard error and exits with error status 2. Consider the first example above, where the user passes ``4x`` to an option -that takes an integer:: +that takes an integer: + +.. code-block:: shell-session $ /usr/bin/foo -n 4x Usage: foo [options] foo: error: option -n: invalid integer value: '4x' -Or, where the user fails to pass a value at all:: +Or, where the user fails to pass a value at all: + +.. code-block:: shell-session $ /usr/bin/foo -n Usage: foo [options] @@ -758,7 +765,7 @@ Putting it all together Here's what :mod:`optparse`\ -based scripts usually look like:: from optparse import OptionParser - [...] + ... def main(): usage = "usage: %prog [options] arg" parser = OptionParser(usage) @@ -768,13 +775,13 @@ Here's what :mod:`optparse`\ -based scripts usually look like:: action="store_true", dest="verbose") parser.add_option("-q", "--quiet", action="store_false", dest="verbose") - [...] + ... (options, args) = parser.parse_args() if len(args) != 1: parser.error("incorrect number of arguments") if options.verbose: print("reading %s..." % options.filename) - [...] + ... if __name__ == "__main__": main() @@ -1409,7 +1416,7 @@ If you're not careful, it's easy to define options with conflicting option strings:: parser.add_option("-n", "--dry-run", ...) - [...] + ... parser.add_option("-n", "--noisy", ...) (This is particularly true if you've defined your own OptionParser subclass with @@ -1450,7 +1457,7 @@ that option. If the user asks for help, the help message will reflect that:: Options: --dry-run do no harm - [...] + ... -n, --noisy be noisy It's possible to whittle away the option strings for a previously-added option @@ -1465,7 +1472,7 @@ At this point, the original ``-n``/``--dry-run`` option is no longer accessible, so :mod:`optparse` removes it, leaving this help text:: Options: - [...] + ... -n, --noisy be noisy --dry-run new dry-run option @@ -1701,7 +1708,7 @@ seen, but blow up if it comes after ``-b`` in the command-line. :: if parser.values.b: raise OptionValueError("can't use -a after -b") parser.values.a = 1 - [...] + ... parser.add_option("-a", action="callback", callback=check_order) parser.add_option("-b", action="store_true", dest="b") @@ -1719,7 +1726,7 @@ message and the flag that it sets must be generalized. :: if parser.values.b: raise OptionValueError("can't use %s after -b" % opt_str) setattr(parser.values, option.dest, 1) - [...] + ... parser.add_option("-a", action="callback", callback=check_order, dest='a') parser.add_option("-b", action="store_true", dest="b") parser.add_option("-c", action="callback", callback=check_order, dest='c') @@ -1739,7 +1746,7 @@ should not be called when the moon is full, all you have to do is this:: raise OptionValueError("%s option invalid when moon is full" % opt_str) setattr(parser.values, option.dest, 1) - [...] + ... parser.add_option("--foo", action="callback", callback=check_moon, dest="foo") @@ -1762,7 +1769,7 @@ Here's an example that just emulates the standard ``"store"`` action:: def store_value(option, opt_str, value, parser): setattr(parser.values, option.dest, value) - [...] + ... parser.add_option("--foo", action="callback", callback=store_value, type="int", nargs=3, dest="foo") @@ -1824,9 +1831,9 @@ arguments:: del parser.rargs[:len(value)] setattr(parser.values, option.dest, value) - [...] - parser.add_option("-c", "--callback", dest="vararg_attr", - action="callback", callback=vararg_callback) + ... + parser.add_option("-c", "--callback", dest="vararg_attr", + action="callback", callback=vararg_callback) .. _optparse-extending-optparse: diff --git a/Doc/library/os.path.rst b/Doc/library/os.path.rst index 33ef564c2c..bf0dface7e 100644 --- a/Doc/library/os.path.rst +++ b/Doc/library/os.path.rst @@ -4,8 +4,14 @@ .. module:: os.path :synopsis: Operations on pathnames. +**Source code:** :source:`Lib/posixpath.py` (for POSIX), +:source:`Lib/ntpath.py` (for Windows NT), +and :source:`Lib/macpath.py` (for Macintosh) + .. index:: single: path; operations +-------------- + This module implements some useful functions on pathnames. To read or write files see :func:`open`, and for accessing the filesystem see the :mod:`os` module. The path parameters can be passed as either strings, @@ -66,11 +72,37 @@ the :mod:`glob` module.) empty string (``''``). +.. function:: commonpath(paths) + + Return the longest common sub-path of each pathname in the sequence + *paths*. Raise ValueError if *paths* contains both absolute and relative + pathnames, or if *paths* is empty. Unlike :func:`commonprefix`, this + returns a valid path. + + Availability: Unix, Windows + + .. versionadded:: 3.5 + + .. function:: commonprefix(list) - Return the longest path prefix (taken character-by-character) that is a prefix - of all paths in *list*. If *list* is empty, return the empty string (``''``). - Note that this may return invalid paths because it works a character at a time. + Return the longest path prefix (taken character-by-character) that is a + prefix of all paths in *list*. If *list* is empty, return the empty string + (``''``). + + .. note:: + + This function may return invalid paths because it works a + character at a time. To obtain a valid path, see + :func:`commonpath`. + + :: + + >>> os.path.commonprefix(['/usr/lib', '/usr/local/lib']) + '/usr/l' + + >>> os.path.commonpath(['/usr/lib', '/usr/local/lib']) + '/usr' .. function:: dirname(path) @@ -188,7 +220,7 @@ the :mod:`glob` module.) .. function:: islink(path) Return ``True`` if *path* refers to a directory entry that is a symbolic link. - Always ``False`` if symbolic links are not supported by the python runtime. + Always ``False`` if symbolic links are not supported by the Python runtime. .. function:: ismount(path) diff --git a/Doc/library/os.rst b/Doc/library/os.rst index 7bc59898ab..4265bc23a1 100644 --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -4,6 +4,9 @@ .. module:: os :synopsis: Miscellaneous operating system interfaces. +**Source code:** :source:`Lib/os.py` + +-------------- This module provides a portable way of using operating system dependent functionality. If you just want to read or write a file see :func:`open`, if @@ -78,9 +81,10 @@ uses the file system encoding to perform this conversion (see .. versionchanged:: 3.1 On some systems, conversion using the file system encoding may fail. In this - case, Python uses the ``surrogateescape`` encoding error handler, which means - that undecodable bytes are replaced by a Unicode character U+DCxx on - decoding, and these are again translated to the original byte on encoding. + case, Python uses the :ref:`surrogateescape encoding error handler + <surrogateescape>`, which means that undecodable bytes are replaced by a + Unicode character U+DCxx on decoding, and these are again translated to the + original byte on encoding. The file system encoding must guarantee to successfully decode all bytes @@ -311,8 +315,6 @@ process and user. Return the current process id. - Availability: Unix, Windows. - .. function:: getppid() @@ -549,8 +551,6 @@ process and user. On platforms where :c:func:`strerror` returns ``NULL`` when given an unknown error number, :exc:`ValueError` is raised. - Availability: Unix, Windows. - .. data:: supports_bytes_environ @@ -564,8 +564,6 @@ process and user. Set the current numeric umask and return the previous umask. - Availability: Unix, Windows. - .. function:: uname() @@ -656,8 +654,6 @@ as internal buffering of data. Close file descriptor *fd*. - Availability: Unix, Windows. - .. note:: This function is intended for low-level I/O and must be applied to a file @@ -677,8 +673,6 @@ as internal buffering of data. except OSError: pass - Availability: Unix, Windows. - .. function:: device_encoding(fd) @@ -695,8 +689,6 @@ as internal buffering of data. 2: stderr), the new file descriptor is :ref:`inheritable <fd_inheritance>`. - Availability: Unix, Windows. - .. versionchanged:: 3.4 The new file descriptor is now non-inheritable. @@ -707,8 +699,6 @@ as internal buffering of data. The file descriptor *fd2* is :ref:`inheritable <fd_inheritance>` by default, or non-inheritable if *inheritable* is ``False``. - Availability: Unix, Windows. - .. versionchanged:: 3.4 Add the optional *inheritable* parameter. @@ -774,8 +764,6 @@ as internal buffering of data. The :func:`.stat` function. - Availability: Unix, Windows. - .. function:: fstatvfs(fd) @@ -804,8 +792,21 @@ as internal buffering of data. most *length* bytes in size. As of Python 3.3, this is equivalent to ``os.truncate(fd, length)``. + Availability: Unix, Windows. + + .. versionchanged:: 3.5 + Added support for Windows + +.. function:: get_blocking(fd) + + Get the blocking mode of the file descriptor: ``False`` if the + :data:`O_NONBLOCK` flag is set, ``True`` if the flag is cleared. + + See also :func:`set_blocking` and :meth:`socket.socket.setblocking`. + Availability: Unix. + .. versionadded:: 3.5 .. function:: isatty(fd) @@ -846,8 +847,6 @@ as internal buffering of data. current position; :const:`SEEK_END` or ``2`` to set it relative to the end of the file. Return the new cursor position in bytes, starting from the beginning. - Availability: Unix, Windows. - .. data:: SEEK_SET SEEK_CUR @@ -856,8 +855,6 @@ as internal buffering of data. Parameters to the :func:`lseek` function. Their values are 0, 1, and 2, respectively. - Availability: Unix, Windows. - .. versionadded:: 3.3 Some operating systems could support additional values, like :data:`os.SEEK_HOLE` or :data:`os.SEEK_DATA`. @@ -878,8 +875,6 @@ as internal buffering of data. This function can support :ref:`paths relative to directory descriptors <dir_fd>` with the *dir_fd* parameter. - Availability: Unix, Windows. - .. versionchanged:: 3.4 The new file descriptor is now non-inheritable. @@ -893,11 +888,16 @@ as internal buffering of data. .. versionadded:: 3.3 The *dir_fd* argument. + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise an + exception, the function now retries the system call instead of raising an + :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + The following constants are options for the *flags* parameter to the :func:`~os.open` function. They can be combined using the bitwise OR operator ``|``. Some of them are not available on all platforms. For descriptions of their availability and use, consult the :manpage:`open(2)` manual page on Unix -or `the MSDN <http://msdn.microsoft.com/en-us/library/z0kc8e3z.aspx>`_ on Windows. +or `the MSDN <https://msdn.microsoft.com/en-us/library/z0kc8e3z.aspx>`_ on Windows. .. data:: O_RDONLY @@ -1060,8 +1060,6 @@ or `the MSDN <http://msdn.microsoft.com/en-us/library/z0kc8e3z.aspx>`_ on Window bytes read. If the end of the file referred to by *fd* has been reached, an empty bytes object is returned. - Availability: Unix, Windows. - .. note:: This function is intended for low-level I/O and must be applied to a file @@ -1070,6 +1068,11 @@ or `the MSDN <http://msdn.microsoft.com/en-us/library/z0kc8e3z.aspx>`_ on Window :func:`popen` or :func:`fdopen`, or :data:`sys.stdin`, use its :meth:`~file.read` or :meth:`~file.readline` methods. + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise an + exception, the function now retries the system call instead of raising an + :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + .. function:: sendfile(out, in, offset, count) sendfile(out, in, offset, count, [headers], [trailers], flags=0) @@ -1099,9 +1102,26 @@ or `the MSDN <http://msdn.microsoft.com/en-us/library/z0kc8e3z.aspx>`_ on Window Availability: Unix. + .. note:: + + For a higher-level wrapper of :func:`sendfile`, see + :meth:`socket.socket.sendfile`. + .. versionadded:: 3.3 +.. function:: set_blocking(fd, blocking) + + Set the blocking mode of the specified file descriptor. Set the + :data:`O_NONBLOCK` flag if blocking is ``False``, clear the flag otherwise. + + See also :func:`get_blocking` and :meth:`socket.socket.setblocking`. + + Availability: Unix. + + .. versionadded:: 3.5 + + .. data:: SF_NODISKIO SF_MNOWAIT SF_SYNC @@ -1158,8 +1178,6 @@ or `the MSDN <http://msdn.microsoft.com/en-us/library/z0kc8e3z.aspx>`_ on Window Write the bytestring in *str* to file descriptor *fd*. Return the number of bytes actually written. - Availability: Unix, Windows. - .. note:: This function is intended for low-level I/O and must be applied to a file @@ -1168,11 +1186,20 @@ or `the MSDN <http://msdn.microsoft.com/en-us/library/z0kc8e3z.aspx>`_ on Window :func:`fdopen`, or :data:`sys.stdout` or :data:`sys.stderr`, use its :meth:`~file.write` method. + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise an + exception, the function now retries the system call instead of raising an + :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + .. function:: writev(fd, buffers) Write the contents of *buffers* to file descriptor *fd*. *buffers* must be a - sequence of :term:`bytes-like objects <bytes-like object>`. + sequence of :term:`bytes-like objects <bytes-like object>`. Buffers are + processed in array order. Entire contents of first buffer is written before + proceeding to second, and so on. The operating system may set a limit + (sysconf() value SC_IOV_MAX) on the number of buffers that can be used. + :func:`~os.writev` writes the contents of each object to the file descriptor and returns the total number of bytes written. @@ -1330,8 +1357,6 @@ features: or not it is available using :data:`os.supports_effective_ids`. If it is unavailable, using it will raise a :exc:`NotImplementedError`. - Availability: Unix, Windows. - .. note:: Using :func:`access` to check if a user is authorized to e.g. open a file @@ -1384,8 +1409,6 @@ features: This function can support :ref:`specifying a file descriptor <path_fd>`. The descriptor must refer to an opened directory, not an open file. - Availability: Unix, Windows. - .. versionadded:: 3.3 Added support for specifying *path* as a file descriptor on some platforms. @@ -1447,8 +1470,6 @@ features: :ref:`paths relative to directory descriptors <dir_fd>` and :ref:`not following symlinks <follow_symlinks>`. - Availability: Unix, Windows. - .. note:: Although Windows supports :func:`chmod`, you can only set the file's @@ -1499,15 +1520,11 @@ features: Return a string representing the current working directory. - Availability: Unix, Windows. - .. function:: getcwdb() Return a bytestring representing the current working directory. - Availability: Unix, Windows. - .. function:: lchflags(path, flags) @@ -1570,7 +1587,11 @@ features: .. note:: To encode ``str`` filenames to ``bytes``, use :func:`~os.fsencode`. - Availability: Unix, Windows. + .. seealso:: + + The :func:`scandir` function returns directory entries along with + file attribute information, giving better performance for many + common use cases. .. versionchanged:: 3.2 The *path* parameter became optional. @@ -1609,9 +1630,15 @@ features: Create a directory named *path* with numeric mode *mode*. + If the directory already exists, :exc:`FileExistsError` is raised. + + .. _mkdir_modebits: + On some systems, *mode* is ignored. Where it is used, the current umask - value is first masked out. If the directory already exists, :exc:`OSError` - is raised. + value is first masked out. If bits other than the last 9 (i.e. the last 3 + digits of the octal representation of the *mode*) are set, their meaning is + platform-dependent. On some platforms, they are ignored and you should call + :func:`chmod` explicitly to set them. This function can also support :ref:`paths relative to directory descriptors <dir_fd>`. @@ -1619,8 +1646,6 @@ features: It is also possible to create temporary directories; see the :mod:`tempfile` module's :func:`tempfile.mkdtemp` function. - Availability: Unix, Windows. - .. versionadded:: 3.3 The *dir_fd* argument. @@ -1634,8 +1659,8 @@ features: Recursive directory creation function. Like :func:`mkdir`, but makes all intermediate-level directories needed to contain the leaf directory. - The default *mode* is ``0o777`` (octal). On some systems, *mode* is - ignored. Where it is used, the current umask value is first masked out. + The *mode* parameter is passed to :func:`mkdir`; see :ref:`the mkdir() + description <mkdir_modebits>` for how it is interpreted. If *exist_ok* is ``False`` (the default), an :exc:`OSError` is raised if the target directory already exists. @@ -1750,7 +1775,7 @@ features: ``os.path.join(os.path.dirname(path), result)``. If the *path* is a string object, the result will also be a string object, - and the call may raise an UnicodeDecodeError. If the *path* is a bytes + and the call may raise a UnicodeDecodeError. If the *path* is a bytes object, the result will be a bytes object. This function can also support :ref:`paths relative to directory descriptors @@ -1777,9 +1802,7 @@ features: be raised; on Unix, the directory entry is removed but the storage allocated to the file is not made available until the original file is no longer in use. - This function is identical to :func:`unlink`. - - Availability: Unix, Windows. + This function is semantically identical to :func:`unlink`. .. versionadded:: 3.3 The *dir_fd* argument. @@ -1814,8 +1837,6 @@ features: If you want cross-platform overwriting of the destination, use :func:`replace`. - Availability: Unix, Windows. - .. versionadded:: 3.3 The *src_dir_fd* and *dst_dir_fd* arguments. @@ -1844,8 +1865,6 @@ features: This function can support specifying *src_dir_fd* and/or *dst_dir_fd* to supply :ref:`paths relative to directory descriptors <dir_fd>`. - Availability: Unix, Windows. - .. versionadded:: 3.3 @@ -1858,12 +1877,188 @@ features: This function can support :ref:`paths relative to directory descriptors <dir_fd>`. - Availability: Unix, Windows. - .. versionadded:: 3.3 The *dir_fd* parameter. +.. function:: scandir(path='.') + + Return an iterator of :class:`DirEntry` objects corresponding to the entries + in the directory given by *path*. The entries are yielded in arbitrary + order, and the special entries ``'.'`` and ``'..'`` are not included. + + Using :func:`scandir` instead of :func:`listdir` can significantly + increase the performance of code that also needs file type or file + attribute information, because :class:`DirEntry` objects expose this + information if the operating system provides it when scanning a directory. + All :class:`DirEntry` methods may perform a system call, but + :func:`~DirEntry.is_dir` and :func:`~DirEntry.is_file` usually only + require a system call for symbolic links; :func:`DirEntry.stat` + always requires a system call on Unix but only requires one for + symbolic links on Windows. + + On Unix, *path* can be of type :class:`str` or :class:`bytes` (use + :func:`~os.fsencode` and :func:`~os.fsdecode` to encode and decode + :class:`bytes` paths). On Windows, *path* must be of type :class:`str`. + On both systems, the type of the :attr:`~DirEntry.name` and + :attr:`~DirEntry.path` attributes of each :class:`DirEntry` will be of + the same type as *path*. + + The following example shows a simple use of :func:`scandir` to display all + the files (excluding directories) in the given *path* that don't start with + ``'.'``. The ``entry.is_file()`` call will generally not make an additional + system call:: + + for entry in os.scandir(path): + if not entry.name.startswith('.') and entry.is_file(): + print(entry.name) + + .. note:: + + On Unix-based systems, :func:`scandir` uses the system's + `opendir() <http://pubs.opengroup.org/onlinepubs/009695399/functions/opendir.html>`_ + and + `readdir() <http://pubs.opengroup.org/onlinepubs/009695399/functions/readdir_r.html>`_ + functions. On Windows, it uses the Win32 + `FindFirstFileW <https://msdn.microsoft.com/en-us/library/windows/desktop/aa364418(v=vs.85).aspx>`_ + and + `FindNextFileW <https://msdn.microsoft.com/en-us/library/windows/desktop/aa364428(v=vs.85).aspx>`_ + functions. + + .. versionadded:: 3.5 + + +.. class:: DirEntry + + Object yielded by :func:`scandir` to expose the file path and other file + attributes of a directory entry. + + :func:`scandir` will provide as much of this information as possible without + making additional system calls. When a ``stat()`` or ``lstat()`` system call + is made, the ``DirEntry`` object will cache the result. + + ``DirEntry`` instances are not intended to be stored in long-lived data + structures; if you know the file metadata has changed or if a long time has + elapsed since calling :func:`scandir`, call ``os.stat(entry.path)`` to fetch + up-to-date information. + + Because the ``DirEntry`` methods can make operating system calls, they may + also raise :exc:`OSError`. If you need very fine-grained + control over errors, you can catch :exc:`OSError` when calling one of the + ``DirEntry`` methods and handle as appropriate. + + Attributes and methods on a ``DirEntry`` instance are as follows: + + .. attribute:: name + + The entry's base filename, relative to the :func:`scandir` *path* + argument. + + The :attr:`name` attribute will be of the same type (``str`` or + ``bytes``) as the :func:`scandir` *path* argument. Use + :func:`~os.fsdecode` to decode byte filenames. + + .. attribute:: path + + The entry's full path name: equivalent to ``os.path.join(scandir_path, + entry.name)`` where *scandir_path* is the :func:`scandir` *path* + argument. The path is only absolute if the :func:`scandir` *path* + argument was absolute. + + The :attr:`path` attribute will be of the same type (``str`` or + ``bytes``) as the :func:`scandir` *path* argument. Use + :func:`~os.fsdecode` to decode byte filenames. + + .. method:: inode() + + Return the inode number of the entry. + + The result is cached on the ``DirEntry`` object. Use ``os.stat(entry.path, + follow_symlinks=False).st_ino`` to fetch up-to-date information. + + On the first, uncached call, a system call is required on Windows but + not on Unix. + + .. method:: is_dir(\*, follow_symlinks=True) + + Return ``True`` if this entry is a directory or a symbolic link pointing + to a directory; return ``False`` if the entry is or points to any other + kind of file, or if it doesn't exist anymore. + + If *follow_symlinks* is ``False``, return ``True`` only if this entry + is a directory (without following symlinks); return ``False`` if the + entry is any other kind of file or if it doesn't exist anymore. + + The result is cached on the ``DirEntry`` object, with a separate cache + for *follow_symlinks* ``True`` and ``False``. Call :func:`os.stat` along + with :func:`stat.S_ISDIR` to fetch up-to-date information. + + On the first, uncached call, no system call is required in most cases. + Specifically, for non-symlinks, neither Windows or Unix require a system + call, except on certain Unix file systems, such as network file systems, + that return ``dirent.d_type == DT_UNKNOWN``. If the entry is a symlink, + a system call will be required to follow the symlink unless + *follow_symlinks* is ``False``. + + This method can raise :exc:`OSError`, such as :exc:`PermissionError`, + but :exc:`FileNotFoundError` is caught and not raised. + + .. method:: is_file(\*, follow_symlinks=True) + + Return ``True`` if this entry is a file or a symbolic link pointing to a + file; return ``False`` if the entry is or points to a directory or other + non-file entry, or if it doesn't exist anymore. + + If *follow_symlinks* is ``False``, return ``True`` only if this entry + is a file (without following symlinks); return ``False`` if the entry is + a directory or other non-file entry, or if it doesn't exist anymore. + + The result is cached on the ``DirEntry`` object. Caching, system calls + made, and exceptions raised are as per :func:`~DirEntry.is_dir`. + + .. method:: is_symlink() + + Return ``True`` if this entry is a symbolic link (even if broken); + return ``False`` if the entry points to a directory or any kind of file, + or if it doesn't exist anymore. + + The result is cached on the ``DirEntry`` object. Call + :func:`os.path.islink` to fetch up-to-date information. + + On the first, uncached call, no system call is required in most cases. + Specifically, neither Windows or Unix require a system call, except on + certain Unix file systems, such as network file systems, that return + ``dirent.d_type == DT_UNKNOWN``. + + This method can raise :exc:`OSError`, such as :exc:`PermissionError`, + but :exc:`FileNotFoundError` is caught and not raised. + + .. method:: stat(\*, follow_symlinks=True) + + Return a :class:`stat_result` object for this entry. This method + follows symbolic links by default; to stat a symbolic link add the + ``follow_symlinks=False`` argument. + + On Unix, this method always requires a system call. On Windows, it + only requires a system call if *follow_symlinks* is ``True`` and the + entry is a symbolic link. + + On Windows, the ``st_ino``, ``st_dev`` and ``st_nlink`` attributes of the + :class:`stat_result` are always set to zero. Call :func:`os.stat` to + get these attributes. + + The result is cached on the ``DirEntry`` object, with a separate cache + for *follow_symlinks* ``True`` and ``False``. Call :func:`os.stat` to + fetch up-to-date information. + + Note that there is a nice correspondence between several attributes + and methods of ``DirEntry`` and of :class:`pathlib.Path`. In + particular, the ``name`` attribute has the same meaning, as do the + ``is_dir()``, ``is_file()``, ``is_symlink()`` and ``stat()`` methods. + + .. versionadded:: 3.5 + + .. function:: stat(path, \*, dir_fd=None, follow_symlinks=True) Get the status of a file or a file descriptor. Perform the equivalent of a @@ -1890,8 +2085,6 @@ features: >>> statinfo.st_size 264 - Availability: Unix, Windows. - .. seealso:: :func:`fstat` and :func:`lstat` functions. @@ -2039,6 +2232,15 @@ features: File type. + On Windows systems, the following attribute is also available: + + .. attribute:: st_file_attributes + + Windows file attributes: ``dwFileAttributes`` member of the + ``BY_HANDLE_FILE_INFORMATION`` structure returned by + :c:func:`GetFileInformationByHandle`. See the ``FILE_ATTRIBUTE_*`` + constants in the :mod:`stat` module. + The standard module :mod:`stat` defines functions and constants that are useful for extracting information from a :c:type:`stat` structure. (On Windows, some items are filled with dummy values.) @@ -2056,6 +2258,9 @@ features: Added the :attr:`st_atime_ns`, :attr:`st_mtime_ns`, and :attr:`st_ctime_ns` members. + .. versionadded:: 3.5 + Added the :attr:`st_file_attributes` member on Windows. + .. function:: stat_float_times([newvalue]) @@ -2259,19 +2464,19 @@ features: This function can support :ref:`specifying a file descriptor <path_fd>`. - Availability: Unix. + Availability: Unix, Windows. .. versionadded:: 3.3 + .. versionchanged:: 3.5 + Added support for Windows .. function:: unlink(path, *, dir_fd=None) - Remove (delete) the file *path*. This function is identical to - :func:`remove`; the ``unlink`` name is its traditional Unix - name. Please see the documentation for :func:`remove` for - further information. - - Availability: Unix, Windows. + Remove (delete) the file *path*. This function is semantically + identical to :func:`remove`; the ``unlink`` name is its + traditional Unix name. Please see the documentation for + :func:`remove` for further information. .. versionadded:: 3.3 The *dir_fd* parameter. @@ -2309,8 +2514,6 @@ features: :ref:`paths relative to directory descriptors <dir_fd>` and :ref:`not following symlinks <follow_symlinks>`. - Availability: Unix, Windows. - .. versionadded:: 3.3 Added support for specifying an open file descriptor for *path*, and the *dir_fd*, *follow_symlinks*, and *ns* parameters. @@ -2401,6 +2604,10 @@ features: for name in dirs: os.rmdir(os.path.join(root, name)) + .. versionchanged:: 3.5 + This function now calls :func:`os.scandir` instead of :func:`os.listdir`, + making it faster by reducing the number of calls to :func:`os.stat`. + .. function:: fwalk(top='.', topdown=True, onerror=None, *, follow_symlinks=False, dir_fd=None) @@ -2557,8 +2764,6 @@ to be ignored. Python signal handler registered for :const:`SIGABRT` with :func:`signal.signal`. - Availability: Unix, Windows. - .. function:: execl(path, arg0, arg1, ...) execle(path, arg0, arg1, ..., env) @@ -2622,8 +2827,6 @@ to be ignored. Exit the process with status *n*, without calling cleanup handlers, flushing stdio buffers, etc. - Availability: Unix, Windows. - .. note:: The standard way to exit is ``sys.exit(n)``. :func:`_exit` should @@ -2986,6 +3189,10 @@ written in Python, such as a mail server's external command delivery program. doesn't work if it is. Use the :func:`os.path.normpath` function to ensure that the path is properly encoded for Win32. + To reduce interpreter startup overhead, the Win32 :c:func:`ShellExecute` + function is not resolved until this function is first called. If the function + cannot be resolved, :exc:`NotImplementedError` will be raised. + Availability: Windows. @@ -3133,6 +3340,11 @@ written in Python, such as a mail server's external command delivery program. id is known, not necessarily a child process. The :func:`spawn\* <spawnl>` functions called with :const:`P_NOWAIT` return suitable process handles. + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise an + exception, the function now retries the system call instead of raising an + :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + .. function:: wait3(options) @@ -3286,7 +3498,7 @@ operating system. .. data:: SCHED_RESET_ON_FORK - This flag can OR'ed with any other scheduling policy. When a process with + This flag can be OR'ed with any other scheduling policy. When a process with this flag set forks, its child's scheduling policy and priority are reset to the default. @@ -3533,10 +3745,23 @@ Miscellaneous Functions This function returns random bytes from an OS-specific randomness source. The returned data should be unpredictable enough for cryptographic applications, - though its exact quality depends on the OS implementation. On a Unix-like - system this will query ``/dev/urandom``, and on Windows it will use - ``CryptGenRandom()``. If a randomness source is not found, + though its exact quality depends on the OS implementation. + + On Linux, ``getrandom()`` syscall is used if available and the urandom + entropy pool is initialized (``getrandom()`` does not block). + On a Unix-like system this will query ``/dev/urandom``. On Windows, it + will use ``CryptGenRandom()``. If a randomness source is not found, :exc:`NotImplementedError` will be raised. For an easy-to-use interface to the random number generator provided by your platform, please see :class:`random.SystemRandom`. + + .. versionchanged:: 3.5.2 + On Linux, if ``getrandom()`` blocks (the urandom entropy pool is not + initialized yet), fall back on reading ``/dev/urandom``. + + .. versionchanged:: 3.5 + On Linux 3.17 and newer, the ``getrandom()`` syscall is now used + when available. On OpenBSD 5.6 and newer, the C ``getentropy()`` + function is now used. These functions avoid the usage of an internal file + descriptor. diff --git a/Doc/library/ossaudiodev.rst b/Doc/library/ossaudiodev.rst index bb5081afde..ec40c0b93a 100644 --- a/Doc/library/ossaudiodev.rst +++ b/Doc/library/ossaudiodev.rst @@ -5,6 +5,7 @@ :platform: Linux, FreeBSD :synopsis: Access to OSS-compatible audio devices. +-------------- This module allows you to access the OSS (Open Sound System) audio interface. OSS is available for a wide range of open-source and commercial Unices, and is @@ -148,21 +149,30 @@ and (read-only) attributes: .. method:: oss_audio_device.write(data) - Write the Python string *data* to the audio device and return the number of - bytes written. If the audio device is in blocking mode (the default), the - entire string is always written (again, this is different from usual Unix device - semantics). If the device is in non-blocking mode, some data may not be written + Write a :term:`bytes-like object` *data* to the audio device and return the + number of bytes written. If the audio device is in blocking mode (the + default), the entire data is always written (again, this is different from + usual Unix device semantics). If the device is in non-blocking mode, some + data may not be written ---see :meth:`writeall`. + .. versionchanged:: 3.5 + Writable :term:`bytes-like object` is now accepted. + .. method:: oss_audio_device.writeall(data) - Write the entire Python string *data* to the audio device: waits until the audio - device is able to accept data, writes as much data as it will accept, and - repeats until *data* has been completely written. If the device is in blocking - mode (the default), this has the same effect as :meth:`write`; :meth:`writeall` - is only useful in non-blocking mode. Has no return value, since the amount of - data written is always equal to the amount of data supplied. + Write a :term:`bytes-like object` *data* to the audio device: waits until + the audio device is able to accept data, writes as much data as it will + accept, and repeats until *data* has been completely written. If the device + is in blocking mode (the default), this has the same effect as + :meth:`write`; :meth:`writeall` is only useful in non-blocking mode. Has + no return value, since the amount of data written is always equal to the + amount of data supplied. + + .. versionchanged:: 3.5 + Writable :term:`bytes-like object` is now accepted. + .. versionchanged:: 3.2 Audio device objects also support the context management protocol, i.e. they can diff --git a/Doc/library/othergui.rst b/Doc/library/othergui.rst index 43721b2688..ee1ce50b6b 100644 --- a/Doc/library/othergui.rst +++ b/Doc/library/othergui.rst @@ -8,40 +8,40 @@ available for Python: .. seealso:: - `PyGObject <https://live.gnome.org/PyGObject>`_ + `PyGObject <https://wiki.gnome.org/Projects/PyGObject>`_ provides introspection bindings for C libraries using `GObject <https://developer.gnome.org/gobject/stable/>`_. One of these libraries is the `GTK+ 3 <http://www.gtk.org/>`_ widget set. GTK+ comes with many more widgets than Tkinter provides. An online - `Python GTK+ 3 Tutorial <http://python-gtk-3-tutorial.readthedocs.org/en/latest/>`_ + `Python GTK+ 3 Tutorial <https://python-gtk-3-tutorial.readthedocs.org/en/latest/>`_ is available. `PyGTK <http://www.pygtk.org/>`_ provides bindings for an older version of the library, GTK+ 2. It provides an object oriented interface that is slightly higher level than the C one. There are also bindings to - `GNOME <http://www.gnome.org>`_. An online `tutorial + `GNOME <https://www.gnome.org/>`_. An online `tutorial <http://www.pygtk.org/pygtk2tutorial/index.html>`_ is available. - `PyQt <http://www.riverbankcomputing.co.uk/software/pyqt/intro>`_ + `PyQt <https://riverbankcomputing.com/software/pyqt/intro>`_ PyQt is a :program:`sip`\ -wrapped binding to the Qt toolkit. Qt is an extensive C++ GUI application development framework that is available for Unix, Windows and Mac OS X. :program:`sip` is a tool for generating bindings for C++ libraries as Python classes, and is specifically designed for Python. The *PyQt3* bindings have a book, `GUI Programming with Python: QT Edition - <http://www.commandprompt.com/community/pyqt/>`_ by Boudewijn + <https://www.commandprompt.com/community/pyqt/>`_ by Boudewijn Rempt. The *PyQt4* bindings also have a book, `Rapid GUI Programming - with Python and Qt <http://www.qtrac.eu/pyqtbook.html>`_, by Mark + with Python and Qt <https://www.qtrac.eu/pyqtbook.html>`_, by Mark Summerfield. - `PySide <http://qt-project.org/wiki/PySide>`_ + `PySide <https://wiki.qt.io/PySide>`_ is a newer binding to the Qt toolkit, provided by Nokia. Compared to PyQt, its licensing scheme is friendlier to non-open source applications. `wxPython <http://www.wxpython.org>`_ wxPython is a cross-platform GUI toolkit for Python that is built around - the popular `wxWidgets <http://www.wxwidgets.org/>`_ (formerly wxWindows) + the popular `wxWidgets <https://www.wxwidgets.org/>`_ (formerly wxWindows) C++ toolkit. It provides a native look and feel for applications on Windows, Mac OS X, and Unix systems by using each platform's native widgets where ever possible, (GTK+ on Unix-like systems). In addition to @@ -50,7 +50,7 @@ available for Python: low-level device context drawing, drag and drop, system clipboard access, an XML-based resource format and more, including an ever growing library of user-contributed modules. wxPython has a book, `wxPython in Action - <http://www.manning.com/rappin/>`_, by Noel Rappin and + <https://www.manning.com/books/wxpython-in-action>`_, by Noel Rappin and Robin Dunn. PyGTK, PyQt, and wxPython, all have a modern look and feel and more diff --git a/Doc/library/parser.rst b/Doc/library/parser.rst index 3e1e31bc79..c3b699a360 100644 --- a/Doc/library/parser.rst +++ b/Doc/library/parser.rst @@ -3,10 +3,10 @@ .. module:: parser :synopsis: Access parse trees for Python source code. + .. moduleauthor:: Fred L. Drake, Jr. <fdrake@acm.org> .. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org> - .. Copyright 1995 Virginia Polytechnic Institute and State University and Fred L. Drake, Jr. This copyright notice must be distributed on all copies, but this document otherwise may be distributed as part of the Python @@ -16,6 +16,8 @@ .. index:: single: parsing; Python source code +-------------- + The :mod:`parser` module provides an interface to Python's internal parser and byte-code compiler. The primary purpose for this interface is to allow Python code to edit the parse tree of a Python expression and create executable code diff --git a/Doc/library/pathlib.rst b/Doc/library/pathlib.rst index 24e2a308df..57a6a848f5 100644 --- a/Doc/library/pathlib.rst +++ b/Doc/library/pathlib.rst @@ -5,9 +5,13 @@ .. module:: pathlib :synopsis: Object-oriented filesystem paths +.. versionadded:: 3.4 + +**Source code:** :source:`Lib/pathlib.py` + .. index:: single: path; operations -.. versionadded:: 3.4 +-------------- This module offers classes representing filesystem paths with semantics appropriate for different operating systems. Path classes are divided @@ -628,6 +632,17 @@ call fails (for example because the path doesn't exist): PosixPath('/home/antoine/pathlib') +.. classmethod:: Path.home() + + Return a new path object representing the user's home directory (as + returned by :func:`os.path.expanduser` with ``~`` construct):: + + >>> Path.home() + PosixPath('/home/antoine') + + .. versionadded:: 3.5 + + .. method:: Path.stat() Return information about this path (similarly to :func:`os.stat`). @@ -670,6 +685,18 @@ call fails (for example because the path doesn't exist): symlink *points to* an existing file or directory. +.. method:: Path.expanduser() + + Return a new path with expanded ``~`` and ``~user`` constructs, + as returned by :meth:`os.path.expanduser`:: + + >>> p = PosixPath('~/films/Monty Python') + >>> p.expanduser() + PosixPath('/home/eric/films/Monty Python') + + .. versionadded:: 3.5 + + .. method:: Path.glob(pattern) Glob the given *pattern* in the directory represented by this path, @@ -791,7 +818,7 @@ call fails (for example because the path doesn't exist): the symbolic link's information rather than its target's. -.. method:: Path.mkdir(mode=0o777, parents=False) +.. method:: Path.mkdir(mode=0o777, parents=False, exist_ok=False) Create a new directory at this given path. If *mode* is given, it is combined with the process' ``umask`` value to determine the file mode @@ -805,6 +832,16 @@ call fails (for example because the path doesn't exist): If *parents* is false (the default), a missing parent raises :exc:`FileNotFoundError`. + If *exist_ok* is false (the default), an :exc:`FileExistsError` is + raised if the target directory already exists. + + If *exist_ok* is true, :exc:`FileExistsError` exceptions will be + ignored (same behavior as the POSIX ``mkdir -p`` command), but only if the + last path component is not an existing non-directory file. + + .. versionchanged:: 3.5 + The *exist_ok* parameter was added. + .. method:: Path.open(mode='r', buffering=-1, encoding=None, errors=None, newline=None) @@ -824,10 +861,39 @@ call fails (for example because the path doesn't exist): if the file's uid isn't found in the system database. +.. method:: Path.read_bytes() + + Return the binary contents of the pointed-to file as a bytes object:: + + >>> p = Path('my_binary_file') + >>> p.write_bytes(b'Binary file contents') + 20 + >>> p.read_bytes() + b'Binary file contents' + + .. versionadded:: 3.5 + + +.. method:: Path.read_text(encoding=None, errors=None) + + Return the decoded contents of the pointed-to file as a string:: + + >>> p = Path('my_text_file') + >>> p.write_text('Text file contents') + 18 + >>> p.read_text() + 'Text file contents' + + The optional parameters have the same meaning as in :func:`open`. + + .. versionadded:: 3.5 + + .. method:: Path.rename(target) - Rename this file or directory to the given *target*. *target* can be - either a string or another path object:: + Rename this file or directory to the given *target*. On Unix, if + *target* exists and is a file, it will be replaced silently if the user + has permission. *target* can be either a string or another path object:: >>> p = Path('foo') >>> p.open('w').write('some text') @@ -884,6 +950,25 @@ call fails (for example because the path doesn't exist): Remove this directory. The directory must be empty. +.. method:: Path.samefile(other_path) + + Return whether this path points to the same file as *other_path*, which + can be either a Path object, or a string. The semantics are similar + to :func:`os.path.samefile` and :func:`os.path.samestat`. + + An :exc:`OSError` can be raised if either file cannot be accessed for some + reason. + + >>> p = Path('spam') + >>> q = Path('eggs') + >>> p.samefile(q) + False + >>> p.samefile('spam') + True + + .. versionadded:: 3.5 + + .. method:: Path.symlink_to(target, target_is_directory=False) Make this path a symbolic link to *target*. Under Windows, @@ -917,3 +1002,33 @@ call fails (for example because the path doesn't exist): Remove this file or symbolic link. If the path points to a directory, use :func:`Path.rmdir` instead. + + +.. method:: Path.write_bytes(data) + + Open the file pointed to in bytes mode, write *data* to it, and close the + file:: + + >>> p = Path('my_binary_file') + >>> p.write_bytes(b'Binary file contents') + 20 + >>> p.read_bytes() + b'Binary file contents' + + An existing file of the same name is overwritten. + + .. versionadded:: 3.5 + + +.. method:: Path.write_text(data, encoding=None, errors=None) + + Open the file pointed to in text mode, write *data* to it, and close the + file:: + + >>> p = Path('my_text_file') + >>> p.write_text('Text file contents') + 18 + >>> p.read_text() + 'Text file contents' + + .. versionadded:: 3.5 diff --git a/Doc/library/pdb.rst b/Doc/library/pdb.rst index c144db6c1a..4520a9b6f3 100644 --- a/Doc/library/pdb.rst +++ b/Doc/library/pdb.rst @@ -8,10 +8,10 @@ **Source code:** :source:`Lib/pdb.py` --------------- - .. index:: single: debugging +-------------- + The module :mod:`pdb` defines an interactive source code debugger for Python programs. It supports setting (conditional) breakpoints and single stepping at the source line level, inspection of stack frames, source code listing, and diff --git a/Doc/library/pickle.rst b/Doc/library/pickle.rst index 3305195853..0d64191d79 100644 --- a/Doc/library/pickle.rst +++ b/Doc/library/pickle.rst @@ -1,6 +1,14 @@ :mod:`pickle` --- Python object serialization ============================================= +.. module:: pickle + :synopsis: Convert Python objects to streams of bytes and back. + +.. sectionauthor:: Jim Kerr <jbkerr@sr.hp.com>. +.. sectionauthor:: Barry Warsaw <barry@python.org> + +**Source code:** :source:`Lib/pickle.py` + .. index:: single: persistence pair: persistent; objects @@ -9,11 +17,7 @@ pair: flattening; objects pair: pickling; objects -.. module:: pickle - :synopsis: Convert Python objects to streams of bytes and back. -.. sectionauthor:: Jim Kerr <jbkerr@sr.hp.com>. -.. sectionauthor:: Barry Warsaw <barry@python.org> - +-------------- The :mod:`pickle` module implements binary protocols for serializing and de-serializing a Python object structure. *"Pickling"* is the process @@ -425,7 +429,7 @@ The following types can be pickled: Attempts to pickle unpicklable objects will raise the :exc:`PicklingError` exception; when this happens, an unspecified number of bytes may have already been written to the underlying file. Trying to pickle a highly recursive data -structure may exceed the maximum recursion depth, a :exc:`RuntimeError` will be +structure may exceed the maximum recursion depth, a :exc:`RecursionError` will be raised in this case. You can carefully raise this limit with :func:`sys.setrecursionlimit`. @@ -859,7 +863,7 @@ For the simplest code, use the :func:`dump` and :func:`load` functions. :: data = { 'a': [1, 2.0, 3, 4+6j], 'b': ("character string", b"byte string"), - 'c': set([None, True, False]) + 'c': {None, True, False} } with open('data.pickle', 'wb') as f: diff --git a/Doc/library/pickletools.rst b/Doc/library/pickletools.rst index 4c0a148bc3..5e5939c7d2 100644 --- a/Doc/library/pickletools.rst +++ b/Doc/library/pickletools.rst @@ -30,7 +30,9 @@ However, when the pickle file that you want to examine comes from an untrusted source, ``-m pickletools`` is a safer option because it does not execute pickle bytecode. -For example, with a tuple ``(1, 2)`` pickled in file ``x.pickle``:: +For example, with a tuple ``(1, 2)`` pickled in file ``x.pickle``: + +.. code-block:: shell-session $ python -m pickle x.pickle (1, 2) @@ -106,4 +108,3 @@ Programmatic Interface Returns a new equivalent pickle string after eliminating unused ``PUT`` opcodes. The optimized pickle is shorter, takes less transmission time, requires less storage space, and unpickles more efficiently. - diff --git a/Doc/library/pipes.rst b/Doc/library/pipes.rst index 69e891db29..0a22da1f55 100644 --- a/Doc/library/pipes.rst +++ b/Doc/library/pipes.rst @@ -4,6 +4,7 @@ .. module:: pipes :platform: Unix :synopsis: A Python interface to Unix shell pipelines. + .. sectionauthor:: Moshe Zadka <moshez@zadka.site.co.il> **Source code:** :source:`Lib/pipes.py` diff --git a/Doc/library/pkgutil.rst b/Doc/library/pkgutil.rst index 13ea7b9b33..26c5ac066b 100644 --- a/Doc/library/pkgutil.rst +++ b/Doc/library/pkgutil.rst @@ -58,7 +58,7 @@ support. .. deprecated:: 3.3 This emulation is no longer needed, as the standard import mechanism - is now fully PEP 302 compliant and available in :mod:`importlib` + is now fully PEP 302 compliant and available in :mod:`importlib`. .. class:: ImpLoader(fullname, file, filename, etc) @@ -67,7 +67,7 @@ support. .. deprecated:: 3.3 This emulation is no longer needed, as the standard import mechanism - is now fully PEP 302 compliant and available in :mod:`importlib` + is now fully PEP 302 compliant and available in :mod:`importlib`. .. function:: find_loader(fullname) @@ -140,7 +140,7 @@ support. .. function:: iter_modules(path=None, prefix='') Yields ``(module_finder, name, ispkg)`` for all submodules on *path*, or, if - path is ``None``, all top-level modules on ``sys.path``. + *path* is ``None``, all top-level modules on ``sys.path``. *path* should be either ``None`` or a list of paths to look for modules in. @@ -161,7 +161,7 @@ support. .. function:: walk_packages(path=None, prefix='', onerror=None) Yields ``(module_finder, name, ispkg)`` for all modules recursively on - *path*, or, if path is ``None``, all accessible modules. + *path*, or, if *path* is ``None``, all accessible modules. *path* should be either ``None`` or a list of paths to look for modules in. diff --git a/Doc/library/platform.rst b/Doc/library/platform.rst index 66b6892f17..eea0abbae4 100644 --- a/Doc/library/platform.rst +++ b/Doc/library/platform.rst @@ -3,6 +3,7 @@ .. module:: platform :synopsis: Retrieves as much platform identifying data as possible. + .. moduleauthor:: Marc-André Lemburg <mal@egenix.com> .. sectionauthor:: Bjorn Pettersen <bpettersen@corp.fairisaac.com> @@ -247,6 +248,8 @@ Unix Platforms This is another name for :func:`linux_distribution`. + .. deprecated-removed:: 3.5 3.7 + .. function:: linux_distribution(distname='', version='', id='', supported_dists=('SuSE','debian','redhat','mandrake',...), full_distribution_name=1) Tries to determine the name of the Linux OS distribution name. @@ -263,6 +266,8 @@ Unix Platforms parameters. ``id`` is the item in parentheses after the version number. It is usually the version codename. + .. deprecated-removed:: 3.5 3.7 + .. function:: libc_ver(executable=sys.executable, lib='', version='', chunksize=2048) Tries to determine the libc version against which the file executable (defaults diff --git a/Doc/library/plistlib.rst b/Doc/library/plistlib.rst index 2c1f3dd79e..9ba226607d 100644 --- a/Doc/library/plistlib.rst +++ b/Doc/library/plistlib.rst @@ -3,16 +3,17 @@ .. module:: plistlib :synopsis: Generate and parse Mac OS X plist files. + .. moduleauthor:: Jack Jansen .. sectionauthor:: Georg Brandl <georg@python.org> .. (harvested from docstrings in the original file) +**Source code:** :source:`Lib/plistlib.py` + .. index:: pair: plist; file single: property list -**Source code:** :source:`Lib/plistlib.py` - -------------- This module provides an interface for reading and writing the "property list" diff --git a/Doc/library/poplib.rst b/Doc/library/poplib.rst index 45baad96af..ffabc32b6e 100644 --- a/Doc/library/poplib.rst +++ b/Doc/library/poplib.rst @@ -3,13 +3,14 @@ .. module:: poplib :synopsis: POP3 protocol client (requires sockets). + .. sectionauthor:: Andrew T. Csillag .. revised by ESR, January 2000 -.. index:: pair: POP3; protocol - **Source code:** :source:`Lib/poplib.py` +.. index:: pair: POP3; protocol + -------------- This module defines a class, :class:`POP3`, which encapsulates a connection to a @@ -194,6 +195,15 @@ An :class:`POP3` instance has the following methods: the unique id for that message in the form ``'response mesgnum uid``, otherwise result is list ``(response, ['mesgnum uid', ...], octets)``. + +.. method:: POP3.utf8() + + Try to switch to UTF-8 mode. Returns the server response if successful, + raises :class:`error_proto` if not. Specified in :RFC:`6856`. + + .. versionadded:: 3.5 + + .. method:: POP3.stls(context=None) Start a TLS session on the active connection as specified in :rfc:`2595`. diff --git a/Doc/library/posix.rst b/Doc/library/posix.rst index 06bab04aaa..9cbc5505ae 100644 --- a/Doc/library/posix.rst +++ b/Doc/library/posix.rst @@ -5,6 +5,7 @@ :platform: Unix :synopsis: The most common POSIX system calls (normally used via module os). +-------------- This module provides access to operating system functionality that is standardized by the C Standard and the POSIX standard (a thinly disguised Unix diff --git a/Doc/library/pprint.rst b/Doc/library/pprint.rst index c0589a31a9..65d94fe386 100644 --- a/Doc/library/pprint.rst +++ b/Doc/library/pprint.rst @@ -3,6 +3,7 @@ .. module:: pprint :synopsis: Data pretty printer. + .. moduleauthor:: Fred L. Drake, Jr. <fdrake@acm.org> .. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org> @@ -197,7 +198,7 @@ are converted to strings. The default implementation uses the internals of the the current presentation context (direct and indirect containers for *object* that are affecting the presentation) as the keys; if an object needs to be presented which is already represented in *context*, the third return value - should be ``True``. Recursive calls to the :meth:`format` method should add + should be ``True``. Recursive calls to the :meth:`.format` method should add additional entries for containers to this dictionary. The third argument, *maxlevels*, gives the requested limit to recursion; this will be ``0`` if there is no requested limit. This argument should be passed unmodified to recursive @@ -235,10 +236,10 @@ In its basic form, :func:`pprint` shows the whole object:: 'classifiers': ['Programming Language :: Python :: 2.6', 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 2 :: Only'], - 'description': 'An extensible framework for Python programming, ' - 'with special focus\r\n' - 'on event-based network programming and ' - 'multiprotocol integration.', + 'description': 'An extensible framework for Python programming, with ' + 'special focus\r\n' + 'on event-based network programming and multiprotocol ' + 'integration.', 'docs_url': '', 'download_url': 'UNKNOWN', 'home_page': 'http://twistedmatrix.com/', @@ -288,10 +289,10 @@ contents):: 'cheesecake_documentation_id': None, 'cheesecake_installability_id': None, 'classifiers': [...], - 'description': 'An extensible framework for Python programming, ' - 'with special focus\r\n' - 'on event-based network programming and ' - 'multiprotocol integration.', + 'description': 'An extensible framework for Python programming, with ' + 'special focus\r\n' + 'on event-based network programming and multiprotocol ' + 'integration.', 'docs_url': '', 'download_url': 'UNKNOWN', 'home_page': 'http://twistedmatrix.com/', @@ -323,13 +324,12 @@ cannot be split, the specified width will be exceeded:: 'cheesecake_installability_id': None, 'classifiers': [...], 'description': 'An extensible ' - 'framework for ' - 'Python programming, ' - 'with special ' - 'focus\r\n' - 'on event-based ' - 'network programming ' - 'and multiprotocol ' + 'framework for Python ' + 'programming, with ' + 'special focus\r\n' + 'on event-based network ' + 'programming and ' + 'multiprotocol ' 'integration.', 'docs_url': '', 'download_url': 'UNKNOWN', @@ -344,8 +344,8 @@ cannot be split, the specified width will be exceeded:: 'release_url': 'http://pypi.python.org/pypi/Twisted/12.3.0', 'requires_python': None, 'stable_version': None, - 'summary': 'An asynchronous ' - 'networking framework ' - 'written in Python', + 'summary': 'An asynchronous networking ' + 'framework written in ' + 'Python', 'version': '12.3.0'}, 'urls': [{...}, {...}]} diff --git a/Doc/library/profile.rst b/Doc/library/profile.rst index 2928821d55..959d9b98a8 100644 --- a/Doc/library/profile.rst +++ b/Doc/library/profile.rst @@ -33,7 +33,7 @@ profiling interface: 2. :mod:`profile`, a pure Python module whose interface is imitated by :mod:`cProfile`, but which adds significant overhead to profiled programs. If you're trying to extend the profiler in some way, the task might be easier - with this module. + with this module. Originally designed and written by Jim Roskind. .. note:: diff --git a/Doc/library/pty.rst b/Doc/library/pty.rst index b8a3897ab8..0ab766065d 100644 --- a/Doc/library/pty.rst +++ b/Doc/library/pty.rst @@ -4,9 +4,13 @@ .. module:: pty :platform: Linux :synopsis: Pseudo-Terminal Handling for Linux. + .. moduleauthor:: Steen Lumholt .. sectionauthor:: Moshe Zadka <moshez@zadka.site.co.il> +**Source code:** :source:`Lib/pty.py` + +-------------- The :mod:`pty` module defines operations for handling the pseudo-terminal concept: starting another process and being able to write to and read from its diff --git a/Doc/library/pwd.rst b/Doc/library/pwd.rst index 2c17d9e036..03ebb02e4e 100644 --- a/Doc/library/pwd.rst +++ b/Doc/library/pwd.rst @@ -5,6 +5,7 @@ :platform: Unix :synopsis: The password database (getpwnam() and friends). +-------------- This module provides access to the Unix user account and password database. It is available on all Unix versions. diff --git a/Doc/library/py_compile.rst b/Doc/library/py_compile.rst index bae8450b7c..65f76b8bb8 100644 --- a/Doc/library/py_compile.rst +++ b/Doc/library/py_compile.rst @@ -3,13 +3,14 @@ .. module:: py_compile :synopsis: Generate byte-code files from Python source files. + .. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org> .. documentation based on module docstrings -.. index:: pair: file; byte-code - **Source code:** :source:`Lib/py_compile.py` +.. index:: pair: file; byte-code + -------------- The :mod:`py_compile` module provides a function to generate a byte-code file @@ -29,9 +30,9 @@ byte-code cache files in the directory containing the source code. .. function:: compile(file, cfile=None, dfile=None, doraise=False, optimize=-1) Compile a source file to byte-code and write out the byte-code cache file. - The source code is loaded from the file name *file*. The byte-code is - written to *cfile*, which defaults to the :PEP:`3147` path, ending in - ``.pyc`` (``.pyo`` if optimization is enabled in the current interpreter). + The source code is loaded from the file name *file*. The byte-code is + written to *cfile*, which defaults to the :pep:`3147`/:pep:`488` path, ending + in ``.pyc``. For example, if *file* is ``/foo/bar/baz.py`` *cfile* will default to ``/foo/bar/__pycache__/baz.cpython-32.pyc`` for Python 3.2. If *dfile* is specified, it is used as the name of the source file in error messages when @@ -68,7 +69,7 @@ byte-code cache files in the directory containing the source code. .. function:: main(args=None) Compile several source files. The files named in *args* (or on the command - line, if *args* is ``None``) are compiled and the resulting bytecode is + line, if *args* is ``None``) are compiled and the resulting byte-code is cached in the normal manner. This function does not search a directory structure to locate source files; it only compiles files named explicitly. If ``'-'`` is the only parameter in args, the list of files is taken from @@ -86,4 +87,3 @@ could not be compiled. Module :mod:`compileall` Utilities to compile all Python source files in a directory tree. - diff --git a/Doc/library/pyclbr.rst b/Doc/library/pyclbr.rst index 13eaabf594..32842717bc 100644 --- a/Doc/library/pyclbr.rst +++ b/Doc/library/pyclbr.rst @@ -3,6 +3,7 @@ .. module:: pyclbr :synopsis: Supports information extraction for a Python class browser. + .. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org> **Source code:** :source:`Lib/pyclbr.py` diff --git a/Doc/library/pydoc.rst b/Doc/library/pydoc.rst index b5e3233619..f1bfab9a3b 100644 --- a/Doc/library/pydoc.rst +++ b/Doc/library/pydoc.rst @@ -3,17 +3,17 @@ .. module:: pydoc :synopsis: Documentation generator and online help system. + .. moduleauthor:: Ka-Ping Yee <ping@lfw.org> .. sectionauthor:: Ka-Ping Yee <ping@lfw.org> +**Source code:** :source:`Lib/pydoc.py` .. index:: single: documentation; generation single: documentation; online single: help; online -**Source code:** :source:`Lib/pydoc.py` - -------------- The :mod:`pydoc` module automatically generates documentation from Python diff --git a/Doc/library/pyexpat.rst b/Doc/library/pyexpat.rst index 620ffb1cd4..075a8b5139 100644 --- a/Doc/library/pyexpat.rst +++ b/Doc/library/pyexpat.rst @@ -3,8 +3,10 @@ .. module:: xml.parsers.expat :synopsis: An interface to the Expat non-validating XML parser. + .. moduleauthor:: Paul Prescod <paul@prescod.net> +-------------- .. Markup notes: @@ -84,7 +86,9 @@ The :mod:`xml.parsers.expat` module contains two functions: separator. For example, if *namespace_separator* is set to a space character (``' '``) and - the following document is parsed:: + the following document is parsed: + + .. code-block:: xml <?xml version="1.0"?> <root xmlns = "http://default-namespace.org/" @@ -242,7 +246,7 @@ XMLParser Objects The following attributes contain values relating to the most recent error encountered by an :class:`xmlparser` object, and will only have correct values -once a call to :meth:`Parse` or :meth:`ParseFile` has raised a +once a call to :meth:`Parse` or :meth:`ParseFile` has raised an :exc:`xml.parsers.expat.ExpatError` exception. @@ -570,9 +574,9 @@ Content Model Descriptions .. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org> -Content modules are described using nested tuples. Each tuple contains four +Content models are described using nested tuples. Each tuple contains four values: the type, the quantifier, the name, and a tuple of children. Children -are simply additional content module descriptions. +are simply additional content model descriptions. The values of the first two fields are constants defined in the :mod:`xml.parsers.expat.model` module. These constants can be collected in two @@ -867,6 +871,6 @@ The ``errors`` module has the following attributes: .. [#] The encoding string included in XML output should conform to the appropriate standards. For example, "UTF-8" is valid, but "UTF8" is - not. See http://www.w3.org/TR/2006/REC-xml11-20060816/#NT-EncodingDecl - and http://www.iana.org/assignments/character-sets/character-sets.xhtml. + not. See https://www.w3.org/TR/2006/REC-xml11-20060816/#NT-EncodingDecl + and https://www.iana.org/assignments/character-sets/character-sets.xhtml. diff --git a/Doc/library/queue.rst b/Doc/library/queue.rst index 680d690052..1cb0935377 100644 --- a/Doc/library/queue.rst +++ b/Doc/library/queue.rst @@ -158,22 +158,32 @@ fully processed by daemon consumer threads. Example of how to wait for enqueued tasks to be completed:: - def worker(): - while True: - item = q.get() - do_work(item) - q.task_done() - - q = Queue() - for i in range(num_worker_threads): - t = Thread(target=worker) - t.daemon = True + def worker(): + while True: + item = q.get() + if item is None: + break + do_work(item) + q.task_done() + + q = queue.Queue() + threads = [] + for i in range(num_worker_threads): + t = threading.Thread(target=worker) t.start() + threads.append(t) - for item in source(): - q.put(item) + for item in source(): + q.put(item) - q.join() # block until all tasks are done + # block until all tasks are done + q.join() + + # stop workers + for i in range(num_worker_threads): + q.put(None) + for t in threads: + t.join() .. seealso:: diff --git a/Doc/library/quopri.rst b/Doc/library/quopri.rst index 3a74cf8511..3c31125435 100644 --- a/Doc/library/quopri.rst +++ b/Doc/library/quopri.rst @@ -4,13 +4,12 @@ .. module:: quopri :synopsis: Encode and decode files using the MIME quoted-printable encoding. +**Source code:** :source:`Lib/quopri.py` .. index:: pair: quoted-printable; encoding single: MIME; quoted-printable encoding -**Source code:** :source:`Lib/quopri.py` - -------------- This module performs quoted-printable transport encoding and decoding, as diff --git a/Doc/library/random.rst b/Doc/library/random.rst index 11dd367f8a..df502a0aff 100644 --- a/Doc/library/random.rst +++ b/Doc/library/random.rst @@ -20,7 +20,7 @@ On the real line, there are functions to compute uniform, normal (Gaussian), lognormal, negative exponential, gamma, and beta distributions. For generating distributions of angles, the von Mises distribution is available. -Almost all module functions depend on the basic function :func:`random`, which +Almost all module functions depend on the basic function :func:`.random`, which generates a random float uniformly in the semi-open range [0.0, 1.0). Python uses the Mersenne Twister as the core generator. It produces 53-bit precision floats and has a period of 2\*\*19937-1. The underlying implementation in C is @@ -34,9 +34,9 @@ instance of the :class:`random.Random` class. You can instantiate your own instances of :class:`Random` to get generators that don't share state. Class :class:`Random` can also be subclassed if you want to use a different -basic generator of your own devising: in that case, override the :meth:`random`, -:meth:`seed`, :meth:`getstate`, and :meth:`setstate` methods. -Optionally, a new generator can supply a :meth:`getrandbits` method --- this +basic generator of your own devising: in that case, override the :meth:`~Random.random`, +:meth:`~Random.seed`, :meth:`~Random.getstate`, and :meth:`~Random.setstate` methods. +Optionally, a new generator can supply a :meth:`~Random.getrandbits` method --- this allows :meth:`randrange` to produce selections over an arbitrarily large range. The :mod:`random` module also provides the :class:`SystemRandom` class which @@ -46,8 +46,7 @@ from sources provided by the operating system. .. warning:: The pseudo-random generators of this module should not be used for - security purposes. Use :func:`os.urandom` or :class:`SystemRandom` if - you require a cryptographically secure pseudo-random number generator. + security purposes. Bookkeeping functions: @@ -126,7 +125,7 @@ Functions for sequences: Shuffle the sequence *x* in place. The optional argument *random* is a 0-argument function returning a random float in [0.0, 1.0); by default, this is - the function :func:`random`. + the function :func:`.random`. Note that for even rather small ``len(x)``, the total number of permutations of *x* is larger than the period of most random number generators; this implies @@ -268,7 +267,7 @@ Alternative Generator: `Complementary-Multiply-with-Carry recipe - <http://code.activestate.com/recipes/576707/>`_ for a compatible alternative + <https://code.activestate.com/recipes/576707/>`_ for a compatible alternative random number generator with a long period and comparatively simple update operations. @@ -286,7 +285,7 @@ change across Python versions, but two aspects are guaranteed not to change: * If a new seeding method is added, then a backward compatible seeder will be offered. -* The generator's :meth:`random` method will continue to produce the same +* The generator's :meth:`~Random.random` method will continue to produce the same sequence when the compatible seeder is given the same seed. .. _random-examples: diff --git a/Doc/library/re.rst b/Doc/library/re.rst index cae6874833..569b522332 100644 --- a/Doc/library/re.rst +++ b/Doc/library/re.rst @@ -3,16 +3,20 @@ .. module:: re :synopsis: Regular expression operations. + .. moduleauthor:: Fredrik Lundh <fredrik@pythonware.com> .. sectionauthor:: Andrew M. Kuchling <amk@amk.ca> +**Source code:** :source:`Lib/re.py` + +-------------- This module provides regular expression matching operations similar to those found in Perl. Both patterns and strings to be searched can be Unicode strings as well as 8-bit strings. However, Unicode strings and 8-bit strings cannot be mixed: -that is, you cannot match an Unicode string with a byte pattern or +that is, you cannot match a Unicode string with a byte pattern or vice-versa; similarly, when asking for a substitution, the replacement string must be of the same type as both the pattern and the search string. @@ -113,11 +117,11 @@ The special characters are: ``*?``, ``+?``, ``??`` The ``'*'``, ``'+'``, and ``'?'`` qualifiers are all :dfn:`greedy`; they match as much text as possible. Sometimes this behaviour isn't desired; if the RE - ``<.*>`` is matched against ``'<H1>title</H1>'``, it will match the entire - string, and not just ``'<H1>'``. Adding ``'?'`` after the qualifier makes it + ``<.*>`` is matched against ``<a> b <c>``, it will match the entire + string, and not just ``<a>``. Adding ``?`` after the qualifier makes it perform the match in :dfn:`non-greedy` or :dfn:`minimal` fashion; as *few* - characters as possible will be matched. Using ``.*?`` in the previous - expression will match only ``'<H1>'``. + characters as possible will be matched. Using the RE ``<.*?>`` will match + only ``<a>``. ``{m}`` Specifies that exactly *m* copies of the previous RE should be matched; fewer @@ -281,9 +285,7 @@ The special characters are: assertion`. ``(?<=abc)def`` will find a match in ``abcdef``, since the lookbehind will back up 3 characters and check if the contained pattern matches. The contained pattern must only match strings of some fixed length, meaning that - ``abc`` or ``a|b`` are allowed, but ``a*`` and ``a{3,4}`` are not. Group - references are not supported even if they match strings of some fixed length. - Note that + ``abc`` or ``a|b`` are allowed, but ``a*`` and ``a{3,4}`` are not. Note that patterns which start with positive lookbehind assertions will not match at the beginning of the string being searched; you will most likely want to use the :func:`search` function rather than the :func:`match` function: @@ -299,12 +301,14 @@ The special characters are: >>> m.group(0) 'egg' + .. versionchanged:: 3.5 + Added support for group references of fixed length. + ``(?<!...)`` Matches if the current position in the string is not preceded by a match for ``...``. This is called a :dfn:`negative lookbehind assertion`. Similar to positive lookbehind assertions, the contained pattern must only match strings of - some fixed length and shouldn't contain group references. - Patterns which start with negative lookbehind assertions may + some fixed length. Patterns which start with negative lookbehind assertions may match at the beginning of the string being searched. ``(?(id/name)yes-pattern|no-pattern)`` @@ -438,6 +442,10 @@ three digits in length. .. versionchanged:: 3.3 The ``'\u'`` and ``'\U'`` escape sequences have been added. +.. deprecated-removed:: 3.5 3.6 + Unknown escapes consisting of ``'\'`` and ASCII letter now raise a + deprecation warning and will be forbidden in Python 3.6. + .. seealso:: @@ -524,7 +532,11 @@ form. current locale. The use of this flag is discouraged as the locale mechanism is very unreliable, and it only handles one "culture" at a time anyway; you should use Unicode matching instead, which is the default in Python 3 - for Unicode (str) patterns. + for Unicode (str) patterns. This flag makes sense only with bytes patterns. + + .. deprecated-removed:: 3.5 3.6 + Deprecated the use of :const:`re.LOCALE` with string patterns or + :const:`re.ASCII`. .. data:: M @@ -627,17 +639,37 @@ form. That way, separator components are always found at the same relative indices within the result list. - Note that *split* will never split a string on an empty pattern match. - For example: + .. note:: + + :func:`split` doesn't currently split a string on an empty pattern match. + For example: + + >>> re.split('x*', 'axbc') + ['a', 'bc'] - >>> re.split('x*', 'foo') - ['foo'] - >>> re.split("(?m)^$", "foo\n\nbar\n") - ['foo\n\nbar\n'] + Even though ``'x*'`` also matches 0 'x' before 'a', between 'b' and 'c', + and after 'c', currently these matches are ignored. The correct behavior + (i.e. splitting on empty matches too and returning ``['', 'a', 'b', 'c', + '']``) will be implemented in future versions of Python, but since this + is a backward incompatible change, a :exc:`FutureWarning` will be raised + in the meanwhile. + + Patterns that can only match empty strings currently never split the + string. Since this doesn't match the expected behavior, a + :exc:`ValueError` will be raised starting from Python 3.5:: + + >>> re.split("^$", "foo\n\nbar\n", flags=re.M) + Traceback (most recent call last): + File "<stdin>", line 1, in <module> + ... + ValueError: split() requires a non-empty pattern match. .. versionchanged:: 3.1 Added the optional flags argument. + .. versionchanged:: 3.5 + Splitting on a pattern that could match an empty string now raises + a warning. Patterns that can only match empty strings are now rejected. .. function:: findall(pattern, string, flags=0) @@ -665,7 +697,7 @@ form. *string* is returned unchanged. *repl* can be a string or a function; if it is a string, any backslash escapes in it are processed. That is, ``\n`` is converted to a single newline character, ``\r`` is converted to a carriage return, and - so forth. Unknown escapes such as ``\j`` are left alone. Backreferences, such + so forth. Unknown escapes such as ``\&`` are left alone. Backreferences, such as ``\6``, are replaced with the substring matched by group 6 in the pattern. For example: @@ -707,6 +739,13 @@ form. .. versionchanged:: 3.1 Added the optional flags argument. + .. versionchanged:: 3.5 + Unmatched groups are replaced with an empty string. + + .. deprecated-removed:: 3.5 3.6 + Unknown escapes consist of ``'\'`` and ASCII letter now raise a + deprecation warning and will be forbidden in Python 3.6. + .. function:: subn(pattern, repl, string, count=0, flags=0) @@ -716,6 +755,9 @@ form. .. versionchanged:: 3.1 Added the optional flags argument. + .. versionchanged:: 3.5 + Unmatched groups are replaced with an empty string. + .. function:: escape(string) @@ -732,13 +774,36 @@ form. Clear the regular expression cache. -.. exception:: error +.. exception:: error(msg, pattern=None, pos=None) Exception raised when a string passed to one of the functions here is not a valid regular expression (for example, it might contain unmatched parentheses) or when some other error occurs during compilation or matching. It is never an - error if a string contains no match for a pattern. + error if a string contains no match for a pattern. The error instance has + the following additional attributes: + + .. attribute:: msg + + The unformatted error message. + + .. attribute:: pattern + + The regular expression pattern. + + .. attribute:: pos + + The index of *pattern* where compilation failed. + + .. attribute:: lineno + + The line corresponding to *pos*. + + .. attribute:: colno + + The column corresponding to *pos*. + .. versionchanged:: 3.5 + Added additional attributes. .. _re-objects: @@ -750,8 +815,8 @@ attributes: .. method:: regex.search(string[, pos[, endpos]]) - Scan through *string* looking for a location where this regular expression - produces a match, and return a corresponding :ref:`match object + Scan through *string* looking for the first location where this regular + expression produces a match, and return a corresponding :ref:`match object <match-objects>`. Return ``None`` if no position in the string matches the pattern; note that this is different from finding a zero-length match at some point in the string. @@ -891,6 +956,8 @@ Match objects support the following methods and attributes: (``\g<1>``, ``\g<name>``) are replaced by the contents of the corresponding group. + .. versionchanged:: 3.5 + Unmatched groups are replaced with an empty string. .. method:: match.group([group1, ...]) @@ -1171,15 +1238,15 @@ does by default). For example:: - >>> re.match("c", "abcdef") # No match - >>> re.search("c", "abcdef") # Match + >>> re.match("c", "abcdef") # No match + >>> re.search("c", "abcdef") # Match <_sre.SRE_Match object; span=(2, 3), match='c'> Regular expressions beginning with ``'^'`` can be used with :func:`search` to restrict the match at the beginning of the string:: - >>> re.match("c", "abcdef") # No match - >>> re.search("^c", "abcdef") # No match + >>> re.match("c", "abcdef") # No match + >>> re.search("^c", "abcdef") # No match >>> re.search("^a", "abcdef") # Match <_sre.SRE_Match object; span=(0, 1), match='a'> @@ -1260,9 +1327,9 @@ a function to "munge" text, or randomize the order of all the characters in each word of a sentence except for the first and last characters:: >>> def repl(m): - ... inner_word = list(m.group(2)) - ... random.shuffle(inner_word) - ... return m.group(1) + "".join(inner_word) + m.group(3) + ... inner_word = list(m.group(2)) + ... random.shuffle(inner_word) + ... return m.group(1) + "".join(inner_word) + m.group(3) >>> text = "Professor Abdolmalek, please report your absences promptly." >>> re.sub(r"(\w)(\w+)(\w)", repl, text) 'Poefsrosr Aealmlobdk, pslaee reorpt your abnseces plmrptoy.' @@ -1326,7 +1393,7 @@ functionally identical: Writing a Tokenizer ^^^^^^^^^^^^^^^^^^^ -A `tokenizer or scanner <http://en.wikipedia.org/wiki/Lexical_analysis>`_ +A `tokenizer or scanner <https://en.wikipedia.org/wiki/Lexical_analysis>`_ analyzes a string to categorize groups of characters. This is a useful first step in writing a compiler or interpreter. @@ -1342,14 +1409,14 @@ successive matches:: def tokenize(code): keywords = {'IF', 'THEN', 'ENDIF', 'FOR', 'NEXT', 'GOSUB', 'RETURN'} token_specification = [ - ('NUMBER', r'\d+(\.\d*)?'), # Integer or decimal number - ('ASSIGN', r':='), # Assignment operator - ('END', r';'), # Statement terminator - ('ID', r'[A-Za-z]+'), # Identifiers - ('OP', r'[+\-*/]'), # Arithmetic operators - ('NEWLINE', r'\n'), # Line endings - ('SKIP', r'[ \t]+'), # Skip over spaces and tabs - ('MISMATCH',r'.'), # Any other character + ('NUMBER', r'\d+(\.\d*)?'), # Integer or decimal number + ('ASSIGN', r':='), # Assignment operator + ('END', r';'), # Statement terminator + ('ID', r'[A-Za-z]+'), # Identifiers + ('OP', r'[+\-*/]'), # Arithmetic operators + ('NEWLINE', r'\n'), # Line endings + ('SKIP', r'[ \t]+'), # Skip over spaces and tabs + ('MISMATCH',r'.'), # Any other character ] tok_regex = '|'.join('(?P<%s>%s)' % pair for pair in token_specification) line_num = 1 diff --git a/Doc/library/readline.rst b/Doc/library/readline.rst index 692310b941..4d3c099ed2 100644 --- a/Doc/library/readline.rst +++ b/Doc/library/readline.rst @@ -4,123 +4,202 @@ .. module:: readline :platform: Unix :synopsis: GNU readline support for Python. + .. sectionauthor:: Skip Montanaro <skip@pobox.com> +-------------- The :mod:`readline` module defines a number of functions to facilitate completion and reading/writing of history files from the Python interpreter. -This module can be used directly or via the :mod:`rlcompleter` module. Settings +This module can be used directly, or via the :mod:`rlcompleter` module, which +supports completion of Python identifiers at the interactive prompt. Settings made using this module affect the behaviour of both the interpreter's interactive prompt and the prompts offered by the built-in :func:`input` function. .. note:: - On MacOS X the :mod:`readline` module can be implemented using + The underlying Readline library API may be implemented by the ``libedit`` library instead of GNU readline. + On MacOS X the :mod:`readline` module detects which library is being used + at run time. The configuration file for ``libedit`` is different from that of GNU readline. If you programmatically load configuration strings you can check for the text "libedit" in :const:`readline.__doc__` to differentiate between GNU readline and libedit. +Readline keybindings may be configured via an initialization file, typically +``.inputrc`` in your home directory. See `Readline Init File +<https://cnswww.cns.cwru.edu/php/chet/readline/rluserman.html#SEC9>`_ +in the GNU Readline manual for information about the format and +allowable constructs of that file, and the capabilities of the +Readline library in general. + -The :mod:`readline` module defines the following functions: +Init file +--------- + +The following functions relate to the init file and user configuration: .. function:: parse_and_bind(string) - Parse and execute single line of a readline init file. + Execute the init line provided in the *string* argument. This calls + :c:func:`rl_parse_and_bind` in the underlying library. + + +.. function:: read_init_file([filename]) + + Execute a readline initialization file. The default filename is the last filename + used. This calls :c:func:`rl_read_init_file` in the underlying library. + + +Line buffer +----------- + +The following functions operate on the line buffer: .. function:: get_line_buffer() - Return the current contents of the line buffer. + Return the current contents of the line buffer (:c:data:`rl_line_buffer` + in the underlying library). .. function:: insert_text(string) - Insert text into the command line. + Insert text into the line buffer at the cursor position. This calls + :c:func:`rl_insert_text` in the underlying library, but ignores + the return value. -.. function:: read_init_file([filename]) +.. function:: redisplay() + + Change what's displayed on the screen to reflect the current contents of the + line buffer. This calls :c:func:`rl_redisplay` in the underlying library. + - Parse a readline initialization file. The default filename is the last filename - used. +History file +------------ + +The following functions operate on a history file: .. function:: read_history_file([filename]) - Load a readline history file. The default filename is :file:`~/.history`. + Load a readline history file, and append it to the history list. + The default filename is :file:`~/.history`. This calls + :c:func:`read_history` in the underlying library. .. function:: write_history_file([filename]) - Save a readline history file. The default filename is :file:`~/.history`. + Save the history list to a readline history file, overwriting any + existing file. The default filename is :file:`~/.history`. This calls + :c:func:`write_history` in the underlying library. -.. function:: clear_history() +.. function:: append_history_file(nelements[, filename]) - Clear the current history. (Note: this function is not available if the - installed version of GNU readline doesn't support it.) + Append the last *nelements* items of history to a file. The default filename is + :file:`~/.history`. The file must already exist. This calls + :c:func:`append_history` in the underlying library. This function + only exists if Python was compiled for a version of the library + that supports it. + + .. versionadded:: 3.5 .. function:: get_history_length() + set_history_length(length) - Return the desired length of the history file. Negative values imply unlimited - history file size. + Set or return the desired number of lines to save in the history file. + The :func:`write_history_file` function uses this value to truncate + the history file, by calling :c:func:`history_truncate_file` in + the underlying library. Negative values imply + unlimited history file size. -.. function:: set_history_length(length) +History list +------------ - Set the number of lines to save in the history file. :func:`write_history_file` - uses this value to truncate the history file when saving. Negative values imply - unlimited history file size. +The following functions operate on a global history list: + + +.. function:: clear_history() + + Clear the current history. This calls :c:func:`clear_history` in the + underlying library. The Python function only exists if Python was + compiled for a version of the library that supports it. .. function:: get_current_history_length() - Return the number of lines currently in the history. (This is different from + Return the number of items currently in the history. (This is different from :func:`get_history_length`, which returns the maximum number of lines that will be written to a history file.) .. function:: get_history_item(index) - Return the current contents of history item at *index*. + Return the current contents of history item at *index*. The item index + is one-based. This calls :c:func:`history_get` in the underlying library. .. function:: remove_history_item(pos) Remove history item specified by its position from the history. + The position is zero-based. This calls :c:func:`remove_history` in + the underlying library. .. function:: replace_history_item(pos, line) - Replace history item specified by its position with the given line. + Replace history item specified by its position with *line*. + The position is zero-based. This calls :c:func:`replace_history_entry` + in the underlying library. -.. function:: redisplay() +.. function:: add_history(line) - Change what's displayed on the screen to reflect the current contents of the - line buffer. + Append *line* to the history buffer, as if it was the last line typed. + This calls :c:func:`add_history` in the underlying library. + + +Startup hooks +------------- .. function:: set_startup_hook([function]) - Set or remove the startup_hook function. If *function* is specified, it will be - used as the new startup_hook function; if omitted or ``None``, any hook function - already installed is removed. The startup_hook function is called with no + Set or remove the function invoked by the :c:data:`rl_startup_hook` + callback of the underlying library. If *function* is specified, it will + be used as the new hook function; if omitted or ``None``, any function + already installed is removed. The hook is called with no arguments just before readline prints the first prompt. .. function:: set_pre_input_hook([function]) - Set or remove the pre_input_hook function. If *function* is specified, it will - be used as the new pre_input_hook function; if omitted or ``None``, any hook - function already installed is removed. The pre_input_hook function is called + Set or remove the function invoked by the :c:data:`rl_pre_input_hook` + callback of the underlying library. If *function* is specified, it will + be used as the new hook function; if omitted or ``None``, any + function already installed is removed. The hook is called with no arguments after the first prompt has been printed and just before - readline starts reading input characters. + readline starts reading input characters. This function only exists + if Python was compiled for a version of the library that supports it. + + +Completion +---------- + +The following functions relate to implementing a custom word completion +function. This is typically operated by the Tab key, and can suggest and +automatically complete a word being typed. By default, Readline is set up +to be used by :mod:`rlcompleter` to complete Python identifiers for +the interactive interpreter. If the :mod:`readline` module is to be used +with a custom completer, a different set of word delimiters should be set. .. function:: set_completer([function]) @@ -132,6 +211,12 @@ The :mod:`readline` module defines the following functions: returns a non-string value. It should return the next possible completion starting with *text*. + The installed completer function is invoked by the *entry_func* callback + passed to :c:func:`rl_completion_matches` in the underlying library. + The *text* string comes from the first parameter to the + :c:data:`rl_attempted_completion_function` callback of the + underlying library. + .. function:: get_completer() @@ -140,27 +225,27 @@ The :mod:`readline` module defines the following functions: .. function:: get_completion_type() - Get the type of completion being attempted. + Get the type of completion being attempted. This returns the + :c:data:`rl_completion_type` variable in the underlying library as + an integer. .. function:: get_begidx() + get_endidx() - Get the beginning index of the readline tab-completion scope. - - -.. function:: get_endidx() - - Get the ending index of the readline tab-completion scope. + Get the beginning or ending index of the completion scope. + These indexes are the *start* and *end* arguments passed to the + :c:data:`rl_attempted_completion_function` callback of the + underlying library. .. function:: set_completer_delims(string) + get_completer_delims() - Set the readline word delimiters for tab-completion. - - -.. function:: get_completer_delims() - - Get the readline word delimiters for tab-completion. + Set or get the word delimiters for completion. These determine the + start of the word to be considered for completion (the completion scope). + These functions access the :c:data:`rl_completer_word_break_characters` + variable in the underlying library. .. function:: set_completion_display_matches_hook([function]) @@ -168,21 +253,13 @@ The :mod:`readline` module defines the following functions: Set or remove the completion display function. If *function* is specified, it will be used as the new completion display function; if omitted or ``None``, any completion display function already - installed is removed. The completion display function is called as + installed is removed. This sets or clears the + :c:data:`rl_completion_display_matches_hook` callback in the + underlying library. The completion display function is called as ``function(substitution, [matches], longest_match_length)`` once each time matches need to be displayed. -.. function:: add_history(line) - - Append a line to the history buffer, as if it was the last line typed. - -.. seealso:: - - Module :mod:`rlcompleter` - Completion of Python identifiers at the interactive prompt. - - .. _readline-example: Example @@ -201,6 +278,8 @@ from the user's :envvar:`PYTHONSTARTUP` file. :: histfile = os.path.join(os.path.expanduser("~"), ".python_history") try: readline.read_history_file(histfile) + # default history len is -1 (infinite), which may grow unruly + readline.set_history_length(1000) except FileNotFoundError: pass @@ -209,6 +288,27 @@ from the user's :envvar:`PYTHONSTARTUP` file. :: This code is actually automatically run when Python is run in :ref:`interactive mode <tut-interactive>` (see :ref:`rlcompleter-config`). +The following example achieves the same goal but supports concurrent interactive +sessions, by only appending the new history. :: + + import atexit + import os + import readline + histfile = os.path.join(os.path.expanduser("~"), ".python_history") + + try: + readline.read_history_file(histfile) + h_len = readline.get_history_length() + except FileNotFoundError: + open(histfile, 'wb').close() + h_len = 0 + + def save(prev_h_len, histfile): + new_h_len = readline.get_history_length() + readline.set_history_length(1000) + readline.append_history_file(new_h_len - prev_h_len, histfile) + atexit.register(save, h_len, histfile) + The following example extends the :class:`code.InteractiveConsole` class to support history save/restore. :: @@ -233,5 +333,5 @@ support history save/restore. :: atexit.register(self.save_history, histfile) def save_history(self, histfile): + readline.set_history_length(1000) readline.write_history_file(histfile) - diff --git a/Doc/library/reprlib.rst b/Doc/library/reprlib.rst index ee9a10d37e..0905b982cd 100644 --- a/Doc/library/reprlib.rst +++ b/Doc/library/reprlib.rst @@ -3,6 +3,7 @@ .. module:: reprlib :synopsis: Alternate repr() implementation with size limits. + .. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org> **Source code:** :source:`Lib/reprlib.py` diff --git a/Doc/library/resource.rst b/Doc/library/resource.rst index a7b229e98a..bdfe5f6d45 100644 --- a/Doc/library/resource.rst +++ b/Doc/library/resource.rst @@ -4,9 +4,11 @@ .. module:: resource :platform: Unix :synopsis: An interface to provide resource usage information on the current process. + .. moduleauthor:: Jeremy Hylton <jeremy@alum.mit.edu> .. sectionauthor:: Jeremy Hylton <jeremy@alum.mit.edu> +-------------- This module provides basic mechanisms for measuring and controlling system resources utilized by a program. @@ -123,8 +125,7 @@ platform. .. data:: RLIMIT_FSIZE - The maximum size of a file which the process may create. This only affects the - stack of the main thread in a multi-threaded process. + The maximum size of a file which the process may create. .. data:: RLIMIT_DATA @@ -134,7 +135,8 @@ platform. .. data:: RLIMIT_STACK - The maximum size (in bytes) of the call stack for the current process. + The maximum size (in bytes) of the call stack for the current process. This only + affects the stack of the main thread in a multi-threaded process. .. data:: RLIMIT_RSS diff --git a/Doc/library/rlcompleter.rst b/Doc/library/rlcompleter.rst index 9ed01c7146..40b09ce897 100644 --- a/Doc/library/rlcompleter.rst +++ b/Doc/library/rlcompleter.rst @@ -3,6 +3,7 @@ .. module:: rlcompleter :synopsis: Python identifier completion, suitable for the GNU readline library. + .. sectionauthor:: Moshe Zadka <moshez@zadka.site.co.il> **Source code:** :source:`Lib/rlcompleter.py` diff --git a/Doc/library/runpy.rst b/Doc/library/runpy.rst index 7293f159e0..af35e81a2d 100644 --- a/Doc/library/runpy.rst +++ b/Doc/library/runpy.rst @@ -3,6 +3,7 @@ .. module:: runpy :synopsis: Locate and run Python modules without importing them first. + .. moduleauthor:: Nick Coghlan <ncoghlan@gmail.com> **Source code:** :source:`Lib/runpy.py` @@ -36,7 +37,8 @@ The :mod:`runpy` module provides two functions: import mechanism (refer to :pep:`302` for details) and then executed in a fresh module namespace. - If the supplied module name refers to a package rather than a normal + The *mod_name* argument should be an absolute module name. + If the module name refers to a package rather than a normal module, then that package is imported and the ``__main__`` submodule within that package is then executed and the resulting module globals dictionary returned. diff --git a/Doc/library/sched.rst b/Doc/library/sched.rst index 26f59c556c..4d4a616105 100644 --- a/Doc/library/sched.rst +++ b/Doc/library/sched.rst @@ -3,12 +3,13 @@ .. module:: sched :synopsis: General purpose event scheduler. -.. sectionauthor:: Moshe Zadka <moshez@zadka.site.co.il> -.. index:: single: event scheduling +.. sectionauthor:: Moshe Zadka <moshez@zadka.site.co.il> **Source code:** :source:`Lib/sched.py` +.. index:: single: event scheduling + -------------- The :mod:`sched` module defines a class which implements a general purpose event diff --git a/Doc/library/select.rst b/Doc/library/select.rst index 5334af8ea4..6cec9f764b 100644 --- a/Doc/library/select.rst +++ b/Doc/library/select.rst @@ -4,6 +4,7 @@ .. module:: select :synopsis: Wait for I/O completion on multiple streams. +-------------- This module provides access to the :c:func:`select` and :c:func:`poll` functions available in most operating systems, :c:func:`devpoll` available on @@ -145,6 +146,13 @@ The module defines the following: library, and does not handle file descriptors that don't originate from WinSock. + .. versionchanged:: 3.5 + The function is now retried with a recomputed timeout when interrupted by + a signal, except if the signal handler raises an exception (see + :pep:`475` for the rationale), instead of raising + :exc:`InterruptedError`. + + .. attribute:: PIPE_BUF The minimum number of bytes which can be written without blocking to a pipe @@ -242,6 +250,12 @@ object. returning. If *timeout* is omitted, -1, or :const:`None`, the call will block until there is an event for this poll object. + .. versionchanged:: 3.5 + The function is now retried with a recomputed timeout when interrupted by + a signal, except if the signal handler raises an exception (see + :pep:`475` for the rationale), instead of raising + :exc:`InterruptedError`. + .. _epoll-objects: @@ -322,6 +336,12 @@ Edge and Level Trigger Polling (epoll) Objects Wait for events. timeout in seconds (float) + .. versionchanged:: 3.5 + The function is now retried with a recomputed timeout when interrupted by + a signal, except if the signal handler raises an exception (see + :pep:`475` for the rationale), instead of raising + :exc:`InterruptedError`. + .. _poll-objects: @@ -401,6 +421,12 @@ linearly scanned again. :c:func:`select` is O(highest file descriptor), while returning. If *timeout* is omitted, negative, or :const:`None`, the call will block until there is an event for this poll object. + .. versionchanged:: 3.5 + The function is now retried with a recomputed timeout when interrupted by + a signal, except if the signal handler raises an exception (see + :pep:`475` for the rationale), instead of raising + :exc:`InterruptedError`. + .. _kqueue-objects: @@ -435,13 +461,19 @@ Kqueue Objects - max_events must be 0 or a positive integer - timeout in seconds (floats possible) + .. versionchanged:: 3.5 + The function is now retried with a recomputed timeout when interrupted by + a signal, except if the signal handler raises an exception (see + :pep:`475` for the rationale), instead of raising + :exc:`InterruptedError`. + .. _kevent-objects: Kevent Objects -------------- -http://www.freebsd.org/cgi/man.cgi?query=kqueue&sektion=2 +https://www.freebsd.org/cgi/man.cgi?query=kqueue&sektion=2 .. attribute:: kevent.ident diff --git a/Doc/library/selectors.rst b/Doc/library/selectors.rst index 98377c890f..44b27f31c1 100644 --- a/Doc/library/selectors.rst +++ b/Doc/library/selectors.rst @@ -6,6 +6,9 @@ .. versionadded:: 3.4 +**Source code:** :source:`Lib/selectors.py` + +-------------- Introduction ------------ @@ -45,12 +48,13 @@ Classes hierarchy:: +-- SelectSelector +-- PollSelector +-- EpollSelector + +-- DevpollSelector +-- KqueueSelector In the following, *events* is a bitwise mask indicating which I/O events should -be waited for on a given file object. It can be a combination of the constants -below: +be waited for on a given file object. It can be a combination of the modules +constants below: +-----------------------+-----------------------------------------------+ | Constant | Meaning | @@ -98,7 +102,7 @@ below: :class:`BaseSelector` and its concrete implementations support the :term:`context manager` protocol. - .. method:: register(fileobj, events, data=None) + .. abstractmethod:: register(fileobj, events, data=None) Register a file object for selection, monitoring it for I/O events. @@ -111,7 +115,7 @@ below: :exc:`ValueError` in case of invalid event mask or file descriptor, or :exc:`KeyError` if the file object is already registered. - .. method:: unregister(fileobj) + .. abstractmethod:: unregister(fileobj) Unregister a file object from selection, removing it from monitoring. A file object shall be unregistered prior to being closed. @@ -135,7 +139,7 @@ below: :exc:`ValueError` in case of invalid event mask or file descriptor, or :exc:`KeyError` if the file object is not registered. - .. method:: select(timeout=None) + .. abstractmethod:: select(timeout=None) Wait until some registered file objects become ready, or the timeout expires. @@ -158,6 +162,12 @@ below: timeout has elapsed if the current process receives a signal: in this case, an empty list will be returned. + .. versionchanged:: 3.5 + The selector is now retried with a recomputed timeout when interrupted + by a signal if the signal handler did not raise an exception (see + :pep:`475` for the rationale), instead of returning an empty list + of events before the timeout. + .. method:: close() Close the selector. @@ -172,7 +182,7 @@ below: This returns the :class:`SelectorKey` instance associated to this file object, or raises :exc:`KeyError` if the file object is not registered. - .. method:: get_map() + .. abstractmethod:: get_map() Return a mapping of file objects to selector keys. @@ -207,6 +217,16 @@ below: This returns the file descriptor used by the underlying :func:`select.epoll` object. +.. class:: DevpollSelector() + + :func:`select.devpoll`-based selector. + + .. method:: fileno() + + This returns the file descriptor used by the underlying + :func:`select.devpoll` object. + + .. versionadded:: 3.5 .. class:: KqueueSelector() diff --git a/Doc/library/shelve.rst b/Doc/library/shelve.rst index 22e202dace..db66a63843 100644 --- a/Doc/library/shelve.rst +++ b/Doc/library/shelve.rst @@ -4,11 +4,10 @@ .. module:: shelve :synopsis: Python object persistence. +**Source code:** :source:`Lib/shelve.py` .. index:: module: pickle -**Source code:** :source:`Lib/shelve.py` - -------------- A "shelf" is a persistent, dictionary-like object. The difference with "dbm" @@ -76,7 +75,7 @@ Two additional methods are supported: .. seealso:: - `Persistent dictionary recipe <http://code.activestate.com/recipes/576642/>`_ + `Persistent dictionary recipe <https://code.activestate.com/recipes/576642/>`_ with widely supported storage formats and having the speed of native dictionaries. @@ -109,7 +108,7 @@ Restrictions A subclass of :class:`collections.abc.MutableMapping` which stores pickled values in the *dict* object. - By default, version 0 pickles are used to serialize values. The version of the + By default, version 3 pickles are used to serialize values. The version of the pickle protocol can be specified with the *protocol* parameter. See the :mod:`pickle` documentation for a discussion of the pickle protocols. @@ -137,7 +136,7 @@ Restrictions A subclass of :class:`Shelf` which exposes :meth:`first`, :meth:`!next`, :meth:`previous`, :meth:`last` and :meth:`set_location` which are available in the third-party :mod:`bsddb` module from `pybsddb - <http://www.jcea.es/programacion/pybsddb.htm>`_ but not in other database + <https://www.jcea.es/programacion/pybsddb.htm>`_ but not in other database modules. The *dict* object passed to the constructor must support those methods. This is generally accomplished by calling one of :func:`bsddb.hashopen`, :func:`bsddb.btopen` or :func:`bsddb.rnopen`. The @@ -165,32 +164,33 @@ object):: import shelve - d = shelve.open(filename) # open -- file may get suffix added by low-level - # library + d = shelve.open(filename) # open -- file may get suffix added by low-level + # library + + d[key] = data # store data at key (overwrites old data if + # using an existing key) + data = d[key] # retrieve a COPY of data at key (raise KeyError + # if no such key) + del d[key] # delete data stored at key (raises KeyError + # if no such key) - d[key] = data # store data at key (overwrites old data if - # using an existing key) - data = d[key] # retrieve a COPY of data at key (raise KeyError if no - # such key) - del d[key] # delete data stored at key (raises KeyError - # if no such key) - flag = key in d # true if the key exists - klist = list(d.keys()) # a list of all existing keys (slow!) + flag = key in d # true if the key exists + klist = list(d.keys()) # a list of all existing keys (slow!) # as d was opened WITHOUT writeback=True, beware: - d['xx'] = [0, 1, 2] # this works as expected, but... - d['xx'].append(3) # *this doesn't!* -- d['xx'] is STILL [0, 1, 2]! + d['xx'] = [0, 1, 2] # this works as expected, but... + d['xx'].append(3) # *this doesn't!* -- d['xx'] is STILL [0, 1, 2]! # having opened d without writeback=True, you need to code carefully: - temp = d['xx'] # extracts the copy - temp.append(5) # mutates the copy - d['xx'] = temp # stores the copy right back, to persist it + temp = d['xx'] # extracts the copy + temp.append(5) # mutates the copy + d['xx'] = temp # stores the copy right back, to persist it # or, d=shelve.open(filename,writeback=True) would let you just code # d['xx'].append(5) and have it work as expected, BUT it would also # consume more memory and make the d.close() operation slower. - d.close() # close it + d.close() # close it .. seealso:: diff --git a/Doc/library/shlex.rst b/Doc/library/shlex.rst index e40a10daa5..e81f9822bb 100644 --- a/Doc/library/shlex.rst +++ b/Doc/library/shlex.rst @@ -3,6 +3,7 @@ .. module:: shlex :synopsis: Simple lexical analysis for Unix shell-like languages. + .. moduleauthor:: Eric S. Raymond <esr@snark.thyrsus.com> .. moduleauthor:: Gustavo Niemeyer <niemeyer@conectiva.com> .. sectionauthor:: Eric S. Raymond <esr@snark.thyrsus.com> @@ -244,7 +245,8 @@ variables which either control lexical analysis or can be used for debugging: This attribute is ``None`` by default. If you assign a string to it, that string will be recognized as a lexical-level inclusion request similar to the ``source`` keyword in various shells. That is, the immediately following token - will opened as a filename and input taken from that stream until EOF, at which + will be opened as a filename and input will + be taken from that stream until EOF, at which point the :meth:`~io.IOBase.close` method of that stream will be called and the input source will again become the original input stream. Source requests may be stacked any number of levels deep. diff --git a/Doc/library/shutil.rst b/Doc/library/shutil.rst index 7566521e2a..a1cf241327 100644 --- a/Doc/library/shutil.rst +++ b/Doc/library/shutil.rst @@ -3,15 +3,16 @@ .. module:: shutil :synopsis: High-level file operations, including copying. + .. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org> .. partly based on the docstrings +**Source code:** :source:`Lib/shutil.py` + .. index:: single: file; copying single: copying files -**Source code:** :source:`Lib/shutil.py` - -------------- The :mod:`shutil` module offers a number of high-level operations on files and @@ -191,7 +192,8 @@ Directory and files operations match one of the glob-style *patterns* provided. See the example below. -.. function:: copytree(src, dst, symlinks=False, ignore=None, copy_function=copy2, ignore_dangling_symlinks=False) +.. function:: copytree(src, dst, symlinks=False, ignore=None, \ + copy_function=copy2, ignore_dangling_symlinks=False) Recursively copy an entire directory tree rooted at *src*, returning the destination directory. The destination @@ -282,7 +284,7 @@ Directory and files operations .. versionadded:: 3.3 -.. function:: move(src, dst) +.. function:: move(src, dst, copy_function=copy2) Recursively move a file or directory (*src*) to another location (*dst*) and return the destination. @@ -292,15 +294,26 @@ Directory and files operations be overwritten depending on :func:`os.rename` semantics. If the destination is on the current filesystem, then :func:`os.rename` is - used. Otherwise, *src* is copied (using :func:`shutil.copy2`) to *dst* and - then removed. In case of symlinks, a new symlink pointing to the target of - *src* will be created in or as *dst* and *src* will be removed. + used. Otherwise, *src* is copied to *dst* using *copy_function* and then + removed. In case of symlinks, a new symlink pointing to the target of *src* + will be created in or as *dst* and *src* will be removed. + + If *copy_function* is given, it must be a callable that takes two arguments + *src* and *dst*, and will be used to copy *src* to *dest* if + :func:`os.rename` cannot be used. If the source is a directory, + :func:`copytree` is called, passing it the :func:`copy_function`. The + default *copy_function* is :func:`copy2`. Using :func:`copy` as the + *copy_function* allows the move to succeed when it is not possible to also + copy the metadata, at the expense of not copying any of the metadata. .. versionchanged:: 3.3 Added explicit symlink handling for foreign filesystems, thus adapting it to the behavior of GNU's :program:`mv`. Now returns *dst*. + .. versionchanged:: 3.5 + Added the *copy_function* keyword argument. + .. function:: disk_usage(path) Return disk usage statistics about the given path as a :term:`named tuple` @@ -330,7 +343,7 @@ Directory and files operations Return the path to an executable which would be run if the given *cmd* was called. If no *cmd* would be called, return ``None``. - *mode* is a permission mask passed a to :func:`os.access`, by default + *mode* is a permission mask passed to :func:`os.access`, by default determining if the file exists and executable. When no *path* is specified, the results of :func:`os.environ` are used, @@ -418,6 +431,26 @@ Another example that uses the *ignore* argument to add a logging call:: copytree(source, destination, ignore=_logpath) +.. _shutil-rmtree-example: + +rmtree example +~~~~~~~~~~~~~~ + +This example shows how to remove a directory tree on Windows where some +of the files have their read-only bit set. It uses the onerror callback +to clear the readonly bit and reattempt the remove. Any subsequent failure +will propagate. :: + + import os, stat + import shutil + + def remove_readonly(func, path, _): + "Clear the readonly bit and reattempt the removal" + os.chmod(path, stat.S_IWRITE) + func(path) + + shutil.rmtree(directory, onerror=remove_readonly) + .. _archiving-operations: Archiving operations @@ -434,7 +467,8 @@ provided. They rely on the :mod:`zipfile` and :mod:`tarfile` modules. *base_name* is the name of the file to create, including the path, minus any format-specific extension. *format* is the archive format: one of - "zip", "tar", "bztar" (if the :mod:`bz2` module is available) or "gztar". + "zip", "tar", "bztar" (if the :mod:`bz2` module is available), "xztar" + (if the :mod:`lzma` module is available) or "gztar". *root_dir* is a directory that will be the root directory of the archive; for example, we typically chdir into *root_dir* before creating the @@ -457,6 +491,9 @@ provided. They rely on the :mod:`zipfile` and :mod:`tarfile` modules. The *verbose* argument is unused and deprecated. + .. versionchanged:: 3.5 + Added support for the *xztar* format. + .. function:: get_archive_formats() @@ -467,6 +504,7 @@ provided. They rely on the :mod:`zipfile` and :mod:`tarfile` modules. - *gztar*: gzip'ed tar-file - *bztar*: bzip2'ed tar-file (if the :mod:`bz2` module is available.) + - *xztar*: xz'ed tar-file (if the :mod:`lzma` module is available.) - *tar*: uncompressed tar file - *zip*: ZIP file @@ -542,6 +580,7 @@ provided. They rely on the :mod:`zipfile` and :mod:`tarfile` modules. - *gztar*: gzip'ed tar-file - *bztar*: bzip2'ed tar-file (if the :mod:`bz2` module is available.) + - *xztar*: xz'ed tar-file (if the :mod:`lzma` module is available.) - *tar*: uncompressed tar file - *zip*: ZIP file @@ -564,7 +603,9 @@ found in the :file:`.ssh` directory of the user:: >>> make_archive(archive_name, 'gztar', root_dir) '/Users/tarek/myarchive.tar.gz' -The resulting archive contains:: +The resulting archive contains: + +.. code-block:: shell-session $ tar -tzvf /Users/tarek/myarchive.tar.gz drwx------ tarek/staff 0 2010-02-01 16:23:40 ./ diff --git a/Doc/library/signal.rst b/Doc/library/signal.rst index 8b90994846..61252ce4e8 100644 --- a/Doc/library/signal.rst +++ b/Doc/library/signal.rst @@ -4,6 +4,7 @@ .. module:: signal :synopsis: Set handlers for asynchronous events. +-------------- This module provides mechanisms to use signal handlers in Python. @@ -11,7 +12,7 @@ This module provides mechanisms to use signal handlers in Python. General rules ------------- -The :func:`signal.signal` function allows to define custom handlers to be +The :func:`signal.signal` function allows defining custom handlers to be executed when a signal is received. A small number of default handlers are installed: :const:`SIGPIPE` is ignored (so write errors on pipes and sockets can be reported as ordinary Python exceptions) and :const:`SIGINT` is @@ -22,9 +23,6 @@ explicitly reset (Python emulates the BSD style interface regardless of the underlying implementation), with the exception of the handler for :const:`SIGCHLD`, which follows the underlying implementation. -There is no way to "block" signals temporarily from critical sections (since -this is not supported by all Unix flavors). - Execution of Python signal handlers ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -65,6 +63,16 @@ Besides, only the main thread is allowed to set a new signal handler. Module contents --------------- +.. versionchanged:: 3.5 + signal (SIG*), handler (:const:`SIG_DFL`, :const:`SIG_IGN`) and sigmask + (:const:`SIG_BLOCK`, :const:`SIG_UNBLOCK`, :const:`SIG_SETMASK`) + related constants listed below were turned into + :class:`enums <enum.IntEnum>`. + :func:`getsignal`, :func:`pthread_sigmask`, :func:`sigpending` and + :func:`sigwait` functions return human-readable + :class:`enums <enum.IntEnum>`. + + The variables defined in the :mod:`signal` module are: @@ -209,21 +217,21 @@ The :mod:`signal` module defines the following functions: :func:`sigpending`. -.. function:: pthread_kill(thread_id, signum) +.. function:: pthread_kill(thread_id, signalnum) - Send the signal *signum* to the thread *thread_id*, another thread in the + Send the signal *signalnum* to the thread *thread_id*, another thread in the same process as the caller. The target thread can be executing any code (Python or not). However, if the target thread is executing the Python interpreter, the Python signal handlers will be :ref:`executed by the main - thread <signals-and-threads>`. Therefore, the only point of sending a signal to a particular - Python thread would be to force a running system call to fail with - :exc:`InterruptedError`. + thread <signals-and-threads>`. Therefore, the only point of sending a + signal to a particular Python thread would be to force a running system call + to fail with :exc:`InterruptedError`. Use :func:`threading.get_ident()` or the :attr:`~threading.Thread.ident` attribute of :class:`threading.Thread` objects to get a suitable value for *thread_id*. - If *signum* is 0, then no signal is sent, but error checking is still + If *signalnum* is 0, then no signal is sent, but error checking is still performed; this can be used to check if the target thread is still running. Availability: Unix (see the man page :manpage:`pthread_kill(3)` for further @@ -308,6 +316,9 @@ The :mod:`signal` module defines the following functions: attempting to call it from other threads will cause a :exc:`ValueError` exception to be raised. + .. versionchanged:: 3.5 + On Windows, the function now also supports socket handles. + .. function:: siginterrupt(signalnum, flag) @@ -341,6 +352,9 @@ The :mod:`signal` module defines the following functions: On Windows, :func:`signal` can only be called with :const:`SIGABRT`, :const:`SIGFPE`, :const:`SIGILL`, :const:`SIGINT`, :const:`SIGSEGV`, or :const:`SIGTERM`. A :exc:`ValueError` will be raised in any other case. + Note that not all systems define the same set of signal names; an + :exc:`AttributeError` will be raised if a signal name is not defined as + ``SIG*`` module level constant. .. function:: sigpending() @@ -395,6 +409,11 @@ The :mod:`signal` module defines the following functions: .. versionadded:: 3.3 + .. versionchanged:: 3.5 + The function is now retried if interrupted by a signal not in *sigset* + and the signal handler does not raise an exception (see :pep:`475` for + the rationale). + .. function:: sigtimedwait(sigset, timeout) @@ -409,6 +428,11 @@ The :mod:`signal` module defines the following functions: .. versionadded:: 3.3 + .. versionchanged:: 3.5 + The function is now retried with the recomputed *timeout* if interrupted + by a signal not in *sigset* and the signal handler does not raise an + exception (see :pep:`475` for the rationale). + .. _signal-example: diff --git a/Doc/library/site.rst b/Doc/library/site.rst index 51e5da81d1..0a73f5abc6 100644 --- a/Doc/library/site.rst +++ b/Doc/library/site.rst @@ -26,24 +26,23 @@ additions, call the :func:`site.main` function. :option:`-S`. .. index:: - pair: site-python; directory pair: site-packages; directory It starts by constructing up to four directories from a head and a tail part. For the head part, it uses ``sys.prefix`` and ``sys.exec_prefix``; empty heads are skipped. For the tail part, it uses the empty string and then :file:`lib/site-packages` (on Windows) or -:file:`lib/python{X.Y}/site-packages` and then :file:`lib/site-python` (on -Unix and Macintosh). For each of the distinct head-tail combinations, it sees -if it refers to an existing directory, and if so, adds it to ``sys.path`` and -also inspects the newly added path for configuration files. +:file:`lib/python{X.Y}/site-packages` (on Unix and Macintosh). For each +of the distinct head-tail combinations, it sees if it refers to an existing +directory, and if so, adds it to ``sys.path`` and also inspects the newly +added path for configuration files. -.. deprecated:: 3.4 - Support for the "site-python" directory will be removed in 3.5. +.. versionchanged:: 3.5 + Support for the "site-python" directory has been removed. If a file named "pyvenv.cfg" exists one directory above sys.executable, sys.prefix and sys.exec_prefix are set to that directory and -it is also checked for site-packages and site-python (sys.base_prefix and +it is also checked for site-packages (sys.base_prefix and sys.base_exec_prefix will always be the "real" prefixes of the Python installation). If "pyvenv.cfg" (a bootstrap configuration file) contains the key "include-system-site-packages" set to anything other than "false" @@ -53,8 +52,7 @@ searched for site-packages; otherwise they won't. A path configuration file is a file whose name has the form :file:`{name}.pth` and exists in one of the four directories mentioned above; its contents are additional items (one per line) to be added to ``sys.path``. Non-existing items -are never added to ``sys.path``, and no check is made that the item refers to a -directory rather than a file. No item is added to ``sys.path`` more than +are never added to ``sys.path``. No item is added to ``sys.path`` more than once. Blank lines and lines beginning with ``#`` are skipped. Lines starting with ``import`` (followed by space or tab) are executed. @@ -195,8 +193,7 @@ Module contents .. function:: getsitepackages() - Return a list containing all global site-packages directories (and possibly - site-python). + Return a list containing all global site-packages directories. .. versionadded:: 3.2 diff --git a/Doc/library/smtpd.rst b/Doc/library/smtpd.rst index 3ebed06260..977f9a8748 100644 --- a/Doc/library/smtpd.rst +++ b/Doc/library/smtpd.rst @@ -20,7 +20,8 @@ specific mail-sending strategies. Additionally the SMTPChannel may be extended to implement very specific interaction behaviour with SMTP clients. -The code supports :RFC:`5321`, plus the :rfc:`1870` SIZE extension. +The code supports :RFC:`5321`, plus the :rfc:`1870` SIZE and :rfc:`6531` +SMTPUTF8 extensions. SMTPServer Objects @@ -28,7 +29,7 @@ SMTPServer Objects .. class:: SMTPServer(localaddr, remoteaddr, data_size_limit=33554432,\ - map=None) + map=None, enable_SMTPUTF8=False, decode_data=True) Create a new :class:`SMTPServer` object, which binds to local address *localaddr*. It will treat *remoteaddr* as an upstream SMTP relayer. It @@ -39,25 +40,77 @@ SMTPServer Objects accepted in a ``DATA`` command. A value of ``None`` or ``0`` means no limit. - A dictionary can be specified in *map* to avoid using a global socket map. - - .. method:: process_message(peer, mailfrom, rcpttos, data) - - Raise :exc:`NotImplementedError` exception. Override this in subclasses to + *map* is the socket map to use for connections (an initially empty + dictionary is a suitable value). If not specified the :mod:`asyncore` + global socket map is used. + + *enable_SMTPUTF8* determins whether the ``SMTPUTF8`` extension (as defined + in :RFC:`6531`) should be enabled. The default is ``False``. If set to + ``True``, *decode_data* must be ``False`` (otherwise an error is raised). + When ``True``, ``SMTPUTF8`` is accepted as a parameter to the ``MAIL`` + command and when present is passed to :meth:`process_message` in the + ``kwargs['mail_options']`` list. + + *decode_data* specifies whether the data portion of the SMTP transaction + should be decoded using UTF-8. The default is ``True`` for backward + compatibility reasons, but will change to ``False`` in Python 3.6; specify + the keyword value explicitly to avoid the :exc:`DeprecationWarning`. When + *decode_data* is set to ``False`` the server advertises the ``8BITMIME`` + extension (:rfc:`6152`), accepts the ``BODY=8BITMIME`` parameter to + the ``MAIL`` command, and when present passes it to :meth:`process_message` + in the ``kwargs['mail_options']`` list. + + .. method:: process_message(peer, mailfrom, rcpttos, data, **kwargs) + + Raise a :exc:`NotImplementedError` exception. Override this in subclasses to do something useful with this message. Whatever was passed in the constructor as *remoteaddr* will be available as the :attr:`_remoteaddr` attribute. *peer* is the remote host's address, *mailfrom* is the envelope originator, *rcpttos* are the envelope recipients and *data* is a string - containing the contents of the e-mail (which should be in :rfc:`2822` + containing the contents of the e-mail (which should be in :rfc:`5321` format). + If the *decode_data* constructor keyword is set to ``True``, the *data* + argument will be a unicode string. If it is set to ``False``, it + will be a bytes object. + + *kwargs* is a dictionary containing additional information. It is empty + unless at least one of ``decode_data=False`` or ``enable_SMTPUTF8=True`` + was given as an init parameter, in which case it contains the following + keys: + + *mail_options*: + a list of all received parameters to the ``MAIL`` + command (the elements are uppercase strings; example: + ``['BODY=8BITMIME', 'SMTPUTF8']``). + + *rcpt_options*: + same as *mail_options* but for the ``RCPT`` command. + Currently no ``RCPT TO`` options are supported, so for now + this will always be an empty list. + + Implementations of ``process_message`` should use the ``**kwargs`` + signature to accept arbitrary keyword arguments, since future feature + enhancements may add keys to the kwargs dictionary. + + Return ``None`` to request a normal ``250 Ok`` response; otherwise + return the desired response string in :RFC:`5321` format. + .. attribute:: channel_class Override this in subclasses to use a custom :class:`SMTPChannel` for managing SMTP clients. - .. versionchanged:: 3.4 - The *map* argument was added. + .. versionadded:: 3.4 + The *map* constructor argument. + + .. versionchanged:: 3.5 + *localaddr* and *remoteaddr* may now contain IPv6 addresses. + + .. versionadded:: 3.5 + the *decode_data* and *enable_SMTPUTF8* constructor arguments, and the + *kwargs* argument to :meth:`process_message` when one or more of these is + specified. DebuggingServer Objects @@ -97,7 +150,7 @@ SMTPChannel Objects ------------------- .. class:: SMTPChannel(server, conn, addr, data_size_limit=33554432,\ - map=None)) + map=None, enable_SMTPUTF8=False, decode_data=True) Create a new :class:`SMTPChannel` object which manages the communication between the server and a single SMTP client. @@ -108,11 +161,24 @@ SMTPChannel Objects accepted in a ``DATA`` command. A value of ``None`` or ``0`` means no limit. + *enable_SMTPUTF8* determins whether the ``SMTPUTF8`` extension (as defined + in :RFC:`6531`) should be enabled. The default is ``False``. A + :exc:`ValueError` is raised if both *enable_SMTPUTF8* and *decode_data* are + set to ``True`` at the same time. + A dictionary can be specified in *map* to avoid using a global socket map. + *decode_data* specifies whether the data portion of the SMTP transaction + should be decoded using UTF-8. The default is ``True`` for backward + compatibility reasons, but will change to ``False`` in Python 3.6. Specify + the keyword value explicitly to avoid the :exc:`DeprecationWarning`. + To use a custom SMTPChannel implementation you need to override the :attr:`SMTPServer.channel_class` of your :class:`SMTPServer`. + .. versionchanged:: 3.5 + the *decode_data* and *enable_SMTPUTF8* arguments were added. + The :class:`SMTPChannel` has the following instance variables: .. attribute:: smtp_server diff --git a/Doc/library/smtplib.rst b/Doc/library/smtplib.rst index b87ad5f326..bdf3805106 100644 --- a/Doc/library/smtplib.rst +++ b/Doc/library/smtplib.rst @@ -3,15 +3,15 @@ .. module:: smtplib :synopsis: SMTP protocol client (requires sockets). + .. sectionauthor:: Eric S. Raymond <esr@snark.thyrsus.com> +**Source code:** :source:`Lib/smtplib.py` .. index:: pair: SMTP; protocol single: Simple Mail Transfer Protocol -**Source code:** :source:`Lib/smtplib.py` - -------------- The :mod:`smtplib` module defines an SMTP client session object that can be used @@ -33,7 +33,7 @@ Protocol) and :rfc:`1869` (SMTP Service Extensions). *timeout* parameter specifies a timeout in seconds for blocking operations like the connection attempt (if not specified, the global default timeout setting will be used). If the timeout expires, :exc:`socket.timeout` is - raised. The optional source_address parameter allows to bind + raised. The optional source_address parameter allows binding to some specific source address in a machine with multiple network interfaces, and/or to some specific source TCP port. It takes a 2-tuple (host, port), for the socket to bind to as its source address before @@ -61,6 +61,10 @@ Protocol) and :rfc:`1869` (SMTP Service Extensions). .. versionchanged:: 3.3 source_address argument was added. + .. versionadded:: 3.5 + The SMTPUTF8 extension (:rfc:`6531`) is now supported. + + .. class:: SMTP_SSL(host='', port=0, local_hostname=None, keyfile=None, \ certfile=None [, timeout], context=None, \ source_address=None) @@ -72,7 +76,7 @@ Protocol) and :rfc:`1869` (SMTP Service Extensions). *port* is zero, the standard SMTP-over-SSL port (465) is used. The optional arguments *local_hostname*, *timeout* and *source_address* have the same meaning as they do in the :class:`SMTP` class. *context*, also optional, - can contain a :class:`~ssl.SSLContext` and allows to configure various + can contain a :class:`~ssl.SSLContext` and allows configuring various aspects of the secure connection. Please read :ref:`ssl-security` for best practices. @@ -161,6 +165,13 @@ A nice selection of exceptions is defined as well: The server refused our ``HELO`` message. +.. exception:: SMTPNotSupportedError + + The command or option attempted is not supported by the server. + + .. versionadded:: 3.5 + + .. exception:: SMTPAuthenticationError SMTP authentication went wrong. Most probably the server didn't accept the @@ -189,8 +200,12 @@ An :class:`SMTP` instance has the following methods: .. method:: SMTP.set_debuglevel(level) - Set the debug output level. A true value for *level* results in debug messages - for connection and for all messages sent to and received from the server. + Set the debug output level. A value of 1 or ``True`` for *level* results in + debug messages for connection and for all messages sent to and received from + the server. A value of 2 for *level* results in these messages being + timestamped. + + .. versionchanged:: 3.5 Added debuglevel 2. .. method:: SMTP.docmd(cmd, args='') @@ -240,8 +255,7 @@ An :class:`SMTP` instance has the following methods: the server is stored as the :attr:`ehlo_resp` attribute, :attr:`does_esmtp` is set to true or false depending on whether the server supports ESMTP, and :attr:`esmtp_features` will be a dictionary containing the names of the - SMTP service extensions this server supports, and their - parameters (if any). + SMTP service extensions this server supports, and their parameters (if any). Unless you wish to use :meth:`has_extn` before sending mail, it should not be necessary to call this method explicitly. It will be implicitly called by @@ -274,7 +288,7 @@ An :class:`SMTP` instance has the following methods: Many sites disable SMTP ``VRFY`` in order to foil spammers. -.. method:: SMTP.login(user, password) +.. method:: SMTP.login(user, password, *, initial_response_ok=True) Log in on an SMTP server that requires authentication. The arguments are the username and the password to authenticate with. If there has been no previous @@ -288,9 +302,68 @@ An :class:`SMTP` instance has the following methods: :exc:`SMTPAuthenticationError` The server didn't accept the username/password combination. + :exc:`SMTPNotSupportedError` + The ``AUTH`` command is not supported by the server. + :exc:`SMTPException` No suitable authentication method was found. + Each of the authentication methods supported by :mod:`smtplib` are tried in + turn if they are advertised as supported by the server. See :meth:`auth` + for a list of supported authentication methods. *initial_response_ok* is + passed through to :meth:`auth`. + + Optional keyword argument *initial_response_ok* specifies whether, for + authentication methods that support it, an "initial response" as specified + in :rfc:`4954` can be sent along with the ``AUTH`` command, rather than + requiring a challenge/response. + + .. versionchanged:: 3.5 + :exc:`SMTPNotSupportedError` may be raised, and the + *initial_response_ok* parameter was added. + + +.. method:: SMTP.auth(mechanism, authobject, *, initial_response_ok=True) + + Issue an ``SMTP`` ``AUTH`` command for the specified authentication + *mechanism*, and handle the challenge response via *authobject*. + + *mechanism* specifies which authentication mechanism is to + be used as argument to the ``AUTH`` command; the valid values are + those listed in the ``auth`` element of :attr:`esmtp_features`. + + *authobject* must be a callable object taking an optional single argument: + + data = authobject(challenge=None) + + If optional keyword argument *initial_response_ok* is true, + ``authobject()`` will be called first with no argument. It can return the + :rfc:`4954` "initial response" bytes which will be encoded and sent with + the ``AUTH`` command as below. If the ``authobject()`` does not support an + initial response (e.g. because it requires a challenge), it should return + None when called with ``challenge=None``. If *initial_response_ok* is + false, then ``authobject()`` will not be called first with None. + + If the initial response check returns None, or if *initial_response_ok* is + false, ``authobject()`` will be called to process the server's challenge + response; the *challenge* argument it is passed will be a ``bytes``. It + should return ``bytes`` *data* that will be base64 encoded and sent to the + server. + + The ``SMTP`` class provides ``authobjects`` for the ``CRAM-MD5``, ``PLAIN``, + and ``LOGIN`` mechanisms; they are named ``SMTP.auth_cram_md5``, + ``SMTP.auth_plain``, and ``SMTP.auth_login`` respectively. They all require + that the ``user`` and ``password`` properties of the ``SMTP`` instance are + set to appropriate values. + + User code does not normally need to call ``auth`` directly, but can instead + call the :meth:`login` method, which will try each of the above mechanisms + in turn, in the order listed. ``auth`` is exposed to facilitate the + implementation of authentication methods not (or not yet) supported + directly by :mod:`smtplib`. + + .. versionadded:: 3.5 + .. method:: SMTP.starttls(keyfile=None, certfile=None, context=None) @@ -310,7 +383,7 @@ An :class:`SMTP` instance has the following methods: :exc:`SMTPHeloError` The server didn't reply properly to the ``HELO`` greeting. - :exc:`SMTPException` + :exc:`SMTPNotSupportedError` The server does not support the STARTTLS extension. :exc:`RuntimeError` @@ -324,6 +397,11 @@ An :class:`SMTP` instance has the following methods: :attr:`SSLContext.check_hostname` and *Server Name Indicator* (see :data:`~ssl.HAS_SNI`). + .. versionchanged:: 3.5 + The error raised for lack of STARTTLS support is now the + :exc:`SMTPNotSupportedError` subclass instead of the base + :exc:`SMTPException`. + .. method:: SMTP.sendmail(from_addr, to_addrs, msg, mail_options=[], rcpt_options=[]) @@ -360,6 +438,9 @@ An :class:`SMTP` instance has the following methods: recipient that was refused. Each entry contains a tuple of the SMTP error code and the accompanying error message sent by the server. + If ``SMTPUTF8`` is included in *mail_options*, and the server supports it, + *from_addr* and *to_addr* may contain non-ASCII characters. + This method may raise the following exceptions: :exc:`SMTPRecipientsRefused` @@ -378,12 +459,20 @@ An :class:`SMTP` instance has the following methods: The server replied with an unexpected error code (other than a refusal of a recipient). + :exc:`SMTPNotSupportedError` + ``SMTPUTF8`` was given in the *mail_options* but is not supported by the + server. + Unless otherwise noted, the connection will be open even after an exception is raised. .. versionchanged:: 3.2 *msg* may be a byte string. + .. versionchanged:: 3.5 + ``SMTPUTF8`` support added, and :exc:`SMTPNotSupportedError` may be + raised if ``SMTPUTF8`` is specified but the server does not support it. + .. method:: SMTP.send_message(msg, from_addr=None, to_addrs=None, \ mail_options=[], rcpt_options=[]) @@ -395,7 +484,7 @@ An :class:`SMTP` instance has the following methods: If *from_addr* is ``None`` or *to_addrs* is ``None``, ``send_message`` fills those arguments with addresses extracted from the headers of *msg* as - specified in :rfc:`2822`\: *from_addr* is set to the :mailheader:`Sender` + specified in :rfc:`5322`\: *from_addr* is set to the :mailheader:`Sender` field if it is present, and otherwise to the :mailheader:`From` field. *to_adresses* combines the values (if any) of the :mailheader:`To`, :mailheader:`Cc`, and :mailheader:`Bcc` fields from *msg*. If exactly one @@ -410,10 +499,18 @@ An :class:`SMTP` instance has the following methods: calls :meth:`sendmail` to transmit the resulting message. Regardless of the values of *from_addr* and *to_addrs*, ``send_message`` does not transmit any :mailheader:`Bcc` or :mailheader:`Resent-Bcc` headers that may appear - in *msg*. + in *msg*. If any of the addresses in *from_addr* and *to_addrs* contain + non-ASCII characters and the server does not advertise ``SMTPUTF8`` support, + an :exc:`SMTPNotSupported` error is raised. Otherwise the ``Message`` is + serialized with a clone of its :mod:`~email.policy` with the + :attr:`~email.policy.EmailPolicy.utf8` attribute set to ``True``, and + ``SMTPUTF8`` and ``BODY=8BITMIME`` are added to *mail_options*. .. versionadded:: 3.2 + .. versionadded:: 3.5 + Support for internationalized addresses (``SMTPUTF8``). + .. method:: SMTP.quit() diff --git a/Doc/library/sndhdr.rst b/Doc/library/sndhdr.rst index f36df68703..6bfa9a9fd2 100644 --- a/Doc/library/sndhdr.rst +++ b/Doc/library/sndhdr.rst @@ -3,21 +3,23 @@ .. module:: sndhdr :synopsis: Determine type of a sound file. + .. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org> .. Based on comments in the module source file. +**Source code:** :source:`Lib/sndhdr.py` + .. index:: single: A-LAW single: u-LAW -**Source code:** :source:`Lib/sndhdr.py` - -------------- The :mod:`sndhdr` provides utility functions which attempt to determine the type of sound data which is in a file. When these functions are able to determine -what type of sound data is stored in a file, they return a tuple ``(type, -sampling_rate, channels, frames, bits_per_sample)``. The value for *type* +what type of sound data is stored in a file, they return a +:func:`~collections.namedtuple`, containing five attributes: (``filetype``, +``framerate``, ``nchannels``, ``nframes``, ``sampwidth``). The value for *type* indicates the data type and will be one of the strings ``'aifc'``, ``'aiff'``, ``'au'``, ``'hcom'``, ``'sndr'``, ``'sndt'``, ``'voc'``, ``'wav'``, ``'8svx'``, ``'sb'``, ``'ub'``, or ``'ul'``. The *sampling_rate* will be either the actual @@ -31,13 +33,19 @@ be the sample size in bits or ``'A'`` for A-LAW or ``'U'`` for u-LAW. .. function:: what(filename) Determines the type of sound data stored in the file *filename* using - :func:`whathdr`. If it succeeds, returns a tuple as described above, otherwise + :func:`whathdr`. If it succeeds, returns a namedtuple as described above, otherwise ``None`` is returned. + .. versionchanged:: 3.5 + Result changed from a tuple to a namedtuple. + .. function:: whathdr(filename) Determines the type of sound data stored in a file based on the file header. - The name of the file is given by *filename*. This function returns a tuple as + The name of the file is given by *filename*. This function returns a namedtuple as described above on success, or ``None``. + .. versionchanged:: 3.5 + Result changed from a tuple to a namedtuple. + diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst index 8577c3c908..02f2350673 100644 --- a/Doc/library/socket.rst +++ b/Doc/library/socket.rst @@ -4,6 +4,9 @@ .. module:: socket :synopsis: Low-level networking interface. +**Source code:** :source:`Lib/socket.py` + +-------------- This module provides access to the BSD *socket* interface. It is available on all modern Unix systems, Windows, MacOS, and probably additional platforms. @@ -46,17 +49,20 @@ created. Socket addresses are represented as follows: - The address of an :const:`AF_UNIX` socket bound to a file system node is represented as a string, using the file system encoding and the ``'surrogateescape'`` error handler (see :pep:`383`). An address in - Linux's abstract namespace is returned as a :class:`bytes` object with + Linux's abstract namespace is returned as a :term:`bytes-like object` with an initial null byte; note that sockets in this namespace can communicate with normal file system sockets, so programs intended to run on Linux may need to deal with both types of address. A string or - :class:`bytes` object can be used for either type of address when + bytes-like object can be used for either type of address when passing it as an argument. .. versionchanged:: 3.3 Previously, :const:`AF_UNIX` socket paths were assumed to use UTF-8 encoding. + .. versionchanged:: 3.5 + Writable :term:`bytes-like object` is now accepted. + - A pair ``(host, port)`` is used for the :const:`AF_INET` address family, where *host* is a string representing either a hostname in Internet domain notation like ``'daring.cwi.nl'`` or an IPv4 address like ``'100.50.200.5'``, @@ -298,6 +304,18 @@ Constants .. versionadded:: 3.4 +.. data:: CAN_RAW_FD_FRAMES + + Enables CAN FD support in a CAN_RAW socket. This is disabled by default. + This allows your application to send both CAN and CAN FD frames; however, + you one must accept both CAN and CAN FD frames when reading from the socket. + + This constant is documented in the Linux documentation. + + Availability: Linux >= 3.6. + + .. versionadded:: 3.5 + .. data:: AF_RDS PF_RDS SOL_RDS @@ -394,7 +412,6 @@ The following functions all create :ref:`socket objects <socket-objects>`. type, and protocol number. Address family, socket type, and protocol number are as for the :func:`.socket` function above. The default family is :const:`AF_UNIX` if defined on the platform; otherwise, the default is :const:`AF_INET`. - Availability: Unix. The newly created sockets are :ref:`non-inheritable <fd_inheritance>`. @@ -405,6 +422,9 @@ The following functions all create :ref:`socket objects <socket-objects>`. .. versionchanged:: 3.4 The returned sockets are now non-inheritable. + .. versionchanged:: 3.5 + Windows support added. + .. function:: create_connection(address[, timeout[, source_address]]) @@ -428,9 +448,6 @@ The following functions all create :ref:`socket objects <socket-objects>`. .. versionchanged:: 3.2 *source_address* was added. - .. versionchanged:: 3.2 - support for the :keyword:`with` statement was added. - .. function:: fromfd(fd, family, type, proto=0) @@ -551,11 +568,6 @@ The :mod:`socket` module also offers various network-related services: Return a string containing the hostname of the machine where the Python interpreter is currently executing. - If you want to know the current machine's IP address, you may want to use - ``gethostbyname(gethostname())``. This operation assumes that there is a - valid address-to-host mapping for the host, and the assumption does not - always hold. - Note: :func:`gethostname` doesn't always return the fully qualified domain name; use :func:`getfqdn` for that. @@ -651,8 +663,8 @@ The :mod:`socket` module also offers various network-related services: .. function:: inet_ntoa(packed_ip) - Convert a 32-bit packed IPv4 address (a bytes object four characters in - length) to its standard dotted-quad string representation (for example, + Convert a 32-bit packed IPv4 address (a :term:`bytes-like object` four + bytes in length) to its standard dotted-quad string representation (for example, '123.45.67.89'). This is useful when conversing with a program that uses the standard C library and needs objects of type :c:type:`struct in_addr`, which is the C type for the 32-bit packed binary data this function takes as an @@ -663,6 +675,9 @@ The :mod:`socket` module also offers various network-related services: support IPv6, and :func:`inet_ntop` should be used instead for IPv4/v6 dual stack support. + .. versionchanged:: 3.5 + Writable :term:`bytes-like object` is now accepted. + .. function:: inet_pton(address_family, ip_string) @@ -685,15 +700,16 @@ The :mod:`socket` module also offers various network-related services: .. function:: inet_ntop(address_family, packed_ip) - Convert a packed IP address (a bytes object of some number of characters) to its - standard, family-specific string representation (for example, ``'7.10.0.5'`` or - ``'5aef:2b::8'``). :func:`inet_ntop` is useful when a library or network protocol - returns an object of type :c:type:`struct in_addr` (similar to :func:`inet_ntoa`) - or :c:type:`struct in6_addr`. + Convert a packed IP address (a :term:`bytes-like object` of some number of + bytes) to its standard, family-specific string representation (for + example, ``'7.10.0.5'`` or ``'5aef:2b::8'``). + :func:`inet_ntop` is useful when a library or network protocol returns an + object of type :c:type:`struct in_addr` (similar to :func:`inet_ntoa`) or + :c:type:`struct in6_addr`. Supported values for *address_family* are currently :const:`AF_INET` and - :const:`AF_INET6`. If the string *packed_ip* is not the correct length for the - specified address family, :exc:`ValueError` will be raised. + :const:`AF_INET6`. If the bytes object *packed_ip* is not the correct + length for the specified address family, :exc:`ValueError` will be raised. :exc:`OSError` is raised for errors from the call to :func:`inet_ntop`. Availability: Unix (maybe not all platforms), Windows. @@ -701,6 +717,9 @@ The :mod:`socket` module also offers various network-related services: .. versionchanged:: 3.4 Windows support added + .. versionchanged:: 3.5 + Writable :term:`bytes-like object` is now accepted. + .. XXX: Are sendmsg(), recvmsg() and CMSG_*() available on any @@ -812,6 +831,10 @@ Socket objects have the following methods. Except for :meth:`~socket.makefile`, these correspond to Unix system calls applicable to sockets. +.. versionchanged:: 3.2 + Support for the :term:`context manager` protocol was added. Exiting the + context manager is equivalent to calling :meth:`~socket.close`. + .. method:: socket.accept() @@ -825,6 +848,11 @@ to sockets. .. versionchanged:: 3.4 The socket is now non-inheritable. + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise + an exception, the method now retries the system call instead of raising + an :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + .. method:: socket.bind(address) @@ -857,6 +885,19 @@ to sockets. Connect to a remote socket at *address*. (The format of *address* depends on the address family --- see above.) + If the connection is interrupted by a signal, the method waits until the + connection completes, or raise a :exc:`socket.timeout` on timeout, if the + signal handler doesn't raise an exception and the socket is blocking or has + a timeout. For non-blocking sockets, the method raises an + :exc:`InterruptedError` exception if the connection is interrupted by a + signal (or the exception raised by the signal handler). + + .. versionchanged:: 3.5 + The method now waits until the connection completes instead of raising an + :exc:`InterruptedError` exception if the connection is interrupted by a + signal, the signal handler doesn't raise an exception and the socket is + blocking or has a timeout (see the :pep:`475` for the rationale). + .. method:: socket.connect_ex(address) @@ -889,14 +930,13 @@ to sockets. .. method:: socket.fileno() - Return the socket's file descriptor (a small integer). This is useful with - :func:`select.select`. + Return the socket's file descriptor (a small integer), or -1 on failure. This + is useful with :func:`select.select`. Under Windows the small integer returned by this method cannot be used where a file descriptor can be used (such as :func:`os.fdopen`). Unix does not have this limitation. - .. method:: socket.get_inheritable() Get the :ref:`inheritable flag <fd_inheritance>` of the socket's file @@ -946,18 +986,21 @@ to sockets. The :meth:`ioctl` method is a limited interface to the WSAIoctl system interface. Please refer to the `Win32 documentation - <http://msdn.microsoft.com/en-us/library/ms741621%28VS.85%29.aspx>`_ for more + <https://msdn.microsoft.com/en-us/library/ms741621%28VS.85%29.aspx>`_ for more information. On other platforms, the generic :func:`fcntl.fcntl` and :func:`fcntl.ioctl` functions may be used; they accept a socket object as their first argument. -.. method:: socket.listen(backlog) +.. method:: socket.listen([backlog]) - Listen for connections made to the socket. The *backlog* argument specifies the - maximum number of queued connections and should be at least 0; the maximum value - is system-dependent (usually 5), the minimum value is forced to 0. + Enable a server to accept connections. If *backlog* is specified, it must + be at least 0 (if it is lower, it is set to 0); it specifies the number of + unaccepted connections that the system will allow before refusing new + connections. If not specified, a default reasonable value is chosen. + .. versionchanged:: 3.5 + The *backlog* parameter is now optional. .. method:: socket.makefile(mode='r', buffering=None, *, encoding=None, \ errors=None, newline=None) @@ -966,7 +1009,8 @@ to sockets. Return a :term:`file object` associated with the socket. The exact returned type depends on the arguments given to :meth:`makefile`. These arguments are - interpreted the same way as by the built-in :func:`open` function. + interpreted the same way as by the built-in :func:`open` function, except + the only supported *mode* values are ``'r'`` (default), ``'w'`` and ``'b'``. The socket must be in blocking mode; it can have a timeout, but the file object's internal buffer may end up in an inconsistent state if a timeout @@ -995,6 +1039,11 @@ to sockets. For best match with hardware and network realities, the value of *bufsize* should be a relatively small power of 2, for example, 4096. + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise + an exception, the method now retries the system call instead of raising + an :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + .. method:: socket.recvfrom(bufsize[, flags]) @@ -1004,6 +1053,11 @@ to sockets. :manpage:`recv(2)` for the meaning of the optional argument *flags*; it defaults to zero. (The format of *address* depends on the address family --- see above.) + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise + an exception, the method now retries the system call instead of raising + an :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + .. method:: socket.recvmsg(bufsize[, ancbufsize[, flags]]) @@ -1070,6 +1124,11 @@ to sockets. .. versionadded:: 3.3 + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise + an exception, the method now retries the system call instead of raising + an :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + .. method:: socket.recvmsg_into(buffers[, ancbufsize[, flags]]) @@ -1136,6 +1195,11 @@ to sockets. application needs to attempt delivery of the remaining data. For further information on this topic, consult the :ref:`socket-howto`. + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise + an exception, the method now retries the system call instead of raising + an :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + .. method:: socket.sendall(bytes[, flags]) @@ -1146,6 +1210,15 @@ to sockets. success. On error, an exception is raised, and there is no way to determine how much data, if any, was successfully sent. + .. versionchanged:: 3.5 + The socket timeout is no more reset each time data is sent successfully. + The socket timeout is now the maximum total duration to send all data. + + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise + an exception, the method now retries the system call instead of raising + an :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + .. method:: socket.sendto(bytes, address) socket.sendto(bytes, flags, address) @@ -1156,6 +1229,11 @@ to sockets. bytes sent. (The format of *address* depends on the address family --- see above.) + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise + an exception, the method now retries the system call instead of raising + an :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + .. method:: socket.sendmsg(buffers[, ancdata[, flags[, address]]]) @@ -1192,6 +1270,26 @@ to sockets. .. versionadded:: 3.3 + .. versionchanged:: 3.5 + If the system call is interrupted and the signal handler does not raise + an exception, the method now retries the system call instead of raising + an :exc:`InterruptedError` exception (see :pep:`475` for the rationale). + +.. method:: socket.sendfile(file, offset=0, count=None) + + Send a file until EOF is reached by using high-performance + :mod:`os.sendfile` and return the total number of bytes which were sent. + *file* must be a regular file object opened in binary mode. If + :mod:`os.sendfile` is not available (e.g. Windows) or *file* is not a + regular file :meth:`send` will be used instead. *offset* tells from where to + start reading the file. If specified, *count* is the total number of bytes + to transmit as opposed to sending the file until EOF is reached. File + position is updated on return or also in case of error in which case + :meth:`file.tell() <io.IOBase.tell>` can be used to figure out the number of + bytes which were sent. The socket must be of :const:`SOCK_STREAM` type. Non- + blocking sockets are not supported. + + .. versionadded:: 3.5 .. method:: socket.set_inheritable(inheritable) @@ -1231,11 +1329,15 @@ to sockets. Set the value of the given socket option (see the Unix manual page :manpage:`setsockopt(2)`). The needed symbolic constants are defined in the - :mod:`socket` module (:const:`SO_\*` etc.). The value can be an integer or a - bytes object representing a buffer. In the latter case it is up to the caller to + :mod:`socket` module (:const:`SO_\*` etc.). The value can be an integer or + a :term:`bytes-like object` representing a buffer. In the latter case it is + up to the caller to ensure that the bytestring contains the proper bits (see the optional built-in module :mod:`struct` for a way to encode C structures as bytestrings). + .. versionchanged:: 3.5 + Writable :term:`bytes-like object` is now accepted. + .. method:: socket.shutdown(how) @@ -1358,16 +1460,16 @@ The first two examples support IPv4 only. :: HOST = '' # Symbolic name meaning all available interfaces PORT = 50007 # Arbitrary non-privileged port - s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) - s.bind((HOST, PORT)) - s.listen(1) - conn, addr = s.accept() - print('Connected by', addr) - while True: - data = conn.recv(1024) - if not data: break - conn.sendall(data) - conn.close() + with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s: + s.bind((HOST, PORT)) + s.listen(1) + conn, addr = s.accept() + with conn: + print('Connected by', addr) + while True: + data = conn.recv(1024) + if not data: break + conn.sendall(data) :: @@ -1376,11 +1478,10 @@ The first two examples support IPv4 only. :: HOST = 'daring.cwi.nl' # The remote host PORT = 50007 # The same port as used by the server - s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) - s.connect((HOST, PORT)) - s.sendall(b'Hello, world') - data = s.recv(1024) - s.close() + with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s: + s.connect((HOST, PORT)) + s.sendall(b'Hello, world') + data = s.recv(1024) print('Received', repr(data)) The next two examples are identical to the above two, but support both IPv4 and @@ -1417,12 +1518,12 @@ sends traffic to the first one connected successfully. :: print('could not open socket') sys.exit(1) conn, addr = s.accept() - print('Connected by', addr) - while True: - data = conn.recv(1024) - if not data: break - conn.send(data) - conn.close() + with conn: + print('Connected by', addr) + while True: + data = conn.recv(1024) + if not data: break + conn.send(data) :: @@ -1450,9 +1551,9 @@ sends traffic to the first one connected successfully. :: if s is None: print('could not open socket') sys.exit(1) - s.sendall(b'Hello, world') - data = s.recv(1024) - s.close() + with s: + s.sendall(b'Hello, world') + data = s.recv(1024) print('Received', repr(data)) diff --git a/Doc/library/socketserver.rst b/Doc/library/socketserver.rst index 3e49af6bba..087f4e0d45 100644 --- a/Doc/library/socketserver.rst +++ b/Doc/library/socketserver.rst @@ -10,16 +10,34 @@ The :mod:`socketserver` module simplifies the task of writing network servers. -There are four basic server classes: :class:`TCPServer` uses the Internet TCP -protocol, which provides for continuous streams of data between the client and -server. :class:`UDPServer` uses datagrams, which are discrete packets of -information that may arrive out of order or be lost while in transit. The more -infrequently used :class:`UnixStreamServer` and :class:`UnixDatagramServer` -classes are similar, but use Unix domain sockets; they're not available on -non-Unix platforms. For more details on network programming, consult a book -such as -W. Richard Steven's UNIX Network Programming or Ralph Davis's Win32 Network -Programming. +There are four basic concrete server classes: + + +.. class:: TCPServer(server_address, RequestHandlerClass, bind_and_activate=True) + + This uses the Internet TCP protocol, which provides for + continuous streams of data between the client and server. + If *bind_and_activate* is true, the constructor automatically attempts to + invoke :meth:`~BaseServer.server_bind` and + :meth:`~BaseServer.server_activate`. The other parameters are passed to + the :class:`BaseServer` base class. + + +.. class:: UDPServer(server_address, RequestHandlerClass, bind_and_activate=True) + + This uses datagrams, which are discrete packets of information that may + arrive out of order or be lost while in transit. The parameters are + the same as for :class:`TCPServer`. + + +.. class:: UnixStreamServer(server_address, RequestHandlerClass, bind_and_activate=True) + UnixDatagramServer(server_address, RequestHandlerClass, bind_and_activate=True) + + These more infrequently used classes are similar to the TCP and + UDP classes, but use Unix domain sockets; they're not available on + non-Unix platforms. The parameters are the same as for + :class:`TCPServer`. + These four classes process requests :dfn:`synchronously`; each request must be completed before the next request can be started. This isn't suitable if each @@ -31,10 +49,12 @@ support asynchronous behaviour. Creating a server requires several steps. First, you must create a request handler class by subclassing the :class:`BaseRequestHandler` class and -overriding its :meth:`handle` method; this method will process incoming +overriding its :meth:`~BaseRequestHandler.handle` method; +this method will process incoming requests. Second, you must instantiate one of the server classes, passing it the server's address and the request handler class. Then call the -:meth:`handle_request` or :meth:`serve_forever` method of the server object to +:meth:`~BaseServer.handle_request` or +:meth:`~BaseServer.serve_forever` method of the server object to process one or many requests. Finally, call :meth:`~BaseServer.server_close` to close the socket. @@ -76,18 +96,33 @@ Note that :class:`UnixDatagramServer` derives from :class:`UDPServer`, not from stream server is the address family, which is simply repeated in both Unix server classes. -Forking and threading versions of each type of server can be created using the -:class:`ForkingMixIn` and :class:`ThreadingMixIn` mix-in classes. For instance, -a threading UDP server class is created as follows:: - class ThreadingUDPServer(ThreadingMixIn, UDPServer): pass +.. class:: ForkingMixIn + ThreadingMixIn + + Forking and threading versions of each type of server can be created + using these mix-in classes. For instance, :class:`ThreadingUDPServer` + is created as follows:: + + class ThreadingUDPServer(ThreadingMixIn, UDPServer): + pass + + The mix-in class comes first, since it overrides a method defined in + :class:`UDPServer`. Setting the various attributes also changes the + behavior of the underlying server mechanism. + + +.. class:: ForkingTCPServer + ForkingUDPServer + ThreadingTCPServer + ThreadingUDPServer + + These classes are pre-defined using the mix-in classes. -The mix-in class must come first, since it overrides a method defined in -:class:`UDPServer`. Setting the various attributes also change the -behavior of the underlying server mechanism. To implement a service, you must derive a class from :class:`BaseRequestHandler` -and redefine its :meth:`handle` method. You can then run various versions of +and redefine its :meth:`~BaseRequestHandler.handle` method. +You can then run various versions of the service by combining one of the server classes with your request handler class. The request handler class must be different for datagram or stream services. This can be hidden by using the handler subclasses @@ -109,12 +144,12 @@ has requested. Here a threading or forking server is appropriate. In some cases, it may be appropriate to process part of a request synchronously, but to finish processing in a forked child depending on the request data. This can be implemented by using a synchronous server and doing an explicit fork in -the request handler class :meth:`handle` method. +the request handler class :meth:`~BaseRequestHandler.handle` method. Another approach to handling multiple simultaneous requests in an environment that supports neither threads nor :func:`~os.fork` (or where these are too expensive or inappropriate for the service) is to maintain an explicit table of -partially finished requests and to use :func:`~select.select` to decide which +partially finished requests and to use :mod:`selectors` to decide which request to work on next (or whether to handle a new incoming request). This is particularly important for stream services where each client can potentially be connected for a long time (if threads or subprocesses cannot be used). See @@ -127,227 +162,240 @@ connected for a long time (if threads or subprocesses cannot be used). See Server Objects -------------- -.. class:: BaseServer +.. class:: BaseServer(server_address, RequestHandlerClass) This is the superclass of all Server objects in the module. It defines the interface, given below, but does not implement most of the methods, which is - done in subclasses. + done in subclasses. The two parameters are stored in the respective + :attr:`server_address` and :attr:`RequestHandlerClass` attributes. + + + .. method:: fileno() + Return an integer file descriptor for the socket on which the server is + listening. This function is most commonly passed to :mod:`selectors`, to + allow monitoring multiple servers in the same process. -.. method:: BaseServer.fileno() - Return an integer file descriptor for the socket on which the server is - listening. This function is most commonly passed to :func:`select.select`, to - allow monitoring multiple servers in the same process. + .. method:: handle_request() + Process a single request. This function calls the following methods in + order: :meth:`get_request`, :meth:`verify_request`, and + :meth:`process_request`. If the user-provided + :meth:`~BaseRequestHandler.handle` method of the + handler class raises an exception, the server's :meth:`handle_error` method + will be called. If no request is received within :attr:`timeout` + seconds, :meth:`handle_timeout` will be called and :meth:`handle_request` + will return. -.. method:: BaseServer.handle_request() - Process a single request. This function calls the following methods in - order: :meth:`get_request`, :meth:`verify_request`, and - :meth:`process_request`. If the user-provided :meth:`handle` method of the - handler class raises an exception, the server's :meth:`handle_error` method - will be called. If no request is received within :attr:`self.timeout` - seconds, :meth:`handle_timeout` will be called and :meth:`handle_request` - will return. + .. method:: serve_forever(poll_interval=0.5) + Handle requests until an explicit :meth:`shutdown` request. Poll for + shutdown every *poll_interval* seconds. + Ignores the :attr:`timeout` attribute. It + also calls :meth:`service_actions`, which may be used by a subclass or mixin + to provide actions specific to a given service. For example, the + :class:`ForkingMixIn` class uses :meth:`service_actions` to clean up zombie + child processes. -.. method:: BaseServer.serve_forever(poll_interval=0.5) + .. versionchanged:: 3.3 + Added ``service_actions`` call to the ``serve_forever`` method. - Handle requests until an explicit :meth:`shutdown` request. Poll for - shutdown every *poll_interval* seconds. Ignores :attr:`self.timeout`. It - also calls :meth:`service_actions`, which may be used by a subclass or mixin - to provide actions specific to a given service. For example, the - :class:`ForkingMixIn` class uses :meth:`service_actions` to clean up zombie - child processes. - .. versionchanged:: 3.3 - Added ``service_actions`` call to the ``serve_forever`` method. + .. method:: service_actions() + This is called in the :meth:`serve_forever` loop. This method can be + overridden by subclasses or mixin classes to perform actions specific to + a given service, such as cleanup actions. -.. method:: BaseServer.service_actions() + .. versionadded:: 3.3 - This is called in the :meth:`serve_forever` loop. This method can be - overridden by subclasses or mixin classes to perform actions specific to - a given service, such as cleanup actions. + .. method:: shutdown() - .. versionadded:: 3.3 + Tell the :meth:`serve_forever` loop to stop and wait until it does. -.. method:: BaseServer.shutdown() - Tell the :meth:`serve_forever` loop to stop and wait until it does. + .. method:: server_close() + Clean up the server. May be overridden. -.. method:: BaseServer.server_close() - Clean up the server. May be overridden. + .. attribute:: address_family - .. versionadded:: 2.6 + The family of protocols to which the server's socket belongs. + Common examples are :const:`socket.AF_INET` and :const:`socket.AF_UNIX`. -.. attribute:: BaseServer.address_family + .. attribute:: RequestHandlerClass - The family of protocols to which the server's socket belongs. - Common examples are :const:`socket.AF_INET` and :const:`socket.AF_UNIX`. + The user-provided request handler class; an instance of this class is created + for each request. -.. attribute:: BaseServer.RequestHandlerClass + .. attribute:: server_address - The user-provided request handler class; an instance of this class is created - for each request. + The address on which the server is listening. The format of addresses varies + depending on the protocol family; + see the documentation for the :mod:`socket` module + for details. For Internet protocols, this is a tuple containing a string giving + the address, and an integer port number: ``('127.0.0.1', 80)``, for example. -.. attribute:: BaseServer.server_address + .. attribute:: socket - The address on which the server is listening. The format of addresses varies - depending on the protocol family; see the documentation for the socket module - for details. For Internet protocols, this is a tuple containing a string giving - the address, and an integer port number: ``('127.0.0.1', 80)``, for example. + The socket object on which the server will listen for incoming requests. -.. attribute:: BaseServer.socket + The server classes support the following class variables: - The socket object on which the server will listen for incoming requests. + .. XXX should class variables be covered before instance variables, or vice versa? + .. attribute:: allow_reuse_address -The server classes support the following class variables: + Whether the server will allow the reuse of an address. This defaults to + :const:`False`, and can be set in subclasses to change the policy. -.. XXX should class variables be covered before instance variables, or vice versa? -.. attribute:: BaseServer.allow_reuse_address + .. attribute:: request_queue_size - Whether the server will allow the reuse of an address. This defaults to - :const:`False`, and can be set in subclasses to change the policy. + The size of the request queue. If it takes a long time to process a single + request, any requests that arrive while the server is busy are placed into a + queue, up to :attr:`request_queue_size` requests. Once the queue is full, + further requests from clients will get a "Connection denied" error. The default + value is usually 5, but this can be overridden by subclasses. -.. attribute:: BaseServer.request_queue_size + .. attribute:: socket_type - The size of the request queue. If it takes a long time to process a single - request, any requests that arrive while the server is busy are placed into a - queue, up to :attr:`request_queue_size` requests. Once the queue is full, - further requests from clients will get a "Connection denied" error. The default - value is usually 5, but this can be overridden by subclasses. + The type of socket used by the server; :const:`socket.SOCK_STREAM` and + :const:`socket.SOCK_DGRAM` are two common values. -.. attribute:: BaseServer.socket_type + .. attribute:: timeout - The type of socket used by the server; :const:`socket.SOCK_STREAM` and - :const:`socket.SOCK_DGRAM` are two common values. + Timeout duration, measured in seconds, or :const:`None` if no timeout is + desired. If :meth:`handle_request` receives no incoming requests within the + timeout period, the :meth:`handle_timeout` method is called. -.. attribute:: BaseServer.timeout + There are various server methods that can be overridden by subclasses of base + server classes like :class:`TCPServer`; these methods aren't useful to external + users of the server object. - Timeout duration, measured in seconds, or :const:`None` if no timeout is - desired. If :meth:`handle_request` receives no incoming requests within the - timeout period, the :meth:`handle_timeout` method is called. + .. XXX should the default implementations of these be documented, or should + it be assumed that the user will look at socketserver.py? + .. method:: finish_request() -There are various server methods that can be overridden by subclasses of base -server classes like :class:`TCPServer`; these methods aren't useful to external -users of the server object. + Actually processes the request by instantiating :attr:`RequestHandlerClass` and + calling its :meth:`~BaseRequestHandler.handle` method. -.. XXX should the default implementations of these be documented, or should - it be assumed that the user will look at socketserver.py? -.. method:: BaseServer.finish_request() + .. method:: get_request() - Actually processes the request by instantiating :attr:`RequestHandlerClass` and - calling its :meth:`handle` method. + Must accept a request from the socket, and return a 2-tuple containing the *new* + socket object to be used to communicate with the client, and the client's + address. -.. method:: BaseServer.get_request() + .. method:: handle_error(request, client_address) - Must accept a request from the socket, and return a 2-tuple containing the *new* - socket object to be used to communicate with the client, and the client's - address. + This function is called if the :meth:`~BaseRequestHandler.handle` + method of a :attr:`RequestHandlerClass` instance raises + an exception. The default action is to print the traceback to + standard output and continue handling further requests. -.. method:: BaseServer.handle_error(request, client_address) + .. method:: handle_timeout() - This function is called if the :attr:`RequestHandlerClass`'s :meth:`handle` - method raises an exception. The default action is to print the traceback to - standard output and continue handling further requests. + This function is called when the :attr:`timeout` attribute has been set to a + value other than :const:`None` and the timeout period has passed with no + requests being received. The default action for forking servers is + to collect the status of any child processes that have exited, while + in threading servers this method does nothing. -.. method:: BaseServer.handle_timeout() + .. method:: process_request(request, client_address) - This function is called when the :attr:`timeout` attribute has been set to a - value other than :const:`None` and the timeout period has passed with no - requests being received. The default action for forking servers is - to collect the status of any child processes that have exited, while - in threading servers this method does nothing. + Calls :meth:`finish_request` to create an instance of the + :attr:`RequestHandlerClass`. If desired, this function can create a new process + or thread to handle the request; the :class:`ForkingMixIn` and + :class:`ThreadingMixIn` classes do this. -.. method:: BaseServer.process_request(request, client_address) + .. Is there any point in documenting the following two functions? + What would the purpose of overriding them be: initializing server + instance variables, adding new network families? - Calls :meth:`finish_request` to create an instance of the - :attr:`RequestHandlerClass`. If desired, this function can create a new process - or thread to handle the request; the :class:`ForkingMixIn` and - :class:`ThreadingMixIn` classes do this. + .. method:: server_activate() + Called by the server's constructor to activate the server. The default behavior + for a TCP server just invokes :meth:`~socket.socket.listen` + on the server's socket. May be overridden. -.. Is there any point in documenting the following two functions? - What would the purpose of overriding them be: initializing server - instance variables, adding new network families? -.. method:: BaseServer.server_activate() + .. method:: server_bind() - Called by the server's constructor to activate the server. The default behavior - just :meth:`listen`\ s to the server's socket. May be overridden. + Called by the server's constructor to bind the socket to the desired address. + May be overridden. -.. method:: BaseServer.server_bind() + .. method:: verify_request(request, client_address) - Called by the server's constructor to bind the socket to the desired address. - May be overridden. + Must return a Boolean value; if the value is :const:`True`, the request will + be processed, and if it's :const:`False`, the request will be denied. This + function can be overridden to implement access controls for a server. The + default implementation always returns :const:`True`. -.. method:: BaseServer.verify_request(request, client_address) +Request Handler Objects +----------------------- - Must return a Boolean value; if the value is :const:`True`, the request will - be processed, and if it's :const:`False`, the request will be denied. This - function can be overridden to implement access controls for a server. The - default implementation always returns :const:`True`. +.. class:: BaseRequestHandler + This is the superclass of all request handler objects. It defines + the interface, given below. A concrete request handler subclass must + define a new :meth:`handle` method, and can override any of + the other methods. A new instance of the subclass is created for each + request. -RequestHandler Objects ----------------------- -The request handler class must define a new :meth:`handle` method, and can -override any of the following methods. A new instance is created for each -request. + .. method:: setup() + Called before the :meth:`handle` method to perform any initialization actions + required. The default implementation does nothing. -.. method:: RequestHandler.finish() - Called after the :meth:`handle` method to perform any clean-up actions - required. The default implementation does nothing. If :meth:`setup` - raises an exception, this function will not be called. + .. method:: handle() + This function must do all the work required to service a request. The + default implementation does nothing. Several instance attributes are + available to it; the request is available as :attr:`self.request`; the client + address as :attr:`self.client_address`; and the server instance as + :attr:`self.server`, in case it needs access to per-server information. -.. method:: RequestHandler.handle() + The type of :attr:`self.request` is different for datagram or stream + services. For stream services, :attr:`self.request` is a socket object; for + datagram services, :attr:`self.request` is a pair of string and socket. - This function must do all the work required to service a request. The - default implementation does nothing. Several instance attributes are - available to it; the request is available as :attr:`self.request`; the client - address as :attr:`self.client_address`; and the server instance as - :attr:`self.server`, in case it needs access to per-server information. - The type of :attr:`self.request` is different for datagram or stream - services. For stream services, :attr:`self.request` is a socket object; for - datagram services, :attr:`self.request` is a pair of string and socket. - However, this can be hidden by using the request handler subclasses - :class:`StreamRequestHandler` or :class:`DatagramRequestHandler`, which - override the :meth:`setup` and :meth:`finish` methods, and provide - :attr:`self.rfile` and :attr:`self.wfile` attributes. :attr:`self.rfile` and - :attr:`self.wfile` can be read or written, respectively, to get the request - data or return data to the client. + .. method:: finish() + Called after the :meth:`handle` method to perform any clean-up actions + required. The default implementation does nothing. If :meth:`setup` + raises an exception, this function will not be called. -.. method:: RequestHandler.setup() - Called before the :meth:`handle` method to perform any initialization actions - required. The default implementation does nothing. +.. class:: StreamRequestHandler + DatagramRequestHandler + + These :class:`BaseRequestHandler` subclasses override the + :meth:`~BaseRequestHandler.setup` and :meth:`~BaseRequestHandler.finish` + methods, and provide :attr:`self.rfile` and :attr:`self.wfile` attributes. + The :attr:`self.rfile` and :attr:`self.wfile` attributes can be + read or written, respectively, to get the request data or return data + to the client. Examples @@ -362,7 +410,7 @@ This is the server side:: class MyTCPHandler(socketserver.BaseRequestHandler): """ - The RequestHandler class for our server. + The request handler class for our server. It is instantiated once per connection to the server, and must override the handle() method to implement communication to the @@ -417,17 +465,13 @@ This is the client side:: data = " ".join(sys.argv[1:]) # Create a socket (SOCK_STREAM means a TCP socket) - sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) - - try: + with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock: # Connect to server and send data sock.connect((HOST, PORT)) sock.sendall(bytes(data + "\n", "utf-8")) # Receive data from the server and shut down received = str(sock.recv(1024), "utf-8") - finally: - sock.close() print("Sent: {}".format(data)) print("Received: {}".format(received)) @@ -435,7 +479,9 @@ This is the client side:: The output of the example should look something like this: -Server:: +Server: + +.. code-block:: shell-session $ python TCPServer.py 127.0.0.1 wrote: @@ -443,7 +489,9 @@ Server:: 127.0.0.1 wrote: b'python is nice' -Client:: +Client: + +.. code-block:: shell-session $ python TCPClient.py hello world with TCP Sent: hello world with TCP @@ -526,14 +574,11 @@ An example for the :class:`ThreadingMixIn` class:: pass def client(ip, port, message): - sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) - sock.connect((ip, port)) - try: + with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock: + sock.connect((ip, port)) sock.sendall(bytes(message, 'ascii')) response = str(sock.recv(1024), 'ascii') print("Received: {}".format(response)) - finally: - sock.close() if __name__ == "__main__": # Port 0 means to select an arbitrary unused port @@ -558,7 +603,9 @@ An example for the :class:`ThreadingMixIn` class:: server.server_close() -The output of the example should look something like this:: +The output of the example should look something like this: + +.. code-block:: shell-session $ python ThreadedTCPServer.py Server loop running in thread: Thread-1 diff --git a/Doc/library/spwd.rst b/Doc/library/spwd.rst index 58be78f017..fd3c9adee4 100644 --- a/Doc/library/spwd.rst +++ b/Doc/library/spwd.rst @@ -5,6 +5,7 @@ :platform: Unix :synopsis: The shadow password database (getspnam() and friends). +-------------- This module provides access to the Unix shadow password database. It is available on various Unix versions. diff --git a/Doc/library/sqlite3.rst b/Doc/library/sqlite3.rst index 715321a19e..605d8d36b7 100644 --- a/Doc/library/sqlite3.rst +++ b/Doc/library/sqlite3.rst @@ -3,8 +3,12 @@ .. module:: sqlite3 :synopsis: A DB-API 2.0 implementation using SQLite 3.x. + .. sectionauthor:: Gerhard Häring <gh@ghaering.de> +**Source code:** :source:`Lib/sqlite3/` + +-------------- SQLite is a C library that provides a lightweight disk-based database that doesn't require a separate server process and allows accessing the database @@ -53,7 +57,7 @@ The data you've saved is persistent and is available in subsequent sessions:: Usually your SQL operations will need to use values from Python variables. You shouldn't assemble your query using Python's string operations because doing so is insecure; it makes your program vulnerable to an SQL injection attack -(see http://xkcd.com/327/ for humorous example of what can go wrong). +(see https://xkcd.com/327/ for humorous example of what can go wrong). Instead, use the DB-API's parameter substitution. Put ``?`` as a placeholder wherever you want to use a value, and then provide a tuple of values as the @@ -99,7 +103,7 @@ This example uses the iterator form:: The pysqlite web page -- sqlite3 is developed externally under the name "pysqlite". - http://www.sqlite.org + https://www.sqlite.org The SQLite web page; the documentation describes the syntax and the available data types for the supported SQL dialect. @@ -190,6 +194,11 @@ Module functions and constants any combination of :const:`PARSE_DECLTYPES` and :const:`PARSE_COLNAMES` to turn type detection on. + By default, *check_same_thread* is :const:`True` and only the creating thread may + use the connection. If set :const:`False`, the returned connection may be shared + across multiple threads. When using multiple threads with the same connection + writing operations should be serialized by the user to avoid data corruption. + By default, the :mod:`sqlite3` module uses its :class:`Connection` class for the connect call. You can, however, subclass the :class:`Connection` class and make :func:`connect` use your class instead by providing your class for the *factory* @@ -209,7 +218,7 @@ Module functions and constants db = sqlite3.connect('file:path/to/database?mode=ro', uri=True) More information about this feature, including a list of recognized options, can - be found in the `SQLite URI documentation <http://www.sqlite.org/uri.html>`_. + be found in the `SQLite URI documentation <https://www.sqlite.org/uri.html>`_. .. versionchanged:: 3.4 Added the *uri* parameter. @@ -300,32 +309,34 @@ Connection Objects call :meth:`commit`. If you just close your database connection without calling :meth:`commit` first, your changes will be lost! - .. method:: execute(sql, [parameters]) + .. method:: execute(sql[, parameters]) - This is a nonstandard shortcut that creates an intermediate cursor object by - calling the cursor method, then calls the cursor's :meth:`execute - <Cursor.execute>` method with the parameters given. + This is a nonstandard shortcut that creates a cursor object by calling + the :meth:`~Connection.cursor` method, calls the cursor's + :meth:`~Cursor.execute` method with the *parameters* given, and returns + the cursor. + .. method:: executemany(sql[, parameters]) - .. method:: executemany(sql, [parameters]) - - This is a nonstandard shortcut that creates an intermediate cursor object by - calling the cursor method, then calls the cursor's :meth:`executemany - <Cursor.executemany>` method with the parameters given. + This is a nonstandard shortcut that creates a cursor object by + calling the :meth:`~Connection.cursor` method, calls the cursor's + :meth:`~Cursor.executemany` method with the *parameters* given, and + returns the cursor. .. method:: executescript(sql_script) - This is a nonstandard shortcut that creates an intermediate cursor object by - calling the cursor method, then calls the cursor's :meth:`executescript - <Cursor.executescript>` method with the parameters given. - + This is a nonstandard shortcut that creates a cursor object by + calling the :meth:`~Connection.cursor` method, calls the cursor's + :meth:`~Cursor.executescript` method with the given *sql_script*, and + returns the cursor. .. method:: create_function(name, num_params, func) Creates a user-defined function that you can later use from within SQL statements under the function name *name*. *num_params* is the number of - parameters the function accepts, and *func* is a Python callable that is called - as the SQL function. + parameters the function accepts (if *num_params* is -1, the function may + take any number of arguments), and *func* is a Python callable that is + called as the SQL function. The function can return any of the types supported by SQLite: bytes, str, int, float and None. @@ -340,7 +351,8 @@ Connection Objects Creates a user-defined aggregate function. The aggregate class must implement a ``step`` method, which accepts the number - of parameters *num_params*, and a ``finalize`` method which will return the + of parameters *num_params* (if *num_params* is -1, the function may take + any number of arguments), and a ``finalize`` method which will return the final result of the aggregate. The ``finalize`` method can return any of the types supported by SQLite: @@ -477,10 +489,6 @@ Connection Objects :mod:`sqlite3` module will return Unicode objects for ``TEXT``. If you want to return bytestrings instead, you can set it to :class:`bytes`. - For efficiency reasons, there's also a way to return :class:`str` objects - only for non-ASCII data, and :class:`bytes` otherwise. To activate it, set - this attribute to :const:`sqlite3.OptimizedUnicode`. - You can also set it to any other callable that accepts a single bytestring parameter and returns the resulting object. @@ -495,7 +503,7 @@ Connection Objects deleted since the database connection was opened. - .. attribute:: iterdump + .. method:: iterdump Returns an iterator to dump the database in an SQL text format. Useful when saving an in-memory database for later restoration. This function provides @@ -505,7 +513,7 @@ Connection Objects Example:: # Convert file existing_db.db to SQL dump file dump.sql - import sqlite3, os + import sqlite3 con = sqlite3.connect('existing_db.db') with open('dump.sql', 'w') as f: @@ -522,7 +530,7 @@ Cursor Objects A :class:`Cursor` instance has the following attributes and methods. - .. method:: execute(sql, [parameters]) + .. method:: execute(sql[, parameters]) Executes an SQL statement. The SQL statement may be parameterized (i. e. placeholders instead of SQL literals). The :mod:`sqlite3` module supports two @@ -534,7 +542,7 @@ Cursor Objects .. literalinclude:: ../includes/sqlite3/execute_1.py :meth:`execute` will only execute a single SQL statement. If you try to execute - more than one statement with it, it will raise a Warning. Use + more than one statement with it, it will raise an ``sqlite3.Warning``. Use :meth:`executescript` if you want to execute multiple SQL statements with one call. @@ -542,8 +550,8 @@ Cursor Objects .. method:: executemany(sql, seq_of_parameters) Executes an SQL command against all parameter sequences or mappings found in - the sequence *sql*. The :mod:`sqlite3` module also allows using an - :term:`iterator` yielding parameters instead of a sequence. + the sequence *seq_of_parameters*. The :mod:`sqlite3` module also allows + using an :term:`iterator` yielding parameters instead of a sequence. .. literalinclude:: ../includes/sqlite3/executemany_1.py @@ -558,7 +566,7 @@ Cursor Objects at once. It issues a ``COMMIT`` statement first, then executes the SQL script it gets as a parameter. - *sql_script* can be an instance of :class:`str` or :class:`bytes`. + *sql_script* can be an instance of :class:`str`. Example: @@ -593,6 +601,12 @@ Cursor Objects the cursor's arraysize attribute can affect the performance of this operation. An empty list is returned when no rows are available. + .. method:: close() + + Close the cursor now (rather than whenever ``__del__`` is called). + + The cursor will be unusable from this point forward; a ``ProgrammingError`` + exception will be raised if any operation is attempted with the cursor. .. attribute:: rowcount @@ -627,6 +641,18 @@ Cursor Objects It is set for ``SELECT`` statements without any matching rows as well. + .. attribute:: connection + + This read-only attribute provides the SQLite database :class:`Connection` + used by the :class:`Cursor` object. A :class:`Cursor` object created by + calling :meth:`con.cursor() <Connection.cursor>` will have a + :attr:`connection` attribute that refers to *con*:: + + >>> con = sqlite3.connect(":memory:") + >>> cur = con.cursor() + >>> cur.connection == con + True + .. _sqlite3-row-objects: Row Objects @@ -649,6 +675,9 @@ Row Objects This method returns a list of column names. Immediately after a query, it is the first member of each tuple in :attr:`Cursor.description`. + .. versionchanged:: 3.5 + Added support of slicing. + Let's assume we initialize a table as in the example given above:: conn = sqlite3.connect(":memory:") diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst index b0cf4ebd83..5792d0d407 100644 --- a/Doc/library/ssl.rst +++ b/Doc/library/ssl.rst @@ -7,13 +7,12 @@ .. moduleauthor:: Bill Janssen <bill.janssen@gmail.com> .. sectionauthor:: Bill Janssen <bill.janssen@gmail.com> +**Source code:** :source:`Lib/ssl.py` .. index:: single: OpenSSL; (use in module ssl) .. index:: TLS, SSL, Transport Layer Security, Secure Sockets Layer -**Source code:** :source:`Lib/ssl.py` - -------------- This module provides access to Transport Layer Security (often known as "Secure @@ -206,7 +205,7 @@ instead. The *ciphers* parameter sets the available ciphers for this SSL object. It should be a string in the `OpenSSL cipher list format - <http://www.openssl.org/docs/apps/ciphers.html#CIPHER_LIST_FORMAT>`_. + <https://www.openssl.org/docs/apps/ciphers.html#CIPHER-LIST-FORMAT>`_. The parameter ``do_handshake_on_connect`` specifies whether to do the SSL handshake automatically after doing a :meth:`socket.connect`, or whether the @@ -296,7 +295,7 @@ Random generation Read the Wikipedia article, `Cryptographically secure pseudorandom number generator (CSPRNG) - <http://en.wikipedia.org/wiki/Cryptographically_secure_pseudorandom_number_generator>`_, + <https://en.wikipedia.org/wiki/Cryptographically_secure_pseudorandom_number_generator>`_, to get the requirements of a cryptographically generator. .. versionadded:: 3.3 @@ -335,6 +334,8 @@ Random generation See http://egd.sourceforge.net/ or http://prngd.sourceforge.net/ for sources of entropy-gathering daemons. + Availability: not available with LibreSSL. + .. function:: RAND_add(bytes, entropy) Mix the given *bytes* into the SSL pseudo-random number generator. The @@ -342,6 +343,9 @@ Random generation string (so you can always use :const:`0.0`). See :rfc:`1750` for more information on sources of entropy. + .. versionchanged:: 3.5 + Writable :term:`bytes-like object` is now accepted. + Certificate handling ^^^^^^^^^^^^^^^^^^^^ @@ -350,10 +354,9 @@ Certificate handling Verify that *cert* (in decoded format as returned by :meth:`SSLSocket.getpeercert`) matches the given *hostname*. The rules applied are those for checking the identity of HTTPS servers as outlined - in :rfc:`2818` and :rfc:`6125`, except that IP addresses are not currently - supported. In addition to HTTPS, this function should be suitable for - checking the identity of servers in various SSL-based protocols such as - FTPS, IMAPS, POPS and others. + in :rfc:`2818` and :rfc:`6125`. In addition to HTTPS, this function + should be suitable for checking the identity of servers in various + SSL-based protocols such as FTPS, IMAPS, POPS and others. :exc:`CertificateError` is raised on failure. On success, the function returns nothing:: @@ -375,22 +378,38 @@ Certificate handling IDN A-labels such as ``www*.xn--pthon-kva.org`` are still supported, but ``x*.python.org`` no longer matches ``xn--tda.python.org``. -.. function:: cert_time_to_seconds(timestring) + .. versionchanged:: 3.5 + Matching of IP addresses, when present in the subjectAltName field + of the certificate, is now supported. - Returns a floating-point value containing a normal seconds-after-the-epoch - time value, given the time-string representing the "notBefore" or "notAfter" - date from a certificate. +.. function:: cert_time_to_seconds(cert_time) - Here's an example:: + Return the time in seconds since the Epoch, given the ``cert_time`` + string representing the "notBefore" or "notAfter" date from a + certificate in ``"%b %d %H:%M:%S %Y %Z"`` strptime format (C + locale). - >>> import ssl - >>> ssl.cert_time_to_seconds("May 9 00:00:00 2007 GMT") - 1178694000.0 - >>> import time - >>> time.ctime(ssl.cert_time_to_seconds("May 9 00:00:00 2007 GMT")) - 'Wed May 9 00:00:00 2007' + Here's an example: -.. function:: get_server_certificate(addr, ssl_version=PROTOCOL_SSLv3, ca_certs=None) + .. doctest:: newcontext + + >>> import ssl + >>> timestamp = ssl.cert_time_to_seconds("Jan 5 09:34:43 2018 GMT") + >>> timestamp + 1515144883 + >>> from datetime import datetime + >>> print(datetime.utcfromtimestamp(timestamp)) + 2018-01-05 09:34:43 + + "notBefore" or "notAfter" dates must use GMT (:rfc:`5280`). + + .. versionchanged:: 3.5 + Interpret the input time as a time in UTC as specified by 'GMT' + timezone in the input string. Local timezone was used + previously. Return an integer (no fractions of a second in the + input format) + +.. function:: get_server_certificate(addr, ssl_version=PROTOCOL_SSLv23, ca_certs=None) Given the address ``addr`` of an SSL-protected server, as a (*hostname*, *port-number*) pair, fetches the server's certificate, and returns it as a @@ -404,6 +423,10 @@ Certificate handling .. versionchanged:: 3.3 This function is now IPv6-compatible. + .. versionchanged:: 3.5 + The default *ssl_version* is changed from :data:`PROTOCOL_SSLv3` to + :data:`PROTOCOL_SSLv23` for maximum compatibility with modern servers. + .. function:: DER_cert_to_PEM_cert(DER_cert_bytes) Given a certificate as a DER-encoded blob of bytes, returns a PEM-encoded @@ -671,6 +694,13 @@ Constants .. versionadded:: 3.3 +.. data:: HAS_ALPN + + Whether the OpenSSL library has built-in support for the *Application-Layer + Protocol Negotiation* TLS extension as described in :rfc:`7301`. + + .. versionadded:: 3.5 + .. data:: HAS_ECDH Whether the OpenSSL library has built-in support for Elliptic Curve-based @@ -690,7 +720,7 @@ Constants Whether the OpenSSL library has built-in support for *Next Protocol Negotiation* as described in the `NPN draft specification - <http://tools.ietf.org/html/draft-agl-tls-nextprotoneg>`_. When true, + <https://tools.ietf.org/html/draft-agl-tls-nextprotoneg>`_. When true, you can use the :meth:`SSLContext.set_npn_protocols` method to advertise which protocols you want to support. @@ -738,7 +768,7 @@ Constants ALERT_DESCRIPTION_* Alert Descriptions from :rfc:`5246` and others. The `IANA TLS Alert Registry - <http://www.iana.org/assignments/tls-parameters/tls-parameters.xml#tls-parameters-6>`_ + <https://www.iana.org/assignments/tls-parameters/tls-parameters.xml#tls-parameters-6>`_ contains this list and references to the RFCs where their meaning is defined. Used as the return value of the callback function in @@ -788,6 +818,8 @@ SSL Sockets (but passing a non-zero ``flags`` argument is not allowed) - :meth:`~socket.socket.send()`, :meth:`~socket.socket.sendall()` (with the same limitation) + - :meth:`~socket.socket.sendfile()` (but :mod:`os.sendfile` will be used + for plain-text sockets only, else :meth:`~socket.socket.send()` will be used) - :meth:`~socket.socket.shutdown()` However, since the SSL (and TLS) protocol has its own framing atop @@ -798,9 +830,18 @@ SSL Sockets Usually, :class:`SSLSocket` are not created directly, but using the :func:`wrap_socket` function or the :meth:`SSLContext.wrap_socket` method. + .. versionchanged:: 3.5 + The :meth:`sendfile` method was added. + + .. versionchanged:: 3.5 + The :meth:`shutdown` does not reset the socket timeout each time bytes + are received or sent. The socket timeout is now to maximum total duration + of the shutdown. + + SSL sockets also have the following additional methods and attributes: -.. method:: SSLSocket.read(len=0, buffer=None) +.. method:: SSLSocket.read(len=1024, buffer=None) Read up to *len* bytes of data from the SSL socket and return the result as a ``bytes`` instance. If *buffer* is specified, then read into the buffer @@ -812,6 +853,11 @@ SSL sockets also have the following additional methods and attributes: As at any time a re-negotiation is possible, a call to :meth:`read` can also cause write operations. + .. versionchanged:: 3.5 + The socket timeout is no more reset each time bytes are received or sent. + The socket timeout is now to maximum total duration to read up to *len* + bytes. + .. method:: SSLSocket.write(buf) Write *buf* to the SSL socket and return the number of bytes written. The @@ -823,6 +869,10 @@ SSL sockets also have the following additional methods and attributes: As at any time a re-negotiation is possible, a call to :meth:`write` can also cause read operations. + .. versionchanged:: 3.5 + The socket timeout is no more reset each time bytes are received or sent. + The socket timeout is now to maximum total duration to write *buf*. + .. note:: The :meth:`~SSLSocket.read` and :meth:`~SSLSocket.write` methods are the @@ -844,6 +894,10 @@ SSL sockets also have the following additional methods and attributes: :attr:`~SSLContext.check_hostname` attribute of the socket's :attr:`~SSLSocket.context` is true. + .. versionchanged:: 3.5 + The socket timeout is no more reset each time bytes are received or sent. + The socket timeout is now to maximum total duration of the handshake. + .. method:: SSLSocket.getpeercert(binary_form=False) If there is no certificate for the peer on the other end of the connection, @@ -917,6 +971,17 @@ SSL sockets also have the following additional methods and attributes: version of the SSL protocol that defines its use, and the number of secret bits being used. If no connection has been established, returns ``None``. +.. method:: SSLSocket.shared_ciphers() + + Return the list of ciphers shared by the client during the handshake. Each + entry of the returned list is a three-value tuple containing the name of the + cipher, the version of the SSL protocol that defines its use, and the number + of secret bits the cipher uses. :meth:`~SSLSocket.shared_ciphers` returns + ``None`` if no connection has been established or the socket is a client + socket. + + .. versionadded:: 3.5 + .. method:: SSLSocket.compression() Return the compression algorithm being used as a string, or ``None`` @@ -940,12 +1005,22 @@ SSL sockets also have the following additional methods and attributes: .. versionadded:: 3.3 +.. method:: SSLSocket.selected_alpn_protocol() + + Return the protocol that was selected during the TLS handshake. If + :meth:`SSLContext.set_alpn_protocols` was not called, if the other party does + not support ALPN, if this socket does not support any of the client's + proposed protocols, or if the handshake has not happened yet, ``None`` is + returned. + + .. versionadded:: 3.5 + .. method:: SSLSocket.selected_npn_protocol() - Returns the protocol that was selected during the TLS/SSL handshake. If - :meth:`SSLContext.set_npn_protocols` was not called, or if the other party - does not support NPN, or if the handshake has not yet happened, this will - return ``None``. + Return the higher-level protocol that was selected during the TLS/SSL + handshake. If :meth:`SSLContext.set_npn_protocols` was not called, or + if the other party does not support NPN, or if the handshake has not yet + happened, this will return ``None``. .. versionadded:: 3.3 @@ -957,6 +1032,16 @@ SSL sockets also have the following additional methods and attributes: returned socket should always be used for further communication with the other side of the connection, rather than the original socket. +.. method:: SSLSocket.version() + + Return the actual SSL protocol version negotiated by the connection + as a string, or ``None`` is no secure connection is established. + As of this writing, possible return values include ``"SSLv2"``, + ``"SSLv3"``, ``"TLSv1"``, ``"TLSv1.1"`` and ``"TLSv1.2"``. + Recent OpenSSL versions may define more return values. + + .. versionadded:: 3.5 + .. method:: SSLSocket.pending() Returns the number of already decrypted bytes available for read, pending on @@ -1088,7 +1173,7 @@ to speed up repeated connections from the same clients. The *capath* string, if present, is the path to a directory containing several CA certificates in PEM format, following an `OpenSSL specific layout - <http://www.openssl.org/docs/ssl/SSL_CTX_load_verify_locations.html>`_. + <https://www.openssl.org/docs/ssl/SSL_CTX_load_verify_locations.html>`_. The *cadata* object, if present, is either an ASCII string of one or more PEM-encoded certificates or a :term:`bytes-like object` of DER-encoded @@ -1126,7 +1211,7 @@ to speed up repeated connections from the same clients. Set the available ciphers for sockets created with this context. It should be a string in the `OpenSSL cipher list format - <http://www.openssl.org/docs/apps/ciphers.html#CIPHER_LIST_FORMAT>`_. + <https://www.openssl.org/docs/apps/ciphers.html#CIPHER-LIST-FORMAT>`_. If no cipher can be selected (because compile-time options or other configuration forbids use of all the specified ciphers), an :class:`SSLError` will be raised. @@ -1135,13 +1220,27 @@ to speed up repeated connections from the same clients. when connected, the :meth:`SSLSocket.cipher` method of SSL sockets will give the currently selected cipher. +.. method:: SSLContext.set_alpn_protocols(protocols) + + Specify which protocols the socket should advertise during the SSL/TLS + handshake. It should be a list of ASCII strings, like ``['http/1.1', + 'spdy/2']``, ordered by preference. The selection of a protocol will happen + during the handshake, and will play out according to :rfc:`7301`. After a + successful handshake, the :meth:`SSLSocket.selected_alpn_protocol` method will + return the agreed-upon protocol. + + This method will raise :exc:`NotImplementedError` if :data:`HAS_ALPN` is + False. + + .. versionadded:: 3.5 + .. method:: SSLContext.set_npn_protocols(protocols) Specify which protocols the socket should advertise during the SSL/TLS handshake. It should be a list of strings, like ``['http/1.1', 'spdy/2']``, ordered by preference. The selection of a protocol will happen during the handshake, and will play out according to the `NPN draft specification - <http://tools.ietf.org/html/draft-agl-tls-nextprotoneg>`_. After a + <https://tools.ietf.org/html/draft-agl-tls-nextprotoneg>`_. After a successful handshake, the :meth:`SSLSocket.selected_npn_protocol` method will return the agreed-upon protocol. @@ -1175,7 +1274,7 @@ to speed up repeated connections from the same clients. Due to the early negotiation phase of the TLS connection, only limited methods and attributes are usable like - :meth:`SSLSocket.selected_npn_protocol` and :attr:`SSLSocket.context`. + :meth:`SSLSocket.selected_alpn_protocol` and :attr:`SSLSocket.context`. :meth:`SSLSocket.getpeercert`, :meth:`SSLSocket.getpeercert`, :meth:`SSLSocket.cipher` and :meth:`SSLSocket.compress` methods require that the TLS connection has progressed beyond the TLS Client Hello and therefore @@ -1251,15 +1350,25 @@ to speed up repeated connections from the same clients. quite similarly to HTTP virtual hosts. Specifying *server_hostname* will raise a :exc:`ValueError` if *server_side* is true. - .. versionchanged:: 3.4.3 + .. versionchanged:: 3.5 Always allow a server_hostname to be passed, even if OpenSSL does not have SNI. +.. method:: SSLContext.wrap_bio(incoming, outgoing, server_side=False, \ + server_hostname=None) + + Create a new :class:`SSLObject` instance by wrapping the BIO objects + *incoming* and *outgoing*. The SSL routines will read input data from the + incoming BIO and write data to the outgoing BIO. + + The *server_side* and *server_hostname* parameters have the same meaning as + in :meth:`SSLContext.wrap_socket`. + .. method:: SSLContext.session_stats() Get statistics about the SSL sessions created or managed by this context. A dictionary is returned which maps the names of each `piece of information - <http://www.openssl.org/docs/ssl/SSL_CTX_sess_number.html>`_ to their + <https://www.openssl.org/docs/ssl/SSL_CTX_sess_number.html>`_ to their numeric values. For example, here is the total number of hits and misses in the session cache since the context was created:: @@ -1475,7 +1584,7 @@ should use the following idiom:: except ImportError: pass else: - ... # do something that requires SSL support + ... # do something that requires SSL support Client-side operation ^^^^^^^^^^^^^^^^^^^^^ @@ -1627,7 +1736,7 @@ are finished with the client (or the client is finished with you):: And go back to listening for new client connections (of course, a real server would probably handle each client connection in a separate thread, or put -the sockets in non-blocking mode and use an event loop). +the sockets in :ref:`non-blocking mode <ssl-nonblocking>` and use an event loop). .. _ssl-nonblocking: @@ -1649,6 +1758,12 @@ thus several things you need to be aware of: socket first, and attempts to *read* from the SSL socket may require a prior *write* to the underlying socket. + .. versionchanged:: 3.5 + + In earlier Python versions, the :meth:`!SSLSocket.send` method + returned zero instead of raising :exc:`SSLWantWriteError` or + :exc:`SSLWantReadError`. + - Calling :func:`~select.select` tells you that the OS-level socket can be read from (or written to), but it does not imply that there is sufficient data at the upper SSL layer. For example, only part of an SSL frame might @@ -1681,13 +1796,143 @@ thus several things you need to be aware of: .. seealso:: - The :mod:`asyncio` module supports non-blocking SSL sockets and provides a + The :mod:`asyncio` module supports :ref:`non-blocking SSL sockets + <ssl-nonblocking>` and provides a higher level API. It polls for events using the :mod:`selectors` module and handles :exc:`SSLWantWriteError`, :exc:`SSLWantReadError` and :exc:`BlockingIOError` exceptions. It runs the SSL handshake asynchronously as well. +Memory BIO Support +------------------ + +.. versionadded:: 3.5 + +Ever since the SSL module was introduced in Python 2.6, the :class:`SSLSocket` +class has provided two related but distinct areas of functionality: + +- SSL protocol handling +- Network IO + +The network IO API is identical to that provided by :class:`socket.socket`, +from which :class:`SSLSocket` also inherits. This allows an SSL socket to be +used as a drop-in replacement for a regular socket, making it very easy to add +SSL support to an existing application. + +Combining SSL protocol handling and network IO usually works well, but there +are some cases where it doesn't. An example is async IO frameworks that want to +use a different IO multiplexing model than the "select/poll on a file +descriptor" (readiness based) model that is assumed by :class:`socket.socket` +and by the internal OpenSSL socket IO routines. This is mostly relevant for +platforms like Windows where this model is not efficient. For this purpose, a +reduced scope variant of :class:`SSLSocket` called :class:`SSLObject` is +provided. + +.. class:: SSLObject + + A reduced-scope variant of :class:`SSLSocket` representing an SSL protocol + instance that does not contain any network IO methods. This class is + typically used by framework authors that want to implement asynchronous IO + for SSL through memory buffers. + + This class implements an interface on top of a low-level SSL object as + implemented by OpenSSL. This object captures the state of an SSL connection + but does not provide any network IO itself. IO needs to be performed through + separate "BIO" objects which are OpenSSL's IO abstraction layer. + + An :class:`SSLObject` instance can be created using the + :meth:`~SSLContext.wrap_bio` method. This method will create the + :class:`SSLObject` instance and bind it to a pair of BIOs. The *incoming* + BIO is used to pass data from Python to the SSL protocol instance, while the + *outgoing* BIO is used to pass data the other way around. + + The following methods are available: + + - :attr:`~SSLSocket.context` + - :attr:`~SSLSocket.server_side` + - :attr:`~SSLSocket.server_hostname` + - :meth:`~SSLSocket.read` + - :meth:`~SSLSocket.write` + - :meth:`~SSLSocket.getpeercert` + - :meth:`~SSLSocket.selected_npn_protocol` + - :meth:`~SSLSocket.cipher` + - :meth:`~SSLSocket.shared_ciphers` + - :meth:`~SSLSocket.compression` + - :meth:`~SSLSocket.pending` + - :meth:`~SSLSocket.do_handshake` + - :meth:`~SSLSocket.unwrap` + - :meth:`~SSLSocket.get_channel_binding` + + When compared to :class:`SSLSocket`, this object lacks the following + features: + + - Any form of network IO incluging methods such as ``recv()`` and + ``send()``. + + - There is no *do_handshake_on_connect* machinery. You must always manually + call :meth:`~SSLSocket.do_handshake` to start the handshake. + + - There is no handling of *suppress_ragged_eofs*. All end-of-file conditions + that are in violation of the protocol are reported via the + :exc:`SSLEOFError` exception. + + - The method :meth:`~SSLSocket.unwrap` call does not return anything, + unlike for an SSL socket where it returns the underlying socket. + + - The *server_name_callback* callback passed to + :meth:`SSLContext.set_servername_callback` will get an :class:`SSLObject` + instance instead of a :class:`SSLSocket` instance as its first parameter. + + Some notes related to the use of :class:`SSLObject`: + + - All IO on an :class:`SSLObject` is :ref:`non-blocking <ssl-nonblocking>`. + This means that for example :meth:`~SSLSocket.read` will raise an + :exc:`SSLWantReadError` if it needs more data than the incoming BIO has + available. + + - There is no module-level ``wrap_bio()`` call like there is for + :meth:`~SSLContext.wrap_socket`. An :class:`SSLObject` is always created + via an :class:`SSLContext`. + +An SSLObject communicates with the outside world using memory buffers. The +class :class:`MemoryBIO` provides a memory buffer that can be used for this +purpose. It wraps an OpenSSL memory BIO (Basic IO) object: + +.. class:: MemoryBIO + + A memory buffer that can be used to pass data between Python and an SSL + protocol instance. + + .. attribute:: MemoryBIO.pending + + Return the number of bytes currently in the memory buffer. + + .. attribute:: MemoryBIO.eof + + A boolean indicating whether the memory BIO is current at the end-of-file + position. + + .. method:: MemoryBIO.read(n=-1) + + Read up to *n* bytes from the memory buffer. If *n* is not specified or + negative, all bytes are returned. + + .. method:: MemoryBIO.write(buf) + + Write the bytes from *buf* to the memory BIO. The *buf* argument must be an + object supporting the buffer protocol. + + The return value is the number of bytes written, which is always equal to + the length of *buf*. + + .. method:: MemoryBIO.write_eof() + + Write an EOF marker to the memory BIO. After this method has been called, it + is illegal to call :meth:`~MemoryBIO.write`. The attribute :attr:`eof` will + become true after all data currently in the buffer has been read. + + .. _ssl-security: Security considerations @@ -1773,7 +2018,7 @@ enabled when negotiating a SSL session is possible through the :meth:`SSLContext.set_ciphers` method. Starting from Python 3.2.3, the ssl module disables certain weak ciphers by default, but you may want to further restrict the cipher choice. Be sure to read OpenSSL's documentation -about the `cipher list format <http://www.openssl.org/docs/apps/ciphers.html#CIPHER_LIST_FORMAT>`_. +about the `cipher list format <https://www.openssl.org/docs/apps/ciphers.html#CIPHER-LIST-FORMAT>`_. If you want to check which ciphers are enabled by a given cipher list, use the ``openssl ciphers`` command on your system. @@ -1794,26 +2039,26 @@ successful call of :func:`~ssl.RAND_add`, :func:`~ssl.RAND_bytes` or Class :class:`socket.socket` Documentation of underlying :mod:`socket` class - `SSL/TLS Strong Encryption: An Introduction <http://httpd.apache.org/docs/trunk/en/ssl/ssl_intro.html>`_ + `SSL/TLS Strong Encryption: An Introduction <https://httpd.apache.org/docs/trunk/en/ssl/ssl_intro.html>`_ Intro from the Apache webserver documentation - `RFC 1422: Privacy Enhancement for Internet Electronic Mail: Part II: Certificate-Based Key Management <http://www.ietf.org/rfc/rfc1422>`_ + `RFC 1422: Privacy Enhancement for Internet Electronic Mail: Part II: Certificate-Based Key Management <https://www.ietf.org/rfc/rfc1422>`_ Steve Kent - `RFC 1750: Randomness Recommendations for Security <http://www.ietf.org/rfc/rfc1750>`_ + `RFC 1750: Randomness Recommendations for Security <https://www.ietf.org/rfc/rfc1750>`_ D. Eastlake et. al. - `RFC 3280: Internet X.509 Public Key Infrastructure Certificate and CRL Profile <http://www.ietf.org/rfc/rfc3280>`_ + `RFC 3280: Internet X.509 Public Key Infrastructure Certificate and CRL Profile <https://www.ietf.org/rfc/rfc3280>`_ Housley et. al. - `RFC 4366: Transport Layer Security (TLS) Extensions <http://www.ietf.org/rfc/rfc4366>`_ + `RFC 4366: Transport Layer Security (TLS) Extensions <https://www.ietf.org/rfc/rfc4366>`_ Blake-Wilson et. al. - `RFC 5246: The Transport Layer Security (TLS) Protocol Version 1.2 <http://tools.ietf.org/html/rfc5246>`_ + `RFC 5246: The Transport Layer Security (TLS) Protocol Version 1.2 <https://tools.ietf.org/html/rfc5246>`_ T. Dierks et. al. - `RFC 6066: Transport Layer Security (TLS) Extensions <http://tools.ietf.org/html/rfc6066>`_ + `RFC 6066: Transport Layer Security (TLS) Extensions <https://tools.ietf.org/html/rfc6066>`_ D. Eastlake - `IANA TLS: Transport Layer Security (TLS) Parameters <http://www.iana.org/assignments/tls-parameters/tls-parameters.xml>`_ + `IANA TLS: Transport Layer Security (TLS) Parameters <https://www.iana.org/assignments/tls-parameters/tls-parameters.xml>`_ IANA diff --git a/Doc/library/stat.rst b/Doc/library/stat.rst index 24769f6893..b256312d91 100644 --- a/Doc/library/stat.rst +++ b/Doc/library/stat.rst @@ -4,10 +4,10 @@ .. module:: stat :synopsis: Utilities for interpreting the results of os.stat(), os.lstat() and os.fstat(). + .. sectionauthor:: Skip Montanaro <skip@automatrix.com> -**Source code:** :source:`Modules/_stat.c` - :source:`Lib/stat.py` +**Source code:** :source:`Lib/stat.py` -------------- @@ -126,7 +126,7 @@ Example:: if __name__ == '__main__': walktree(sys.argv[1], visitfile) -An additional utility function is provided to covert a file's mode in a human +An additional utility function is provided to convert a file's mode in a human readable string: .. function:: filemode(mode) @@ -399,3 +399,29 @@ The following flags can be used in the *flags* argument of :func:`os.chflags`: The file is a snapshot file. See the \*BSD or Mac OS systems man page :manpage:`chflags(2)` for more information. + +On Windows, the following file attribute constants are available for use when +testing bits in the ``st_file_attributes`` member returned by :func:`os.stat`. +See the `Windows API documentation +<https://msdn.microsoft.com/en-us/library/windows/desktop/gg258117.aspx>`_ +for more detail on the meaning of these constants. + +.. data:: FILE_ATTRIBUTE_ARCHIVE + FILE_ATTRIBUTE_COMPRESSED + FILE_ATTRIBUTE_DEVICE + FILE_ATTRIBUTE_DIRECTORY + FILE_ATTRIBUTE_ENCRYPTED + FILE_ATTRIBUTE_HIDDEN + FILE_ATTRIBUTE_INTEGRITY_STREAM + FILE_ATTRIBUTE_NORMAL + FILE_ATTRIBUTE_NOT_CONTENT_INDEXED + FILE_ATTRIBUTE_NO_SCRUB_DATA + FILE_ATTRIBUTE_OFFLINE + FILE_ATTRIBUTE_READONLY + FILE_ATTRIBUTE_REPARSE_POINT + FILE_ATTRIBUTE_SPARSE_FILE + FILE_ATTRIBUTE_SYSTEM + FILE_ATTRIBUTE_TEMPORARY + FILE_ATTRIBUTE_VIRTUAL + + .. versionadded:: 3.5 diff --git a/Doc/library/statistics.rst b/Doc/library/statistics.rst index 0c9d88c8de..ea3d7dab0f 100644 --- a/Doc/library/statistics.rst +++ b/Doc/library/statistics.rst @@ -3,18 +3,19 @@ .. module:: statistics :synopsis: mathematical statistics functions + .. moduleauthor:: Steven D'Aprano <steve+python@pearwood.info> .. sectionauthor:: Steven D'Aprano <steve+python@pearwood.info> .. versionadded:: 3.4 +**Source code:** :source:`Lib/statistics.py` + .. testsetup:: * from statistics import * __name__ = '<doctest>' -**Source code:** :source:`Lib/statistics.py` - -------------- This module provides functions for calculating mathematical statistics of @@ -223,7 +224,7 @@ However, for reading convenience, most of the examples show sorted sequences. * "Statistics for the Behavioral Sciences", Frederick J Gravetter and Larry B Wallnau (8th Edition). - * Calculating the `median <http://www.ualberta.ca/~opscan/median.html>`_. + * Calculating the `median <https://www.ualberta.ca/~opscan/median.html>`_. * The `SSMEDIAN <https://help.gnome.org/users/gnumeric/stable/gnumeric.html#gnumeric-function-SSMEDIAN>`_ diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst index 8815d38921..2996eef27c 100644 --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -354,26 +354,29 @@ Notes: The numeric literals accepted include the digits ``0`` to ``9`` or any Unicode equivalent (code points with the ``Nd`` property). - See http://www.unicode.org/Public/6.3.0/ucd/extracted/DerivedNumericType.txt + See http://www.unicode.org/Public/8.0.0/ucd/extracted/DerivedNumericType.txt for a complete list of code points with the ``Nd`` property. All :class:`numbers.Real` types (:class:`int` and :class:`float`) also include the following operations: -+--------------------+------------------------------------+--------+ -| Operation | Result | Notes | -+====================+====================================+========+ -| ``math.trunc(x)`` | *x* truncated to Integral | | -+--------------------+------------------------------------+--------+ -| ``round(x[, n])`` | *x* rounded to n digits, | | -| | rounding half to even. If n is | | -| | omitted, it defaults to 0. | | -+--------------------+------------------------------------+--------+ -| ``math.floor(x)`` | the greatest integral float <= *x* | | -+--------------------+------------------------------------+--------+ -| ``math.ceil(x)`` | the least integral float >= *x* | | -+--------------------+------------------------------------+--------+ ++--------------------+---------------------------------------------+ +| Operation | Result | ++====================+=============================================+ +| :func:`math.trunc(\| *x* truncated to :class:`~numbers.Integral` | +| x) <math.trunc>` | | ++--------------------+---------------------------------------------+ +| :func:`round(x[, | *x* rounded to *n* digits, | +| n]) <round>` | rounding half to even. If *n* is | +| | omitted, it defaults to 0. | ++--------------------+---------------------------------------------+ +| :func:`math.floor(\| the greatest :class:`~numbers.Integral` | +| x) <math.floor>` | <= *x* | ++--------------------+---------------------------------------------+ +| :func:`math.ceil(x)| the least :class:`~numbers.Integral` >= *x* | +| <math.ceil>` | | ++--------------------+---------------------------------------------+ For additional numeric operations see the :mod:`math` and :mod:`cmath` modules. @@ -397,8 +400,8 @@ Bitwise Operations on Integer Types operator: >> Bitwise operations only make sense for integers. Negative numbers are treated -as their 2's complement value (this assumes a sufficiently large number of bits -that no overflow occurs during the operation). +as their 2's complement value (this assumes that there are enough bits so that +no overflow occurs during the operation). The priorities of the binary bitwise operations are all lower than the numeric operations and higher than the comparisons; the unary operation ``~`` has the @@ -689,16 +692,16 @@ number, :class:`float`, or :class:`complex`:: m, n = m // P, n // P if n % P == 0: - hash_ = sys.hash_info.inf + hash_value = sys.hash_info.inf else: # Fermat's Little Theorem: pow(n, P-1, P) is 1, so # pow(n, P-2, P) gives the inverse of n modulo P. - hash_ = (abs(m) % P) * pow(n, P - 2, P) % P + hash_value = (abs(m) % P) * pow(n, P - 2, P) % P if m < 0: - hash_ = -hash_ - if hash_ == -1: - hash_ = -2 - return hash_ + hash_value = -hash_value + if hash_value == -1: + hash_value = -2 + return hash_value def hash_float(x): """Compute the hash of a float x.""" @@ -713,13 +716,13 @@ number, :class:`float`, or :class:`complex`:: def hash_complex(z): """Compute the hash of a complex number z.""" - hash_ = hash_float(z.real) + sys.hash_info.imag * hash_float(z.imag) + hash_value = hash_float(z.real) + sys.hash_info.imag * hash_float(z.imag) # do a signed reduction modulo 2**sys.hash_info.width M = 2**(sys.hash_info.width - 1) - hash_ = (hash_ & (M - 1)) - (hash & M) - if hash_ == -1: - hash_ == -2 - return hash_ + hash_value = (hash_value & (M - 1)) - (hash_value & M) + if hash_value == -1: + hash_value = -2 + return hash_value .. _typeiter: @@ -1298,16 +1301,16 @@ loops. only represent sequences that follow a strict pattern and repetition and concatenation will usually violate that pattern). - .. data: start + .. attribute:: start The value of the *start* parameter (or ``0`` if the parameter was not supplied) - .. data: stop + .. attribute:: stop The value of the *stop* parameter - .. data: step + .. attribute:: step The value of the *step* parameter (or ``1`` if the parameter was not supplied) @@ -1450,7 +1453,7 @@ multiple fragments. For more information on the ``str`` class and its methods, see :ref:`textseq` and the :ref:`string-methods` section below. To output - formatted strings, see the :ref:`string-formatting` section. In addition, + formatted strings, see the :ref:`formatstrings` section. In addition, see the :ref:`stringservices` section. @@ -1949,6 +1952,16 @@ expression support in the :mod:`re` module). >>> 'www.example.com'.strip('cmowz.') 'example' + The outermost leading and trailing *chars* argument values are stripped + from the string. Characters are removed from the leading end until + reaching a string character that is not contained in the set of + characters in *chars*. A similar action takes place on the trailing end. + For example:: + + >>> comment_string = '#....... Section 3.2.1 Issue #32 .......' + >>> comment_string.strip('.#! ') + 'Section 3.2.1 Issue #32' + .. method:: str.swapcase() @@ -2302,6 +2315,19 @@ the bytes type has an additional class method to read data in that format: >>> bytes.fromhex('2Ef0 F1f2 ') b'.\xf0\xf1\xf2' +A reverse conversion function exists to transform a bytes object into its +hexadecimal representation. + +.. method:: bytes.hex() + + Return a string object containing two hexadecimal digits for each + byte in the instance. + + >>> b'\xf0\xf1\xf2'.hex() + 'f0f1f2' + + .. versionadded:: 3.5 + Since bytes objects are sequences of integers (akin to a tuple), for a bytes object *b*, ``b[0]`` will be an integer, while ``b[0:1]`` will be a bytes object of length 1. (This contrasts with text strings, where both indexing @@ -2357,6 +2383,19 @@ the bytearray type has an additional class method to read data in that format: >>> bytearray.fromhex('2Ef0 F1f2 ') bytearray(b'.\xf0\xf1\xf2') +A reverse conversion function exists to transform a bytearray object into its +hexadecimal representation. + +.. method:: bytearray.hex() + + Return a string object containing two hexadecimal digits for each + byte in the instance. + + >>> bytearray(b'\xf0\xf1\xf2').hex() + 'f0f1f2' + + .. versionadded:: 3.5 + Since bytearray objects are sequences of integers (akin to a list), for a bytearray object *b*, ``b[0]`` will be an integer, while ``b[0:1]`` will be a bytearray object of length 1. (This contrasts with text strings, where @@ -3102,6 +3141,203 @@ place, and instead produce new objects. always produces a new object, even if no changes were made. +.. _bytes-formatting: + +``printf``-style Bytes Formatting +---------------------------------- + +.. index:: + single: formatting, bytes (%) + single: formatting, bytearray (%) + single: interpolation, bytes (%) + single: interpolation, bytearray (%) + single: bytes; formatting + single: bytearray; formatting + single: bytes; interpolation + single: bytearray; interpolation + single: printf-style formatting + single: sprintf-style formatting + single: % formatting + single: % interpolation + +.. note:: + + The formatting operations described here exhibit a variety of quirks that + lead to a number of common errors (such as failing to display tuples and + dictionaries correctly). If the value being printed may be a tuple or + dictionary, wrap it in a tuple. + +Bytes objects (``bytes``/``bytearray``) have one unique built-in operation: +the ``%`` operator (modulo). +This is also known as the bytes *formatting* or *interpolation* operator. +Given ``format % values`` (where *format* is a bytes object), ``%`` conversion +specifications in *format* are replaced with zero or more elements of *values*. +The effect is similar to using the :c:func:`sprintf` in the C language. + +If *format* requires a single argument, *values* may be a single non-tuple +object. [5]_ Otherwise, *values* must be a tuple with exactly the number of +items specified by the format bytes object, or a single mapping object (for +example, a dictionary). + +A conversion specifier contains two or more characters and has the following +components, which must occur in this order: + +#. The ``'%'`` character, which marks the start of the specifier. + +#. Mapping key (optional), consisting of a parenthesised sequence of characters + (for example, ``(somename)``). + +#. Conversion flags (optional), which affect the result of some conversion + types. + +#. Minimum field width (optional). If specified as an ``'*'`` (asterisk), the + actual width is read from the next element of the tuple in *values*, and the + object to convert comes after the minimum field width and optional precision. + +#. Precision (optional), given as a ``'.'`` (dot) followed by the precision. If + specified as ``'*'`` (an asterisk), the actual precision is read from the next + element of the tuple in *values*, and the value to convert comes after the + precision. + +#. Length modifier (optional). + +#. Conversion type. + +When the right argument is a dictionary (or other mapping type), then the +formats in the bytes object *must* include a parenthesised mapping key into that +dictionary inserted immediately after the ``'%'`` character. The mapping key +selects the value to be formatted from the mapping. For example: + + >>> print(b'%(language)s has %(number)03d quote types.' % + ... {b'language': b"Python", b"number": 2}) + b'Python has 002 quote types.' + +In this case no ``*`` specifiers may occur in a format (since they require a +sequential parameter list). + +The conversion flag characters are: + ++---------+---------------------------------------------------------------------+ +| Flag | Meaning | ++=========+=====================================================================+ +| ``'#'`` | The value conversion will use the "alternate form" (where defined | +| | below). | ++---------+---------------------------------------------------------------------+ +| ``'0'`` | The conversion will be zero padded for numeric values. | ++---------+---------------------------------------------------------------------+ +| ``'-'`` | The converted value is left adjusted (overrides the ``'0'`` | +| | conversion if both are given). | ++---------+---------------------------------------------------------------------+ +| ``' '`` | (a space) A blank should be left before a positive number (or empty | +| | string) produced by a signed conversion. | ++---------+---------------------------------------------------------------------+ +| ``'+'`` | A sign character (``'+'`` or ``'-'``) will precede the conversion | +| | (overrides a "space" flag). | ++---------+---------------------------------------------------------------------+ + +A length modifier (``h``, ``l``, or ``L``) may be present, but is ignored as it +is not necessary for Python -- so e.g. ``%ld`` is identical to ``%d``. + +The conversion types are: + ++------------+-----------------------------------------------------+-------+ +| Conversion | Meaning | Notes | ++============+=====================================================+=======+ +| ``'d'`` | Signed integer decimal. | | ++------------+-----------------------------------------------------+-------+ +| ``'i'`` | Signed integer decimal. | | ++------------+-----------------------------------------------------+-------+ +| ``'o'`` | Signed octal value. | \(1) | ++------------+-----------------------------------------------------+-------+ +| ``'u'`` | Obsolete type -- it is identical to ``'d'``. | \(8) | ++------------+-----------------------------------------------------+-------+ +| ``'x'`` | Signed hexadecimal (lowercase). | \(2) | ++------------+-----------------------------------------------------+-------+ +| ``'X'`` | Signed hexadecimal (uppercase). | \(2) | ++------------+-----------------------------------------------------+-------+ +| ``'e'`` | Floating point exponential format (lowercase). | \(3) | ++------------+-----------------------------------------------------+-------+ +| ``'E'`` | Floating point exponential format (uppercase). | \(3) | ++------------+-----------------------------------------------------+-------+ +| ``'f'`` | Floating point decimal format. | \(3) | ++------------+-----------------------------------------------------+-------+ +| ``'F'`` | Floating point decimal format. | \(3) | ++------------+-----------------------------------------------------+-------+ +| ``'g'`` | Floating point format. Uses lowercase exponential | \(4) | +| | format if exponent is less than -4 or not less than | | +| | precision, decimal format otherwise. | | ++------------+-----------------------------------------------------+-------+ +| ``'G'`` | Floating point format. Uses uppercase exponential | \(4) | +| | format if exponent is less than -4 or not less than | | +| | precision, decimal format otherwise. | | ++------------+-----------------------------------------------------+-------+ +| ``'c'`` | Single byte (accepts integer or single | | +| | byte objects). | | ++------------+-----------------------------------------------------+-------+ +| ``'b'`` | Bytes (any object that follows the | \(5) | +| | :ref:`buffer protocol <bufferobjects>` or has | | +| | :meth:`__bytes__`). | | ++------------+-----------------------------------------------------+-------+ +| ``'s'`` | ``'s'`` is an alias for ``'b'`` and should only | \(6) | +| | be used for Python2/3 code bases. | | ++------------+-----------------------------------------------------+-------+ +| ``'a'`` | Bytes (converts any Python object using | \(5) | +| | ``repr(obj).encode('ascii','backslashreplace)``). | | ++------------+-----------------------------------------------------+-------+ +| ``'r'`` | ``'r'`` is an alias for ``'a'`` and should only | \(7) | +| | be used for Python2/3 code bases. | | ++------------+-----------------------------------------------------+-------+ +| ``'%'`` | No argument is converted, results in a ``'%'`` | | +| | character in the result. | | ++------------+-----------------------------------------------------+-------+ + +Notes: + +(1) + The alternate form causes a leading zero (``'0'``) to be inserted between + left-hand padding and the formatting of the number if the leading character + of the result is not already a zero. + +(2) + The alternate form causes a leading ``'0x'`` or ``'0X'`` (depending on whether + the ``'x'`` or ``'X'`` format was used) to be inserted between left-hand padding + and the formatting of the number if the leading character of the result is not + already a zero. + +(3) + The alternate form causes the result to always contain a decimal point, even if + no digits follow it. + + The precision determines the number of digits after the decimal point and + defaults to 6. + +(4) + The alternate form causes the result to always contain a decimal point, and + trailing zeroes are not removed as they would otherwise be. + + The precision determines the number of significant digits before and after the + decimal point and defaults to 6. + +(5) + If precision is ``N``, the output is truncated to ``N`` characters. + +(6) + ``b'%s'`` is deprecated, but will not be removed during the 3.x series. + +(7) + ``b'%r'`` is deprecated, but will not be removed during the 3.x series. + +(8) + See :pep:`237`. + +.. note:: + + The bytearray version of this method does *not* operate in place - it + always produces a new object, even if no changes were made. + +.. seealso:: :pep:`461`. +.. versionadded:: 3.5 + .. _typememoryview: Memory Views @@ -3130,10 +3366,8 @@ copying. the view. The :class:`~memoryview.itemsize` attribute will give you the number of bytes in a single element. - A :class:`memoryview` supports slicing to expose its data. If - :class:`~memoryview.format` is one of the native format specifiers - from the :mod:`struct` module, indexing will return a single element - with the correct type. Full slicing will result in a subview:: + A :class:`memoryview` supports slicing and indexing to expose its data. + One-dimensional slicing will result in a subview:: >>> v = memoryview(b'abcefg') >>> v[1] @@ -3145,25 +3379,29 @@ copying. >>> bytes(v[1:4]) b'bce' - Other native formats:: + If :class:`~memoryview.format` is one of the native format specifiers + from the :mod:`struct` module, indexing with an integer or a tuple of + integers is also supported and returns a single *element* with + the correct type. One-dimensional memoryviews can be indexed + with an integer or a one-integer tuple. Multi-dimensional memoryviews + can be indexed with tuples of exactly *ndim* integers where *ndim* is + the number of dimensions. Zero-dimensional memoryviews can be indexed + with the empty tuple. + + Here is an example with a non-byte format:: >>> import array >>> a = array.array('l', [-11111111, 22222222, -33333333, 44444444]) - >>> a[0] + >>> m = memoryview(a) + >>> m[0] -11111111 - >>> a[-1] + >>> m[-1] 44444444 - >>> a[2:3].tolist() - [-33333333] - >>> a[::2].tolist() + >>> m[::2].tolist() [-11111111, -33333333] - >>> a[::-1].tolist() - [44444444, -33333333, 22222222, -11111111] - .. versionadded:: 3.3 - - If the underlying object is writable, the memoryview supports slice - assignment. Resizing is not allowed:: + If the underlying object is writable, the memoryview supports + one-dimensional slice assignment. Resizing is not allowed:: >>> data = bytearray(b'abcefg') >>> v = memoryview(data) @@ -3196,12 +3434,16 @@ copying. True .. versionchanged:: 3.3 + One-dimensional memoryviews can now be sliced. One-dimensional memoryviews with formats 'B', 'b' or 'c' are now hashable. .. versionchanged:: 3.4 memoryview is now registered automatically with :class:`collections.abc.Sequence` + .. versionchanged:: 3.5 + memoryviews can now be indexed with tuple of integers. + :class:`memoryview` has several methods: .. method:: __eq__(exporter) @@ -3268,6 +3510,17 @@ copying. supports all format strings, including those that are not in :mod:`struct` module syntax. + .. method:: hex() + + Return a string object containing two hexadecimal digits for each + byte in the buffer. :: + + >>> m = memoryview(b"abc") + >>> m.hex() + '616263' + + .. versionadded:: 3.5 + .. method:: tolist() Return the data in the buffer as a list of elements. :: @@ -3323,10 +3576,10 @@ copying. Cast a memoryview to a new format or shape. *shape* defaults to ``[byte_length//new_itemsize]``, which means that the result view will be one-dimensional. The return value is a new memoryview, but - the buffer itself is not copied. Supported casts are 1D -> C-contiguous + the buffer itself is not copied. Supported casts are 1D -> C-:term:`contiguous` and C-contiguous -> 1D. - Both formats are restricted to single element native formats in + The destination format is restricted to a single element native format in :mod:`struct` syntax. One of the formats must be a byte format ('B', 'b' or 'c'). The byte length of the result must be the same as the original length. @@ -3407,6 +3660,9 @@ copying. .. versionadded:: 3.3 + .. versionchanged:: 3.5 + The source format is no longer restricted when casting to a byte view. + There are also several readonly attributes available: .. attribute:: obj @@ -3511,19 +3767,19 @@ copying. .. attribute:: c_contiguous - A bool indicating whether the memory is C-contiguous. + A bool indicating whether the memory is C-:term:`contiguous`. .. versionadded:: 3.3 .. attribute:: f_contiguous - A bool indicating whether the memory is Fortran contiguous. + A bool indicating whether the memory is Fortran :term:`contiguous`. .. versionadded:: 3.3 .. attribute:: contiguous - A bool indicating whether the memory is contiguous. + A bool indicating whether the memory is :term:`contiguous`. .. versionadded:: 3.3 @@ -3575,7 +3831,7 @@ The constructors for both classes work the same: .. describe:: len(s) - Return the cardinality of set *s*. + Return the number of elements in set *s* (cardinality of *s*). .. describe:: x in s @@ -4104,9 +4360,10 @@ an (external) *definition* for a module named *foo* somewhere.) A special attribute of every module is :attr:`~object.__dict__`. This is the dictionary containing the module's symbol table. Modifying this dictionary will actually change the module's symbol table, but direct assignment to the -:attr:`__dict__` attribute is not possible (you can write +:attr:`~object.__dict__` attribute is not possible (you can write ``m.__dict__['a'] = 1``, which defines ``m.a`` to be ``1``, but you can't write -``m.__dict__ = {}``). Modifying :attr:`__dict__` directly is not recommended. +``m.__dict__ = {}``). Modifying :attr:`~object.__dict__` directly is +not recommended. Modules built into the interpreter are written like this: ``<module 'sys' (built-in)>``. If loaded from a file, they are written as ``<module 'os' from @@ -4180,13 +4437,13 @@ attribute, you need to explicitly set it on the underlying function object:: See :ref:`types` for more information. +.. index:: object; code, code object + .. _bltin-code-objects: Code Objects ------------ -.. index:: object: code - .. index:: builtin: compile single: __code__ (function object attribute) @@ -4319,14 +4576,16 @@ types, where they are relevant. Some of these are not reported by the The tuple of base classes of a class object. -.. attribute:: class.__name__ +.. attribute:: definition.__name__ - The name of the class or type. + The name of the class, function, method, descriptor, or + generator instance. -.. attribute:: class.__qualname__ +.. attribute:: definition.__qualname__ - The :term:`qualified name` of the class or type. + The :term:`qualified name` of the class, function, method, descriptor, + or generator instance. .. versionadded:: 3.3 diff --git a/Doc/library/string.rst b/Doc/library/string.rst index 19bdb21b3e..d5d24301b6 100644 --- a/Doc/library/string.rst +++ b/Doc/library/string.rst @@ -75,14 +75,14 @@ The constants defined in this module are: .. _string-formatting: -String Formatting ------------------ +Custom String Formatting +------------------------ The built-in string class provides the ability to do complex variable -substitutions and value formatting via the :func:`format` method described in +substitutions and value formatting via the :meth:`~str.format` method described in :pep:`3101`. The :class:`Formatter` class in the :mod:`string` module allows you to create and customize your own string formatting behaviors using the same -implementation as the built-in :meth:`format` method. +implementation as the built-in :meth:`~str.format` method. .. class:: Formatter @@ -91,9 +91,13 @@ implementation as the built-in :meth:`format` method. .. method:: format(format_string, *args, **kwargs) - :meth:`format` is the primary API method. It takes a format string and + The primary API method. It takes a format string and an arbitrary set of positional and keyword arguments. - :meth:`format` is just a wrapper that calls :meth:`vformat`. + It is just a wrapper that calls :meth:`vformat`. + + .. deprecated:: 3.5 + Passing a format string as keyword argument *format_string* has been + deprecated. .. method:: vformat(format_string, args, kwargs) @@ -230,12 +234,12 @@ does an index lookup using :func:`__getitem__`. Some simple format string examples:: - "First, thou shalt count to {0}" # References first positional argument - "Bring me a {}" # Implicitly references the first positional argument - "From {} to {}" # Same as "From {0} to {1}" - "My quest is {name}" # References keyword argument 'name' - "Weight in tons {0.weight}" # 'weight' attribute of first positional arg - "Units destroyed: {players[0]}" # First element of keyword argument 'players'. + "First, thou shalt count to {0}" # References first positional argument + "Bring me a {}" # Implicitly references the first positional argument + "From {} to {}" # Same as "From {0} to {1}" + "My quest is {name}" # References keyword argument 'name' + "Weight in tons {0.weight}" # 'weight' attribute of first positional arg + "Units destroyed: {players[0]}" # First element of keyword argument 'players'. The *conversion* field causes a type coercion before formatting. Normally, the job of formatting a value is done by the :meth:`__format__` method of the value @@ -263,8 +267,9 @@ Most built-in types support a common formatting mini-language, which is described in the next section. A *format_spec* field can also include nested replacement fields within it. -These nested replacement fields can contain only a field name; conversion flags -and format specifications are not allowed. The replacement fields within the +These nested replacement fields may contain a field name, conversion flag +and format specification, but deeper nesting is +not allowed. The replacement fields within the format_spec are substituted before the *format_spec* string is interpreted. This allows the formatting of a value to be dynamically specified. @@ -302,8 +307,10 @@ The general form of a *standard format specifier* is: If a valid *align* value is specified, it can be preceded by a *fill* character that can be any character and defaults to a space if omitted. -Note that it is not possible to use ``{`` and ``}`` as *fill* char while -using the :meth:`str.format` method; this limitation however doesn't +It is not possible to use a literal curly brace ("``{``" or "``}``") as +the *fill* character when using the :meth:`str.format` +method. However, it is possible to insert a curly brace +with a nested replacement field. This limitation doesn't affect the :func:`format` function. The meaning of the various alignment options is as follows: @@ -320,7 +327,8 @@ The meaning of the various alignment options is as follows: | ``'='`` | Forces the padding to be placed after the sign (if any) | | | but before the digits. This is used for printing fields | | | in the form '+000000120'. This alignment option is only | - | | valid for numeric types. | + | | valid for numeric types. It becomes the default when '0'| + | | immediately precedes the field width. | +---------+----------------------------------------------------------+ | ``'^'`` | Forces the field to be centered within the available | | | space. | @@ -369,7 +377,8 @@ instead. *width* is a decimal integer defining the minimum field width. If not specified, then the field width will be determined by the content. -Preceding the *width* field by a zero (``'0'``) character enables +When no explicit alignment is given, preceding the *width* field by a zero +(``'0'``) character enables sign-aware zero-padding for numeric types. This is equivalent to a *fill* character of ``'0'`` with an *alignment* type of ``'='``. @@ -492,8 +501,8 @@ The available presentation types for floating point and decimal values are: Format examples ^^^^^^^^^^^^^^^ -This section contains examples of the new format syntax and comparison with -the old ``%``-formatting. +This section contains examples of the :meth:`str.format` syntax and +comparison with the old ``%``-formatting. In most of the cases the syntax is similar to the old ``%``-formatting, with the addition of the ``{}`` and with ``:`` used instead of ``%``. diff --git a/Doc/library/stringprep.rst b/Doc/library/stringprep.rst index fc890cb232..e7fae5631d 100644 --- a/Doc/library/stringprep.rst +++ b/Doc/library/stringprep.rst @@ -3,9 +3,13 @@ .. module:: stringprep :synopsis: String preparation, as per RFC 3453 + .. moduleauthor:: Martin v. Löwis <martin@v.loewis.de> .. sectionauthor:: Martin v. Löwis <martin@v.loewis.de> +**Source code:** :source:`Lib/stringprep.py` + +-------------- When identifying things (such as host names) in the internet, it is often necessary to compare such identifications for "equality". Exactly how this diff --git a/Doc/library/struct.rst b/Doc/library/struct.rst index 12d4fbcd52..ae2e38fdc0 100644 --- a/Doc/library/struct.rst +++ b/Doc/library/struct.rst @@ -4,10 +4,14 @@ .. module:: struct :synopsis: Interpret bytes as packed binary data. +**Source code:** :source:`Lib/struct.py` + .. index:: pair: C; structures triple: packing; binary; data +-------------- + This module performs conversions between Python values and C structs represented as Python :class:`bytes` objects. This can be used in handling binary data stored in files or from network connections, among other sources. It uses @@ -62,16 +66,16 @@ The module defines the following exception and functions: Unpack from the buffer *buffer* (presumably packed by ``pack(fmt, ...)``) according to the format string *fmt*. The result is a tuple even if it - contains exactly one item. The buffer must contain exactly the amount of - data required by the format (``len(bytes)`` must equal ``calcsize(fmt)``). + contains exactly one item. The buffer's size in bytes must match the + size required by the format, as reflected by :func:`calcsize`. .. function:: unpack_from(fmt, buffer, offset=0) Unpack from *buffer* starting at position *offset*, according to the format string *fmt*. The result is a tuple even if it contains exactly one - item. *buffer* must contain at least the amount of data required by the - format (``len(buffer[offset:])`` must be at least ``calcsize(fmt)``). + item. The buffer's size in bytes, minus *offset*, must be at least + the size required by the format, as reflected by :func:`calcsize`. .. function:: iter_unpack(fmt, buffer) @@ -79,8 +83,8 @@ The module defines the following exception and functions: Iteratively unpack from the buffer *buffer* according to the format string *fmt*. This function returns an iterator which will read equally-sized chunks from the buffer until all its contents have been - consumed. The buffer's size in bytes must be a multiple of the amount - of data required by the format, as reflected by :func:`calcsize`. + consumed. The buffer's size in bytes must be a multiple of the size + required by the format, as reflected by :func:`calcsize`. Each iteration yields a tuple as specified by the format string. @@ -389,7 +393,7 @@ The :mod:`struct` module also defines the following type: .. method:: pack(v1, v2, ...) Identical to the :func:`pack` function, using the compiled format. - (``len(result)`` will equal :attr:`self.size`.) + (``len(result)`` will equal :attr:`size`.) .. method:: pack_into(buffer, offset, v1, v2, ...) @@ -400,19 +404,20 @@ The :mod:`struct` module also defines the following type: .. method:: unpack(buffer) Identical to the :func:`unpack` function, using the compiled format. - (``len(buffer)`` must equal :attr:`self.size`). + The buffer's size in bytes must equal :attr:`size`. .. method:: unpack_from(buffer, offset=0) Identical to the :func:`unpack_from` function, using the compiled format. - (``len(buffer[offset:])`` must be at least :attr:`self.size`). + The buffer's size in bytes, minus *offset*, must be at least + :attr:`size`. .. method:: iter_unpack(buffer) Identical to the :func:`iter_unpack` function, using the compiled format. - (``len(buffer)`` must be a multiple of :attr:`self.size`). + The buffer's size in bytes must be a multiple of :attr:`size`. .. versionadded:: 3.4 diff --git a/Doc/library/subprocess.rst b/Doc/library/subprocess.rst index 9ac8882d32..356605f76c 100644 --- a/Doc/library/subprocess.rst +++ b/Doc/library/subprocess.rst @@ -3,9 +3,13 @@ .. module:: subprocess :synopsis: Subprocess management. + .. moduleauthor:: Peter Åstrand <astrand@lysator.liu.se> .. sectionauthor:: Peter Åstrand <astrand@lysator.liu.se> +**Source code:** :source:`Lib/subprocess.py` + +-------------- The :mod:`subprocess` module allows you to spawn new processes, connect to their input/output/error pipes, and obtain their return codes. This module intends to @@ -25,160 +29,99 @@ modules and functions can be found in the following sections. Using the :mod:`subprocess` Module ---------------------------------- -The recommended approach to invoking subprocesses is to use the following -convenience functions for all use cases they can handle. For more advanced -use cases, the underlying :class:`Popen` interface can be used directly. +The recommended approach to invoking subprocesses is to use the :func:`run` +function for all use cases it can handle. For more advanced use cases, the +underlying :class:`Popen` interface can be used directly. +The :func:`run` function was added in Python 3.5; if you need to retain +compatibility with older versions, see the :ref:`call-function-trio` section. -.. function:: call(args, *, stdin=None, stdout=None, stderr=None, shell=False, timeout=None) + +.. function:: run(args, *, stdin=None, input=None, stdout=None, stderr=None,\ + shell=False, timeout=None, check=False) Run the command described by *args*. Wait for command to complete, then - return the :attr:`returncode` attribute. + return a :class:`CompletedProcess` instance. The arguments shown above are merely the most common ones, described below in :ref:`frequently-used-arguments` (hence the use of keyword-only notation in the abbreviated signature). The full function signature is largely the - same as that of the :class:`Popen` constructor - this function passes all - supplied arguments other than *timeout* directly through to that interface. + same as that of the :class:`Popen` constructor - apart from *timeout*, + *input* and *check*, all the arguments to this function are passed through to + that interface. + + This does not capture stdout or stderr by default. To do so, pass + :data:`PIPE` for the *stdout* and/or *stderr* arguments. - The *timeout* argument is passed to :meth:`Popen.wait`. If the timeout - expires, the child process will be killed and then waited for again. The + The *timeout* argument is passed to :meth:`Popen.communicate`. If the timeout + expires, the child process will be killed and waited for. The :exc:`TimeoutExpired` exception will be re-raised after the child process has terminated. - Examples:: - - >>> subprocess.call(["ls", "-l"]) - 0 - - >>> subprocess.call("exit 1", shell=True) - 1 - - .. note:: - - Do not use ``stdout=PIPE`` or ``stderr=PIPE`` with this - function. The child process will block if it generates enough - output to a pipe to fill up the OS pipe buffer as the pipes are - not being read from. - - .. versionchanged:: 3.3 - *timeout* was added. - - -.. function:: check_call(args, *, stdin=None, stdout=None, stderr=None, shell=False, timeout=None) - - Run command with arguments. Wait for command to complete. If the return - code was zero then return, otherwise raise :exc:`CalledProcessError`. The - :exc:`CalledProcessError` object will have the return code in the - :attr:`~CalledProcessError.returncode` attribute. - - The arguments shown above are merely the most common ones, described below - in :ref:`frequently-used-arguments` (hence the use of keyword-only notation - in the abbreviated signature). The full function signature is largely the - same as that of the :class:`Popen` constructor - this function passes all - supplied arguments other than *timeout* directly through to that interface. + The *input* argument is passed to :meth:`Popen.communicate` and thus to the + subprocess's stdin. If used it must be a byte sequence, or a string if + ``universal_newlines=True``. When used, the internal :class:`Popen` object + is automatically created with ``stdin=PIPE``, and the *stdin* argument may + not be used as well. - The *timeout* argument is passed to :meth:`Popen.wait`. If the timeout - expires, the child process will be killed and then waited for again. The - :exc:`TimeoutExpired` exception will be re-raised after the child process - has terminated. + If *check* is True, and the process exits with a non-zero exit code, a + :exc:`CalledProcessError` exception will be raised. Attributes of that + exception hold the arguments, the exit code, and stdout and stderr if they + were captured. Examples:: - >>> subprocess.check_call(["ls", "-l"]) - 0 + >>> subprocess.run(["ls", "-l"]) # doesn't capture output + CompletedProcess(args=['ls', '-l'], returncode=0) - >>> subprocess.check_call("exit 1", shell=True) + >>> subprocess.run("exit 1", shell=True, check=True) Traceback (most recent call last): - ... + ... subprocess.CalledProcessError: Command 'exit 1' returned non-zero exit status 1 - .. note:: + >>> subprocess.run(["ls", "-l", "/dev/null"], stdout=subprocess.PIPE) + CompletedProcess(args=['ls', '-l', '/dev/null'], returncode=0, + stdout=b'crw-rw-rw- 1 root root 1, 3 Jan 23 16:23 /dev/null\n') - Do not use ``stdout=PIPE`` or ``stderr=PIPE`` with this - function. The child process will block if it generates enough - output to a pipe to fill up the OS pipe buffer as the pipes are - not being read from. + .. versionadded:: 3.5 - .. versionchanged:: 3.3 - *timeout* was added. +.. class:: CompletedProcess + The return value from :func:`run`, representing a process that has finished. -.. function:: check_output(args, *, input=None, stdin=None, stderr=None, shell=False, universal_newlines=False, timeout=None) + .. attribute:: args - Run command with arguments and return its output. + The arguments used to launch the process. This may be a list or a string. - If the return code was non-zero it raises a :exc:`CalledProcessError`. The - :exc:`CalledProcessError` object will have the return code in the - :attr:`~CalledProcessError.returncode` attribute and any output in the - :attr:`~CalledProcessError.output` attribute. - - The arguments shown above are merely the most common ones, described below - in :ref:`frequently-used-arguments` (hence the use of keyword-only notation - in the abbreviated signature). The full function signature is largely the - same as that of the :class:`Popen` constructor - this functions passes all - supplied arguments other than *input* and *timeout* directly through to - that interface. In addition, *stdout* is not permitted as an argument, as - it is used internally to collect the output from the subprocess. - - The *timeout* argument is passed to :meth:`Popen.wait`. If the timeout - expires, the child process will be killed and then waited for again. The - :exc:`TimeoutExpired` exception will be re-raised after the child process - has terminated. - - The *input* argument is passed to :meth:`Popen.communicate` and thus to the - subprocess's stdin. If used it must be a byte sequence, or a string if - ``universal_newlines=True``. When used, the internal :class:`Popen` object - is automatically created with ``stdin=PIPE``, and the *stdin* argument may - not be used as well. - - Examples:: - - >>> subprocess.check_output(["echo", "Hello World!"]) - b'Hello World!\n' - - >>> subprocess.check_output(["echo", "Hello World!"], universal_newlines=True) - 'Hello World!\n' + .. attribute:: returncode - >>> subprocess.check_output(["sed", "-e", "s/foo/bar/"], - ... input=b"when in the course of fooman events\n") - b'when in the course of barman events\n' - - >>> subprocess.check_output("exit 1", shell=True) - Traceback (most recent call last): - ... - subprocess.CalledProcessError: Command 'exit 1' returned non-zero exit status 1 + Exit status of the child process. Typically, an exit status of 0 indicates + that it ran successfully. - By default, this function will return the data as encoded bytes. The actual - encoding of the output data may depend on the command being invoked, so the - decoding to text will often need to be handled at the application level. + A negative value ``-N`` indicates that the child was terminated by signal + ``N`` (POSIX only). - This behaviour may be overridden by setting *universal_newlines* to - ``True`` as described below in :ref:`frequently-used-arguments`. + .. attribute:: stdout - To also capture standard error in the result, use - ``stderr=subprocess.STDOUT``:: + Captured stdout from the child process. A bytes sequence, or a string if + :func:`run` was called with ``universal_newlines=True``. None if stdout + was not captured. - >>> subprocess.check_output( - ... "ls non_existent_file; exit 0", - ... stderr=subprocess.STDOUT, - ... shell=True) - 'ls: non_existent_file: No such file or directory\n' + If you ran the process with ``stderr=subprocess.STDOUT``, stdout and + stderr will be combined in this attribute, and :attr:`stderr` will be + None. - .. note:: + .. attribute:: stderr - Do not use ``stdout=PIPE`` or ``stderr=PIPE`` with this - function. The child process will block if it generates enough - output to a pipe to fill up the OS pipe buffer as the pipes are - not being read from. + Captured stderr from the child process. A bytes sequence, or a string if + :func:`run` was called with ``universal_newlines=True``. None if stderr + was not captured. - .. versionadded:: 3.1 + .. method:: check_returncode() - .. versionchanged:: 3.3 - *timeout* was added. + If :attr:`returncode` is non-zero, raise a :exc:`CalledProcessError`. - .. versionchanged:: 3.4 - *input* was added. + .. versionadded:: 3.5 .. data:: DEVNULL @@ -225,11 +168,22 @@ use cases, the underlying :class:`Popen` interface can be used directly. .. attribute:: output - Output of the child process if this exception is raised by + Output of the child process if it was captured by :func:`run` or :func:`check_output`. Otherwise, ``None``. + .. attribute:: stdout + + Alias for output, for symmetry with :attr:`stderr`. + + .. attribute:: stderr + + Stderr output of the child process if it was captured by :func:`run`. + Otherwise, ``None``. + .. versionadded:: 3.3 + .. versionchanged:: 3.5 + *stdout* and *stderr* attributes added .. exception:: CalledProcessError @@ -238,7 +192,8 @@ use cases, the underlying :class:`Popen` interface can be used directly. .. attribute:: returncode - Exit status of the child process. + Exit status of the child process. If the process exited due to a + signal, this will be the negative signal number. .. attribute:: cmd @@ -246,9 +201,20 @@ use cases, the underlying :class:`Popen` interface can be used directly. .. attribute:: output - Output of the child process if this exception is raised by + Output of the child process if it was captured by :func:`run` or :func:`check_output`. Otherwise, ``None``. + .. attribute:: stdout + + Alias for output, for symmetry with :attr:`stderr`. + + .. attribute:: stderr + + Stderr output of the child process if it was captured by :func:`run`. + Otherwise, ``None``. + + .. versionchanged:: 3.5 + *stdout* and *stderr* attributes added .. _frequently-used-arguments: @@ -514,7 +480,7 @@ functions. execute. On Windows, in order to run a `side-by-side assembly`_ the specified *env* **must** include a valid :envvar:`SystemRoot`. - .. _side-by-side assembly: http://en.wikipedia.org/wiki/Side-by-Side_Assembly + .. _side-by-side assembly: https://en.wikipedia.org/wiki/Side-by-Side_Assembly If *universal_newlines* is ``True``, the file objects *stdin*, *stdout* and *stderr* are opened as text streams in universal newlines mode, as @@ -575,7 +541,7 @@ including shell metacharacters, can safely be passed to child processes. If the shell is invoked explicitly, via ``shell=True``, it is the application's responsibility to ensure that all whitespace and metacharacters are quoted appropriately to avoid -`shell injection <http://en.wikipedia.org/wiki/Shell_injection#Shell_injection>`_ +`shell injection <https://en.wikipedia.org/wiki/Shell_injection#Shell_injection>`_ vulnerabilities. When using ``shell=True``, the :func:`shlex.quote` function can be @@ -635,6 +601,7 @@ Instances of the :class:`Popen` class have the following methods: must be bytes or, if *universal_newlines* was ``True``, a string. :meth:`communicate` returns a tuple ``(stdout_data, stderr_data)``. + The data will be bytes or, if *universal_newlines* was ``True``, strings. Note that if you want to send data to the process's stdin, you need to create the Popen object with ``stdin=PIPE``. Similarly, to get anything other than @@ -759,7 +726,7 @@ on Windows. .. class:: STARTUPINFO() Partial support of the Windows - `STARTUPINFO <http://msdn.microsoft.com/en-us/library/ms686331(v=vs.85).aspx>`__ + `STARTUPINFO <https://msdn.microsoft.com/en-us/library/ms686331(v=vs.85).aspx>`__ structure is used for :class:`Popen` creation. .. attribute:: dwFlags @@ -795,7 +762,7 @@ on Windows. If :attr:`dwFlags` specifies :data:`STARTF_USESHOWWINDOW`, this attribute can be any of the values that can be specified in the ``nCmdShow`` parameter for the - `ShowWindow <http://msdn.microsoft.com/en-us/library/ms633548(v=vs.85).aspx>`__ + `ShowWindow <https://msdn.microsoft.com/en-us/library/ms633548(v=vs.85).aspx>`__ function, except for ``SW_SHOWDEFAULT``. Otherwise, this attribute is ignored. @@ -851,6 +818,112 @@ The :mod:`subprocess` module exposes the following constants. This flag is ignored if :data:`CREATE_NEW_CONSOLE` is specified. +.. _call-function-trio: + +Older high-level API +-------------------- + +Prior to Python 3.5, these three functions comprised the high level API to +subprocess. You can now use :func:`run` in many cases, but lots of existing code +calls these functions. + +.. function:: call(args, *, stdin=None, stdout=None, stderr=None, shell=False, timeout=None) + + Run the command described by *args*. Wait for command to complete, then + return the :attr:`~Popen.returncode` attribute. + + This is equivalent to:: + + run(...).returncode + + (except that the *input* and *check* parameters are not supported) + + The arguments shown above are merely the most + common ones. The full function signature is largely the + same as that of the :class:`Popen` constructor - this function passes all + supplied arguments other than *timeout* directly through to that interface. + + .. note:: + + Do not use ``stdout=PIPE`` or ``stderr=PIPE`` with this + function. The child process will block if it generates enough + output to a pipe to fill up the OS pipe buffer as the pipes are + not being read from. + + .. versionchanged:: 3.3 + *timeout* was added. + +.. function:: check_call(args, *, stdin=None, stdout=None, stderr=None, shell=False, timeout=None) + + Run command with arguments. Wait for command to complete. If the return + code was zero then return, otherwise raise :exc:`CalledProcessError`. The + :exc:`CalledProcessError` object will have the return code in the + :attr:`~CalledProcessError.returncode` attribute. + + This is equivalent to:: + + run(..., check=True) + + (except that the *input* parameter is not supported) + + The arguments shown above are merely the most + common ones. The full function signature is largely the + same as that of the :class:`Popen` constructor - this function passes all + supplied arguments other than *timeout* directly through to that interface. + + .. note:: + + Do not use ``stdout=PIPE`` or ``stderr=PIPE`` with this + function. The child process will block if it generates enough + output to a pipe to fill up the OS pipe buffer as the pipes are + not being read from. + + .. versionchanged:: 3.3 + *timeout* was added. + + +.. function:: check_output(args, *, stdin=None, stderr=None, shell=False, universal_newlines=False, timeout=None) + + Run command with arguments and return its output. + + If the return code was non-zero it raises a :exc:`CalledProcessError`. The + :exc:`CalledProcessError` object will have the return code in the + :attr:`~CalledProcessError.returncode` attribute and any output in the + :attr:`~CalledProcessError.output` attribute. + + This is equivalent to:: + + run(..., check=True, stdout=PIPE).stdout + + The arguments shown above are merely the most common ones. + The full function signature is largely the same as that of :func:`run` - + most arguments are passed directly through to that interface. + However, explicitly passing ``input=None`` to inherit the parent's + standard input file handle is not supported. + + By default, this function will return the data as encoded bytes. The actual + encoding of the output data may depend on the command being invoked, so the + decoding to text will often need to be handled at the application level. + + This behaviour may be overridden by setting *universal_newlines* to + ``True`` as described above in :ref:`frequently-used-arguments`. + + To also capture standard error in the result, use + ``stderr=subprocess.STDOUT``:: + + >>> subprocess.check_output( + ... "ls non_existent_file; exit 0", + ... stderr=subprocess.STDOUT, + ... shell=True) + 'ls: non_existent_file: No such file or directory\n' + + .. versionadded:: 3.1 + + .. versionchanged:: 3.3 + *timeout* was added. + + .. versionchanged:: 3.4 + Support for the *input* keyword argument was added. .. _subprocess-replacements: @@ -877,20 +950,23 @@ been imported from the :mod:`subprocess` module. Replacing /bin/sh shell backquote ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -:: +.. code-block:: bash output=`mycmd myarg` - # becomes - output = check_output(["mycmd", "myarg"]) +becomes:: + + output = check_output(["mycmd", "myarg"]) Replacing shell pipeline ^^^^^^^^^^^^^^^^^^^^^^^^ -:: +.. code-block:: bash output=`dmesg | grep hda` - # becomes + +becomes:: + p1 = Popen(["dmesg"], stdout=PIPE) p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE) p1.stdout.close() # Allow p1 to receive a SIGPIPE if p2 exits. @@ -900,10 +976,14 @@ The p1.stdout.close() call after starting the p2 is important in order for p1 to receive a SIGPIPE if p2 exits before p1. Alternatively, for trusted input, the shell's own pipeline support may still -be used directly:: +be used directly: + +.. code-block:: bash output=`dmesg | grep hda` - # becomes + +becomes:: + output=check_output("dmesg | grep hda", shell=True) diff --git a/Doc/library/sunau.rst b/Doc/library/sunau.rst index bd37ee29cc..d0fd0a3cc7 100644 --- a/Doc/library/sunau.rst +++ b/Doc/library/sunau.rst @@ -3,6 +3,7 @@ .. module:: sunau :synopsis: Provide an interface to the Sun AU sound format. + .. sectionauthor:: Moshe Zadka <moshez@zadka.site.co.il> **Source code:** :source:`Lib/sunau.py` diff --git a/Doc/library/symbol.rst b/Doc/library/symbol.rst index ef9ef1e129..44996936e2 100644 --- a/Doc/library/symbol.rst +++ b/Doc/library/symbol.rst @@ -3,6 +3,7 @@ .. module:: symbol :synopsis: Constants representing internal nodes of the parse tree. + .. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org> **Source code:** :source:`Lib/symbol.py` diff --git a/Doc/library/symtable.rst b/Doc/library/symtable.rst index 2503d335e8..ba2caff589 100644 --- a/Doc/library/symtable.rst +++ b/Doc/library/symtable.rst @@ -71,10 +71,6 @@ Examining Symbol Tables Return ``True`` if the block uses ``exec``. - .. method:: has_import_star() - - Return ``True`` if the block uses a starred from-import. - .. method:: get_identifiers() Return a list of names of symbols in this table. diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst index c0378e09b0..ed5db05516 100644 --- a/Doc/library/sys.rst +++ b/Doc/library/sys.rst @@ -4,6 +4,7 @@ .. module:: sys :synopsis: Access system-specific parameters and functions. +-------------- This module provides access to some variables used or maintained by the interpreter and to functions that interact strongly with the interpreter. It is @@ -167,7 +168,7 @@ always available. .. data:: dont_write_bytecode - If this is true, Python won't try to write ``.pyc`` or ``.pyo`` files on the + If this is true, Python won't try to write ``.pyc`` files on the import of source modules. This value is initially set to ``True`` or ``False`` depending on the :option:`-B` command line option and the :envvar:`PYTHONDONTWRITEBYTECODE` environment variable, but you can set it @@ -474,7 +475,7 @@ always available. additional garbage collector overhead if the object is managed by the garbage collector. - See `recursive sizeof recipe <http://code.activestate.com/recipes/577504>`_ + See `recursive sizeof recipe <https://code.activestate.com/recipes/577504>`_ for an example of using :func:`getsizeof` recursively to find the size of containers and all their contents. @@ -576,6 +577,18 @@ always available. *service_pack_major*, *suite_mask*, and *product_type*. +.. function:: get_coroutine_wrapper() + + Returns ``None``, or a wrapper set by :func:`set_coroutine_wrapper`. + + .. versionadded:: 3.5 + See :pep:`492` for more details. + + .. note:: + This function has been added on a provisional basis (see :pep:`411` + for details.) Use it only for debugging purposes. + + .. data:: hash_info A :term:`struct sequence` giving parameters of the numeric hash @@ -718,6 +731,14 @@ always available. value of :func:`intern` around to benefit from it. +.. function:: is_finalizing() + + Return :const:`True` if the Python interpreter is + :term:`shutting down <interpreter shutdown>`, :const:`False` otherwise. + + .. versionadded:: 3.5 + + .. data:: last_type last_value last_traceback @@ -754,19 +775,32 @@ always available. .. data:: meta_path - A list of :term:`finder` objects that have their :meth:`find_module` - methods called to see if one of the objects can find the module to be - imported. The :meth:`find_module` method is called at least with the - absolute name of the module being imported. If the module to be imported is - contained in package then the parent package's :attr:`__path__` attribute - is passed in as a second argument. The method returns ``None`` if - the module cannot be found, else returns a :term:`loader`. - - :data:`sys.meta_path` is searched before any implicit default finders or - :data:`sys.path`. - - See :pep:`302` for the original specification. - + A list of :term:`meta path finder` objects that have their + :meth:`~importlib.abc.MetaPathFinder.find_spec` methods called to see if one + of the objects can find the module to be imported. The + :meth:`~importlib.abc.MetaPathFinder.find_spec` method is called with at + least the absolute name of the module being imported. If the module to be + imported is contained in a package, then the parent package's :attr:`__path__` + attribute is passed in as a second argument. The method returns a + :term:`module spec`, or ``None`` if the module cannot be found. + + .. seealso:: + + :class:`importlib.abc.MetaPathFinder` + The abstract base class defining the interface of finder objects on + :data:`meta_path`. + :class:`importlib.machinery.ModuleSpec` + The concrete class which + :meth:`~importlib.abc.MetaPathFinder.find_spec` should return + instances of. + + .. versionchanged:: 3.4 + + :term:`Module specs <module spec>` were introduced in Python 3.4, by + :pep:`451`. Earlier versions of Python looked for a method called + :meth:`~importlib.abc.MetaPathFinder.find_module`. + This is still called as a fallback if a :data:`meta_path` entry doesn't + have a :meth:`~importlib.abc.MetaPathFinder.find_spec` method. .. data:: modules @@ -955,6 +989,13 @@ always available. that supports a higher limit. This should be done with care, because a too-high limit can lead to a crash. + If the new limit is too low at the current recursion depth, a + :exc:`RecursionError` exception is raised. + + .. versionchanged:: 3.5.1 + A :exc:`RecursionError` exception is now raised if the new limit is too + low at the current recursion depth. + .. function:: setswitchinterval(interval) @@ -1053,6 +1094,46 @@ always available. thus not likely to be implemented elsewhere. +.. function:: set_coroutine_wrapper(wrapper) + + Allows intercepting creation of :term:`coroutine` objects (only ones that + are created by an :keyword:`async def` function; generators decorated with + :func:`types.coroutine` or :func:`asyncio.coroutine` will not be + intercepted). + + The *wrapper* argument must be either: + + * a callable that accepts one argument (a coroutine object); + * ``None``, to reset the wrapper. + + If called twice, the new wrapper replaces the previous one. The function + is thread-specific. + + The *wrapper* callable cannot define new coroutines directly or indirectly:: + + def wrapper(coro): + async def wrap(coro): + return await coro + return wrap(coro) + sys.set_coroutine_wrapper(wrapper) + + async def foo(): + pass + + # The following line will fail with a RuntimeError, because + # ``wrapper`` creates a ``wrap(coro)`` coroutine: + foo() + + See also :func:`get_coroutine_wrapper`. + + .. versionadded:: 3.5 + See :pep:`492` for more details. + + .. note:: + This function has been added on a provisional basis (see :pep:`411` + for details.) Use it only for debugging purposes. + + .. data:: stdin stdout stderr @@ -1201,7 +1282,9 @@ always available. A dictionary of the various implementation-specific flags passed through the :option:`-X` command-line option. Option names are either mapped to - their values, if given explicitly, or to :const:`True`. Example:: + their values, if given explicitly, or to :const:`True`. Example: + + .. code-block:: shell-session $ ./python -Xa=b -Xc Python 3.2a3+ (py3k, Oct 16 2010, 20:14:50) @@ -1223,4 +1306,3 @@ always available. .. rubric:: Citations .. [C99] ISO/IEC 9899:1999. "Programming languages -- C." A public draft of this standard is available at http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1256.pdf\ . - diff --git a/Doc/library/sysconfig.rst b/Doc/library/sysconfig.rst index 535ac54b3c..c51567a81a 100644 --- a/Doc/library/sysconfig.rst +++ b/Doc/library/sysconfig.rst @@ -3,16 +3,17 @@ .. module:: sysconfig :synopsis: Python's configuration information + .. moduleauthor:: Tarek Ziadé <tarek@ziade.org> .. sectionauthor:: Tarek Ziadé <tarek@ziade.org> -.. index:: - single: configuration information - .. versionadded:: 3.2 **Source code:** :source:`Lib/sysconfig.py` +.. index:: + single: configuration information + -------------- The :mod:`sysconfig` module provides access to Python's configuration @@ -228,7 +229,9 @@ Other functions Using :mod:`sysconfig` as a script ---------------------------------- -You can use :mod:`sysconfig` as a script with Python's *-m* option:: +You can use :mod:`sysconfig` as a script with Python's *-m* option: + +.. code-block:: shell-session $ python -m sysconfig Platform: "macosx-10.4-i386" diff --git a/Doc/library/syslog.rst b/Doc/library/syslog.rst index 6e90dc05d9..af3fb9b57f 100644 --- a/Doc/library/syslog.rst +++ b/Doc/library/syslog.rst @@ -5,6 +5,7 @@ :platform: Unix :synopsis: An interface to the Unix syslog library routines. +-------------- This module provides an interface to the Unix ``syslog`` library routines. Refer to the Unix manual pages for a detailed description of the ``syslog`` diff --git a/Doc/library/tabnanny.rst b/Doc/library/tabnanny.rst index 4f3e705cab..1edb0fbabb 100644 --- a/Doc/library/tabnanny.rst +++ b/Doc/library/tabnanny.rst @@ -4,6 +4,7 @@ .. module:: tabnanny :synopsis: Tool for detecting white space related problems in Python source files in a directory tree. + .. moduleauthor:: Tim Peters <tim_one@users.sourceforge.net> .. sectionauthor:: Peter Funk <pf@artcom-gmbh.de> diff --git a/Doc/library/tarfile.rst b/Doc/library/tarfile.rst index 05f29adb0b..5b95ef3107 100644 --- a/Doc/library/tarfile.rst +++ b/Doc/library/tarfile.rst @@ -4,7 +4,6 @@ .. module:: tarfile :synopsis: Read and write tar-format archive files. - .. moduleauthor:: Lars Gustäbel <lars@gustaebel.de> .. sectionauthor:: Lars Gustäbel <lars@gustaebel.de> @@ -62,6 +61,23 @@ Some facts and figures: +------------------+---------------------------------------------+ | ``'r:xz'`` | Open for reading with lzma compression. | +------------------+---------------------------------------------+ + | ``'x'`` or | Create a tarfile exclusively without | + | ``'x:'`` | compression. | + | | Raise an :exc:`FileExistsError` exception | + | | if it already exists. | + +------------------+---------------------------------------------+ + | ``'x:gz'`` | Create a tarfile with gzip compression. | + | | Raise an :exc:`FileExistsError` exception | + | | if it already exists. | + +------------------+---------------------------------------------+ + | ``'x:bz2'`` | Create a tarfile with bzip2 compression. | + | | Raise an :exc:`FileExistsError` exception | + | | if it already exists. | + +------------------+---------------------------------------------+ + | ``'x:xz'`` | Create a tarfile with lzma compression. | + | | Raise an :exc:`FileExistsError` exception | + | | if it already exists. | + +------------------+---------------------------------------------+ | ``'a' or 'a:'`` | Open for appending with no compression. The | | | file is created if it does not exist. | +------------------+---------------------------------------------+ @@ -82,9 +98,9 @@ Some facts and figures: If *fileobj* is specified, it is used as an alternative to a :term:`file object` opened in binary mode for *name*. It is supposed to be at position 0. - For modes ``'w:gz'``, ``'r:gz'``, ``'w:bz2'``, ``'r:bz2'``, :func:`tarfile.open` - accepts the keyword argument *compresslevel* to specify the compression level of - the file. + For modes ``'w:gz'``, ``'r:gz'``, ``'w:bz2'``, ``'r:bz2'``, ``'x:gz'``, + ``'x:bz2'``, :func:`tarfile.open` accepts the keyword argument + *compresslevel* (default ``9``) to specify the compression level of the file. For special purposes, there is a second format for *mode*: ``'filemode|[compression]'``. :func:`tarfile.open` will return a :class:`TarFile` @@ -94,7 +110,7 @@ Some facts and figures: specifies the blocksize and defaults to ``20 * 512`` bytes. Use this variant in combination with e.g. ``sys.stdin``, a socket :term:`file object` or a tape device. However, such a :class:`TarFile` object is limited in that it does - not allow to be accessed randomly, see :ref:`tar-examples`. The currently + not allow random access, see :ref:`tar-examples`. The currently possible modes: +-------------+--------------------------------------------+ @@ -112,7 +128,7 @@ Some facts and figures: | ``'r|bz2'`` | Open a bzip2 compressed *stream* for | | | reading. | +-------------+--------------------------------------------+ - | ``'r|xz'`` | Open a lzma compressed *stream* for | + | ``'r|xz'`` | Open an lzma compressed *stream* for | | | reading. | +-------------+--------------------------------------------+ | ``'w|'`` | Open an uncompressed *stream* for writing. | @@ -127,11 +143,13 @@ Some facts and figures: | | writing. | +-------------+--------------------------------------------+ + .. versionchanged:: 3.5 + The ``'x'`` (exclusive creation) mode was added. .. class:: TarFile - Class for reading and writing tar archives. Do not use this class directly, - better use :func:`tarfile.open` instead. See :ref:`tarfile-objects`. + Class for reading and writing tar archives. Do not use this class directly: + use :func:`tarfile.open` instead. See :ref:`tarfile-objects`. .. function:: is_tarfile(name) @@ -219,7 +237,7 @@ details. Documentation of the higher-level archiving facilities provided by the standard :mod:`shutil` module. - `GNU tar manual, Basic Tar Format <http://www.gnu.org/software/tar/manual/html_node/Standard.html>`_ + `GNU tar manual, Basic Tar Format <https://www.gnu.org/software/tar/manual/html_node/Standard.html>`_ Documentation for tar archive files, including GNU tar extensions. @@ -252,8 +270,8 @@ be finalized; only the internally used file object will be closed. See the In this case, the file object's :attr:`name` attribute is used if it exists. *mode* is either ``'r'`` to read from an existing archive, ``'a'`` to append - data to an existing file or ``'w'`` to create a new file overwriting an existing - one. + data to an existing file, ``'w'`` to create a new file overwriting an existing + one, or ``'x'`` to create a new file only if it does not already exist. If *fileobj* is given, it is used for reading or writing data. If it can be determined, *mode* is overridden by *fileobj*'s mode. *fileobj* will be used @@ -292,12 +310,14 @@ be finalized; only the internally used file object will be closed. See the to be handled. The default settings will work for most users. See section :ref:`tar-unicode` for in-depth information. - .. versionchanged:: 3.2 - Use ``'surrogateescape'`` as the default for the *errors* argument. - The *pax_headers* argument is an optional dictionary of strings which will be added as a pax global header if *format* is :const:`PAX_FORMAT`. + .. versionchanged:: 3.2 + Use ``'surrogateescape'`` as the default for the *errors* argument. + + .. versionchanged:: 3.5 + The ``'x'`` (exclusive creation) mode was added. .. classmethod:: TarFile.open(...) @@ -328,11 +348,15 @@ be finalized; only the internally used file object will be closed. See the returned by :meth:`getmembers`. -.. method:: TarFile.list(verbose=True) +.. method:: TarFile.list(verbose=True, *, members=None) Print a table of contents to ``sys.stdout``. If *verbose* is :const:`False`, only the names of the members are printed. If it is :const:`True`, output - similar to that of :program:`ls -l` is produced. + similar to that of :program:`ls -l` is produced. If optional *members* is + given, it must be a subset of the list returned by :meth:`getmembers`. + + .. versionchanged:: 3.5 + Added the *members* parameter. .. method:: TarFile.next() @@ -342,7 +366,7 @@ be finalized; only the internally used file object will be closed. See the available. -.. method:: TarFile.extractall(path=".", members=None) +.. method:: TarFile.extractall(path=".", members=None, *, numeric_owner=False) Extract all members from the archive to the current working directory or directory *path*. If optional *members* is given, it must be a subset of the @@ -352,6 +376,10 @@ be finalized; only the internally used file object will be closed. See the reset each time a file is created in it. And, if a directory's permissions do not allow writing, extracting files to it will fail. + If *numeric_owner* is :const:`True`, the uid and gid numbers from the tarfile + are used to set the owner/group for the extracted files. Otherwise, the named + values from the tarfile are used. + .. warning:: Never extract archives from untrusted sources without prior inspection. @@ -359,8 +387,11 @@ be finalized; only the internally used file object will be closed. See the that have absolute filenames starting with ``"/"`` or filenames with two dots ``".."``. + .. versionchanged:: 3.5 + Added the *numeric_only* parameter. -.. method:: TarFile.extract(member, path="", set_attrs=True) + +.. method:: TarFile.extract(member, path="", set_attrs=True, *, numeric_owner=False) Extract a member from the archive to the current working directory, using its full name. Its file information is extracted as accurately as possible. *member* @@ -368,6 +399,10 @@ be finalized; only the internally used file object will be closed. See the directory using *path*. File attributes (owner, mtime, mode) are set unless *set_attrs* is false. + If *numeric_owner* is :const:`True`, the uid and gid numbers from the tarfile + are used to set the owner/group for the extracted files. Otherwise, the named + values from the tarfile are used. + .. note:: The :meth:`extract` method does not take care of several extraction issues. @@ -380,6 +415,9 @@ be finalized; only the internally used file object will be closed. See the .. versionchanged:: 3.2 Added the *set_attrs* parameter. + .. versionchanged:: 3.5 + Added the *numeric_only* parameter. + .. method:: TarFile.extractfile(member) Extract a member from the archive as a file object. *member* may be a filename @@ -417,21 +455,28 @@ be finalized; only the internally used file object will be closed. See the .. method:: TarFile.addfile(tarinfo, fileobj=None) Add the :class:`TarInfo` object *tarinfo* to the archive. If *fileobj* is given, + it should be a :term:`binary file`, and ``tarinfo.size`` bytes are read from it and added to the archive. You can - create :class:`TarInfo` objects using :meth:`gettarinfo`. - - .. note:: - - On Windows platforms, *fileobj* should always be opened with mode ``'rb'`` to - avoid irritation about the file size. + create :class:`TarInfo` objects directly, or by using :meth:`gettarinfo`. .. method:: TarFile.gettarinfo(name=None, arcname=None, fileobj=None) - Create a :class:`TarInfo` object for either the file *name* or the :term:`file - object` *fileobj* (using :func:`os.fstat` on its file descriptor). You can modify - some of the :class:`TarInfo`'s attributes before you add it using :meth:`addfile`. - If given, *arcname* specifies an alternative name for the file in the archive. + Create a :class:`TarInfo` object from the result of :func:`os.stat` or + equivalent on an existing file. The file is either named by *name*, or + specified as a :term:`file object` *fileobj* with a file descriptor. If + given, *arcname* specifies an alternative name for the file in the + archive, otherwise, the name is taken from *fileobj*’s + :attr:`~io.FileIO.name` attribute, or the *name* argument. The name + should be a text string. + + You can modify + some of the :class:`TarInfo`’s attributes before you add it using :meth:`addfile`. + If the file object is not an ordinary file object positioned at the + beginning of the file, attributes such as :attr:`~TarInfo.size` may need + modifying. This is the case for objects such as :class:`~gzip.GzipFile`. + The :attr:`~TarInfo.name` may also be modified, in which case *arcname* + could be a dummy string. .. method:: TarFile.close() @@ -609,25 +654,35 @@ The :mod:`tarfile` module provides a simple command line interface to interact with tar archives. If you want to create a new tar archive, specify its name after the :option:`-c` -option and then list the filename(s) that should be included:: +option and then list the filename(s) that should be included: + +.. code-block:: shell-session $ python -m tarfile -c monty.tar spam.txt eggs.txt -Passing a directory is also acceptable:: +Passing a directory is also acceptable: + +.. code-block:: shell-session $ python -m tarfile -c monty.tar life-of-brian_1979/ If you want to extract a tar archive into the current directory, use -the :option:`-e` option:: +the :option:`-e` option: + +.. code-block:: shell-session $ python -m tarfile -e monty.tar You can also extract a tar archive into a different directory by passing the -directory's name:: +directory's name: + +.. code-block:: shell-session $ python -m tarfile -e monty.tar other-dir/ -For a list of the files in a tar archive, use the :option:`-l` option:: +For a list of the files in a tar archive, use the :option:`-l` option: + +.. code-block:: shell-session $ python -m tarfile -l monty.tar @@ -802,4 +857,3 @@ In case of :const:`PAX_FORMAT` archives, *encoding* is generally not needed because all the metadata is stored using *UTF-8*. *encoding* is only used in the rare cases when binary pax headers are decoded or when strings with surrogate characters are stored. - diff --git a/Doc/library/telnetlib.rst b/Doc/library/telnetlib.rst index 4040f72ee8..b950e41dc1 100644 --- a/Doc/library/telnetlib.rst +++ b/Doc/library/telnetlib.rst @@ -3,13 +3,13 @@ .. module:: telnetlib :synopsis: Telnet client class. + .. sectionauthor:: Skip Montanaro <skip@pobox.com> +**Source code:** :source:`Lib/telnetlib.py` .. index:: single: protocol; Telnet -**Source code:** :source:`Lib/telnetlib.py` - -------------- The :mod:`telnetlib` module provides a :class:`Telnet` class that implements the diff --git a/Doc/library/tempfile.rst b/Doc/library/tempfile.rst index 1efb5e6bde..665261ffb2 100644 --- a/Doc/library/tempfile.rst +++ b/Doc/library/tempfile.rst @@ -1,66 +1,79 @@ :mod:`tempfile` --- Generate temporary files and directories ============================================================ -.. sectionauthor:: Zack Weinberg <zack@codesourcery.com> - - .. module:: tempfile :synopsis: Generate temporary files and directories. +.. sectionauthor:: Zack Weinberg <zack@codesourcery.com> + +**Source code:** :source:`Lib/tempfile.py` .. index:: pair: temporary; file name pair: temporary; file -**Source code:** :source:`Lib/tempfile.py` - -------------- -This module generates temporary files and directories. It works on all -supported platforms. It provides three new functions, -:func:`NamedTemporaryFile`, :func:`mkstemp`, and :func:`mkdtemp`, which should -eliminate all remaining need to use the insecure :func:`mktemp` function. -Temporary file names created by this module no longer contain the process ID; -instead a string of six random characters is used. - -Also, all the user-callable functions now take additional arguments which -allow direct control over the location and name of temporary files. It is -no longer necessary to use the global *tempdir* variable. +This module creates temporary files and directories. It works on all +supported platforms. :class:`TemporaryFile`, :class:`NamedTemporaryFile`, +:class:`TemporaryDirectory`, and :class:`SpooledTemporaryFile` are high-level +interfaces which provide automatic cleanup and can be used as +context managers. :func:`mkstemp` and +:func:`mkdtemp` are lower-level functions which require manual cleanup. + +All the user-callable functions and constructors take additional arguments which +allow direct control over the location and name of temporary files and +directories. Files names used by this module include a string of +random characters which allows those files to be securely created in +shared temporary directories. To maintain backward compatibility, the argument order is somewhat odd; it is recommended to use keyword arguments for clarity. The module defines the following user-callable items: -.. function:: TemporaryFile(mode='w+b', buffering=None, encoding=None, newline=None, suffix='', prefix='tmp', dir=None) +.. function:: TemporaryFile(mode='w+b', buffering=None, encoding=None, newline=None, suffix=None, prefix=None, dir=None) Return a :term:`file-like object` that can be used as a temporary storage area. - The file is created using :func:`mkstemp`. It will be destroyed as soon + The file is created securely, using the same rules as :func:`mkstemp`. It will be destroyed as soon as it is closed (including an implicit close when the object is garbage - collected). Under Unix, the directory entry for the file is removed + collected). Under Unix, the directory entry for the file is either not created at all or is removed immediately after the file is created. Other platforms do not support this; your code should not rely on a temporary file created using this function having or not having a visible name in the file system. + The resulting object can be used as a context manager (see + :ref:`tempfile-examples`). On completion of the context or + destruction of the file object the temporary file will be removed + from the filesystem. + The *mode* parameter defaults to ``'w+b'`` so that the file created can be read and written without being closed. Binary mode is used so that it behaves consistently on all platforms without regard for the data that is stored. *buffering*, *encoding* and *newline* are interpreted as for :func:`open`. - The *dir*, *prefix* and *suffix* parameters are passed to :func:`mkstemp`. + The *dir*, *prefix* and *suffix* parameters have the same meaning and + defaults as with :func:`mkstemp`. The returned object is a true file object on POSIX platforms. On other platforms, it is a file-like object whose :attr:`!file` attribute is the - underlying true file object. This file-like object can be used in a - :keyword:`with` statement, just like a normal file. + underlying true file object. + The :py:data:`os.O_TMPFILE` flag is used if it is available and works + (Linux-specific, requires Linux kernel 3.11 or later). -.. function:: NamedTemporaryFile(mode='w+b', buffering=None, encoding=None, newline=None, suffix='', prefix='tmp', dir=None, delete=True) + .. versionchanged:: 3.5 + + The :py:data:`os.O_TMPFILE` flag is now used if available. + + +.. function:: NamedTemporaryFile(mode='w+b', buffering=None, encoding=None, newline=None, suffix=None, prefix=None, dir=None, delete=True) This function operates exactly as :func:`TemporaryFile` does, except that the file is guaranteed to have a visible name in the file system (on Unix, the directory entry is not unlinked). That name can be retrieved - from the :attr:`name` attribute of the file object. Whether the name can be + from the :attr:`name` attribute of the returned + file-like object. Whether the name can be used to open the file a second time, while the named temporary file is still open, varies across platforms (it can be so used on Unix; it cannot on Windows NT or later). If *delete* is true (the default), the file is @@ -70,7 +83,7 @@ The module defines the following user-callable items: be used in a :keyword:`with` statement, just like a normal file. -.. function:: SpooledTemporaryFile(max_size=0, mode='w+b', buffering=None, encoding=None, newline=None, suffix='', prefix='tmp', dir=None) +.. function:: SpooledTemporaryFile(max_size=0, mode='w+b', buffering=None, encoding=None, newline=None, suffix=None, prefix=None, dir=None) This function operates exactly as :func:`TemporaryFile` does, except that data is spooled in memory until the file size exceeds *max_size*, or @@ -92,12 +105,11 @@ The module defines the following user-callable items: the truncate method now accepts a ``size`` argument. -.. function:: TemporaryDirectory(suffix='', prefix='tmp', dir=None) +.. function:: TemporaryDirectory(suffix=None, prefix=None, dir=None) - This function creates a temporary directory using :func:`mkdtemp` - (the supplied arguments are passed directly to the underlying function). + This function securely creates a temporary directory using the same rules as :func:`mkdtemp`. The resulting object can be used as a context manager (see - :ref:`context-managers`). On completion of the context or destruction + :ref:`tempfile-examples`). On completion of the context or destruction of the temporary directory object the newly created temporary directory and all its contents are removed from the filesystem. @@ -112,7 +124,7 @@ The module defines the following user-callable items: .. versionadded:: 3.2 -.. function:: mkstemp(suffix='', prefix='tmp', dir=None, text=False) +.. function:: mkstemp(suffix=None, prefix=None, dir=None, text=False) Creates a temporary file in the most secure manner possible. There are no race conditions in the file's creation, assuming that the platform @@ -125,15 +137,16 @@ The module defines the following user-callable items: Unlike :func:`TemporaryFile`, the user of :func:`mkstemp` is responsible for deleting the temporary file when done with it. - If *suffix* is specified, the file name will end with that suffix, + If *suffix* is not ``None``, the file name will end with that suffix, otherwise there will be no suffix. :func:`mkstemp` does not put a dot between the file name and the suffix; if you need one, put it at the beginning of *suffix*. - If *prefix* is specified, the file name will begin with that prefix; - otherwise, a default prefix is used. + If *prefix* is not ``None``, the file name will begin with that prefix; + otherwise, a default prefix is used. The default is the return value of + :func:`gettempprefix` or :func:`gettempprefixb`, as appropriate. - If *dir* is specified, the file will be created in that directory; + If *dir* is not ``None``, the file will be created in that directory; otherwise, a default directory is used. The default directory is chosen from a platform-dependent list, but the user of the application can control the directory location by setting the *TMPDIR*, *TEMP* or *TMP* @@ -141,6 +154,12 @@ The module defines the following user-callable items: filename will have any nice properties, such as not requiring quoting when passed to external commands via ``os.popen()``. + If any of *suffix*, *prefix*, and *dir* are not + ``None``, they must be the same type. + If they are bytes, the returned name will be bytes instead of str. + If you want to force a bytes return value with otherwise default behavior, + pass ``suffix=b''``. + If *text* is specified, it indicates whether to open the file in binary mode (the default) or text mode. On some platforms, this makes no difference. @@ -149,8 +168,14 @@ The module defines the following user-callable items: file (as would be returned by :func:`os.open`) and the absolute pathname of that file, in that order. + .. versionchanged:: 3.5 + *suffix*, *prefix*, and *dir* may now be supplied in bytes in order to + obtain a bytes return value. Prior to this, only str was allowed. + *suffix* and *prefix* now accept and default to ``None`` to cause + an appropriate default value to be used. + -.. function:: mkdtemp(suffix='', prefix='tmp', dir=None) +.. function:: mkdtemp(suffix=None, prefix=None, dir=None) Creates a temporary directory in the most secure manner possible. There are no race conditions in the directory's creation. The directory is @@ -164,50 +189,21 @@ The module defines the following user-callable items: :func:`mkdtemp` returns the absolute pathname of the new directory. - -.. function:: mktemp(suffix='', prefix='tmp', dir=None) - - .. deprecated:: 2.3 - Use :func:`mkstemp` instead. - - Return an absolute pathname of a file that did not exist at the time the - call is made. The *prefix*, *suffix*, and *dir* arguments are the same - as for :func:`mkstemp`. - - .. warning:: - - Use of this function may introduce a security hole in your program. By - the time you get around to doing anything with the file name it returns, - someone else may have beaten you to the punch. :func:`mktemp` usage can - be replaced easily with :func:`NamedTemporaryFile`, passing it the - ``delete=False`` parameter:: - - >>> f = NamedTemporaryFile(delete=False) - >>> f.name - '/tmp/tmptjujjt' - >>> f.write(b"Hello World!\n") - 13 - >>> f.close() - >>> os.unlink(f.name) - >>> os.path.exists(f.name) - False - -The module uses a global variable that tell it how to construct a -temporary name. They are initialized at the first call to any of the -functions above. The caller may change them, but this is discouraged; use -the appropriate function arguments, instead. + .. versionchanged:: 3.5 + *suffix*, *prefix*, and *dir* may now be supplied in bytes in order to + obtain a bytes return value. Prior to this, only str was allowed. + *suffix* and *prefix* now accept and default to ``None`` to cause + an appropriate default value to be used. -.. data:: tempdir +.. function:: gettempdir() - When set to a value other than ``None``, this variable defines the - default value for the *dir* argument to all the functions defined in this - module. + Return the name of the directory used for temporary files. This + defines the default value for the *dir* argument to all functions + in this module. - If ``tempdir`` is unset or ``None`` at any call to any of the above - functions, Python searches a standard list of directories and sets - *tempdir* to the first one which the calling user can create files in. - The list is: + Python searches a standard list of directories to find one which + the calling user can create files in. The list is: #. The directory named by the :envvar:`TMPDIR` environment variable. @@ -225,19 +221,43 @@ the appropriate function arguments, instead. #. As a last resort, the current working directory. + The result of this search is cached, see the description of + :data:`tempdir` below. -.. function:: gettempdir() +.. function:: gettempdirb() - Return the directory currently selected to create temporary files in. If - :data:`tempdir` is not ``None``, this simply returns its contents; otherwise, - the search described above is performed, and the result returned. + Same as :func:`gettempdir` but the return value is in bytes. + .. versionadded:: 3.5 .. function:: gettempprefix() Return the filename prefix used to create temporary files. This does not contain the directory component. +.. function:: gettempprefixb() + + Same as :func:`gettempprefix` but the return value is in bytes. + + .. versionadded:: 3.5 + +The module uses a global variable to store the name of the directory +used for temporary files returned by :func:`gettempdir`. It can be +set directly to override the selection process, but this is discouraged. +All functions in this module take a *dir* argument which can be used +to specify the directory and this is the recommend approach. + +.. data:: tempdir + + When set to a value other than ``None``, this variable defines the + default value for the *dir* argument to all the functions defined in this + module. + + If ``tempdir`` is unset or ``None`` at any call to any of the above + functions except :func:`gettempprefix` it is initialized following the + algorithm described in :func:`gettempdir`. + +.. _tempfile-examples: Examples -------- @@ -271,3 +291,43 @@ Here are some examples of typical usage of the :mod:`tempfile` module:: >>> # directory and contents have been removed + +Deprecated functions and variables +---------------------------------- + +A historical way to create temporary files was to first generate a +file name with the :func:`mktemp` function and then create a file +using this name. Unfortunately this is not secure, because a different +process may create a file with this name in the time between the call +to :func:`mktemp` and the subsequent attempt to create the file by the +first process. The solution is to combine the two steps and create the +file immediately. This approach is used by :func:`mkstemp` and the +other functions described above. + +.. function:: mktemp(suffix='', prefix='tmp', dir=None) + + .. deprecated:: 2.3 + Use :func:`mkstemp` instead. + + Return an absolute pathname of a file that did not exist at the time the + call is made. The *prefix*, *suffix*, and *dir* arguments are similar + to those of :func:`mkstemp`, except that bytes file names, ``suffix=None`` + and ``prefix=None`` are not supported. + + .. warning:: + + Use of this function may introduce a security hole in your program. By + the time you get around to doing anything with the file name it returns, + someone else may have beaten you to the punch. :func:`mktemp` usage can + be replaced easily with :func:`NamedTemporaryFile`, passing it the + ``delete=False`` parameter:: + + >>> f = NamedTemporaryFile(delete=False) + >>> f.name + '/tmp/tmptjujjt' + >>> f.write(b"Hello World!\n") + 13 + >>> f.close() + >>> os.unlink(f.name) + >>> os.path.exists(f.name) + False diff --git a/Doc/library/termios.rst b/Doc/library/termios.rst index a90a825be4..ad6a9f7c97 100644 --- a/Doc/library/termios.rst +++ b/Doc/library/termios.rst @@ -5,15 +5,16 @@ :platform: Unix :synopsis: POSIX style tty control. - .. index:: pair: POSIX; I/O control pair: tty; I/O control -This module provides an interface to the POSIX calls for tty I/O control. For a -complete description of these calls, see the POSIX or Unix manual pages. It is -only available for those Unix versions that support POSIX *termios* style tty -I/O control (and then only if configured at installation time). +-------------- + +This module provides an interface to the POSIX calls for tty I/O control. For a +complete description of these calls, see :manpage:`termios(2)` Unix manual +page. It is only available for those Unix versions that support POSIX +*termios* style tty I/O control configured during installation. All functions in this module take a file descriptor *fd* as their first argument. This can be an integer file descriptor, such as returned by diff --git a/Doc/library/test.rst b/Doc/library/test.rst index 2fdaf8cb37..2ea9c27e49 100644 --- a/Doc/library/test.rst +++ b/Doc/library/test.rst @@ -3,6 +3,7 @@ .. module:: test :synopsis: Regression tests package containing the testing suite for Python. + .. sectionauthor:: Brett Cannon <brett@python.org> .. note:: @@ -12,6 +13,7 @@ mentioned here can change or be removed without notice between releases of Python. +-------------- The :mod:`test` package contains all regression tests for Python as well as the modules :mod:`test.support` and :mod:`test.regrtest`. @@ -550,7 +552,7 @@ The :mod:`test.support` module defines the following functions: or passed to an external program (i.e. the ``-accept`` argument to openssl's s_server mode). Always prefer :func:`bind_port` over :func:`find_unused_port` where possible. Using a hard coded port is - discouraged since it can makes multiple instances of the test impossible to + discouraged since it can make multiple instances of the test impossible to run simultaneously, which is a problem for buildbots. @@ -568,6 +570,17 @@ The :mod:`test.support` module defines the following functions: def load_tests(*args): return load_package_tests(os.path.dirname(__file__), *args) +.. function:: detect_api_mismatch(ref_api, other_api, *, ignore=()): + + Returns the set of attributes, functions or methods of *ref_api* not + found on *other_api*, except for a defined list of items to be + ignored in this check specified in *ignore*. + + By default this skips private attributes beginning with '_' but + includes all magic methods, i.e. those starting and ending in '__'. + + .. versionadded:: 3.5 + The :mod:`test.support` module defines the following classes: @@ -608,7 +621,7 @@ The :mod:`test.support` module defines the following classes: are expected to crash a subprocess. On Windows, it disables Windows Error Reporting dialogs using - `SetErrorMode <http://msdn.microsoft.com/en-us/library/windows/desktop/ms680621.aspx>`_. + `SetErrorMode <https://msdn.microsoft.com/en-us/library/windows/desktop/ms680621.aspx>`_. On UNIX, :func:`resource.setrlimit` is used to set :attr:`resource.RLIMIT_CORE`'s soft limit to 0 to prevent coredump file diff --git a/Doc/library/textwrap.rst b/Doc/library/textwrap.rst index 9fe7a3589a..438007d002 100644 --- a/Doc/library/textwrap.rst +++ b/Doc/library/textwrap.rst @@ -3,6 +3,7 @@ .. module:: textwrap :synopsis: Text wrapping and filling + .. moduleauthor:: Greg Ward <gward@python.net> .. sectionauthor:: Greg Ward <gward@python.net> diff --git a/Doc/library/threading.rst b/Doc/library/threading.rst index c56d707342..3066496fa4 100644 --- a/Doc/library/threading.rst +++ b/Doc/library/threading.rst @@ -847,7 +847,7 @@ For example:: print("hello, world") t = Timer(30.0, hello) - t.start() # after 30 seconds, "hello, world" will be printed + t.start() # after 30 seconds, "hello, world" will be printed .. class:: Timer(interval, function, args=None, kwargs=None) diff --git a/Doc/library/time.rst b/Doc/library/time.rst index 0a3d9772b3..e6626f262d 100644 --- a/Doc/library/time.rst +++ b/Doc/library/time.rst @@ -4,6 +4,7 @@ .. module:: time :synopsis: Time access and conversions. +-------------- This module provides various time-related functions. For related functionality, see also the :mod:`datetime` and :mod:`calendar` modules. @@ -314,9 +315,9 @@ The module defines the following functions and data items: processes running for more than 49 days. On more recent versions of Windows and on other operating systems, :func:`monotonic` is system-wide. - Availability: Windows, Mac OS X, Linux, FreeBSD, OpenBSD, Solaris. - .. versionadded:: 3.3 + .. versionchanged:: 3.5 + The function is now always available. .. function:: perf_counter() @@ -350,6 +351,11 @@ The module defines the following functions and data items: requested by an arbitrary amount because of the scheduling of other activity in the system. + .. versionchanged:: 3.5 + The function now sleeps at least *secs* even if the sleep is interrupted + by a signal, except if the signal handler raises an exception (see + :pep:`475` for the rationale). + .. function:: strftime(format[, t]) diff --git a/Doc/library/timeit.rst b/Doc/library/timeit.rst index 70df409fac..57a4834c90 100644 --- a/Doc/library/timeit.rst +++ b/Doc/library/timeit.rst @@ -4,17 +4,16 @@ .. module:: timeit :synopsis: Measure the execution time of small code snippets. +**Source code:** :source:`Lib/timeit.py` .. index:: single: Benchmarking single: Performance -**Source code:** :source:`Lib/timeit.py` - -------------- This module provides a simple way to time small bits of Python code. It has both -a :ref:`command-line-interface` as well as a :ref:`callable <python-interface>` +a :ref:`timeit-command-line-interface` as well as a :ref:`callable <python-interface>` one. It avoids a number of common traps for measuring execution times. See also Tim Peters' introduction to the "Algorithms" chapter in the *Python Cookbook*, published by O'Reilly. @@ -23,7 +22,7 @@ Cookbook*, published by O'Reilly. Basic Examples -------------- -The following example shows how the :ref:`command-line-interface` +The following example shows how the :ref:`timeit-command-line-interface` can be used to compare three different expressions: .. code-block:: sh @@ -59,18 +58,26 @@ Python Interface The module defines three convenience functions and a public class: -.. function:: timeit(stmt='pass', setup='pass', timer=<default timer>, number=1000000) +.. function:: timeit(stmt='pass', setup='pass', timer=<default timer>, number=1000000, globals=None) Create a :class:`Timer` instance with the given statement, *setup* code and *timer* function and run its :meth:`.timeit` method with *number* executions. + The optional *globals* argument specifies a namespace in which to execute the + code. + .. versionchanged:: 3.5 + The optional *globals* parameter was added. -.. function:: repeat(stmt='pass', setup='pass', timer=<default timer>, repeat=3, number=1000000) + +.. function:: repeat(stmt='pass', setup='pass', timer=<default timer>, repeat=3, number=1000000, globals=None) Create a :class:`Timer` instance with the given statement, *setup* code and *timer* function and run its :meth:`.repeat` method with the given *repeat* - count and *number* executions. + count and *number* executions. The optional *globals* argument specifies a + namespace in which to execute the code. + .. versionchanged:: 3.5 + The optional *globals* parameter was added. .. function:: default_timer() @@ -80,7 +87,7 @@ The module defines three convenience functions and a public class: :func:`time.perf_counter` is now the default timer. -.. class:: Timer(stmt='pass', setup='pass', timer=<timer function>) +.. class:: Timer(stmt='pass', setup='pass', timer=<timer function>, globals=None) Class for timing execution speed of small code snippets. @@ -88,7 +95,9 @@ The module defines three convenience functions and a public class: for setup, and a timer function. Both statements default to ``'pass'``; the timer function is platform-dependent (see the module doc string). *stmt* and *setup* may also contain multiple statements separated by ``;`` - or newlines, as long as they don't contain multi-line string literals. + or newlines, as long as they don't contain multi-line string literals. The + statement will by default be executed within timeit's namespace; this behavior + can be controlled by passing a namespace to *globals*. To measure the execution time of the first statement, use the :meth:`.timeit` method. The :meth:`.repeat` method is a convenience to call :meth:`.timeit` @@ -101,6 +110,8 @@ The module defines three convenience functions and a public class: will then be executed by :meth:`.timeit`. Note that the timing overhead is a little larger in this case because of the extra function calls. + .. versionchanged:: 3.5 + The optional *globals* parameter was added. .. method:: Timer.timeit(number=1000000) @@ -162,14 +173,14 @@ The module defines three convenience functions and a public class: where the traceback is sent; it defaults to :data:`sys.stderr`. -.. _command-line-interface: +.. _timeit-command-line-interface: Command-Line Interface ---------------------- When called as a program from the command line, the following form is used:: - python -m timeit [-n N] [-r N] [-s S] [-t] [-c] [-h] [statement ...] + python -m timeit [-n N] [-r N] [-u U] [-s S] [-t] [-c] [-h] [statement ...] Where the following options are understood: @@ -198,6 +209,12 @@ Where the following options are understood: use :func:`time.time` (deprecated) +.. cmdoption:: -u, --unit=U + + specify a time unit for timer output; can select usec, msec, or sec + + .. versionadded:: 3.5 + .. cmdoption:: -c, --clock use :func:`time.clock` (deprecated) @@ -320,3 +337,17 @@ To give the :mod:`timeit` module access to functions you define, you can pass a if __name__ == '__main__': import timeit print(timeit.timeit("test()", setup="from __main__ import test")) + +Another option is to pass :func:`globals` to the *globals* parameter, which will cause the code +to be executed within your current global namespace. This can be more convenient +than individually specifying imports:: + + def f(x): + return x**2 + def g(x): + return x**4 + def h(x): + return x**8 + + import timeit + print(timeit.timeit('[func(42) for func in (f,g,h)]', globals=globals())) diff --git a/Doc/library/tkinter.rst b/Doc/library/tkinter.rst index 8b738c3481..3e1faed8c7 100644 --- a/Doc/library/tkinter.rst +++ b/Doc/library/tkinter.rst @@ -3,8 +3,12 @@ .. module:: tkinter :synopsis: Interface to Tcl/Tk for graphical user interfaces + .. moduleauthor:: Guido van Rossum <guido@Python.org> +**Source code:** :source:`Lib/tkinter/__init__.py` + +-------------- The :mod:`tkinter` package ("Tk interface") is the standard Python interface to the Tk GUI toolkit. Both Tk and :mod:`tkinter` are available on most Unix @@ -22,22 +26,22 @@ this should open a window demonstrating a simple Tk interface. `TKDocs <http://www.tkdocs.com/>`_ Extensive tutorial plus friendlier widget pages for some of the widgets. - `Tkinter reference: a GUI for Python <http://infohost.nmt.edu/tcc/help/pubs/tkinter/web/index.html>`_ + `Tkinter reference: a GUI for Python <https://infohost.nmt.edu/tcc/help/pubs/tkinter/web/index.html>`_ On-line reference material. `Tkinter docs from effbot <http://effbot.org/tkinterbook/>`_ Online reference for tkinter supported by effbot.org. - `Tcl/Tk manual <http://www.tcl.tk/man/tcl8.5/>`_ + `Tcl/Tk manual <https://www.tcl.tk/man/tcl8.5/>`_ Official manual for the latest tcl/tk version. - `Programming Python <http://www.rmi.net/~lutz/about-pp4e.html>`_ + `Programming Python <http://learning-python.com/books/about-pp4e.html>`_ Book by Mark Lutz, has excellent coverage of Tkinter. `Modern Tkinter for Busy Python Developers <http://www.amazon.com/Modern-Tkinter-Python-Developers-ebook/dp/B0071QDNLO/>`_ Book by Mark Rozerman about building attractive and modern graphical user interfaces with Python and Tkinter. - `Python and Tkinter Programming <http://www.manning.com/grayson/>`_ + `Python and Tkinter Programming <https://www.manning.com/books/python-and-tkinter-programming>`_ The book by John Grayson (ISBN 1-884777-81-3). @@ -173,7 +177,7 @@ documentation that exists. Here are some hints: .. seealso:: - `Tcl/Tk 8.6 man pages <http://www.tcl.tk/man/tcl8.6/>`_ + `Tcl/Tk 8.6 man pages <https://www.tcl.tk/man/tcl8.6/>`_ The Tcl/Tk manual on www.tcl.tk. `ActiveState Tcl Home Page <http://tcl.activestate.com/>`_ @@ -195,19 +199,19 @@ A Simple Hello World Program class Application(tk.Frame): def __init__(self, master=None): - tk.Frame.__init__(self, master) + super().__init__(master) self.pack() - self.createWidgets() + self.create_widgets() - def createWidgets(self): + def create_widgets(self): self.hi_there = tk.Button(self) self.hi_there["text"] = "Hello World\n(click me)" self.hi_there["command"] = self.say_hi self.hi_there.pack(side="top") - self.QUIT = tk.Button(self, text="QUIT", fg="red", - command=root.destroy) - self.QUIT.pack(side="bottom") + self.quit = tk.Button(self, text="QUIT", fg="red", + command=root.destroy) + self.quit.pack(side="bottom") def say_hi(self): print("hi there, everyone!") @@ -532,7 +536,7 @@ For example:: class App(Frame): def __init__(self, master=None): - Frame.__init__(self, master) + super().__init__(master) self.pack() self.entrythingy = Entry() @@ -577,13 +581,13 @@ part of the implementation, and not an interface to Tk functionality. Here are some examples of typical usage:: - from tkinter import * - class App(Frame): + import tkinter as tk + + class App(tk.Frame): def __init__(self, master=None): - Frame.__init__(self, master) + super().__init__(master) self.pack() - # create the application myapp = App() @@ -704,13 +708,13 @@ add For example:: - def turnRed(self, event): + def turn_red(self, event): event.widget["activeforeground"] = "red" - self.button.bind("<Enter>", self.turnRed) + self.button.bind("<Enter>", self.turn_red) Notice how the widget field of the event is being accessed in the -:meth:`turnRed` callback. This field contains the widget that caught the X +``turn_red()`` callback. This field contains the widget that caught the X event. The following table lists the other event fields you can access, and how they are denoted in Tk, which can be useful when referring to the Tk man pages. diff --git a/Doc/library/tkinter.scrolledtext.rst b/Doc/library/tkinter.scrolledtext.rst index 520970f168..138720e478 100644 --- a/Doc/library/tkinter.scrolledtext.rst +++ b/Doc/library/tkinter.scrolledtext.rst @@ -4,8 +4,12 @@ .. module:: tkinter.scrolledtext :platform: Tk :synopsis: Text widget with a vertical scroll bar. + .. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org> +**Source code:** :source:`Lib/tkinter/scrolledtext.py` + +-------------- The :mod:`tkinter.scrolledtext` module provides a class of the same name which implements a basic text widget which has a vertical scroll bar configured to do diff --git a/Doc/library/tkinter.tix.rst b/Doc/library/tkinter.tix.rst index 9de73ade42..41f20ddf4e 100644 --- a/Doc/library/tkinter.tix.rst +++ b/Doc/library/tkinter.tix.rst @@ -3,11 +3,15 @@ .. module:: tkinter.tix :synopsis: Tk Extension Widgets for Tkinter + .. sectionauthor:: Mike Clarkson <mikeclarkson@users.sourceforge.net> +**Source code:** :source:`Lib/tkinter/tix.py` .. index:: single: Tix +-------------- + The :mod:`tkinter.tix` (Tk Interface Extension) module provides an additional rich set of widgets. Although the standard Tk library has many useful widgets, they are far from complete. The :mod:`tkinter.tix` library provides most of the @@ -142,7 +146,7 @@ Basic Widgets The `LabelEntry <http://tix.sourceforge.net/dist/current/man/html/TixCmd/tixLabelEntry.htm>`_ - widget packages an entry widget and a label into one mega widget. It can be used + widget packages an entry widget and a label into one mega widget. It can be used to simplify the creation of "entry-form" type of interface. .. Python Demo of: @@ -267,7 +271,7 @@ File Selectors The `ExFileSelectBox <http://tix.sourceforge.net/dist/current/man/html/TixCmd/tixExFileSelectBox.htm>`_ - widget is usually embedded in a tixExFileSelectDialog widget. It provides an + widget is usually embedded in a tixExFileSelectDialog widget. It provides a convenient method for the user to select files. The style of the :class:`ExFileSelectBox` widget is very similar to the standard file dialog on MS Windows 3.1. diff --git a/Doc/library/tkinter.ttk.rst b/Doc/library/tkinter.ttk.rst index 4601171b59..dbb5bd2505 100644 --- a/Doc/library/tkinter.ttk.rst +++ b/Doc/library/tkinter.ttk.rst @@ -3,11 +3,15 @@ .. module:: tkinter.ttk :synopsis: Tk themed widget set + .. sectionauthor:: Guilherme Polo <ggpolo@gmail.com> +**Source code:** :source:`Lib/tkinter/ttk.py` .. index:: single: ttk +-------------- + The :mod:`tkinter.ttk` module provides access to the Tk themed widget set, introduced in Tk 8.5. If Python has not been compiled against Tk 8.5, this module can still be accessed if *Tile* has been installed. The former @@ -22,7 +26,7 @@ appearance. .. seealso:: - `Tk Widget Styling Support <http://www.tcl.tk/cgi-bin/tct/tip/48>`_ + `Tk Widget Styling Support <https://www.tcl.tk/cgi-bin/tct/tip/48>`_ A document introducing theming support for Tk @@ -299,7 +303,7 @@ Besides the methods inherited from :class:`Widget`: :meth:`Widget.cget`, :meth:`Widget.configure`, :meth:`Widget.identify`, :meth:`Widget.instate` and :meth:`Widget.state`, and the following inherited from :class:`Entry`: :meth:`Entry.bbox`, :meth:`Entry.delete`, :meth:`Entry.icursor`, -:meth:`Entry.index`, :meth:`Entry.inset`, :meth:`Entry.selection`, +:meth:`Entry.index`, :meth:`Entry.insert`, :meth:`Entry.selection`, :meth:`Entry.xview`, it has some other methods, described at :class:`ttk.Combobox`. @@ -701,7 +705,7 @@ the widget option ``displaycolumns``. The tree widget can also display column headings. Columns may be accessed by number or symbolic names listed in the widget option columns. See `Column Identifiers`_. -Each item is identified by an unique name. The widget will generate item IDs +Each item is identified by a unique name. The widget will generate item IDs if they are not supplied by the caller. There is a distinguished root item, named ``{}``. The root item itself is not displayed; its children appear at the top level of the hierarchy. diff --git a/Doc/library/token.rst b/Doc/library/token.rst index 4cd709814c..116efca715 100644 --- a/Doc/library/token.rst +++ b/Doc/library/token.rst @@ -3,6 +3,7 @@ .. module:: token :synopsis: Constants representing terminal nodes of the parse tree. + .. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org> **Source code:** :source:`Lib/token.py` @@ -93,13 +94,20 @@ The token constants are: DOUBLESLASH DOUBLESLASHEQUAL AT + ATEQUAL RARROW ELLIPSIS OP + AWAIT + ASYNC ERRORTOKEN N_TOKENS NT_OFFSET + .. versionchanged:: 3.5 + Added :data:`AWAIT` and :data:`ASYNC` tokens. Starting with + Python 3.7, "async" and "await" will be tokenized as :data:`NAME` + tokens, and :data:`AWAIT` and :data:`ASYNC` will be removed. .. seealso:: diff --git a/Doc/library/tokenize.rst b/Doc/library/tokenize.rst index c9cb51896e..ff55aacbd4 100644 --- a/Doc/library/tokenize.rst +++ b/Doc/library/tokenize.rst @@ -3,6 +3,7 @@ .. module:: tokenize :synopsis: Lexical scanner for Python source code. + .. moduleauthor:: Ka Ping Yee .. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org> @@ -27,7 +28,7 @@ The primary entry point is a :term:`generator`: .. function:: tokenize(readline) - The :func:`tokenize` generator requires one argument, *readline*, which + The :func:`.tokenize` generator requires one argument, *readline*, which must be a callable object which provides the same interface as the :meth:`io.IOBase.readline` method of file objects. Each call to the function should return one line of input as bytes. @@ -52,7 +53,7 @@ The primary entry point is a :term:`generator`: .. versionchanged:: 3.3 Added support for ``exact_type``. - :func:`tokenize` determines the source encoding of the file by looking for a + :func:`.tokenize` determines the source encoding of the file by looking for a UTF-8 BOM or encoding cookie, according to :pep:`263`. @@ -74,7 +75,7 @@ All constants from the :mod:`token` module are also exported from .. data:: ENCODING Token value that indicates the encoding used to decode the source bytes - into text. The first token returned by :func:`tokenize` will always be an + into text. The first token returned by :func:`.tokenize` will always be an ENCODING token. @@ -96,17 +97,17 @@ write back the modified script. positions) may change. It returns bytes, encoded using the ENCODING token, which is the first - token sequence output by :func:`tokenize`. + token sequence output by :func:`.tokenize`. -:func:`tokenize` needs to detect the encoding of source files it tokenizes. The +:func:`.tokenize` needs to detect the encoding of source files it tokenizes. The function it uses to do this is available: .. function:: detect_encoding(readline) The :func:`detect_encoding` function is used to detect the encoding that should be used to decode a Python source file. It requires one argument, - readline, in the same way as the :func:`tokenize` generator. + readline, in the same way as the :func:`.tokenize` generator. It will call readline a maximum of twice, and return the encoding used (as a string) and a list of any lines (not decoded from bytes) it has read @@ -120,7 +121,7 @@ function it uses to do this is available: If no encoding is specified, then the default of ``'utf-8'`` will be returned. - Use :func:`open` to open Python source files: it uses + Use :func:`.open` to open Python source files: it uses :func:`detect_encoding` to detect the file encoding. @@ -201,7 +202,7 @@ objects:: we're only showing 12 digits, and the 13th isn't close to 5, the rest of the output should be platform-independent. - >>> exec(s) #doctest: +ELLIPSIS + >>> exec(s) #doctest: +ELLIPSIS -3.21716034272e-0...7 Output from calculations with Decimal should be identical across all @@ -211,8 +212,8 @@ objects:: -3.217160342717258261933904529E-7 """ result = [] - g = tokenize(BytesIO(s.encode('utf-8')).readline) # tokenize the string - for toknum, tokval, _, _, _ in g: + g = tokenize(BytesIO(s.encode('utf-8')).readline) # tokenize the string + for toknum, tokval, _, _, _ in g: if toknum == NUMBER and '.' in tokval: # replace NUMBER tokens result.extend([ (NAME, 'Decimal'), diff --git a/Doc/library/traceback.rst b/Doc/library/traceback.rst index 15fbedcf33..6ec9ada530 100644 --- a/Doc/library/traceback.rst +++ b/Doc/library/traceback.rst @@ -4,6 +4,9 @@ .. module:: traceback :synopsis: Print or retrieve a stack traceback. +**Source code:** :source:`Lib/traceback.py` + +-------------- This module provides a standard interface to extract, format and print stack traces of Python programs. It exactly mimics the behavior of the Python @@ -20,27 +23,33 @@ the :data:`sys.last_traceback` variable and returned as the third item from The module defines the following functions: -.. function:: print_tb(traceback, limit=None, file=None) +.. function:: print_tb(tb, limit=None, file=None) + + Print up to *limit* stack trace entries from traceback object *tb* (starting + from the caller's frame) if *limit* is positive. Otherwise, print the last + ``abs(limit)`` entries. If *limit* is omitted or ``None``, all entries are + printed. If *file* is omitted or ``None``, the output goes to + ``sys.stderr``; otherwise it should be an open file or file-like object to + receive the output. - Print up to *limit* stack trace entries from *traceback*. If *limit* is omitted - or ``None``, all entries are printed. If *file* is omitted or ``None``, the - output goes to ``sys.stderr``; otherwise it should be an open file or file-like - object to receive the output. + .. versionchanged:: 3.5 + Added negative *limit* support. -.. function:: print_exception(type, value, traceback, limit=None, file=None, chain=True) +.. function:: print_exception(etype, value, tb, limit=None, file=None, chain=True) - Print exception information and up to *limit* stack trace entries from - *traceback* to *file*. This differs from :func:`print_tb` in the following + Print exception information and stack trace entries from traceback object + *tb* to *file*. This differs from :func:`print_tb` in the following ways: - * if *traceback* is not ``None``, it prints a header ``Traceback (most recent + * if *tb* is not ``None``, it prints a header ``Traceback (most recent call last):`` - * it prints the exception *type* and *value* after the stack trace - * if *type* is :exc:`SyntaxError` and *value* has the appropriate format, it + * it prints the exception *etype* and *value* after the stack trace + * if *etype* is :exc:`SyntaxError` and *value* has the appropriate format, it prints the line where the syntax error occurred with a caret indicating the approximate position of the error. + The optional *limit* argument has the same meaning as for :func:`print_tb`. If *chain* is true (the default), then chained exceptions (the :attr:`__cause__` or :attr:`__context__` attributes of the exception) will be printed as well, like the interpreter itself does when printing an unhandled @@ -49,33 +58,41 @@ The module defines the following functions: .. function:: print_exc(limit=None, file=None, chain=True) - This is a shorthand for ``print_exception(*sys.exc_info())``. + This is a shorthand for ``print_exception(*sys.exc_info(), limit, file, + chain)``. .. function:: print_last(limit=None, file=None, chain=True) This is a shorthand for ``print_exception(sys.last_type, sys.last_value, - sys.last_traceback, limit, file)``. In general it will work only after - an exception has reached an interactive prompt (see :data:`sys.last_type`). + sys.last_traceback, limit, file, chain)``. In general it will work only + after an exception has reached an interactive prompt (see + :data:`sys.last_type`). .. function:: print_stack(f=None, limit=None, file=None) - This function prints a stack trace from its invocation point. The optional *f* - argument can be used to specify an alternate stack frame to start. The optional - *limit* and *file* arguments have the same meaning as for - :func:`print_exception`. + Print up to *limit* stack trace entries (starting from the invocation + point) if *limit* is positive. Otherwise, print the last ``abs(limit)`` + entries. If *limit* is omitted or ``None``, all entries are printed. + The optional *f* argument can be used to specify an alternate stack frame + to start. The optional *file* argument has the same meaning as for + :func:`print_tb`. + .. versionchanged:: 3.5 + Added negative *limit* support. -.. function:: extract_tb(traceback, limit=None) - Return a list of up to *limit* "pre-processed" stack trace entries extracted - from the traceback object *traceback*. It is useful for alternate formatting of - stack traces. If *limit* is omitted or ``None``, all entries are extracted. A - "pre-processed" stack trace entry is a 4-tuple (*filename*, *line number*, - *function name*, *text*) representing the information that is usually printed - for a stack trace. The *text* is a string with leading and trailing whitespace - stripped; if the source is not available it is ``None``. +.. function:: extract_tb(tb, limit=None) + + Return a list of "pre-processed" stack trace entries extracted from the + traceback object *tb*. It is useful for alternate formatting of + stack traces. The optional *limit* argument has the same meaning as for + :func:`print_tb`. A "pre-processed" stack trace entry is a 4-tuple + (*filename*, *line number*, *function name*, *text*) representing the + information that is usually printed for a stack trace. The *text* is a + string with leading and trailing whitespace stripped; if the source is + not available it is ``None``. .. function:: extract_stack(f=None, limit=None) @@ -85,39 +102,40 @@ The module defines the following functions: arguments have the same meaning as for :func:`print_stack`. -.. function:: format_list(list) +.. function:: format_list(extracted_list) Given a list of tuples as returned by :func:`extract_tb` or - :func:`extract_stack`, return a list of strings ready for printing. Each string - in the resulting list corresponds to the item with the same index in the - argument list. Each string ends in a newline; the strings may contain internal - newlines as well, for those items whose source text line is not ``None``. + :func:`extract_stack`, return a list of strings ready for printing. Each + string in the resulting list corresponds to the item with the same index in + the argument list. Each string ends in a newline; the strings may contain + internal newlines as well, for those items whose source text line is not + ``None``. -.. function:: format_exception_only(type, value) +.. function:: format_exception_only(etype, value) - Format the exception part of a traceback. The arguments are the exception type - and value such as given by ``sys.last_type`` and ``sys.last_value``. The return - value is a list of strings, each ending in a newline. Normally, the list - contains a single string; however, for :exc:`SyntaxError` exceptions, it - contains several lines that (when printed) display detailed information about - where the syntax error occurred. The message indicating which exception - occurred is the always last string in the list. + Format the exception part of a traceback. The arguments are the exception + type and value such as given by ``sys.last_type`` and ``sys.last_value``. + The return value is a list of strings, each ending in a newline. Normally, + the list contains a single string; however, for :exc:`SyntaxError` + exceptions, it contains several lines that (when printed) display detailed + information about where the syntax error occurred. The message indicating + which exception occurred is the always last string in the list. -.. function:: format_exception(type, value, tb, limit=None, chain=True) +.. function:: format_exception(etype, value, tb, limit=None, chain=True) Format a stack trace and the exception information. The arguments have the same meaning as the corresponding arguments to :func:`print_exception`. The - return value is a list of strings, each ending in a newline and some containing - internal newlines. When these lines are concatenated and printed, exactly the - same text is printed as does :func:`print_exception`. + return value is a list of strings, each ending in a newline and some + containing internal newlines. When these lines are concatenated and printed, + exactly the same text is printed as does :func:`print_exception`. .. function:: format_exc(limit=None, chain=True) - This is like ``print_exc(limit)`` but returns a string instead of printing to a - file. + This is like ``print_exc(limit)`` but returns a string instead of printing to + a file. .. function:: format_tb(tb, limit=None) @@ -136,6 +154,162 @@ The module defines the following functions: .. versionadded:: 3.4 +.. function:: walk_stack(f) + + Walk a stack following ``f.f_back`` from the given frame, yielding the frame + and line number for each frame. If *f* is ``None``, the current stack is + used. This helper is used with :meth:`StackSummary.extract`. + + .. versionadded:: 3.5 + +.. function:: walk_tb(tb) + + Walk a traceback following ``tb_next`` yielding the frame and line number + for each frame. This helper is used with :meth:`StackSummary.extract`. + + .. versionadded:: 3.5 + +The module also defines the following classes: + +:class:`TracebackException` Objects +----------------------------------- + +.. versionadded:: 3.5 + +:class:`TracebackException` objects are created from actual exceptions to +capture data for later printing in a lightweight fashion. + +.. class:: TracebackException(exc_type, exc_value, exc_traceback, *, limit=None, lookup_lines=True, capture_locals=False) + + Capture an exception for later rendering. *limit*, *lookup_lines* and + *capture_locals* are as for the :class:`StackSummary` class. + + Note that when locals are captured, they are also shown in the traceback. + + .. attribute:: __cause__ + + A :class:`TracebackException` of the original ``__cause__``. + + .. attribute:: __context__ + + A :class:`TracebackException` of the original ``__context__``. + + .. attribute:: __suppress_context__ + + The ``__suppress_context__`` value from the original exception. + + .. attribute:: stack + + A :class:`StackSummary` representing the traceback. + + .. attribute:: exc_type + + The class of the original traceback. + + .. attribute:: filename + + For syntax errors - the file name where the error occurred. + + .. attribute:: lineno + + For syntax errors - the line number where the error occurred. + + .. attribute:: text + + For syntax errors - the text where the error occurred. + + .. attribute:: offset + + For syntax errors - the offset into the text where the error occurred. + + .. attribute:: msg + + For syntax errors - the compiler error message. + + .. classmethod:: from_exception(exc, *, limit=None, lookup_lines=True, capture_locals=False) + + Capture an exception for later rendering. *limit*, *lookup_lines* and + *capture_locals* are as for the :class:`StackSummary` class. + + Note that when locals are captured, they are also shown in the traceback. + + .. method:: format(*, chain=True) + + Format the exception. + + If *chain* is not ``True``, ``__cause__`` and ``__context__`` will not + be formatted. + + The return value is a generator of strings, each ending in a newline and + some containing internal newlines. :func:`~traceback.print_exception` + is a wrapper around this method which just prints the lines to a file. + + The message indicating which exception occurred is always the last + string in the output. + + .. method:: format_exception_only() + + Format the exception part of the traceback. + + The return value is a generator of strings, each ending in a newline. + + Normally, the generator emits a single string; however, for + :exc:`SyntaxError` exceptions, it emits several lines that (when + printed) display detailed information about where the syntax + error occurred. + + The message indicating which exception occurred is always the last + string in the output. + + +:class:`StackSummary` Objects +----------------------------- + +.. versionadded:: 3.5 + +:class:`StackSummary` objects represent a call stack ready for formatting. + +.. class:: StackSummary + + .. classmethod:: extract(frame_gen, *, limit=None, lookup_lines=True, capture_locals=False) + + Construct a :class:`StackSummary` object from a frame generator (such as + is returned by :func:`~traceback.walk_stack` or + :func:`~traceback.walk_tb`). + + If *limit* is supplied, only this many frames are taken from *frame_gen*. + If *lookup_lines* is ``False``, the returned :class:`FrameSummary` + objects will not have read their lines in yet, making the cost of + creating the :class:`StackSummary` cheaper (which may be valuable if it + may not actually get formatted). If *capture_locals* is ``True`` the + local variables in each :class:`FrameSummary` are captured as object + representations. + + .. classmethod:: from_list(a_list) + + Construct a :class:`StackSummary` object from a supplied old-style list + of tuples. Each tuple should be a 4-tuple with filename, lineno, name, + line as the elements. + + +:class:`FrameSummary` Objects +----------------------------- + +.. versionadded:: 3.5 + +:class:`FrameSummary` objects represent a single frame in a traceback. + +.. class:: FrameSummary(filename, lineno, name, lookup_line=True, locals=None, line=None) + + Represent a single frame in the traceback or stack that is being formatted + or printed. It may optionally have a stringified version of the frames + locals included in it. If *lookup_line* is ``False``, the source code is not + looked up until the :class:`FrameSummary` has the :attr:`~FrameSummary.line` + attribute accessed (which also happens when casting it to a tuple). + :attr:`~FrameSummary.line` may be directly provided, and will prevent line + lookups happening at all. *locals* is an optional local variable + dictionary, and if supplied the variable representations are stored in the + summary for later display. .. _traceback-example: diff --git a/Doc/library/tracemalloc.rst b/Doc/library/tracemalloc.rst index cd62e556fb..3a0b1e0797 100644 --- a/Doc/library/tracemalloc.rst +++ b/Doc/library/tracemalloc.rst @@ -6,6 +6,10 @@ .. versionadded:: 3.4 +**Source code:** :source:`Lib/tracemalloc.py` + +-------------- + The tracemalloc module is a debug tool to trace memory blocks allocated by Python. It provides the following information: @@ -363,7 +367,7 @@ Filter Filter on traces of memory blocks. See the :func:`fnmatch.fnmatch` function for the syntax of - *filename_pattern*. The ``'.pyc'`` and ``'.pyo'`` file extensions are + *filename_pattern*. The ``'.pyc'`` file extension is replaced with ``'.py'``. Examples: @@ -374,6 +378,10 @@ Filter :mod:`tracemalloc` module * ``Filter(False, "<unknown>")`` excludes empty tracebacks + + .. versionchanged:: 3.5 + The ``'.pyo'`` file extension is no longer replaced with ``'.py'``. + .. attribute:: inclusive If *inclusive* is ``True`` (include), only trace memory blocks allocated @@ -616,7 +624,7 @@ Traceback *limit* is set, only format the *limit* most recent frames. Similar to the :func:`traceback.format_tb` function, except that - :meth:`format` does not include newlines. + :meth:`.format` does not include newlines. Example:: @@ -631,4 +639,3 @@ Traceback obj = Object() File "test.py", line 12 tb = tracemalloc.get_object_traceback(f()) - diff --git a/Doc/library/tty.rst b/Doc/library/tty.rst index d13c6f92d3..b30bc3c7ac 100644 --- a/Doc/library/tty.rst +++ b/Doc/library/tty.rst @@ -4,9 +4,13 @@ .. module:: tty :platform: Unix :synopsis: Utility functions that perform common terminal control operations. + .. moduleauthor:: Steen Lumholt .. sectionauthor:: Moshe Zadka <moshez@zadka.site.co.il> +**Source code:** :source:`Lib/tty.py` + +-------------- The :mod:`tty` module defines functions for putting the tty into cbreak and raw modes. diff --git a/Doc/library/tulip_coro.dia b/Doc/library/tulip_coro.dia Binary files differindex eccf4fa7f5..70a33e3c00 100644 --- a/Doc/library/tulip_coro.dia +++ b/Doc/library/tulip_coro.dia diff --git a/Doc/library/tulip_coro.png b/Doc/library/tulip_coro.png Binary files differindex 65b6951550..36ced8ddbf 100644 --- a/Doc/library/tulip_coro.png +++ b/Doc/library/tulip_coro.png diff --git a/Doc/library/turtle.rst b/Doc/library/turtle.rst index efe5c54edf..e4a82eaada 100644 --- a/Doc/library/turtle.rst +++ b/Doc/library/turtle.rst @@ -4,13 +4,18 @@ .. module:: turtle :synopsis: An educational framework for simple graphics applications + .. sectionauthor:: Gregor Lingl <gregor.lingl@aon.at> +**Source code:** :source:`Lib/turtle.py` + .. testsetup:: default from turtle import * turtle = Turtle() +-------------- + Introduction ============ @@ -1879,7 +1884,7 @@ Settings and special methods >>> cv = screen.getcanvas() >>> cv - <turtle.ScrolledCanvas object at ...> + <turtle.ScrolledCanvas object ...> .. function:: getshapes() @@ -2351,6 +2356,9 @@ The demo scripts are: | | pairwise in opposite | shapesize, tilt, | | | direction | get_shapepoly, update | +----------------+------------------------------+-----------------------+ +| sorting_animate| visual demonstration of | simple alignment, | +| | different sorting methods | randomization | ++----------------+------------------------------+-----------------------+ | tree | a (graphical) breadth | :func:`clone` | | | first tree (using generators)| | +----------------+------------------------------+-----------------------+ diff --git a/Doc/library/types.rst b/Doc/library/types.rst index abdb939d1a..118bc4c29d 100644 --- a/Doc/library/types.rst +++ b/Doc/library/types.rst @@ -86,8 +86,16 @@ Standard names are defined for the following types: .. data:: GeneratorType - The type of :term:`generator`-iterator objects, produced by calling a - generator function. + The type of :term:`generator`-iterator objects, created by + generator functions. + + +.. data:: CoroutineType + + The type of :term:`coroutine` objects, created by + :keyword:`async def` functions. + + .. versionadded:: 3.5 .. data:: CodeType @@ -115,6 +123,10 @@ Standard names are defined for the following types: The type of :term:`modules <module>`. Constructor takes the name of the module to be created and optionally its :term:`docstring`. + .. note:: + Use :func:`importlib.util.module_from_spec` to create a new module if you + wish to set the various import-controlled attributes. + .. attribute:: __doc__ The :term:`docstring` of the module. Defaults to ``None``. @@ -240,10 +252,12 @@ Additional Utility Classes and Functions class SimpleNamespace: def __init__(self, **kwargs): self.__dict__.update(kwargs) + def __repr__(self): keys = sorted(self.__dict__) items = ("{}={!r}".format(k, self.__dict__[k]) for k in keys) return "{}({})".format(type(self).__name__, ", ".join(items)) + def __eq__(self, other): return self.__dict__ == other.__dict__ @@ -267,3 +281,25 @@ Additional Utility Classes and Functions attributes on the class with the same name (see Enum for an example). .. versionadded:: 3.4 + + +Coroutine Utility Functions +--------------------------- + +.. function:: coroutine(gen_func) + + This function transforms a :term:`generator` function into a + :term:`coroutine function` which returns a generator-based coroutine. + The generator-based coroutine is still a :term:`generator iterator`, + but is also considered to be a :term:`coroutine` object and is + :term:`awaitable`. However, it may not necessarily implement + the :meth:`__await__` method. + + If *gen_func* is a generator function, it will be modified in-place. + + If *gen_func* is not a generator function, it will be wrapped. If it + returns an instance of :class:`collections.abc.Generator`, the instance + will be wrapped in an *awaitable* proxy object. All other types + of objects will be returned as is. + + .. versionadded:: 3.5 diff --git a/Doc/library/typing.rst b/Doc/library/typing.rst new file mode 100644 index 0000000000..d797aeca5a --- /dev/null +++ b/Doc/library/typing.rst @@ -0,0 +1,628 @@ +:mod:`typing` --- Support for type hints +======================================== + +.. module:: typing + :synopsis: Support for type hints (see PEP 484). + +.. versionadded:: 3.5 + +**Source code:** :source:`Lib/typing.py` + +-------------- + +This module supports type hints as specified by :pep:`484`. The most +fundamental support consists of the type :class:`Any`, :class:`Union`, +:class:`Tuple`, :class:`Callable`, :class:`TypeVar`, and +:class:`Generic`. For full specification please see :pep:`484`. For +a simplified introduction to type hints see :pep:`483`. + + +The function below takes and returns a string and is annotated as follows:: + + def greeting(name: str) -> str: + return 'Hello ' + name + +In the function ``greeting``, the argument ``name`` is expected to be of type +:class:`str` and the return type :class:`str`. Subtypes are accepted as +arguments. + +Type aliases +------------ + +A type alias is defined by assigning the type to the alias. In this example, +``Vector`` and ``List[float]`` will be treated as interchangeable synonyms:: + + from typing import List + Vector = List[float] + + def scale(scalar: float, vector: Vector) -> Vector: + return [scalar * num for num in vector] + + # typechecks; a list of floats qualifies as a Vector. + new_vector = scale(2.0, [1.0, -4.2, 5.4]) + +Type aliases are useful for simplifying complex type signatures. For example:: + + from typing import Dict, Tuple, List + + ConnectionOptions = Dict[str, str] + Address = Tuple[str, int] + Server = Tuple[Address, ConnectionOptions] + + def broadcast_message(message: str, servers: List[Server]) -> None: + ... + + # The static type checker will treat the previous type signature as + # being exactly equivalent to this one. + def broadcast_message( + message: str, + servers: List[Tuple[Tuple[str, int], Dict[str, str]]]) -> None: + ... + +NewType +------- + +Use the ``NewType`` helper function to create distinct types:: + + from typing import NewType + + UserId = NewType('UserId', int) + some_id = UserId(524313) + +The static type checker will treat the new type as if it were a subclass +of the original type. This is useful in helping catch logical errors:: + + def get_user_name(user_id: UserId) -> str: + ... + + # typechecks + user_a = get_user_name(UserId(42351)) + + # does not typecheck; an int is not a UserId + user_b = get_user_name(-1) + +You may still perform all ``int`` operations on a variable of type ``UserId``, +but the result will always be of type ``int``. This lets you pass in a +``UserId`` wherever an ``int`` might be expected, but will prevent you from +accidentally creating a ``UserId`` in an invalid way:: + + # 'output' is of type 'int', not 'UserId' + output = UserId(23413) + UserId(54341) + +Note that these checks are enforced only by the static type checker. At runtime +the statement ``Derived = NewType('Derived', Base)`` will make ``Derived`` a +function that immediately returns whatever parameter you pass it. That means +the expression ``Derived(some_value)`` does not create a new class or introduce +any overhead beyond that of a regular function call. + +More precisely, the expression ``some_value is Derived(some_value)`` is always +true at runtime. + +This also means that it is not possible to create a subtype of ``Derived`` +since it is an identity function at runtime, not an actual type. Similarly, it +is not possible to create another ``NewType`` based on a ``Derived`` type:: + + from typing import NewType + + UserId = NewType('UserId', int) + + # Fails at runtime and does not typecheck + class AdminUserId(UserId): pass + + # Also does not typecheck + ProUserId = NewType('ProUserId', UserId) + +See :pep:`484` for more details. + +.. note:: + + Recall that the use of a type alias declares two types to be *equivalent* to + one another. Doing ``Alias = Original`` will make the static type checker + treat ``Alias`` as being *exactly equivalent* to ``Original`` in all cases. + This is useful when you want to simplify complex type signatures. + + In contrast, ``NewType`` declares one type to be a *subtype* of another. + Doing ``Derived = NewType('Derived', Original)`` will make the static type + checker treat ``Derived`` as a *subclass* of ``Original``, which means a + value of type ``Original`` cannot be used in places where a value of type + ``Derived`` is expected. This is useful when you want to prevent logic + errors with minimal runtime cost. + +Callable +-------- + +Frameworks expecting callback functions of specific signatures might be +type hinted using ``Callable[[Arg1Type, Arg2Type], ReturnType]``. + +For example:: + + from typing import Callable + + def feeder(get_next_item: Callable[[], str]) -> None: + # Body + + def async_query(on_success: Callable[[int], None], + on_error: Callable[[int, Exception], None]) -> None: + # Body + +It is possible to declare the return type of a callable without specifying +the call signature by substituting a literal ellipsis +for the list of arguments in the type hint: ``Callable[..., ReturnType]``. +``None`` as a type hint is a special case and is replaced by ``type(None)``. + +Generics +-------- + +Since type information about objects kept in containers cannot be statically +inferred in a generic way, abstract base classes have been extended to support +subscription to denote expected types for container elements. + +:: + + from typing import Mapping, Sequence + + def notify_by_email(employees: Sequence[Employee], + overrides: Mapping[str, str]) -> None: ... + +Generics can be parametrized by using a new factory available in typing +called :class:`TypeVar`. + +:: + + from typing import Sequence, TypeVar + + T = TypeVar('T') # Declare type variable + + def first(l: Sequence[T]) -> T: # Generic function + return l[0] + + +User-defined generic types +-------------------------- + +A user-defined class can be defined as a generic class. + +:: + + from typing import TypeVar, Generic + from logging import Logger + + T = TypeVar('T') + + class LoggedVar(Generic[T]): + def __init__(self, value: T, name: str, logger: Logger) -> None: + self.name = name + self.logger = logger + self.value = value + + def set(self, new: T) -> None: + self.log('Set ' + repr(self.value)) + self.value = new + + def get(self) -> T: + self.log('Get ' + repr(self.value)) + return self.value + + def log(self, message: str) -> None: + self.logger.info('{}: {}'.format(self.name, message)) + +``Generic[T]`` as a base class defines that the class ``LoggedVar`` takes a +single type parameter ``T`` . This also makes ``T`` valid as a type within the +class body. + +The :class:`Generic` base class uses a metaclass that defines +:meth:`__getitem__` so that ``LoggedVar[t]`` is valid as a type:: + + from typing import Iterable + + def zero_all_vars(vars: Iterable[LoggedVar[int]]) -> None: + for var in vars: + var.set(0) + +A generic type can have any number of type variables, and type variables may +be constrained:: + + from typing import TypeVar, Generic + ... + + T = TypeVar('T') + S = TypeVar('S', int, str) + + class StrangePair(Generic[T, S]): + ... + +Each type variable argument to :class:`Generic` must be distinct. +This is thus invalid:: + + from typing import TypeVar, Generic + ... + + T = TypeVar('T') + + class Pair(Generic[T, T]): # INVALID + ... + +You can use multiple inheritance with :class:`Generic`:: + + from typing import TypeVar, Generic, Sized + + T = TypeVar('T') + + class LinkedList(Sized, Generic[T]): + ... + +When inheriting from generic classes, some type variables could be fixed:: + + from typing import TypeVar, Mapping + + T = TypeVar('T') + + class MyDict(Mapping[str, T]): + ... + +In this case ``MyDict`` has a single parameter, ``T``. + +Subclassing a generic class without specifying type parameters assumes +:class:`Any` for each position. In the following example, ``MyIterable`` is +not generic but implicitly inherits from ``Iterable[Any]``:: + + from typing import Iterable + + class MyIterable(Iterable): # Same as Iterable[Any] + +The metaclass used by :class:`Generic` is a subclass of :class:`abc.ABCMeta`. +A generic class can be an ABC by including abstract methods or properties, +and generic classes can also have ABCs as base classes without a metaclass +conflict. Generic metaclasses are not supported. + + +The :class:`Any` type +--------------------- + +A special kind of type is :class:`Any`. Every type is a subtype of +:class:`Any`. This is also true for the builtin type object. However, to the +static type checker these are completely different. + +When the type of a value is :class:`object`, the type checker will reject +almost all operations on it, and assigning it to a variable (or using it as a +return value) of a more specialized type is a type error. On the other hand, +when a value has type :class:`Any`, the type checker will allow all operations +on it, and a value of type :class:`Any` can be assigned to a variable (or used +as a return value) of a more constrained type. + + +Classes, functions, and decorators +---------------------------------- + +The module defines the following classes, functions and decorators: + +.. class:: Any + + Special type indicating an unconstrained type. + + * Any object is an instance of :class:`Any`. + * Any class is a subclass of :class:`Any`. + * As a special case, :class:`Any` and :class:`object` are subclasses of + each other. + +.. class:: TypeVar + + Type variable. + + Usage:: + + T = TypeVar('T') # Can be anything + A = TypeVar('A', str, bytes) # Must be str or bytes + + Type variables exist primarily for the benefit of static type + checkers. They serve as the parameters for generic types as well + as for generic function definitions. See class Generic for more + information on generic types. Generic functions work as follows:: + + def repeat(x: T, n: int) -> Sequence[T]: + """Return a list containing n references to x.""" + return [x]*n + + def longest(x: A, y: A) -> A: + """Return the longest of two strings.""" + return x if len(x) >= len(y) else y + + The latter example's signature is essentially the overloading + of ``(str, str) -> str`` and ``(bytes, bytes) -> bytes``. Also note + that if the arguments are instances of some subclass of :class:`str`, + the return type is still plain :class:`str`. + + At runtime, ``isinstance(x, T)`` will raise :exc:`TypeError`. In general, + :func:`isinstance` and :func:`issubclass` should not be used with types. + + Type variables may be marked covariant or contravariant by passing + ``covariant=True`` or ``contravariant=True``. See :pep:`484` for more + details. By default type variables are invariant. Alternatively, + a type variable may specify an upper bound using ``bound=<type>``. + This means that an actual type substituted (explicitly or implicitly) + for the type variable must be a subclass of the boundary type, + see :pep:`484`. + +.. class:: Union + + Union type; ``Union[X, Y]`` means either X or Y. + + To define a union, use e.g. ``Union[int, str]``. Details: + + * The arguments must be types and there must be at least one. + + * Unions of unions are flattened, e.g.:: + + Union[Union[int, str], float] == Union[int, str, float] + + * Unions of a single argument vanish, e.g.:: + + Union[int] == int # The constructor actually returns int + + * Redundant arguments are skipped, e.g.:: + + Union[int, str, int] == Union[int, str] + + * When comparing unions, the argument order is ignored, e.g.:: + + Union[int, str] == Union[str, int] + + * If :class:`Any` is present it is the sole survivor, e.g.:: + + Union[int, Any] == Any + + * You cannot subclass or instantiate a union. + + * You cannot write ``Union[X][Y]``. + + * You can use ``Optional[X]`` as a shorthand for ``Union[X, None]``. + +.. class:: Optional + + Optional type. + + ``Optional[X]`` is equivalent to ``Union[X, type(None)]``. + + Note that this is not the same concept as an optional argument, + which is one that has a default. An optional argument with a + default needn't use the ``Optional`` qualifier on its type + annotation (although it is inferred if the default is ``None``). + A mandatory argument may still have an ``Optional`` type if an + explicit value of ``None`` is allowed. + +.. class:: Tuple + + Tuple type; ``Tuple[X, Y]`` is the type of a tuple of two items + with the first item of type X and the second of type Y. + + Example: ``Tuple[T1, T2]`` is a tuple of two elements corresponding + to type variables T1 and T2. ``Tuple[int, float, str]`` is a tuple + of an int, a float and a string. + + To specify a variable-length tuple of homogeneous type, + use literal ellipsis, e.g. ``Tuple[int, ...]``. + +.. class:: Callable + + Callable type; ``Callable[[int], str]`` is a function of (int) -> str. + + The subscription syntax must always be used with exactly two + values: the argument list and the return type. The argument list + must be a list of types; the return type must be a single type. + + There is no syntax to indicate optional or keyword arguments, + such function types are rarely used as callback types. + ``Callable[..., ReturnType]`` could be used to type hint a callable + taking any number of arguments and returning ``ReturnType``. + A plain :class:`Callable` is equivalent to ``Callable[..., Any]``. + +.. class:: Generic + + Abstract base class for generic types. + + A generic type is typically declared by inheriting from an + instantiation of this class with one or more type variables. + For example, a generic mapping type might be defined as:: + + class Mapping(Generic[KT, VT]): + def __getitem__(self, key: KT) -> VT: + ... + # Etc. + + This class can then be used as follows:: + + X = TypeVar('X') + Y = TypeVar('Y') + + def lookup_name(mapping: Mapping[X, Y], key: X, default: Y) -> Y: + try: + return mapping[key] + except KeyError: + return default + +.. class:: Iterable(Generic[T_co]) + + A generic version of the :class:`collections.abc.Iterable`. + +.. class:: Iterator(Iterable[T_co]) + + A generic version of the :class:`collections.abc.Iterator`. + +.. class:: SupportsInt + + An ABC with one abstract method ``__int__``. + +.. class:: SupportsFloat + + An ABC with one abstract method ``__float__``. + +.. class:: SupportsAbs + + An ABC with one abstract method ``__abs__`` that is covariant + in its return type. + +.. class:: SupportsRound + + An ABC with one abstract method ``__round__`` + that is covariant in its return type. + +.. class:: Reversible + + An ABC with one abstract method ``__reversed__`` returning + an ``Iterator[T_co]``. + +.. class:: Container(Generic[T_co]) + + A generic version of :class:`collections.abc.Container`. + +.. class:: AbstractSet(Sized, Iterable[T_co], Container[T_co]) + + A generic version of :class:`collections.abc.Set`. + +.. class:: MutableSet(AbstractSet[T]) + + A generic version of :class:`collections.abc.MutableSet`. + +.. class:: Mapping(Sized, Iterable[KT], Container[KT], Generic[VT_co]) + + A generic version of :class:`collections.abc.Mapping`. + +.. class:: MutableMapping(Mapping[KT, VT]) + + A generic version of :class:`collections.abc.MutableMapping`. + +.. class:: Sequence(Sized, Iterable[T_co], Container[T_co]) + + A generic version of :class:`collections.abc.Sequence`. + +.. class:: MutableSequence(Sequence[T]) + + A generic version of :class:`collections.abc.MutableSequence`. + +.. class:: ByteString(Sequence[int]) + + A generic version of :class:`collections.abc.ByteString`. + + This type represents the types :class:`bytes`, :class:`bytearray`, + and :class:`memoryview`. + + As a shorthand for this type, :class:`bytes` can be used to + annotate arguments of any of the types mentioned above. + +.. class:: List(list, MutableSequence[T]) + + Generic version of :class:`list`. + Useful for annotating return types. To annotate arguments it is preferred + to use abstract collection types such as :class:`Mapping`, :class:`Sequence`, + or :class:`AbstractSet`. + + This type may be used as follows:: + + T = TypeVar('T', int, float) + + def vec2(x: T, y: T) -> List[T]: + return [x, y] + + def keep_positives(vector: Sequence[T]) -> List[T]: + return [item for item in vector if item > 0] + +.. class:: Set(set, MutableSet[T]) + + A generic version of :class:`builtins.set <set>`. + +.. class:: MappingView(Sized, Iterable[T_co]) + + A generic version of :class:`collections.abc.MappingView`. + +.. class:: KeysView(MappingView[KT_co], AbstractSet[KT_co]) + + A generic version of :class:`collections.abc.KeysView`. + +.. class:: ItemsView(MappingView, Generic[KT_co, VT_co]) + + A generic version of :class:`collections.abc.ItemsView`. + +.. class:: ValuesView(MappingView[VT_co]) + + A generic version of :class:`collections.abc.ValuesView`. + +.. class:: Dict(dict, MutableMapping[KT, VT]) + + A generic version of :class:`dict`. + The usage of this type is as follows:: + + def get_position_in_index(word_list: Dict[str, int], word: str) -> int: + return word_list[word] + +.. class:: Generator(Iterator[T_co], Generic[T_co, T_contra, V_co]) + +.. class:: io + + Wrapper namespace for I/O stream types. + + This defines the generic type ``IO[AnyStr]`` and aliases ``TextIO`` + and ``BinaryIO`` for respectively ``IO[str]`` and ``IO[bytes]``. + These representing the types of I/O streams such as returned by + :func:`open`. + +.. class:: re + + Wrapper namespace for regular expression matching types. + + This defines the type aliases ``Pattern`` and ``Match`` which + correspond to the return types from :func:`re.compile` and + :func:`re.match`. These types (and the corresponding functions) + are generic in ``AnyStr`` and can be made specific by writing + ``Pattern[str]``, ``Pattern[bytes]``, ``Match[str]``, or + ``Match[bytes]``. + +.. function:: NamedTuple(typename, fields) + + Typed version of namedtuple. + + Usage:: + + Employee = typing.NamedTuple('Employee', [('name', str), ('id', int)]) + + This is equivalent to:: + + Employee = collections.namedtuple('Employee', ['name', 'id']) + + The resulting class has one extra attribute: _field_types, + giving a dict mapping field names to types. (The field names + are in the _fields attribute, which is part of the namedtuple + API.) + +.. function:: cast(typ, val) + + Cast a value to a type. + + This returns the value unchanged. To the type checker this + signals that the return value has the designated type, but at + runtime we intentionally don't check anything (we want this + to be as fast as possible). + +.. function:: get_type_hints(obj) + + Return type hints for a function or method object. + + This is often the same as ``obj.__annotations__``, but it handles + forward references encoded as string literals, and if necessary + adds ``Optional[t]`` if a default value equal to None is set. + +.. decorator:: no_type_check(arg) + + Decorator to indicate that annotations are not type hints. + + The argument must be a class or function; if it is a class, it + applies recursively to all methods defined in that class (but not + to methods defined in its superclasses or subclasses). + + This mutates the function(s) in place. + +.. decorator:: no_type_check_decorator(decorator) + + Decorator to give another decorator the :func:`no_type_check` effect. + + This wraps the decorator with something that wraps the decorated + function in :func:`no_type_check`. diff --git a/Doc/library/unicodedata.rst b/Doc/library/unicodedata.rst index 3b3d3a0a8f..6cd8132f8a 100644 --- a/Doc/library/unicodedata.rst +++ b/Doc/library/unicodedata.rst @@ -3,20 +3,22 @@ .. module:: unicodedata :synopsis: Access the Unicode Database. + .. moduleauthor:: Marc-André Lemburg <mal@lemburg.com> .. sectionauthor:: Marc-André Lemburg <mal@lemburg.com> .. sectionauthor:: Martin v. Löwis <martin@v.loewis.de> - .. index:: single: Unicode single: character pair: Unicode; database +-------------- + This module provides access to the Unicode Character Database (UCD) which defines character properties for all Unicode characters. The data contained in -this database is compiled from the `UCD version 6.3.0 -<http://www.unicode.org/Public/6.3.0/ucd>`_. +this database is compiled from the `UCD version 8.0.0 +<http://www.unicode.org/Public/8.0.0/ucd>`_. The module uses the same names and symbols as defined by Unicode Standard Annex #44, `"Unicode Character Database" @@ -166,6 +168,6 @@ Examples: .. rubric:: Footnotes -.. [#] http://www.unicode.org/Public/6.3.0/ucd/NameAliases.txt +.. [#] http://www.unicode.org/Public/8.0.0/ucd/NameAliases.txt -.. [#] http://www.unicode.org/Public/6.3.0/ucd/NamedSequences.txt +.. [#] http://www.unicode.org/Public/8.0.0/ucd/NamedSequences.txt diff --git a/Doc/library/unittest.mock-examples.rst b/Doc/library/unittest.mock-examples.rst index 055abe0de1..05f33740d7 100644 --- a/Doc/library/unittest.mock-examples.rst +++ b/Doc/library/unittest.mock-examples.rst @@ -359,7 +359,7 @@ The module name can be 'dotted', in the form ``package.module`` if needed: A nice pattern is to actually decorate test methods themselves: - >>> class MyTest(unittest2.TestCase): + >>> class MyTest(unittest.TestCase): ... @patch.object(SomeClass, 'attribute', sentinel.attribute) ... def test_something(self): ... self.assertEqual(SomeClass.attribute, sentinel.attribute) @@ -372,7 +372,7 @@ If you want to patch with a Mock, you can use :func:`patch` with only one argume (or :func:`patch.object` with two arguments). The mock will be created for you and passed into the test function / method: - >>> class MyTest(unittest2.TestCase): + >>> class MyTest(unittest.TestCase): ... @patch.object(SomeClass, 'static_method') ... def test_something(self, mock_method): ... SomeClass.static_method() @@ -382,7 +382,7 @@ passed into the test function / method: You can stack up multiple patch decorators using this pattern: - >>> class MyTest(unittest2.TestCase): + >>> class MyTest(unittest.TestCase): ... @patch('package.module.ClassName1') ... @patch('package.module.ClassName2') ... def test_something(self, MockClass2, MockClass1): @@ -549,7 +549,7 @@ Calls to the date constructor are recorded in the ``mock_date`` attributes An alternative way of dealing with mocking dates, or other builtin classes, is discussed in `this blog entry -<http://www.williamjohnbert.com/2011/07/how-to-unit-testing-in-django-with-mocking-and-patching/>`_. +<https://williambert.online/2011/07/how-to-unit-testing-in-django-with-mocking-and-patching/>`_. Mocking a Generator Method @@ -1010,7 +1010,7 @@ subclass. Sometimes this is inconvenient. For example, `one user <https://code.google.com/p/mock/issues/detail?id=105>`_ is subclassing mock to created a `Twisted adaptor -<http://twistedmatrix.com/documents/11.0.0/api/twisted.python.components.html>`_. +<https://twistedmatrix.com/documents/11.0.0/api/twisted.python.components.html>`_. Having this applied to attributes too actually causes errors. ``Mock`` (in all its flavours) uses a method called ``_get_child_mock`` to create @@ -1251,7 +1251,7 @@ With a bit of tweaking you could have the comparison function raise the :exc:`AssertionError` directly and provide a more useful failure message. As of version 1.5, the Python testing library `PyHamcrest -<https://pypi.python.org/pypi/PyHamcrest>`_ provides similar functionality, +<https://pyhamcrest.readthedocs.org/>`_ provides similar functionality, that may be useful here, in the form of its equality matcher (`hamcrest.library.integration.match_equality -<http://pythonhosted.org/PyHamcrest/integration.html#hamcrest.library.integration.match_equality.match_equality>`_). +<https://pyhamcrest.readthedocs.org/en/release-1.8/integration/#module-hamcrest.library.integration.match_equality>`_). diff --git a/Doc/library/unittest.mock.rst b/Doc/library/unittest.mock.rst index d7d735c5f5..c13f09512c 100644 --- a/Doc/library/unittest.mock.rst +++ b/Doc/library/unittest.mock.rst @@ -4,11 +4,16 @@ .. module:: unittest.mock :synopsis: Mock object library. + .. moduleauthor:: Michael Foord <michael@python.org> .. currentmodule:: unittest.mock .. versionadded:: 3.3 +**Source code:** :source:`Lib/unittest/mock.py` + +-------------- + :mod:`unittest.mock` is a library for testing in Python. It allows you to replace parts of your system under test with mock objects and make assertions about how they have been used. @@ -32,8 +37,6 @@ used by many mocking frameworks. There is a backport of :mod:`unittest.mock` for earlier versions of Python, available as `mock on PyPI <https://pypi.python.org/pypi/mock>`_. -**Source code:** :source:`Lib/unittest/mock.py` - Quick Guide ----------- @@ -198,7 +201,7 @@ a :class:`MagicMock` for you. You can specify an alternative class of :class:`Mo the *new_callable* argument to :func:`patch`. -.. class:: Mock(spec=None, side_effect=None, return_value=DEFAULT, wraps=None, name=None, spec_set=None, **kwargs) +.. class:: Mock(spec=None, side_effect=None, return_value=DEFAULT, wraps=None, name=None, spec_set=None, unsafe=False, **kwargs) Create a new :class:`Mock` object. :class:`Mock` takes several optional arguments that specify the behaviour of the Mock object: @@ -235,6 +238,12 @@ the *new_callable* argument to :func:`patch`. this is a new Mock (created on first access). See the :attr:`return_value` attribute. + * *unsafe*: By default if any attribute starts with *assert* or + *assret* will raise an :exc:`AttributeError`. Passing ``unsafe=True`` + will allow access to these attributes. + + .. versionadded:: 3.5 + * *wraps*: Item for the mock object to wrap. If *wraps* is not None then calling the Mock will pass the call through to the wrapped object (returning the real result). Attribute access on the mock will return a @@ -315,6 +324,20 @@ the *new_callable* argument to :func:`patch`. >>> calls = [call(4), call(2), call(3)] >>> mock.assert_has_calls(calls, any_order=True) + .. method:: assert_not_called() + + Assert the mock was never called. + + >>> m = Mock() + >>> m.hello.assert_not_called() + >>> obj = m.hello() + >>> m.hello.assert_not_called() + Traceback (most recent call last): + ... + AssertionError: Expected 'hello' to not have been called. Called 1 times. + + .. versionadded:: 3.5 + .. method:: reset_mock() @@ -1032,6 +1055,12 @@ patch default because it can be dangerous. With it switched on you can write passing tests against APIs that don't actually exist! + .. note:: + + .. versionchanged:: 3.5 + If you are patching builtins in a module then you don't + need to pass ``create=True``, it will be added by default. + Patch can be used as a :class:`TestCase` class decorator. It works by decorating each test method in the class. This reduces the boilerplate code when your test methods share a common patchings set. :func:`patch` finds @@ -1403,6 +1432,22 @@ It is also possible to stop all patches which have been started by using Stop all active patches. Only stops patches started with ``start``. +.. _patch-builtins: + +patch builtins +~~~~~~~~~~~~~~ +You can patch any builtins within a module. The following example patches +builtin :func:`ord`: + + >>> @patch('__main__.ord') + ... def test(mock_ord): + ... mock_ord.return_value = 101 + ... print(ord('c')) + ... + >>> test() + 101 + + TEST_PREFIX ~~~~~~~~~~~ @@ -1587,7 +1632,7 @@ The full list of supported magic methods is: * Context manager: ``__enter__`` and ``__exit__`` * Unary numeric methods: ``__neg__``, ``__pos__`` and ``__invert__`` * The numeric methods (including right hand and in-place variants): - ``__add__``, ``__sub__``, ``__mul__``, ``__div__``, ``__truediv__``, + ``__add__``, ``__sub__``, ``__mul__``, ``__matmul__``, ``__div__``, ``__truediv__``, ``__floordiv__``, ``__mod__``, ``__divmod__``, ``__lshift__``, ``__rshift__``, ``__and__``, ``__xor__``, ``__or__``, and ``__pow__`` * Numeric conversion methods: ``__complex__``, ``__int__``, ``__float__`` @@ -1655,7 +1700,7 @@ Methods and their defaults: * ``__ge__``: NotImplemented * ``__int__``: 1 * ``__contains__``: False -* ``__len__``: 1 +* ``__len__``: 0 * ``__iter__``: iter([]) * ``__exit__``: False * ``__complex__``: 1j @@ -2006,7 +2051,7 @@ mock_open The mock of :meth:`~io.IOBase.read` changed to consume *read_data* rather than returning it on each call. - .. versionchanged:: 3.4.4 + .. versionchanged:: 3.5 *read_data* is now reset on each call to the *mock*. Using :func:`open` as a context manager is a great way to ensure your file handles @@ -2023,7 +2068,7 @@ Mocking context managers with a :class:`MagicMock` is common enough and fiddly enough that a helper function is useful. >>> m = mock_open() - >>> with patch('__main__.open', m, create=True): + >>> with patch('__main__.open', m): ... with open('foo', 'w') as h: ... h.write('some stuff') ... @@ -2038,7 +2083,7 @@ enough that a helper function is useful. And for reading files: - >>> with patch('__main__.open', mock_open(read_data='bibble'), create=True) as m: + >>> with patch('__main__.open', mock_open(read_data='bibble')) as m: ... with open('foo') as h: ... result = h.read() ... diff --git a/Doc/library/unittest.rst b/Doc/library/unittest.rst index a896fc165a..0fc02c4dbe 100644 --- a/Doc/library/unittest.rst +++ b/Doc/library/unittest.rst @@ -3,11 +3,16 @@ .. module:: unittest :synopsis: Unit testing framework for Python. + .. moduleauthor:: Steve Purcell <stephen_purcell@yahoo.com> .. sectionauthor:: Steve Purcell <stephen_purcell@yahoo.com> .. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org> .. sectionauthor:: Raymond Hettinger <python@rcn.com> +**Source code:** :source:`Lib/unittest/__init__.py` + +-------------- + (If you are already familiar with the basic concepts of testing, you might want to skip to :ref:`the list of assert methods <assert-methods>`.) @@ -67,7 +72,7 @@ test runner a GUI tool for test discovery and execution. This is intended largely for ease of use for those new to unit testing. For production environments it is recommended that tests be driven by a continuous integration system such as - `Buildbot <http://buildbot.net/>`_, `Jenkins <http://jenkins-ci.org/>`_ + `Buildbot <https://buildbot.net/>`_, `Jenkins <https://jenkins.io/>`_ or `Hudson <http://hudson-ci.org/>`_. @@ -86,19 +91,19 @@ Here is a short script to test three string methods:: class TestStringMethods(unittest.TestCase): - def test_upper(self): - self.assertEqual('foo'.upper(), 'FOO') + def test_upper(self): + self.assertEqual('foo'.upper(), 'FOO') - def test_isupper(self): - self.assertTrue('FOO'.isupper()) - self.assertFalse('Foo'.isupper()) + def test_isupper(self): + self.assertTrue('FOO'.isupper()) + self.assertFalse('Foo'.isupper()) - def test_split(self): - s = 'hello world' - self.assertEqual(s.split(), ['hello', 'world']) - # check that s.split fails when the separator is not a string - with self.assertRaises(TypeError): - s.split(2) + def test_split(self): + s = 'hello world' + self.assertEqual(s.split(), ['hello', 'world']) + # check that s.split fails when the separator is not a string + with self.assertRaises(TypeError): + s.split(2) if __name__ == '__main__': unittest.main() @@ -214,9 +219,16 @@ Command-line options Stop the test run on the first error or failure. +.. cmdoption:: --locals + + Show local variables in tracebacks. + .. versionadded:: 3.2 The command-line options ``-b``, ``-c`` and ``-f`` were added. +.. versionadded:: 3.5 + The command-line option ``--locals``. + The command line can also be used for test discovery, for running all of the tests in a project or just a subset. @@ -673,10 +685,12 @@ Test cases Method called immediately after the test method has been called and the result recorded. This is called even if the test method raised an exception, so the implementation in subclasses may need to be particularly - careful about checking internal state. Any exception, other than :exc:`AssertionError` - or :exc:`SkipTest`, raised by this method will be considered an error rather than a - test failure. This method will only be called if the :meth:`setUp` succeeds, - regardless of the outcome of the test method. The default implementation does nothing. + careful about checking internal state. Any exception, other than + :exc:`AssertionError` or :exc:`SkipTest`, raised by this method will be + considered an additional error rather than a test failure (thus increasing + the total number of reported errors). This method will only be called if + the :meth:`setUp` succeeds, regardless of the outcome of the test method. + The default implementation does nothing. .. method:: setUpClass() @@ -755,8 +769,9 @@ Test cases .. _assert-methods: - The :class:`TestCase` class provides a number of methods to check for and - report failures, such as: + The :class:`TestCase` class provides several assert methods to check for and + report failures. The following table lists the most commonly used methods + (see the tables below for more assert methods): +-----------------------------------------+-----------------------------+---------------+ | Method | Checks that | New in | @@ -877,7 +892,7 @@ Test cases - It is also possible to check the production of exceptions, warnings and + It is also possible to check the production of exceptions, warnings, and log messages using the following methods: +---------------------------------------------------------+--------------------------------------+------------+ @@ -1541,6 +1556,20 @@ Loading and running tests :data:`unittest.defaultTestLoader`. Using a subclass or instance, however, allows customization of some configurable properties. + :class:`TestLoader` objects have the following attributes: + + + .. attribute:: errors + + A list of the non-fatal errors encountered while loading tests. Not reset + by the loader at any point. Fatal errors are signalled by the relevant + a method raising an exception to the caller. Non-fatal errors are also + indicated by a synthetic test that will raise the original error when + run. + + .. versionadded:: 3.5 + + :class:`TestLoader` objects have the following methods: @@ -1556,7 +1585,7 @@ Loading and running tests case is created for that method instead. - .. method:: loadTestsFromModule(module) + .. method:: loadTestsFromModule(module, pattern=None) Return a suite of all tests cases contained in the given module. This method searches *module* for classes derived from :class:`TestCase` and @@ -1573,11 +1602,18 @@ Loading and running tests If a module provides a ``load_tests`` function it will be called to load the tests. This allows modules to customize test loading. - This is the `load_tests protocol`_. + This is the `load_tests protocol`_. The *pattern* argument is passed as + the third argument to ``load_tests``. .. versionchanged:: 3.2 Support for ``load_tests`` added. + .. versionchanged:: 3.5 + The undocumented and unofficial *use_load_tests* default argument is + deprecated and ignored, although it is still accepted for backward + compatibility. The method also now accepts a keyword-only argument + *pattern* which is passed to ``load_tests`` as the third argument. + .. method:: loadTestsFromName(name, module=None) @@ -1603,6 +1639,12 @@ Loading and running tests The method optionally resolves *name* relative to the given *module*. + .. versionchanged:: 3.5 + If an :exc:`ImportError` or :exc:`AttributeError` occurs while traversing + *name* then a synthetic test that raises that error when run will be + returned. These errors are included in the errors accumulated by + self.errors. + .. method:: loadTestsFromNames(names, module=None) @@ -1629,18 +1671,22 @@ Loading and running tests the start directory is not the top level directory then the top level directory must be specified separately. - If importing a module fails, for example due to a syntax error, then this - will be recorded as a single error and discovery will continue. If the - import failure is due to :exc:`SkipTest` being raised, it will be recorded - as a skip instead of an error. + If importing a module fails, for example due to a syntax error, then + this will be recorded as a single error and discovery will continue. If + the import failure is due to :exc:`SkipTest` being raised, it will be + recorded as a skip instead of an error. - If a test package name (directory with :file:`__init__.py`) matches the - pattern then the package will be checked for a ``load_tests`` - function. If this exists then it will be called with *loader*, *tests*, - *pattern*. + If a package (a directory containing a file named :file:`__init__.py`) is + found, the package will be checked for a ``load_tests`` function. If this + exists then it will be called + ``package.load_tests(loader, tests, pattern)``. Test discovery takes care + to ensure that a package is only checked for tests once during an + invocation, even if the load_tests function itself calls + ``loader.discover``. - If ``load_tests`` exists then discovery does *not* recurse into the package, - ``load_tests`` is responsible for loading all tests in the package. + If ``load_tests`` exists then discovery does *not* recurse into the + package, ``load_tests`` is responsible for loading all tests in the + package. The pattern is deliberately not stored as a loader attribute so that packages can continue discovery themselves. *top_level_dir* is stored so @@ -1659,6 +1705,11 @@ Loading and running tests the same even if the underlying file system's ordering is not dependent on file name. + .. versionchanged:: 3.5 + Found packages are now checked for ``load_tests`` regardless of + whether their path matches *pattern*, because it is impossible for + a package name to match the default pattern. + The following attributes of a :class:`TestLoader` can be configured either by subclassing or assignment on an instance: @@ -1741,12 +1792,10 @@ Loading and running tests Set to ``True`` when the execution of tests should stop by :meth:`stop`. - .. attribute:: testsRun The total number of tests run so far. - .. attribute:: buffer If set to true, ``sys.stdout`` and ``sys.stderr`` will be buffered in between @@ -1756,7 +1805,6 @@ Loading and running tests .. versionadded:: 3.2 - .. attribute:: failfast If set to true :meth:`stop` will be called on the first failure or error, @@ -1764,6 +1812,11 @@ Loading and running tests .. versionadded:: 3.2 + .. attribute:: tb_locals + + If set to true then local variables will be shown in tracebacks. + + .. versionadded:: 3.5 .. method:: wasSuccessful() @@ -1774,7 +1827,6 @@ Loading and running tests Returns ``False`` if there were any :attr:`unexpectedSuccesses` from tests marked with the :func:`expectedFailure` decorator. - .. method:: stop() This method can be called to signal that the set of tests being run should @@ -1906,12 +1958,14 @@ Loading and running tests .. class:: TextTestRunner(stream=None, descriptions=True, verbosity=1, failfast=False, \ - buffer=False, resultclass=None, warnings=None) + buffer=False, resultclass=None, warnings=None, *, tb_locals=False) A basic test runner implementation that outputs results to a stream. If *stream* is ``None``, the default, :data:`sys.stderr` is used as the output stream. This class has a few configurable parameters, but is essentially very simple. Graphical - applications which run test suites should provide alternate implementations. + applications which run test suites should provide alternate implementations. Such + implementations should accept ``**kwargs`` as the interface to construct runners + changes when features are added to unittest. By default this runner shows :exc:`DeprecationWarning`, :exc:`PendingDeprecationWarning`, :exc:`ResourceWarning` and @@ -1930,6 +1984,9 @@ Loading and running tests The default stream is set to :data:`sys.stderr` at instantiation time rather than import time. + .. versionchanged:: 3.5 + Added the tb_locals parameter. + .. method:: _makeResult() This method returns the instance of ``TestResult`` used by :meth:`run`. @@ -2027,7 +2084,10 @@ test runs or test discovery by implementing a function called ``load_tests``. If a test module defines ``load_tests`` it will be called by :meth:`TestLoader.loadTestsFromModule` with the following arguments:: - load_tests(loader, standard_tests, None) + load_tests(loader, standard_tests, pattern) + +where *pattern* is passed straight through from ``loadTestsFromModule``. It +defaults to ``None``. It should return a :class:`TestSuite`. @@ -2049,21 +2109,12 @@ A typical ``load_tests`` function that loads tests from a specific set of suite.addTests(tests) return suite -If discovery is started, either from the command line or by calling -:meth:`TestLoader.discover`, with a pattern that matches a package -name then the package :file:`__init__.py` will be checked for ``load_tests``. - -.. note:: - - The default pattern is ``'test*.py'``. This matches all Python files - that start with ``'test'`` but *won't* match any test directories. - - A pattern like ``'test*'`` will match test packages as well as - modules. - -If the package :file:`__init__.py` defines ``load_tests`` then it will be -called and discovery not continued into the package. ``load_tests`` -is called with the following arguments:: +If discovery is started in a directory containing a package, either from the +command line or by calling :meth:`TestLoader.discover`, then the package +:file:`__init__.py` will be checked for ``load_tests``. If that function does +not exist, discovery will recurse into the package as though it were just +another directory. Otherwise, discovery of the package's tests will be left up +to ``load_tests`` which is called with the following arguments:: load_tests(loader, standard_tests, pattern) @@ -2082,6 +2133,11 @@ continue (and potentially modify) test discovery. A 'do nothing' standard_tests.addTests(package_tests) return standard_tests +.. versionchanged:: 3.5 + Discovery no longer checks package names for matching *pattern* due to the + impossibility of package names matching the default pattern. + + Class and Module Fixtures ------------------------- diff --git a/Doc/library/urllib.error.rst b/Doc/library/urllib.error.rst index f7f0c9739d..5517b04f5d 100644 --- a/Doc/library/urllib.error.rst +++ b/Doc/library/urllib.error.rst @@ -3,9 +3,13 @@ .. module:: urllib.error :synopsis: Exception classes raised by urllib.request. + .. moduleauthor:: Jeremy Hylton <jeremy@alum.mit.edu> .. sectionauthor:: Senthil Kumaran <orsenthil@gmail.com> +**Source code:** :source:`Lib/urllib/error.py` + +-------------- The :mod:`urllib.error` module defines the exception classes for exceptions raised by :mod:`urllib.request`. The base exception class is :exc:`URLError`. diff --git a/Doc/library/urllib.parse.rst b/Doc/library/urllib.parse.rst index ac04f99deb..c6de2303c6 100644 --- a/Doc/library/urllib.parse.rst +++ b/Doc/library/urllib.parse.rst @@ -4,6 +4,7 @@ .. module:: urllib.parse :synopsis: Parse URLs into or assemble them from components. +**Source code:** :source:`Lib/urllib/parse.py` .. index:: single: WWW @@ -12,8 +13,6 @@ pair: URL; parsing pair: relative; URL -**Source code:** :source:`Lib/urllib/parse.py` - -------------- This module defines a standard interface to break Uniform Resource Locator (URL) @@ -270,6 +269,11 @@ or on combining URL components into a URL string. :func:`urlunsplit`, removing possible *scheme* and *netloc* parts. + .. versionchanged:: 3.5 + + Behaviour updated to match the semantics defined in :rfc:`3986`. + + .. function:: urldefrag(url) If *url* contains a fragment identifier, return a modified version of *url* @@ -516,7 +520,8 @@ task isn't already covered by the URL parsing functions above. Example: ``unquote_to_bytes('a%26%EF')`` yields ``b'a&\xef'``. -.. function:: urlencode(query, doseq=False, safe='', encoding=None, errors=None) +.. function:: urlencode(query, doseq=False, safe='', encoding=None, \ + errors=None, quote_via=quote_plus) Convert a mapping object or a sequence of two-element tuples, which may contain :class:`str` or :class:`bytes` objects, to a percent-encoded ASCII @@ -526,8 +531,16 @@ task isn't already covered by the URL parsing functions above. :exc:`TypeError`. The resulting string is a series of ``key=value`` pairs separated by ``'&'`` - characters, where both *key* and *value* are quoted using :func:`quote_plus` - above. When a sequence of two-element tuples is used as the *query* + characters, where both *key* and *value* are quoted using the *quote_via* + function. By default, :func:`quote_plus` is used to quote the values, which + means spaces are quoted as a ``'+'`` character and '/' characters are + encoded as ``%2F``, which follows the standard for GET requests + (``application/x-www-form-urlencoded``). An alternate function that can be + passed as *quote_via* is :func:`quote`, which will encode spaces as ``%20`` + and not encode '/' characters. For maximum control of what is quoted, use + ``quote`` and specify a value for *safe*. + + When a sequence of two-element tuples is used as the *query* argument, the first element of each tuple is a key and the second is a value. The value element in itself can be a sequence and in that case, if the optional parameter *doseq* is evaluates to *True*, individual @@ -536,7 +549,7 @@ task isn't already covered by the URL parsing functions above. string will match the order of parameter tuples in the sequence. The *safe*, *encoding*, and *errors* parameters are passed down to - :func:`quote_plus` (the *encoding* and *errors* parameters are only passed + *quote_via* (the *encoding* and *errors* parameters are only passed when a query element is a :class:`str`). To reverse this encoding process, :func:`parse_qs` and :func:`parse_qsl` are @@ -548,6 +561,9 @@ task isn't already covered by the URL parsing functions above. .. versionchanged:: 3.2 Query parameter supports bytes and string objects. + .. versionadded:: 3.5 + *quote_via* parameter. + .. seealso:: @@ -565,7 +581,7 @@ task isn't already covered by the URL parsing functions above. Names (URNs) and Uniform Resource Locators (URLs). :rfc:`2368` - The mailto URL scheme. - Parsing requirements for mailto url schemes. + Parsing requirements for mailto URL schemes. :rfc:`1808` - Relative Uniform Resource Locators This Request For Comments includes the rules for joining an absolute and a diff --git a/Doc/library/urllib.request.rst b/Doc/library/urllib.request.rst index 44db3e1325..1291aebcd2 100644 --- a/Doc/library/urllib.request.rst +++ b/Doc/library/urllib.request.rst @@ -3,10 +3,14 @@ .. module:: urllib.request :synopsis: Extensible library for opening URLs. + .. moduleauthor:: Jeremy Hylton <jeremy@alum.mit.edu> .. sectionauthor:: Moshe Zadka <moshez@users.sourceforge.net> .. sectionauthor:: Senthil Kumaran <senthil@uthcode.com> +**Source code:** :source:`Lib/urllib/request.py` + +-------------- The :mod:`urllib.request` module defines functions and classes which help in opening URLs (mostly HTTP) in a complex world --- basic and digest @@ -14,8 +18,8 @@ authentication, redirections, cookies and more. .. seealso:: - The `Requests package <http://requests.readthedocs.org/>`_ - is recommended for a higher-level http client interface. + The `Requests package <https://requests.readthedocs.org/>`_ + is recommended for a higher-level HTTP client interface. The :mod:`urllib.request` module defines the following functions: @@ -59,7 +63,7 @@ The :mod:`urllib.request` module defines the following functions: The *cadefault* parameter is ignored. - This function always returns an object which can work as + This function always returns an object which can work as a :term:`context manager` and has methods such as * :meth:`~urllib.response.addinfourl.geturl` --- return the URL of the resource retrieved, @@ -67,11 +71,11 @@ The :mod:`urllib.request` module defines the following functions: * :meth:`~urllib.response.addinfourl.info` --- return the meta-information of the page, such as headers, in the form of an :func:`email.message_from_string` instance (see - `Quick Reference to HTTP Headers <http://www.cs.tut.fi/~jkorpela/http.html>`_) + `Quick Reference to HTTP Headers <https://www.cs.tut.fi/~jkorpela/http.html>`_) * :meth:`~urllib.response.addinfourl.getcode` -- return the HTTP status code of the response. - For http and https urls, this function returns a + For HTTP and HTTPS URLs, this function returns a :class:`http.client.HTTPResponse` object slightly modified. In addition to the three new methods above, the msg attribute contains the same information as the :attr:`~http.client.HTTPResponse.reason` @@ -79,11 +83,11 @@ The :mod:`urllib.request` module defines the following functions: the response headers as it is specified in the documentation for :class:`~http.client.HTTPResponse`. - For ftp, file, and data urls and requests explicitly handled by legacy + For FTP, file, and data URLs and requests explicitly handled by legacy :class:`URLopener` and :class:`FancyURLopener` classes, this function returns a :class:`urllib.response.addinfourl` object. - Raises :exc:`~urllib.error.URLError` on errors. + Raises :exc:`~urllib.error.URLError` on protocol errors. Note that ``None`` may be returned if no handler handles the request (though the default installed global :class:`OpenerDirector` uses @@ -116,6 +120,7 @@ The :mod:`urllib.request` module defines the following functions: .. versionchanged:: 3.4.3 *context* was added. + .. function:: install_opener(opener) Install an :class:`OpenerDirector` instance as the default global opener. @@ -165,15 +170,19 @@ The :mod:`urllib.request` module defines the following functions: in a case insensitive approach, for all operating systems first, and when it cannot find it, looks for proxy information from Mac OSX System Configuration for Mac OS X and Windows Systems Registry for Windows. + If both lowercase and uppercase environment variables exist (and disagree), + lowercase is preferred. - .. note:: + .. note:: + + If the environment variable ``REQUEST_METHOD`` is set, which usually + indicates your script is running in a CGI environment, the environment + variable ``HTTP_PROXY`` (uppercase ``_PROXY``) will be ignored. This is + because that variable can be injected by a client using the "Proxy:" HTTP + header. If you need to use an HTTP proxy in a CGI environment, either use + ``ProxyHandler`` explicitly, or make sure the variable name is in + lowercase (or at least the ``_proxy`` suffix). - If the environment variable ``REQUEST_METHOD`` is set, which usually - indicates your script is running in a CGI environment, the environment - variable ``HTTP_PROXY`` (uppercase ``_PROXY``) will be ignored. This is - because that variable can be injected by a client using the "Proxy:" HTTP - header. If you need to use an HTTP proxy in a CGI environment use - ``ProxyHandler`` explicitly. The following classes are provided: @@ -194,7 +203,7 @@ The following classes are provided: *headers* should be a dictionary, and will be treated as if :meth:`add_header` was called with each key and value as arguments. - This is often used to "spoof" the ``User-Agent`` header, which is + This is often used to "spoof" the ``User-Agent`` header value, which is used by a browser to identify itself -- some HTTP servers only allow requests coming from common browsers as opposed to scripts. For example, Mozilla Firefox may identify itself as ``"Mozilla/5.0 @@ -276,10 +285,15 @@ The following classes are provided: To disable autodetected proxy pass an empty dictionary. - .. note:: + The :envvar:`no_proxy` environment variable can be used to specify hosts + which shouldn't be reached via proxy; if set, it should be a comma-separated + list of hostname suffixes, optionally with ``:port`` appended, for example + ``cern.ch,ncsa.uiuc.edu,some.host:8080``. - ``HTTP_PROXY`` will be ignored if a variable ``REQUEST_METHOD`` is set; - see the documentation on :func:`~urllib.request.getproxies`. + .. note:: + + ``HTTP_PROXY`` will be ignored if a variable ``REQUEST_METHOD`` is set; + see the documentation on :func:`~urllib.request.getproxies`. .. class:: HTTPPasswordMgr() @@ -294,13 +308,37 @@ The following classes are provided: fits. +.. class:: HTTPPasswordMgrWithPriorAuth() + + A variant of :class:`HTTPPasswordMgrWithDefaultRealm` that also has a + database of ``uri -> is_authenticated`` mappings. Can be used by a + BasicAuth handler to determine when to send authentication credentials + immediately instead of waiting for a ``401`` response first. + + .. versionadded:: 3.5 + + .. class:: AbstractBasicAuthHandler(password_mgr=None) This is a mixin class that helps with HTTP authentication, both to the remote host and to a proxy. *password_mgr*, if given, should be something that is compatible with :class:`HTTPPasswordMgr`; refer to section :ref:`http-password-mgr` for information on the interface that must be - supported. + supported. If *passwd_mgr* also provides ``is_authenticated`` and + ``update_authenticated`` methods (see + :ref:`http-password-mgr-with-prior-auth`), then the handler will use the + ``is_authenticated`` result for a given URI to determine whether or not to + send authentication credentials with the request. If ``is_authenticated`` + returns ``True`` for the URI, credentials are sent. If ``is_authenticated`` + is ``False``, credentials are not sent, and then if a ``401`` response is + received the request is re-sent with the authentication credentials. If + authentication succeeds, ``update_authenticated`` is called to set + ``is_authenticated`` ``True`` for the URI, so that subsequent requests to + the URI or any of its super-URIs will automatically include the + authentication credentials. + + .. versionadded:: 3.5 + Added ``is_authenticated`` support. .. class:: HTTPBasicAuthHandler(password_mgr=None) @@ -434,7 +472,7 @@ request. .. attribute:: Request.selector The URI path. If the :class:`Request` uses a proxy, then selector - will be the full url that is passed to the proxy. + will be the full URL that is passed to the proxy. .. attribute:: Request.data @@ -753,8 +791,8 @@ HTTPRedirectHandler Objects details of the precise meanings of the various redirection codes. An :class:`HTTPError` exception raised as a security consideration if the - HTTPRedirectHandler is presented with a redirected url which is not an HTTP, - HTTPS or FTP url. + HTTPRedirectHandler is presented with a redirected URL which is not an HTTP, + HTTPS or FTP URL. .. method:: HTTPRedirectHandler.redirect_request(req, fp, code, msg, hdrs, newurl) @@ -852,6 +890,42 @@ These methods are available on :class:`HTTPPasswordMgr` and searched if the given *realm* has no matching user/password. +.. _http-password-mgr-with-prior-auth: + +HTTPPasswordMgrWithPriorAuth Objects +------------------------------------ + +This password manager extends :class:`HTTPPasswordMgrWithDefaultRealm` to support +tracking URIs for which authentication credentials should always be sent. + + +.. method:: HTTPPasswordMgrWithPriorAuth.add_password(realm, uri, user, \ + passwd, is_authenticated=False) + + *realm*, *uri*, *user*, *passwd* are as for + :meth:`HTTPPasswordMgr.add_password`. *is_authenticated* sets the initial + value of the ``is_authenticated`` flag for the given URI or list of URIs. + If *is_authenticated* is specified as ``True``, *realm* is ignored. + + +.. method:: HTTPPasswordMgr.find_user_password(realm, authuri) + + Same as for :class:`HTTPPasswordMgrWithDefaultRealm` objects + + +.. method:: HTTPPasswordMgrWithPriorAuth.update_authenticated(self, uri, \ + is_authenticated=False) + + Update the ``is_authenticated`` flag for the given *uri* or list + of URIs. + + +.. method:: HTTPPasswordMgrWithPriorAuth.is_authenticated(self, authuri) + + Returns the current state of the ``is_authenticated`` flag for + the given URI. + + .. _abstract-basic-auth-handler: AbstractBasicAuthHandler Objects @@ -1056,6 +1130,9 @@ HTTPErrorProcessor Objects Examples -------- +In addition to the examples below, more examples are given in +:ref:`urllib-howto`. + This example gets the python.org main page and displays the first 300 bytes of it. :: @@ -1071,12 +1148,12 @@ it. :: Note that urlopen returns a bytes object. This is because there is no way for urlopen to automatically determine the encoding of the byte stream -it receives from the http server. In general, a program will decode +it receives from the HTTP server. In general, a program will decode the returned bytes object to string once it determines or guesses the appropriate encoding. -The following W3C document, http://www.w3.org/International/O-charset\ , lists -the various ways in which a (X)HTML or a XML document could have specified its +The following W3C document, https://www.w3.org/International/O-charset\ , lists +the various ways in which an (X)HTML or an XML document could have specified its encoding information. As the python.org website uses *utf-8* encoding as specified in its meta tag, we @@ -1119,7 +1196,7 @@ The code for the sample CGI used in the above example is:: Here is an example of doing a ``PUT`` request using :class:`Request`:: import urllib.request - DATA=b'some data' + DATA = b'some data' req = urllib.request.Request(url='http://localhost:8080', data=DATA,method='PUT') with urllib.request.urlopen(req) as f: pass @@ -1165,6 +1242,8 @@ Use the *headers* argument to the :class:`Request` constructor, or:: import urllib.request req = urllib.request.Request('http://www.example.com/') req.add_header('Referer', 'http://www.python.org/') + # Customize the default User-Agent header value: + req.add_header('User-Agent', 'urllib-example/0.1 (Contact: . . .)') r = urllib.request.urlopen(req) :class:`OpenerDirector` automatically adds a :mailheader:`User-Agent` header to diff --git a/Doc/library/urllib.robotparser.rst b/Doc/library/urllib.robotparser.rst index f179de2f92..ba701c3b68 100644 --- a/Doc/library/urllib.robotparser.rst +++ b/Doc/library/urllib.robotparser.rst @@ -4,8 +4,10 @@ .. module:: urllib.robotparser :synopsis: Load a robots.txt file and answer questions about fetchability of other URLs. + .. sectionauthor:: Skip Montanaro <skip@pobox.com> +**Source code:** :source:`Lib/urllib/robotparser.py` .. index:: single: WWW @@ -13,6 +15,8 @@ single: URL single: robots.txt +-------------- + This module provides a single class, :class:`RobotFileParser`, which answers questions about whether or not a particular user agent can fetch a URL on the Web site that published the :file:`robots.txt` file. For more details on the diff --git a/Doc/library/urllib.rst b/Doc/library/urllib.rst index 8e308bc589..624e164625 100644 --- a/Doc/library/urllib.rst +++ b/Doc/library/urllib.rst @@ -3,6 +3,10 @@ .. module:: urllib +**Source code:** :source:`Lib/urllib/` + +-------------- + ``urllib`` is a package that collects several modules for working with URLs: * :mod:`urllib.request` for opening and reading URLs diff --git a/Doc/library/uu.rst b/Doc/library/uu.rst index d61c178831..33fb36d0b8 100644 --- a/Doc/library/uu.rst +++ b/Doc/library/uu.rst @@ -3,6 +3,7 @@ .. module:: uu :synopsis: Encode and decode files in uuencode format. + .. moduleauthor:: Lance Ellinghouse **Source code:** :source:`Lib/uu.py` diff --git a/Doc/library/uuid.rst b/Doc/library/uuid.rst index 7dc46acd05..53cbd6c770 100644 --- a/Doc/library/uuid.rst +++ b/Doc/library/uuid.rst @@ -6,6 +6,9 @@ .. moduleauthor:: Ka-Ping Yee <ping@zesty.ca> .. sectionauthor:: George Yoshida <quiver@users.sourceforge.net> +**Source code:** :source:`Lib/uuid.py` + +-------------- This module provides immutable :class:`UUID` objects (the :class:`UUID` class) and the functions :func:`uuid1`, :func:`uuid3`, :func:`uuid4`, :func:`uuid5` for diff --git a/Doc/library/venv.rst b/Doc/library/venv.rst index e9ede8b415..af4a6d1ab4 100644 --- a/Doc/library/venv.rst +++ b/Doc/library/venv.rst @@ -3,15 +3,15 @@ .. module:: venv :synopsis: Creation of virtual environments. + .. moduleauthor:: Vinay Sajip <vinay_sajip@yahoo.co.uk> .. sectionauthor:: Vinay Sajip <vinay_sajip@yahoo.co.uk> - -.. index:: pair: Environments; virtual - .. versionadded:: 3.3 -**Source code:** :source:`Lib/venv` +**Source code:** :source:`Lib/venv/` + +.. index:: pair: Environments; virtual -------------- @@ -43,11 +43,6 @@ Creating virtual environments Common installation tools such as ``Setuptools`` and ``pip`` work as expected with venvs - i.e. when a venv is active, they install Python packages into the venv without needing to be told to do so explicitly. - Of course, you need to install them into the venv first: this could be - done by running ``ez_setup.py`` with the venv activated, - followed by running ``easy_install pip``. Alternatively, you could download - the source tarballs and run ``python setup.py install`` after unpacking, - with the venv activated. When a venv is active (i.e. the venv's Python interpreter is running), the attributes :attr:`sys.prefix` and :attr:`sys.exec_prefix` point to the base diff --git a/Doc/library/warnings.rst b/Doc/library/warnings.rst index 8a538adb9b..37f6874a02 100644 --- a/Doc/library/warnings.rst +++ b/Doc/library/warnings.rst @@ -1,13 +1,13 @@ :mod:`warnings` --- Warning control =================================== -.. index:: single: warnings - .. module:: warnings :synopsis: Issue warning messages and control their disposition. **Source code:** :source:`Lib/warnings.py` +.. index:: single: warnings + -------------- Warning messages are typically issued in situations where it is useful to alert @@ -141,14 +141,15 @@ the disposition of the match. Each entry is a tuple of the form (*action*, | | warnings, regardless of location | +---------------+----------------------------------------------+ -* *message* is a string containing a regular expression that the warning message - must match (the match is compiled to always be case-insensitive). +* *message* is a string containing a regular expression that the start of + the warning message must match. The expression is compiled to always be + case-insensitive. * *category* is a class (a subclass of :exc:`Warning`) of which the warning category must be a subclass in order to match. * *module* is a string containing a regular expression that the module name must - match (the match is compiled to be case-sensitive). + match. The expression is compiled to be case-sensitive. * *lineno* is an integer that the line number where the warning occurred must match, or ``0`` to match all line numbers. @@ -265,7 +266,7 @@ Updating Code For New Versions of Python Warnings that are only of interest to the developer are ignored by default. As such you should make sure to test your code with typically ignored warnings -made visible. You can do this from the command-line by passing :option:`-Wd` +made visible. You can do this from the command-line by passing :option:`-Wd <-W>` to the interpreter (this is shorthand for :option:`-W default`). This enables default handling for all warnings, including those that are ignored by default. To change what action is taken for encountered warnings you simply change what diff --git a/Doc/library/wave.rst b/Doc/library/wave.rst index ab64978cfd..7144379784 100644 --- a/Doc/library/wave.rst +++ b/Doc/library/wave.rst @@ -3,6 +3,7 @@ .. module:: wave :synopsis: Provide an interface to the WAV sound format. + .. sectionauthor:: Moshe Zadka <moshez@zadka.site.co.il> .. Documentations stolen from comments in file. diff --git a/Doc/library/weakref.rst b/Doc/library/weakref.rst index cc883b1b51..0470bd113a 100644 --- a/Doc/library/weakref.rst +++ b/Doc/library/weakref.rst @@ -3,6 +3,7 @@ .. module:: weakref :synopsis: Support for weak references and weak dictionaries. + .. moduleauthor:: Fred L. Drake, Jr. <fdrake@acm.org> .. moduleauthor:: Neil Schemenauer <nas@arctrix.com> .. moduleauthor:: Martin von Löwis <martin@loewis.home.cs.tu-berlin.de> @@ -258,7 +259,7 @@ These method have the same issues as the and :meth:`keyrefs` method of are called in reverse order of creation. A finalizer will never invoke its callback during the later part of - the interpreter shutdown when module globals are liable to have + the :term:`interpreter shutdown` when module globals are liable to have been replaced by :const:`None`. .. method:: __call__() @@ -329,7 +330,7 @@ These method have the same issues as the and :meth:`keyrefs` method of .. seealso:: - :pep:`0205` - Weak References + :pep:`205` - Weak References The proposal and rationale for this feature, including links to earlier implementations and information about similar features in other languages. @@ -527,8 +528,8 @@ follows:: Starting with Python 3.4, :meth:`__del__` methods no longer prevent reference cycles from being garbage collected, and module globals are -no longer forced to :const:`None` during interpreter shutdown. So this -code should work without any issues on CPython. +no longer forced to :const:`None` during :term:`interpreter shutdown`. +So this code should work without any issues on CPython. However, handling of :meth:`__del__` methods is notoriously implementation specific, since it depends on internal details of the interpreter's garbage diff --git a/Doc/library/webbrowser.rst b/Doc/library/webbrowser.rst index aa5e4ad15d..85d3636722 100644 --- a/Doc/library/webbrowser.rst +++ b/Doc/library/webbrowser.rst @@ -3,6 +3,7 @@ .. module:: webbrowser :synopsis: Easy-to-use controller for Web browsers. + .. moduleauthor:: Fred L. Drake, Jr. <fdrake@acm.org> .. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org> @@ -33,7 +34,7 @@ browsers are not available on Unix, the controlling process will launch a new browser and wait. The script :program:`webbrowser` can be used as a command-line interface for the -module. It accepts an URL as the argument. It accepts the following optional +module. It accepts a URL as the argument. It accepts the following optional parameters: ``-n`` opens the URL in a new browser window, if possible; ``-t`` opens the URL in a new browser page ("tab"). The options are, naturally, mutually exclusive. Usage example:: diff --git a/Doc/library/winreg.rst b/Doc/library/winreg.rst index 6c920b4882..52d591ad75 100644 --- a/Doc/library/winreg.rst +++ b/Doc/library/winreg.rst @@ -4,8 +4,10 @@ .. module:: winreg :platform: Windows :synopsis: Routines and objects for manipulating the Windows registry. + .. sectionauthor:: Mark Hammond <MarkH@ActiveState.com> +-------------- These functions expose the Windows registry API to Python. Instead of using an integer as the registry handle, a :ref:`handle object <handle-object>` is used @@ -134,7 +136,7 @@ This module offers the following functions: The :func:`DeleteKeyEx` function is implemented with the RegDeleteKeyEx Windows API function, which is specific to 64-bit versions of Windows. See the `RegDeleteKeyEx documentation - <http://msdn.microsoft.com/en-us/library/ms724847%28VS.85%29.aspx>`__. + <https://msdn.microsoft.com/en-us/library/ms724847%28VS.85%29.aspx>`__. *key* is an already open key, or one of the predefined :ref:`HKEY_* constants <hkey-constants>`. @@ -268,7 +270,7 @@ This module offers the following functions: A call to :func:`LoadKey` fails if the calling process does not have the :const:`SE_RESTORE_PRIVILEGE` privilege. Note that privileges are different from permissions -- see the `RegLoadKey documentation - <http://msdn.microsoft.com/en-us/library/ms724889%28v=VS.85%29.aspx>`__ for + <https://msdn.microsoft.com/en-us/library/ms724889%28v=VS.85%29.aspx>`__ for more details. If *key* is a handle returned by :func:`ConnectRegistry`, then the path @@ -383,7 +385,7 @@ This module offers the following functions: possess the :const:`SeBackupPrivilege` security privilege. Note that privileges are different than permissions -- see the `Conflicts Between User Rights and Permissions documentation - <http://msdn.microsoft.com/en-us/library/ms724878%28v=VS.85%29.aspx>`__ + <https://msdn.microsoft.com/en-us/library/ms724878%28v=VS.85%29.aspx>`__ for more details. This function passes NULL for *security_attributes* to the API. @@ -547,7 +549,7 @@ Access Rights +++++++++++++ For more information, see `Registry Key Security and Access -<http://msdn.microsoft.com/en-us/library/ms724878%28v=VS.85%29.aspx>`__. +<https://msdn.microsoft.com/en-us/library/ms724878%28v=VS.85%29.aspx>`__. .. data:: KEY_ALL_ACCESS @@ -602,7 +604,7 @@ For more information, see `Registry Key Security and Access *************** For more information, see `Accessing an Alternate Registry View -<http://msdn.microsoft.com/en-us/library/aa384129(v=VS.85).aspx>`__. +<https://msdn.microsoft.com/en-us/library/aa384129(v=VS.85).aspx>`__. .. data:: KEY_WOW64_64KEY @@ -621,7 +623,7 @@ Value Types +++++++++++ For more information, see `Registry Value Types -<http://msdn.microsoft.com/en-us/library/ms724884%28v=VS.85%29.aspx>`__. +<https://msdn.microsoft.com/en-us/library/ms724884%28v=VS.85%29.aspx>`__. .. data:: REG_BINARY diff --git a/Doc/library/winsound.rst b/Doc/library/winsound.rst index 61f34cddc1..d2c421047e 100644 --- a/Doc/library/winsound.rst +++ b/Doc/library/winsound.rst @@ -4,9 +4,11 @@ .. module:: winsound :platform: Windows :synopsis: Access to the sound-playing machinery for Windows. + .. moduleauthor:: Toby Dickenson <htrd90@zepler.org> .. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org> +-------------- The :mod:`winsound` module provides access to the basic sound-playing machinery provided by Windows platforms. It includes functions and several constants. diff --git a/Doc/library/wsgiref.rst b/Doc/library/wsgiref.rst index 4ed945417b..aad27a88c9 100644 --- a/Doc/library/wsgiref.rst +++ b/Doc/library/wsgiref.rst @@ -3,9 +3,11 @@ .. module:: wsgiref :synopsis: WSGI Utilities and Reference Implementation. + .. moduleauthor:: Phillip J. Eby <pje@telecommunity.com> .. sectionauthor:: Phillip J. Eby <pje@telecommunity.com> +-------------- The Web Server Gateway Interface (WSGI) is a standard interface between web server software and web applications written in Python. Having a standard @@ -24,8 +26,8 @@ for implementing WSGI servers, a demo HTTP server that serves WSGI applications, and a validation tool that checks WSGI servers and applications for conformance to the WSGI specification (:pep:`3333`). -See http://www.wsgi.org for more information about WSGI, and links to tutorials -and other resources. +See https://wsgi.readthedocs.org/ for more information about WSGI, and links to +tutorials and other resources. .. XXX If you're just trying to write a web application... @@ -184,10 +186,11 @@ This module provides a single class, :class:`Headers`, for convenient manipulation of WSGI response headers using a mapping-like interface. -.. class:: Headers(headers) +.. class:: Headers([headers]) Create a mapping-like object wrapping *headers*, which must be a list of header - name/value tuples as described in :pep:`3333`. + name/value tuples as described in :pep:`3333`. The default value of *headers* is + an empty list. :class:`Headers` objects support typical mapping operations including :meth:`__getitem__`, :meth:`get`, :meth:`__setitem__`, :meth:`setdefault`, @@ -251,6 +254,10 @@ manipulation of WSGI response headers using a mapping-like interface. Content-Disposition: attachment; filename="bud.gif" + .. versionchanged:: 3.5 + *headers* parameter is optional. + + :mod:`wsgiref.simple_server` -- a simple WSGI HTTP server --------------------------------------------------------- @@ -414,8 +421,8 @@ Paste" library. # Our callable object which is intentionally not compliant to the # standard, so the validator is going to break def simple_app(environ, start_response): - status = '200 OK' # HTTP Status - headers = [('Content-type', 'text/plain')] # HTTP Headers + status = '200 OK' # HTTP Status + headers = [('Content-type', 'text/plain')] # HTTP Headers start_response(status, headers) # This is going to break because we need to return a list, and @@ -510,6 +517,9 @@ input, output, and error streams. streams are stored in the :attr:`stdin`, :attr:`stdout`, :attr:`stderr`, and :attr:`environ` attributes. + The :meth:`~io.BufferedIOBase.write` method of *stdout* should write + each chunk in full, like :class:`io.BufferedIOBase`. + .. class:: BaseHandler() @@ -757,8 +767,8 @@ This is a working "Hello World" WSGI application:: # is a dictionary containing CGI-style environment variables and the # second variable is the callable object (see PEP 333). def hello_world_app(environ, start_response): - status = '200 OK' # HTTP Status - headers = [('Content-type', 'text/plain; charset=utf-8')] # HTTP Headers + status = '200 OK' # HTTP Status + headers = [('Content-type', 'text/plain; charset=utf-8')] # HTTP Headers start_response(status, headers) # The returned object is going to be printed diff --git a/Doc/library/xdrlib.rst b/Doc/library/xdrlib.rst index 5c7dfa454b..42a03a4675 100644 --- a/Doc/library/xdrlib.rst +++ b/Doc/library/xdrlib.rst @@ -4,13 +4,12 @@ .. module:: xdrlib :synopsis: Encoders and decoders for the External Data Representation (XDR). +**Source code:** :source:`Lib/xdrlib.py` .. index:: single: XDR single: External Data Representation -**Source code:** :source:`Lib/xdrlib.py` - -------------- The :mod:`xdrlib` module supports the External Data Representation Standard as diff --git a/Doc/library/xml.dom.minidom.rst b/Doc/library/xml.dom.minidom.rst index 6762e9177f..2e9e814693 100644 --- a/Doc/library/xml.dom.minidom.rst +++ b/Doc/library/xml.dom.minidom.rst @@ -3,6 +3,7 @@ .. module:: xml.dom.minidom :synopsis: Minimal Document Object Model (DOM) implementation. + .. moduleauthor:: Paul Prescod <paul@prescod.net> .. sectionauthor:: Paul Prescod <paul@prescod.net> .. sectionauthor:: Martin v. Löwis <martin@v.loewis.de> @@ -30,10 +31,10 @@ DOM applications typically start by parsing some XML into a DOM. With from xml.dom.minidom import parse, parseString - dom1 = parse('c:\\temp\\mydata.xml') # parse an XML file by name + dom1 = parse('c:\\temp\\mydata.xml') # parse an XML file by name datasource = open('c:\\temp\\mydata.xml') - dom2 = parse(datasource) # parse an open file + dom2 = parse(datasource) # parse an open file dom3 = parseString('<myxml>Some data<empty/> some more data</myxml>') @@ -93,14 +94,14 @@ document: the one that holds all others. Here is an example program:: When you are finished with a DOM tree, you may optionally call the :meth:`unlink` method to encourage early cleanup of the now-unneeded -objects. :meth:`unlink` is a :mod:`xml.dom.minidom`\ -specific +objects. :meth:`unlink` is an :mod:`xml.dom.minidom`\ -specific extension to the DOM API that renders the node and its descendants are essentially useless. Otherwise, Python's garbage collector will eventually take care of the objects in the tree. .. seealso:: - `Document Object Model (DOM) Level 1 Specification <http://www.w3.org/TR/REC-DOM-Level-1/>`_ + `Document Object Model (DOM) Level 1 Specification <https://www.w3.org/TR/REC-DOM-Level-1/>`_ The W3C recommendation for the DOM supported by :mod:`xml.dom.minidom`. @@ -251,5 +252,5 @@ utility to most DOM users. the appropriate standards. For example, "UTF-8" is valid, but "UTF8" is not valid in an XML document's declaration, even though Python accepts it as an encoding name. - See http://www.w3.org/TR/2006/REC-xml11-20060816/#NT-EncodingDecl - and http://www.iana.org/assignments/character-sets/character-sets.xhtml. + See https://www.w3.org/TR/2006/REC-xml11-20060816/#NT-EncodingDecl + and https://www.iana.org/assignments/character-sets/character-sets.xhtml. diff --git a/Doc/library/xml.dom.pulldom.rst b/Doc/library/xml.dom.pulldom.rst index a3b8bc166a..c3339edd54 100644 --- a/Doc/library/xml.dom.pulldom.rst +++ b/Doc/library/xml.dom.pulldom.rst @@ -3,6 +3,7 @@ .. module:: xml.dom.pulldom :synopsis: Support for building partial DOM trees from SAX events. + .. moduleauthor:: Paul Prescod <paul@prescod.net> **Source code:** :source:`Lib/xml/dom/pulldom.py` @@ -114,13 +115,15 @@ DOMEventStream Objects Expands all children of *node* into *node*. Example:: + from xml.dom import pulldom + xml = '<html><title>Foo</title> <p>Some text <div>and more</div></p> </html>' doc = pulldom.parseString(xml) for event, node in doc: if event == pulldom.START_ELEMENT and node.tagName == 'p': # Following statement only prints '<p/>' print(node.toxml()) - doc.exandNode(node) + doc.expandNode(node) # Following statement prints node with all its children '<p>Some text <div>and more</div></p>' print(node.toxml()) diff --git a/Doc/library/xml.dom.rst b/Doc/library/xml.dom.rst index a432202ec0..b037ff6b82 100644 --- a/Doc/library/xml.dom.rst +++ b/Doc/library/xml.dom.rst @@ -3,9 +3,13 @@ .. module:: xml.dom :synopsis: Document Object Model API for Python. + .. sectionauthor:: Paul Prescod <paul@prescod.net> .. sectionauthor:: Martin v. Löwis <martin@v.loewis.de> +**Source code:** :source:`Lib/xml/dom/__init__.py` + +-------------- The Document Object Model, or "DOM," is a cross-language API from the World Wide Web Consortium (W3C) for accessing and modifying XML documents. A DOM @@ -63,10 +67,10 @@ implementations are free to support the strict mapping from IDL). See section .. seealso:: - `Document Object Model (DOM) Level 2 Specification <http://www.w3.org/TR/DOM-Level-2-Core/>`_ + `Document Object Model (DOM) Level 2 Specification <https://www.w3.org/TR/DOM-Level-2-Core/>`_ The W3C recommendation upon which the Python DOM API is based. - `Document Object Model (DOM) Level 1 Specification <http://www.w3.org/TR/REC-DOM-Level-1/>`_ + `Document Object Model (DOM) Level 1 Specification <https://www.w3.org/TR/REC-DOM-Level-1/>`_ The W3C recommendation for the DOM supported by :mod:`xml.dom.minidom`. `Python Language Mapping Specification <http://www.omg.org/spec/PYTH/1.2/PDF>`_ @@ -115,20 +119,20 @@ Some convenience constants are also provided: .. data:: XML_NAMESPACE The namespace URI associated with the reserved prefix ``xml``, as defined by - `Namespaces in XML <http://www.w3.org/TR/REC-xml-names/>`_ (section 4). + `Namespaces in XML <https://www.w3.org/TR/REC-xml-names/>`_ (section 4). .. data:: XMLNS_NAMESPACE The namespace URI for namespace declarations, as defined by `Document Object Model (DOM) Level 2 Core Specification - <http://www.w3.org/TR/DOM-Level-2-Core/core.html>`_ (section 1.1.8). + <https://www.w3.org/TR/DOM-Level-2-Core/core.html>`_ (section 1.1.8). .. data:: XHTML_NAMESPACE The URI of the XHTML namespace as defined by `XHTML 1.0: The Extensible - HyperText Markup Language <http://www.w3.org/TR/xhtml1/>`_ (section 3.1.1). + HyperText Markup Language <https://www.w3.org/TR/xhtml1/>`_ (section 3.1.1). In addition, :mod:`xml.dom` contains a base :class:`Node` class and the DOM @@ -874,7 +878,7 @@ attribute. .. exception:: NamespaceErr If an attempt is made to change any object in a way that is not permitted with - regard to the `Namespaces in XML <http://www.w3.org/TR/REC-xml-names/>`_ + regard to the `Namespaces in XML <https://www.w3.org/TR/REC-xml-names/>`_ recommendation, this exception is raised. diff --git a/Doc/library/xml.etree.elementtree.rst b/Doc/library/xml.etree.elementtree.rst index dc0274eb0b..99d7e8b7ce 100644 --- a/Doc/library/xml.etree.elementtree.rst +++ b/Doc/library/xml.etree.elementtree.rst @@ -3,8 +3,13 @@ .. module:: xml.etree.ElementTree :synopsis: Implementation of the ElementTree API. + .. moduleauthor:: Fredrik Lundh <fredrik@pythonware.com> +**Source code:** :source:`Lib/xml/etree/ElementTree.py` + +-------------- + The :mod:`xml.etree.ElementTree` module implements a simple and efficient API for parsing and creating XML data. @@ -94,7 +99,7 @@ As an :class:`Element`, ``root`` has a tag and a dictionary of attributes:: It also has children nodes over which we can iterate:: >>> for child in root: - ... print(child.tag, child.attrib) + ... print(child.tag, child.attrib) ... country {'name': 'Liechtenstein'} country {'name': 'Singapore'} @@ -143,8 +148,8 @@ elements, call :meth:`XMLPullParser.read_events`. Here is an example:: [('start', <Element 'mytag' at 0x7fa66db2be58>)] >>> parser.feed(' more text</mytag>') >>> for event, elem in parser.read_events(): - ... print(event) - ... print(elem.tag, 'text=', elem.text) + ... print(event) + ... print(elem.tag, 'text=', elem.text) ... end @@ -166,7 +171,7 @@ the sub-tree below it (its children, their children, and so on). For example, :meth:`Element.iter`:: >>> for neighbor in root.iter('neighbor'): - ... print(neighbor.attrib) + ... print(neighbor.attrib) ... {'name': 'Austria', 'direction': 'E'} {'name': 'Switzerland', 'direction': 'W'} @@ -180,9 +185,9 @@ with a particular tag, and :attr:`Element.text` accesses the element's text content. :meth:`Element.get` accesses the element's attributes:: >>> for country in root.findall('country'): - ... rank = country.find('rank').text - ... name = country.get('name') - ... print(name, rank) + ... rank = country.find('rank').text + ... name = country.get('name') + ... print(name, rank) ... Liechtenstein 1 Singapore 4 @@ -206,9 +211,9 @@ Let's say we want to add one to each country's rank, and add an ``updated`` attribute to the rank element:: >>> for rank in root.iter('rank'): - ... new_rank = int(rank.text) + 1 - ... rank.text = str(new_rank) - ... rank.set('updated', 'yes') + ... new_rank = int(rank.text) + 1 + ... rank.text = str(new_rank) + ... rank.set('updated', 'yes') ... >>> tree.write('output.xml') @@ -244,9 +249,9 @@ We can remove elements using :meth:`Element.remove`. Let's say we want to remove all countries with a rank higher than 50:: >>> for country in root.findall('country'): - ... rank = int(country.find('rank').text) - ... if rank > 50: - ... root.remove(country) + ... rank = int(country.find('rank').text) + ... if rank > 50: + ... root.remove(country) ... >>> tree.write('output.xml') @@ -292,7 +297,7 @@ If the XML input has `namespaces with prefixes in the form ``prefix:sometag`` get expanded to ``{uri}sometag`` where the *prefix* is replaced by the full *URI*. Also, if there is a `default namespace -<http://www.w3.org/TR/2006/REC-xml-names-20060816/#defaulting>`__, +<https://www.w3.org/TR/2006/REC-xml-names-20060816/#defaulting>`__, that full URI gets prepended to all of the non-prefixed tags. Here is an XML example that incorporates two namespaces, one with the @@ -363,7 +368,7 @@ XPath support ------------- This module provides limited support for -`XPath expressions <http://www.w3.org/TR/xpath>`_ for locating elements in a +`XPath expressions <https://www.w3.org/TR/xpath>`_ for locating elements in a tree. The goal is to support a small subset of the abbreviated syntax; a full XPath engine is outside the scope of the module. @@ -978,7 +983,7 @@ QName Objects to get proper namespace handling on output. *text_or_uri* is a string containing the QName value, in the form {uri}local, or, if the tag argument is given, the URI part of a QName. If *tag* is given, the first argument is - interpreted as an URI, and this argument is interpreted as a local name. + interpreted as a URI, and this argument is interpreted as a local name. :class:`QName` instances are opaque. @@ -1044,16 +1049,16 @@ XMLParser Objects This class is the low-level building block of the module. It uses :mod:`xml.parsers.expat` for efficient, event-based parsing of XML. It can - be fed XML data incrementall with the :meth:`feed` method, and parsing events - are translated to a push API - by invoking callbacks on the *target* object. - If *target* is omitted, the standard :class:`TreeBuilder` is used. The - *html* argument was historically used for backwards compatibility and is now - deprecated. If *encoding* [1]_ is given, the value overrides the encoding - specified in the XML file. + be fed XML data incrementally with the :meth:`feed` method, and parsing + events are translated to a push API - by invoking callbacks on the *target* + object. If *target* is omitted, the standard :class:`TreeBuilder` is used. + The *html* argument was historically used for backwards compatibility and is + now deprecated. If *encoding* [1]_ is given, the value overrides the + encoding specified in the XML file. .. deprecated:: 3.4 The *html* argument. The remaining arguments should be passed via - keywword to prepare for the removal of the *html* argument. + keyword to prepare for the removal of the *html* argument. .. method:: close() @@ -1189,5 +1194,5 @@ Exceptions .. [#] The encoding string included in XML output should conform to the appropriate standards. For example, "UTF-8" is valid, but "UTF8" is - not. See http://www.w3.org/TR/2006/REC-xml11-20060816/#NT-EncodingDecl - and http://www.iana.org/assignments/character-sets/character-sets.xhtml. + not. See https://www.w3.org/TR/2006/REC-xml11-20060816/#NT-EncodingDecl + and https://www.iana.org/assignments/character-sets/character-sets.xhtml. diff --git a/Doc/library/xml.rst b/Doc/library/xml.rst index 0188219092..3c2fc89d50 100644 --- a/Doc/library/xml.rst +++ b/Doc/library/xml.rst @@ -5,9 +5,13 @@ XML Processing Modules .. module:: xml :synopsis: Package containing XML processing modules + .. sectionauthor:: Christian Heimes <christian@python.org> .. sectionauthor:: Georg Brandl <georg@python.org> +**Source code:** :source:`Lib/xml/` + +-------------- Python's interfaces for processing XML are grouped in the ``xml`` package. @@ -62,7 +66,7 @@ kind sax etree minidom pulldom xmlrpc billion laughs **Yes** **Yes** **Yes** **Yes** **Yes** quadratic blowup **Yes** **Yes** **Yes** **Yes** **Yes** external entity expansion **Yes** No (1) No (2) **Yes** No (3) -DTD retrieval **Yes** No No **Yes** No +`DTD`_ retrieval **Yes** No No **Yes** No decompression bomb No No No No **Yes** ========================= ======== ========= ========= ======== ========= @@ -92,7 +96,7 @@ external entity expansion also point to external resources or local files. The XML parser accesses the resource and embeds the content into the XML document. -DTD retrieval +`DTD`_ retrieval Some XML libraries like Python's :mod:`xml.dom.pulldom` retrieve document type definitions from remote or local locations. The feature has similar implications as the external entity expansion issue. @@ -128,6 +132,6 @@ Python because they break backward compatibility. .. _defusedxml: https://pypi.python.org/pypi/defusedxml/ .. _defusedexpat: https://pypi.python.org/pypi/defusedexpat/ -.. _Billion Laughs: http://en.wikipedia.org/wiki/Billion_laughs -.. _ZIP bomb: http://en.wikipedia.org/wiki/Zip_bomb -.. _DTD: http://en.wikipedia.org/wiki/Document_Type_Definition +.. _Billion Laughs: https://en.wikipedia.org/wiki/Billion_laughs +.. _ZIP bomb: https://en.wikipedia.org/wiki/Zip_bomb +.. _DTD: https://en.wikipedia.org/wiki/Document_type_definition diff --git a/Doc/library/xml.sax.handler.rst b/Doc/library/xml.sax.handler.rst index 0fb3341c61..ae0877ca90 100644 --- a/Doc/library/xml.sax.handler.rst +++ b/Doc/library/xml.sax.handler.rst @@ -3,9 +3,13 @@ .. module:: xml.sax.handler :synopsis: Base classes for SAX event handlers. + .. moduleauthor:: Lars Marius Garshol <larsga@garshol.priv.no> .. sectionauthor:: Martin v. Löwis <martin@v.loewis.de> +**Source code:** :source:`Lib/xml/sax/handler.py` + +-------------- The SAX API defines four kinds of handlers: content handlers, DTD handlers, error handlers, and entity resolvers. Applications normally only need to diff --git a/Doc/library/xml.sax.reader.rst b/Doc/library/xml.sax.reader.rst index 3ab60636c5..c368fc2e5a 100644 --- a/Doc/library/xml.sax.reader.rst +++ b/Doc/library/xml.sax.reader.rst @@ -3,9 +3,13 @@ .. module:: xml.sax.xmlreader :synopsis: Interface which SAX-compliant XML parsers must implement. + .. moduleauthor:: Lars Marius Garshol <larsga@garshol.priv.no> .. sectionauthor:: Martin v. Löwis <martin@v.loewis.de> +**Source code:** :source:`Lib/xml/sax/xmlreader.py` + +-------------- SAX parsers implement the :class:`XMLReader` interface. They are implemented in a Python module, which must provide a function :func:`create_parser`. This @@ -98,10 +102,12 @@ The :class:`XMLReader` interface supports the following methods: Process an input source, producing SAX events. The *source* object can be a system identifier (a string identifying the input source -- typically a file - name or an URL), a file-like object, or an :class:`InputSource` object. When + name or a URL), a file-like object, or an :class:`InputSource` object. When :meth:`parse` returns, the input is completely processed, and the parser object - can be discarded or reset. As a limitation, the current implementation only - accepts byte streams; processing of character streams is for further study. + can be discarded or reset. + + .. versionchanged:: 3.5 + Added support of character streams. .. method:: XMLReader.getContentHandler() @@ -226,12 +232,12 @@ Instances of :class:`Locator` provide these methods: .. method:: Locator.getColumnNumber() - Return the column number where the current event ends. + Return the column number where the current event begins. .. method:: Locator.getLineNumber() - Return the line number where the current event ends. + Return the line number where the current event begins. .. method:: Locator.getPublicId() @@ -288,8 +294,7 @@ InputSource Objects .. method:: InputSource.setByteStream(bytefile) - Set the byte stream (a Python file-like object which does not perform - byte-to-character conversion) for this input source. + Set the byte stream (a :term:`binary file`) for this input source. The SAX parser will ignore this if there is also a character stream specified, but it will use a byte stream in preference to opening a URI connection itself. @@ -308,8 +313,7 @@ InputSource Objects .. method:: InputSource.setCharacterStream(charfile) - Set the character stream for this input source. (The stream must be a Python 1.6 - Unicode-wrapped file-like that performs conversion to strings.) + Set the character stream (a :term:`text file`) for this input source. If there is a character stream specified, the SAX parser will ignore any byte stream and will not attempt to open a URI connection to the system identifier. diff --git a/Doc/library/xml.sax.rst b/Doc/library/xml.sax.rst index e95d6b0f85..78d6633e09 100644 --- a/Doc/library/xml.sax.rst +++ b/Doc/library/xml.sax.rst @@ -3,10 +3,14 @@ .. module:: xml.sax :synopsis: Package containing SAX2 base classes and convenience functions. + .. moduleauthor:: Lars Marius Garshol <larsga@garshol.priv.no> .. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org> .. sectionauthor:: Martin v. Löwis <martin@v.loewis.de> +**Source code:** :source:`Lib/xml/sax/__init__.py` + +-------------- The :mod:`xml.sax` package provides a number of modules which implement the Simple API for XML (SAX) interface for Python. The package itself provides the @@ -47,7 +51,11 @@ The convenience functions are: .. function:: parseString(string, handler, error_handler=handler.ErrorHandler()) Similar to :func:`parse`, but parses from a buffer *string* received as a - parameter. + parameter. *string* must be a :class:`str` instance or a + :term:`bytes-like object`. + + .. versionchanged:: 3.5 + Added support of :class:`str` instances. A typical SAX application uses three kinds of objects: readers, handlers and input sources. "Reader" in this context is another term for parser, i.e. some diff --git a/Doc/library/xml.sax.utils.rst b/Doc/library/xml.sax.utils.rst index 14cf07839b..538b7980bd 100644 --- a/Doc/library/xml.sax.utils.rst +++ b/Doc/library/xml.sax.utils.rst @@ -3,9 +3,13 @@ .. module:: xml.sax.saxutils :synopsis: Convenience functions and classes for use with SAX. + .. moduleauthor:: Lars Marius Garshol <larsga@garshol.priv.no> .. sectionauthor:: Martin v. Löwis <martin@v.loewis.de> +**Source code:** :source:`Lib/xml/sax/saxutils.py` + +-------------- The module :mod:`xml.sax.saxutils` contains a number of classes and functions that are commonly useful when creating SAX applications, either in direct use, diff --git a/Doc/library/xmlrpc.client.rst b/Doc/library/xmlrpc.client.rst index cc5e83a706..e7916d2bff 100644 --- a/Doc/library/xmlrpc.client.rst +++ b/Doc/library/xmlrpc.client.rst @@ -3,18 +3,18 @@ .. module:: xmlrpc.client :synopsis: XML-RPC client access. + .. moduleauthor:: Fredrik Lundh <fredrik@pythonware.com> .. sectionauthor:: Eric S. Raymond <esr@snark.thyrsus.com> +**Source code:** :source:`Lib/xmlrpc/client.py` .. XXX Not everything is documented yet. It might be good to describe Marshaller, Unmarshaller, getparser and Transport. -**Source code:** :source:`Lib/xmlrpc/client.py` - -------------- -XML-RPC is a Remote Procedure Call method that uses XML passed via HTTP as a +XML-RPC is a Remote Procedure Call method that uses XML passed via HTTP(S) as a transport. With it, a client can call methods with parameters on a remote server (the server is named by a URI) and get back structured data. This module supports writing XML-RPC client code; it handles all the details of translating @@ -27,10 +27,10 @@ between conformable Python objects and XML on the wire. constructed data. If you need to parse untrusted or unauthenticated data see :ref:`xml-vulnerabilities`. -.. versionchanged:: 3.4.3 +.. versionchanged:: 3.5 - For https URIs, :mod:`xmlrpc.client` now performs all the necessary - certificate and hostname checks by default + For HTTPS URIs, :mod:`xmlrpc.client` now performs all the necessary + certificate and hostname checks by default. .. class:: ServerProxy(uri, transport=None, encoding=None, verbose=False, \ allow_none=False, use_datetime=False, \ @@ -46,15 +46,19 @@ between conformable Python objects and XML on the wire. :class:`SafeTransport` instance for https: URLs and an internal HTTP :class:`Transport` instance otherwise. The optional third argument is an encoding, by default UTF-8. The optional fourth argument is a debugging flag. + + The following parameters govern the use of the returned proxy instance. If *allow_none* is true, the Python constant ``None`` will be translated into XML; the default behaviour is for ``None`` to raise a :exc:`TypeError`. This is a commonly-used extension to the XML-RPC specification, but isn't supported by - all clients and servers; see http://ontosys.com/xml-rpc/extensions.php for a - description. The *use_builtin_types* flag can be used to cause date/time values + all clients and servers; see `http://ontosys.com/xml-rpc/extensions.php + <https://web.archive.org/web/20130120074804/http://ontosys.com/xml-rpc/extensions.php>`_ + for a description. + The *use_builtin_types* flag can be used to cause date/time values to be presented as :class:`datetime.datetime` objects and binary data to be presented as :class:`bytes` objects; this flag is false by default. - :class:`datetime.datetime` and :class:`bytes` objects may be passed to calls. - + :class:`datetime.datetime`, :class:`bytes` and :class:`bytearray` objects + may be passed to calls. The obsolete *use_datetime* flag is similar to *use_builtin_types* but it applies only to date/time values. @@ -63,7 +67,7 @@ between conformable Python objects and XML on the wire. portion will be base64-encoded as an HTTP 'Authorization' header, and sent to the remote server as part of the connection process when invoking an XML-RPC method. You only need to use this if the remote server requires a Basic - Authentication user and password. If an HTTPS url is provided, *context* may + Authentication user and password. If an HTTPS URL is provided, *context* may be :class:`ssl.SSLContext` and configures the SSL settings of the underlying HTTPS connection. @@ -73,42 +77,43 @@ between conformable Python objects and XML on the wire. methods it supports (service discovery) and fetch other server-associated metadata. - :class:`ServerProxy` instance methods take Python basic types and objects as - arguments and return Python basic types and classes. Types that are conformable - (e.g. that can be marshalled through XML), include the following (and except - where noted, they are unmarshalled as the same Python type): + Types that are conformable (e.g. that can be marshalled through XML), + include the following (and except where noted, they are unmarshalled + as the same Python type): .. tabularcolumns:: |l|L| - +---------------------------------+---------------------------------------------+ - | Name | Meaning | - +=================================+=============================================+ - | :const:`boolean` | The :const:`True` and :const:`False` | - | | constants | - +---------------------------------+---------------------------------------------+ - | :const:`integers` | Pass in directly | - +---------------------------------+---------------------------------------------+ - | :const:`floating-point numbers` | Pass in directly | - +---------------------------------+---------------------------------------------+ - | :const:`strings` | Pass in directly | - +---------------------------------+---------------------------------------------+ - | :const:`arrays` | Any Python sequence type containing | - | | conformable elements. Arrays are returned | - | | as lists | - +---------------------------------+---------------------------------------------+ - | :const:`structures` | A Python dictionary. Keys must be strings, | - | | values may be any conformable type. Objects | - | | of user-defined classes can be passed in; | - | | only their *__dict__* attribute is | - | | transmitted. | - +---------------------------------+---------------------------------------------+ - | :const:`dates` | In seconds since the epoch. Pass in an | - | | instance of the :class:`DateTime` class or | - | | a :class:`datetime.datetime` instance. | - +---------------------------------+---------------------------------------------+ - | :const:`binary data` | Pass in an instance of the :class:`Binary` | - | | wrapper class or a :class:`bytes` instance. | - +---------------------------------+---------------------------------------------+ + +----------------------+-------------------------------------------------------+ + | XML-RPC type | Python type | + +======================+=======================================================+ + | ``boolean`` | :class:`bool` | + +----------------------+-------------------------------------------------------+ + | ``int`` or ``i4`` | :class:`int` in range from -2147483648 to 2147483647. | + +----------------------+-------------------------------------------------------+ + | ``double`` | :class:`float` | + +----------------------+-------------------------------------------------------+ + | ``string`` | :class:`str` | + +----------------------+-------------------------------------------------------+ + | ``array`` | :class:`list` or :class:`tuple` containing | + | | conformable elements. Arrays are returned as | + | | :class:`lists <list>`. | + +----------------------+-------------------------------------------------------+ + | ``struct`` | :class:`dict`. Keys must be strings, values may be | + | | any conformable type. Objects of user-defined | + | | classes can be passed in; only their | + | | :attr:`~object.__dict__` attribute is transmitted. | + +----------------------+-------------------------------------------------------+ + | ``dateTime.iso8601`` | :class:`DateTime` or :class:`datetime.datetime`. | + | | Returned type depends on values of | + | | *use_builtin_types* and *use_datetime* flags. | + +----------------------+-------------------------------------------------------+ + | ``base64`` | :class:`Binary`, :class:`bytes` or | + | | :class:`bytearray`. Returned type depends on the | + | | value of the *use_builtin_types* flag. | + +----------------------+-------------------------------------------------------+ + | ``nil`` | The ``None`` constant. Passing is allowed only if | + | | *allow_none* is true. | + +----------------------+-------------------------------------------------------+ This is the full set of data types supported by XML-RPC. Method calls may also raise a special :exc:`Fault` instance, used to signal XML-RPC server errors, or @@ -123,13 +128,13 @@ between conformable Python objects and XML on the wire. the control characters with ASCII values between 0 and 31 (except, of course, tab, newline and carriage return); failing to do this will result in an XML-RPC request that isn't well-formed XML. If you have to pass arbitrary bytes - via XML-RPC, use the :class:`bytes` class or the class:`Binary` wrapper class - described below. + via XML-RPC, use :class:`bytes` or :class:`bytearray` classes or the + :class:`Binary` wrapper class described below. :class:`Server` is retained as an alias for :class:`ServerProxy` for backwards compatibility. New code should use :class:`ServerProxy`. - .. versionchanged:: 3.4.3 + .. versionchanged:: 3.5 Added the *context* argument. @@ -142,7 +147,7 @@ between conformable Python objects and XML on the wire. `XML-RPC Introspection <http://xmlrpc-c.sourceforge.net/introspection.html>`_ Describes the XML-RPC protocol extension for introspection. - `XML-RPC Specification <http://www.xmlrpc.com/spec>`_ + `XML-RPC Specification <http://xmlrpc.scripting.com/spec.html>`_ The official specification. `Unofficial XML-RPC Errata <http://effbot.org/zone/xmlrpc-errata.htm>`_ @@ -164,7 +169,7 @@ returning a value, which may be either returned data in a conformant type or a :class:`Fault` or :class:`ProtocolError` object indicating an error. Servers that support the XML introspection API support some common methods -grouped under the reserved :attr:`system` attribute: +grouped under the reserved :attr:`~ServerProxy.system` attribute: .. method:: ServerProxy.system.listMethods() @@ -200,13 +205,18 @@ grouped under the reserved :attr:`system` attribute: no such string is available, an empty string is returned. The documentation string may contain HTML markup. +.. versionchanged:: 3.5 + + Instances of :class:`ServerProxy` support the :term:`context manager` protocol + for closing the underlying transport. + A working example follows. The server code:: from xmlrpc.server import SimpleXMLRPCServer def is_even(n): - return n%2 == 0 + return n % 2 == 0 server = SimpleXMLRPCServer(("localhost", 8000)) print("Listening on port 8000...") @@ -217,33 +227,35 @@ The client code for the preceding server:: import xmlrpc.client - proxy = xmlrpc.client.ServerProxy("http://localhost:8000/") - print("3 is even: %s" % str(proxy.is_even(3))) - print("100 is even: %s" % str(proxy.is_even(100))) + with xmlrpc.client.ServerProxy("http://localhost:8000/") as proxy: + print("3 is even: %s" % str(proxy.is_even(3))) + print("100 is even: %s" % str(proxy.is_even(100))) .. _datetime-objects: DateTime Objects ---------------- -This class may be initialized with seconds since the epoch, a time -tuple, an ISO 8601 time/date string, or a :class:`datetime.datetime` -instance. It has the following methods, supported mainly for internal -use by the marshalling/unmarshalling code: +.. class:: DateTime + This class may be initialized with seconds since the epoch, a time + tuple, an ISO 8601 time/date string, or a :class:`datetime.datetime` + instance. It has the following methods, supported mainly for internal + use by the marshalling/unmarshalling code: -.. method:: DateTime.decode(string) - Accept a string as the instance's new time value. + .. method:: decode(string) + Accept a string as the instance's new time value. -.. method:: DateTime.encode(out) - Write the XML-RPC encoding of this :class:`DateTime` item to the *out* stream - object. + .. method:: encode(out) -It also supports certain of Python's built-in operators through rich comparison -and :meth:`__repr__` methods. + Write the XML-RPC encoding of this :class:`DateTime` item to the *out* stream + object. + + It also supports certain of Python's built-in operators through rich comparison + and :meth:`__repr__` methods. A working example follows. The server code:: @@ -277,36 +289,38 @@ The client code for the preceding server:: Binary Objects -------------- -This class may be initialized from bytes data (which may include NULs). The -primary access to the content of a :class:`Binary` object is provided by an -attribute: +.. class:: Binary + + This class may be initialized from bytes data (which may include NULs). The + primary access to the content of a :class:`Binary` object is provided by an + attribute: -.. attribute:: Binary.data + .. attribute:: data - The binary data encapsulated by the :class:`Binary` instance. The data is - provided as a :class:`bytes` object. + The binary data encapsulated by the :class:`Binary` instance. The data is + provided as a :class:`bytes` object. -:class:`Binary` objects have the following methods, supported mainly for -internal use by the marshalling/unmarshalling code: + :class:`Binary` objects have the following methods, supported mainly for + internal use by the marshalling/unmarshalling code: -.. method:: Binary.decode(bytes) + .. method:: decode(bytes) - Accept a base64 :class:`bytes` object and decode it as the instance's new data. + Accept a base64 :class:`bytes` object and decode it as the instance's new data. -.. method:: Binary.encode(out) + .. method:: encode(out) - Write the XML-RPC base 64 encoding of this binary item to the out stream object. + Write the XML-RPC base 64 encoding of this binary item to the *out* stream object. - The encoded data will have newlines every 76 characters as per - `RFC 2045 section 6.8 <http://tools.ietf.org/html/rfc2045#section-6.8>`_, - which was the de facto standard base64 specification when the - XML-RPC spec was written. + The encoded data will have newlines every 76 characters as per + `RFC 2045 section 6.8 <https://tools.ietf.org/html/rfc2045#section-6.8>`_, + which was the de facto standard base64 specification when the + XML-RPC spec was written. -It also supports certain of Python's built-in operators through :meth:`__eq__` -and :meth:`__ne__` methods. + It also supports certain of Python's built-in operators through :meth:`__eq__` + and :meth:`__ne__` methods. Example usage of the binary objects. We're going to transfer an image over XMLRPC:: @@ -337,18 +351,20 @@ The client gets the image and saves it to a file:: Fault Objects ------------- -A :class:`Fault` object encapsulates the content of an XML-RPC fault tag. Fault -objects have the following attributes: +.. class:: Fault + + A :class:`Fault` object encapsulates the content of an XML-RPC fault tag. Fault + objects have the following attributes: -.. attribute:: Fault.faultCode + .. attribute:: faultCode - A string indicating the fault type. + A string indicating the fault type. -.. attribute:: Fault.faultString + .. attribute:: faultString - A string containing a diagnostic message associated with the fault. + A string containing a diagnostic message associated with the fault. In the following example we're going to intentionally cause a :exc:`Fault` by returning a complex type object. The server code:: @@ -357,7 +373,7 @@ returning a complex type object. The server code:: # A marshalling error is going to occur because we're returning a # complex number - def add(x,y): + def add(x, y): return x+y+0j server = SimpleXMLRPCServer(("localhost", 8000)) @@ -385,37 +401,39 @@ The client code for the preceding server:: ProtocolError Objects --------------------- -A :class:`ProtocolError` object describes a protocol error in the underlying -transport layer (such as a 404 'not found' error if the server named by the URI -does not exist). It has the following attributes: +.. class:: ProtocolError + + A :class:`ProtocolError` object describes a protocol error in the underlying + transport layer (such as a 404 'not found' error if the server named by the URI + does not exist). It has the following attributes: -.. attribute:: ProtocolError.url + .. attribute:: url - The URI or URL that triggered the error. + The URI or URL that triggered the error. -.. attribute:: ProtocolError.errcode + .. attribute:: errcode - The error code. + The error code. -.. attribute:: ProtocolError.errmsg + .. attribute:: errmsg - The error message or diagnostic string. + The error message or diagnostic string. -.. attribute:: ProtocolError.headers + .. attribute:: headers - A dict containing the headers of the HTTP/HTTPS request that triggered the - error. + A dict containing the headers of the HTTP/HTTPS request that triggered the + error. In the following example we're going to intentionally cause a :exc:`ProtocolError` by providing an invalid URI:: import xmlrpc.client - # create a ServerProxy with an URI that doesn't respond to XMLRPC requests + # create a ServerProxy with a URI that doesn't respond to XMLRPC requests proxy = xmlrpc.client.ServerProxy("http://google.com/") try: @@ -527,16 +545,16 @@ Example of Client Usage from xmlrpc.client import ServerProxy, Error # server = ServerProxy("http://localhost:8000") # local server - server = ServerProxy("http://betty.userland.com") + with ServerProxy("http://betty.userland.com") as proxy: - print(server) + print(proxy) - try: - print(server.examples.getStateName(41)) - except Error as v: - print("ERROR", v) + try: + print(proxy.examples.getStateName(41)) + except Error as v: + print("ERROR", v) -To access an XML-RPC server through a proxy, you need to define a custom +To access an XML-RPC server through a HTTP proxy, you need to define a custom transport. The following example shows how: .. Example taken from http://lowlife.jp/nobonobo/wiki/xmlrpcwithproxy.html @@ -548,18 +566,21 @@ transport. The following example shows how: class ProxiedTransport(xmlrpc.client.Transport): def set_proxy(self, proxy): self.proxy = proxy + def make_connection(self, host): self.realhost = host - h = http.client.HTTP(self.proxy) + h = http.client.HTTPConnection(self.proxy) return h - def send_request(self, connection, handler, request_body): + + def send_request(self, connection, handler, request_body, debug): connection.putrequest("POST", 'http://%s%s' % (self.realhost, handler)) + def send_host(self, connection, host): connection.putheader('Host', self.realhost) p = ProxiedTransport() p.set_proxy('proxy-server:8080') - server = xmlrpc.client.Server('http://time.xmlrpc.com/RPC2', transport=p) + server = xmlrpc.client.ServerProxy('http://time.xmlrpc.com/RPC2', transport=p) print(server.currentTime.getCurrentTime()) @@ -572,7 +593,7 @@ See :ref:`simplexmlrpcserver-example`. .. rubric:: Footnotes .. [#] This approach has been first presented in `a discussion on xmlrpc.com - <http://web.archive.org/web/20060624230303/http://www.xmlrpc.com/discuss/msgReader$1208?mode=topic>`_. + <https://web.archive.org/web/20060624230303/http://www.xmlrpc.com/discuss/msgReader$1208?mode=topic>`_. .. the link now points to webarchive since the one at .. http://www.xmlrpc.com/discuss/msgReader%241208 is broken (and webadmin .. doesn't reply) diff --git a/Doc/library/xmlrpc.server.rst b/Doc/library/xmlrpc.server.rst index 37d1393eff..1c77e84c9b 100644 --- a/Doc/library/xmlrpc.server.rst +++ b/Doc/library/xmlrpc.server.rst @@ -3,6 +3,7 @@ .. module:: xmlrpc.server :synopsis: Basic XML-RPC server implementations. + .. moduleauthor:: Brian Quinlan <brianq@activestate.com> .. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org> @@ -18,7 +19,7 @@ servers written in Python. Servers can either be free standing, using .. warning:: - The :mod:`xmlrpc.client` module is not secure against maliciously + The :mod:`xmlrpc.server` module is not secure against maliciously constructed data. If you need to parse untrusted or unauthenticated data see :ref:`xml-vulnerabilities`. @@ -292,7 +293,7 @@ requests sent to Python CGI scripts. .. method:: CGIXMLRPCRequestHandler.handle_request(request_text=None) - Handle a XML-RPC request. If *request_text* is given, it should be the POST + Handle an XML-RPC request. If *request_text* is given, it should be the POST data provided by the HTTP server, otherwise the contents of stdin will be used. Example:: diff --git a/Doc/library/zipapp.rst b/Doc/library/zipapp.rst new file mode 100644 index 0000000000..7c7767f0d4 --- /dev/null +++ b/Doc/library/zipapp.rst @@ -0,0 +1,258 @@ +:mod:`zipapp` --- Manage executable python zip archives +======================================================= + +.. module:: zipapp + :synopsis: Manage executable python zip archives + +.. versionadded:: 3.5 + +**Source code:** :source:`Lib/zipapp.py` + +.. index:: + single: Executable Zip Files + +-------------- + +This module provides tools to manage the creation of zip files containing +Python code, which can be :ref:`executed directly by the Python interpreter +<using-on-interface-options>`. The module provides both a +:ref:`zipapp-command-line-interface` and a :ref:`zipapp-python-api`. + + +Basic Example +------------- + +The following example shows how the :ref:`zipapp-command-line-interface` +can be used to create an executable archive from a directory containing +Python code. When run, the archive will execute the ``main`` function from +the module ``myapp`` in the archive. + +.. code-block:: sh + + $ python -m zipapp myapp -m "myapp:main" + $ python myapp.pyz + <output from myapp> + + +.. _zipapp-command-line-interface: + +Command-Line Interface +---------------------- + +When called as a program from the command line, the following form is used: + +.. code-block:: sh + + $ python -m zipapp source [options] + +If *source* is a directory, this will create an archive from the contents of +*source*. If *source* is a file, it should be an archive, and it will be +copied to the target archive (or the contents of its shebang line will be +displayed if the --info option is specified). + +The following options are understood: + +.. program:: zipapp + +.. cmdoption:: -o <output>, --output=<output> + + Write the output to a file named *output*. If this option is not specified, + the output filename will be the same as the input *source*, with the + extension ``.pyz`` added. If an explicit filename is given, it is used as + is (so a ``.pyz`` extension should be included if required). + + An output filename must be specified if the *source* is an archive (and in + that case, *output* must not be the same as *source*). + +.. cmdoption:: -p <interpreter>, --python=<interpreter> + + Add a ``#!`` line to the archive specifying *interpreter* as the command + to run. Also, on POSIX, make the archive executable. The default is to + write no ``#!`` line, and not make the file executable. + +.. cmdoption:: -m <mainfn>, --main=<mainfn> + + Write a ``__main__.py`` file to the archive that executes *mainfn*. The + *mainfn* argument should have the form "pkg.mod:fn", where "pkg.mod" is a + package/module in the archive, and "fn" is a callable in the given module. + The ``__main__.py`` file will execute that callable. + + :option:`--main` cannot be specified when copying an archive. + +.. cmdoption:: --info + + Display the interpreter embedded in the archive, for diagnostic purposes. In + this case, any other options are ignored and SOURCE must be an archive, not a + directory. + +.. cmdoption:: -h, --help + + Print a short usage message and exit. + + +.. _zipapp-python-api: + +Python API +---------- + +The module defines two convenience functions: + + +.. function:: create_archive(source, target=None, interpreter=None, main=None) + + Create an application archive from *source*. The source can be any + of the following: + + * The name of a directory, or a :class:`pathlib.Path` object referring + to a directory, in which case a new application archive will be + created from the content of that directory. + * The name of an existing application archive file, or a :class:`pathlib.Path` + object referring to such a file, in which case the file is copied to + the target (modifying it to reflect the value given for the *interpreter* + argument). The file name should include the ``.pyz`` extension, if required. + * A file object open for reading in bytes mode. The content of the + file should be an application archive, and the file object is + assumed to be positioned at the start of the archive. + + The *target* argument determines where the resulting archive will be + written: + + * If it is the name of a file, or a :class:`pathlb.Path` object, + the archive will be written to that file. + * If it is an open file object, the archive will be written to that + file object, which must be open for writing in bytes mode. + * If the target is omitted (or None), the source must be a directory + and the target will be a file with the same name as the source, with + a ``.pyz`` extension added. + + The *interpreter* argument specifies the name of the Python + interpreter with which the archive will be executed. It is written as + a "shebang" line at the start of the archive. On POSIX, this will be + interpreted by the OS, and on Windows it will be handled by the Python + launcher. Omitting the *interpreter* results in no shebang line being + written. If an interpreter is specified, and the target is a + filename, the executable bit of the target file will be set. + + The *main* argument specifies the name of a callable which will be + used as the main program for the archive. It can only be specified if + the source is a directory, and the source does not already contain a + ``__main__.py`` file. The *main* argument should take the form + "pkg.module:callable" and the archive will be run by importing + "pkg.module" and executing the given callable with no arguments. It + is an error to omit *main* if the source is a directory and does not + contain a ``__main__.py`` file, as otherwise the resulting archive + would not be executable. + + If a file object is specified for *source* or *target*, it is the + caller's responsibility to close it after calling create_archive. + + When copying an existing archive, file objects supplied only need + ``read`` and ``readline``, or ``write`` methods. When creating an + archive from a directory, if the target is a file object it will be + passed to the ``zipfile.ZipFile`` class, and must supply the methods + needed by that class. + +.. function:: get_interpreter(archive) + + Return the interpreter specified in the ``#!`` line at the start of the + archive. If there is no ``#!`` line, return :const:`None`. + The *archive* argument can be a filename or a file-like object open + for reading in bytes mode. It is assumed to be at the start of the archive. + + +.. _zipapp-examples: + +Examples +-------- + +Pack up a directory into an archive, and run it. + +.. code-block:: sh + + $ python -m zipapp myapp + $ python myapp.pyz + <output from myapp> + +The same can be done using the :func:`create_archive` functon:: + + >>> import zipapp + >>> zipapp.create_archive('myapp.pyz', 'myapp') + +To make the application directly executable on POSIX, specify an interpreter +to use. + +.. code-block:: sh + + $ python -m zipapp myapp -p "/usr/bin/env python" + $ ./myapp.pyz + <output from myapp> + +To replace the shebang line on an existing archive, create a modified archive +using the :func:`create_archive` function:: + + >>> import zipapp + >>> zipapp.create_archive('old_archive.pyz', 'new_archive.pyz', '/usr/bin/python3') + +To update the file in place, do the replacement in memory using a :class:`BytesIO` +object, and then overwrite the source afterwards. Note that there is a risk +when overwriting a file in place that an error will result in the loss of +the original file. This code does not protect against such errors, but +production code should do so. Also, this method will only work if the archive +fits in memory:: + + >>> import zipapp + >>> import io + >>> temp = io.BytesIO() + >>> zipapp.create_archive('myapp.pyz', temp, '/usr/bin/python2') + >>> with open('myapp.pyz', 'wb') as f: + >>> f.write(temp.getvalue()) + +Note that if you specify an interpreter and then distribute your application +archive, you need to ensure that the interpreter used is portable. The Python +launcher for Windows supports most common forms of POSIX ``#!`` line, but there +are other issues to consider: + +* If you use "/usr/bin/env python" (or other forms of the "python" command, + such as "/usr/bin/python"), you need to consider that your users may have + either Python 2 or Python 3 as their default, and write your code to work + under both versions. +* If you use an explicit version, for example "/usr/bin/env python3" your + application will not work for users who do not have that version. (This + may be what you want if you have not made your code Python 2 compatible). +* There is no way to say "python X.Y or later", so be careful of using an + exact version like "/usr/bin/env python3.4" as you will need to change your + shebang line for users of Python 3.5, for example. + +The Python Zip Application Archive Format +----------------------------------------- + +Python has been able to execute zip files which contain a ``__main__.py`` file +since version 2.6. In order to be executed by Python, an application archive +simply has to be a standard zip file containing a ``__main__.py`` file which +will be run as the entry point for the application. As usual for any Python +script, the parent of the script (in this case the zip file) will be placed on +:data:`sys.path` and thus further modules can be imported from the zip file. + +The zip file format allows arbitrary data to be prepended to a zip file. The +zip application format uses this ability to prepend a standard POSIX "shebang" +line to the file (``#!/path/to/interpreter``). + +Formally, the Python zip application format is therefore: + +1. An optional shebang line, containing the characters ``b'#!'`` followed by an + interpreter name, and then a newline (``b'\n'``) character. The interpreter + name can be anything acceptable to the OS "shebang" processing, or the Python + launcher on Windows. The interpreter should be encoded in UTF-8 on Windows, + and in :func:`sys.getfilesystemencoding()` on POSIX. +2. Standard zipfile data, as generated by the :mod:`zipfile` module. The + zipfile content *must* include a file called ``__main__.py`` (which must be + in the "root" of the zipfile - i.e., it cannot be in a subdirectory). The + zipfile data can be compressed or uncompressed. + +If an application archive has a shebang line, it may have the executable bit set +on POSIX systems, to allow it to be executed directly. + +There is no requirement that the tools in this module are used to create +application archives - the module is a convenience, but archives in the above +format created by any means are acceptable to Python. + diff --git a/Doc/library/zipfile.rst b/Doc/library/zipfile.rst index 10a094f8b1..abe38c42f1 100644 --- a/Doc/library/zipfile.rst +++ b/Doc/library/zipfile.rst @@ -3,6 +3,7 @@ .. module:: zipfile :synopsis: Read and write ZIP-format archive files. + .. moduleauthor:: James C. Ahlstrom <jim@interet.com> .. sectionauthor:: James C. Ahlstrom <jim@interet.com> @@ -13,8 +14,7 @@ The ZIP file format is a common archive and compression standard. This module provides tools to create, read, write, append, and list a ZIP file. Any advanced use of this module will require an understanding of the format, as -defined in `PKZIP Application Note -<http://www.pkware.com/documents/casestudies/APPNOTE.TXT>`_. +defined in `PKZIP Application Note`_. This module does not currently handle multi-disk ZIP files. It can handle ZIP files that use the ZIP64 extensions @@ -115,7 +115,7 @@ The module defines the following items: .. seealso:: - `PKZIP Application Note <http://www.pkware.com/documents/casestudies/APPNOTE.TXT>`_ + `PKZIP Application Note`_ Documentation on the ZIP file format by Phil Katz, the creator of the format and algorithms used. @@ -134,12 +134,16 @@ ZipFile Objects Open a ZIP file, where *file* can be either a path to a file (a string) or a file-like object. The *mode* parameter should be ``'r'`` to read an existing - file, ``'w'`` to truncate and write a new file, or ``'a'`` to append to an - existing file. If *mode* is ``'a'`` and *file* refers to an existing ZIP + file, ``'w'`` to truncate and write a new file, ``'a'`` to append to an + existing file, or ``'x'`` to exclusively create and write a new file. + If *mode* is ``'x'`` and *file* refers to an existing file, + a :exc:`FileExistsError` will be raised. + If *mode* is ``'a'`` and *file* refers to an existing ZIP file, then additional files are added to it. If *file* does not refer to a ZIP file, then a new ZIP archive is appended to the file. This is meant for adding a ZIP archive to another file (such as :file:`python.exe`). If *mode* is ``a`` and the file does not exist at all, it is created. + If *mode* is ``r`` or ``a``, the file should be seekable. *compression* is the ZIP compression method to use when writing the archive, and should be :const:`ZIP_STORED`, :const:`ZIP_DEFLATED`, :const:`ZIP_BZIP2` or :const:`ZIP_LZMA`; unrecognized @@ -151,7 +155,7 @@ ZipFile Objects extensions when the zipfile is larger than 2 GiB. If it is false :mod:`zipfile` will raise an exception when the ZIP file would require ZIP64 extensions. - If the file is created with mode ``'a'`` or ``'w'`` and then + If the file is created with mode ``'w'``, ``'x'`` or ``'a'`` and then :meth:`closed <close>` without adding any files to the archive, the appropriate ZIP structures for an empty archive will be written to the file. @@ -171,6 +175,10 @@ ZipFile Objects .. versionchanged:: 3.4 ZIP64 extensions are enabled by default. + .. versionchanged:: 3.5 + Added support for writing to unseekable streams. + Added support for the ``'x'`` mode. + .. method:: ZipFile.close() @@ -226,14 +234,8 @@ ZipFile Objects .. note:: - If the ZipFile was created by passing in a file-like object as the first - argument to the constructor, then the object returned by :meth:`.open` shares the - ZipFile's file pointer. Under these circumstances, the object returned by - :meth:`.open` should not be used after any additional operations are performed - on the ZipFile object. If the ZipFile was created by passing in a string (the - filename) as the first argument to the constructor, then :meth:`.open` will - create a new file object that will be held by the ZipExtFile, allowing it to - operate independently of the ZipFile. + Objects returned by :meth:`.open` can operate independently of the + ZipFile. .. note:: @@ -248,7 +250,7 @@ ZipFile Objects .. method:: ZipFile.extract(member, path=None, pwd=None) Extract a member from the archive to the current working directory; *member* - must be its full name or a :class:`ZipInfo` object). Its file information is + must be its full name or a :class:`ZipInfo` object. Its file information is extracted as accurately as possible. *path* specifies a different directory to extract to. *member* can be a filename or a :class:`ZipInfo` object. *pwd* is the password used for encrypted files. @@ -318,7 +320,8 @@ ZipFile Objects *arcname* (by default, this will be the same as *filename*, but without a drive letter and with leading path separators removed). If given, *compress_type* overrides the value given for the *compression* parameter to the constructor for - the new entry. The archive must be open with mode ``'w'`` or ``'a'`` -- calling + the new entry. + The archive must be open with mode ``'w'``, ``'x'`` or ``'a'`` -- calling :meth:`write` on a ZipFile created with mode ``'r'`` will raise a :exc:`RuntimeError`. Calling :meth:`write` on a closed ZipFile will raise a :exc:`RuntimeError`. @@ -340,16 +343,16 @@ ZipFile Objects If ``arcname`` (or ``filename``, if ``arcname`` is not given) contains a null byte, the name of the file in the archive will be truncated at the null byte. +.. method:: ZipFile.writestr(zinfo_or_arcname, data[, compress_type]) -.. method:: ZipFile.writestr(zinfo_or_arcname, bytes[, compress_type]) - - Write the string *bytes* to the archive; *zinfo_or_arcname* is either the file + Write the string *data* to the archive; *zinfo_or_arcname* is either the file name it will be given in the archive, or a :class:`ZipInfo` instance. If it's an instance, at least the filename, date, and time must be given. If it's a - name, the date and time is set to the current date and time. The archive must be - opened with mode ``'w'`` or ``'a'`` -- calling :meth:`writestr` on a ZipFile - created with mode ``'r'`` will raise a :exc:`RuntimeError`. Calling - :meth:`writestr` on a closed ZipFile will raise a :exc:`RuntimeError`. + name, the date and time is set to the current date and time. + The archive must be opened with mode ``'w'``, ``'x'`` or ``'a'`` -- calling + :meth:`writestr` on a ZipFile created with mode ``'r'`` will raise a + :exc:`RuntimeError`. Calling :meth:`writestr` on a closed ZipFile will + raise a :exc:`RuntimeError`. If given, *compress_type* overrides the value given for the *compression* parameter to the constructor for the new entry, or in the *zinfo_or_arcname* @@ -377,7 +380,8 @@ The following data attributes are also available: .. attribute:: ZipFile.comment The comment text associated with the ZIP file. If assigning a comment to a - :class:`ZipFile` instance created with mode 'a' or 'w', this should be a + :class:`ZipFile` instance created with mode ``'w'``, ``'x'`` or ``'a'``, + this should be a string no longer than 65535 bytes. Comments longer than this will be truncated in the written archive when :meth:`close` is called. @@ -407,8 +411,7 @@ The :class:`PyZipFile` constructor takes the same parameters as the archive. If the *optimize* parameter to :class:`PyZipFile` was not given or ``-1``, - the corresponding file is a :file:`\*.pyo` file if available, else a - :file:`\*.pyc` file, compiling if necessary. + the corresponding file is a :file:`\*.pyc` file, compiling if necessary. If the *optimize* parameter to :class:`PyZipFile` was ``0``, ``1`` or ``2``, only files with that optimization level (see :func:`compile`) are @@ -508,8 +511,7 @@ Instances have the following attributes: .. attribute:: ZipInfo.extra - Expansion field data. The `PKZIP Application Note - <http://www.pkware.com/documents/casestudies/APPNOTE.TXT>`_ contains + Expansion field data. The `PKZIP Application Note`_ contains some comments on the internal structure of the data contained in this string. @@ -572,3 +574,4 @@ Instances have the following attributes: Size of the uncompressed file. +.. _PKZIP Application Note: https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT diff --git a/Doc/library/zipimport.rst b/Doc/library/zipimport.rst index 2cf508b85c..46b8c245f7 100644 --- a/Doc/library/zipimport.rst +++ b/Doc/library/zipimport.rst @@ -3,8 +3,10 @@ .. module:: zipimport :synopsis: support for importing Python modules from ZIP archives. + .. moduleauthor:: Just van Rossum <just@letterror.com> +-------------- This module adds the ability to import Python modules (:file:`\*.py`, :file:`\*.py[co]`) and packages from ZIP-format archives. It is usually not @@ -20,17 +22,17 @@ subdirectory. For example, the path :file:`example.zip/lib/` would only import from the :file:`lib/` subdirectory within the archive. Any files may be present in the ZIP archive, but only files :file:`.py` and -:file:`.py[co]` are available for import. ZIP import of dynamic modules +:file:`.pyc` are available for import. ZIP import of dynamic modules (:file:`.pyd`, :file:`.so`) is disallowed. Note that if an archive only contains :file:`.py` files, Python will not attempt to modify the archive by adding the -corresponding :file:`.pyc` or :file:`.pyo` file, meaning that if a ZIP archive +corresponding :file:`.pyc` file, meaning that if a ZIP archive doesn't contain :file:`.pyc` files, importing may be rather slow. ZIP archives with an archive comment are currently not supported. .. seealso:: - `PKZIP Application Note <http://www.pkware.com/documents/casestudies/APPNOTE.TXT>`_ + `PKZIP Application Note <https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT>`_ Documentation on the ZIP file format by Phil Katz, the creator of the format and algorithms used. @@ -145,7 +147,9 @@ Examples -------- Here is an example that imports a module from a ZIP archive - note that the -:mod:`zipimport` module is not explicitly used. :: +:mod:`zipimport` module is not explicitly used. + +.. code-block:: shell-session $ unzip -l example.zip Archive: example.zip @@ -161,4 +165,3 @@ Here is an example that imports a module from a ZIP archive - note that the >>> import jwzthreading >>> jwzthreading.__file__ 'example.zip/jwzthreading.py' - diff --git a/Doc/library/zlib.rst b/Doc/library/zlib.rst index 58fc31b095..1de7baee54 100644 --- a/Doc/library/zlib.rst +++ b/Doc/library/zlib.rst @@ -5,6 +5,7 @@ :synopsis: Low-level interface to compression and decompression routines compatible with gzip. +-------------- For applications that require data compression, the functions in this module allow compression and decompression, using the zlib library. The zlib library @@ -31,22 +32,19 @@ The available exception and functions in this module are: .. function:: adler32(data[, value]) Computes an Adler-32 checksum of *data*. (An Adler-32 checksum is almost as - reliable as a CRC32 but can be computed much more quickly.) If *value* is - present, it is used as the starting value of the checksum; otherwise, a fixed - default value is used. This allows computing a running checksum over the + reliable as a CRC32 but can be computed much more quickly.) The result + is an unsigned 32-bit integer. If *value* is present, it is used as + the starting value of the checksum; otherwise, a default value of 1 + is used. Passing in *value* allows computing a running checksum over the concatenation of several inputs. The algorithm is not cryptographically strong, and should not be used for authentication or digital signatures. Since the algorithm is designed for use as a checksum algorithm, it is not suitable for use as a general hash algorithm. - Always returns an unsigned 32-bit integer. - -.. note:: - To generate the same numeric value across all Python versions and - platforms use adler32(data) & 0xffffffff. If you are only using - the checksum in packed binary format this is not necessary as the - return value is the correct 32bit binary representation - regardless of sign. + .. versionchanged:: 3.0 + Always returns an unsigned value. + To generate the same numeric value across all Python versions and + platforms, use ``adler32(data) & 0xffffffff``. .. function:: compress(data[, level]) @@ -63,17 +61,31 @@ The available exception and functions in this module are: Returns a compression object, to be used for compressing data streams that won't fit into memory at once. - *level* is the compression level -- an integer from ``0`` to ``9``. A value - of ``1`` is fastest and produces the least compression, while a value of + *level* is the compression level -- an integer from ``0`` to ``9`` or ``-1``. + A value of ``1`` is fastest and produces the least compression, while a value of ``9`` is slowest and produces the most. ``0`` is no compression. The default - value is ``6``. + value is ``-1`` (Z_DEFAULT_COMPRESSION). Z_DEFAULT_COMPRESSION represents a default + compromise between speed and compression (currently equivalent to level 6). *method* is the compression algorithm. Currently, the only supported value is ``DEFLATED``. - *wbits* is the base two logarithm of the size of the window buffer. This - should be an integer from ``8`` to ``15``. Higher values give better - compression, but use more memory. + The *wbits* argument controls the size of the history buffer (or the + "window size") used when compressing data, and whether a header and + trailer is included in the output. It can take several ranges of values: + + * +9 to +15: The base-two logarithm of the window size, which + therefore ranges between 512 and 32768. Larger values produce + better compression at the expense of greater memory usage. The + resulting output will include a zlib-specific header and trailer. + + * −9 to −15: Uses the absolute value of *wbits* as the + window size logarithm, while producing a raw output stream with no + header or trailing checksum. + + * +25 to +31 = 16 + (9 to 15): Uses the low 4 bits of the value as the + window size logarithm, while including a basic :program:`gzip` header + and trailing checksum in the output. The *memLevel* argument controls the amount of memory used for the internal compression state. Valid values range from ``1`` to ``9``. @@ -97,42 +109,58 @@ The available exception and functions in this module are: single: Cyclic Redundancy Check single: checksum; Cyclic Redundancy Check - Computes a CRC (Cyclic Redundancy Check) checksum of *data*. If *value* is - present, it is used as the starting value of the checksum; otherwise, a fixed - default value is used. This allows computing a running checksum over the + Computes a CRC (Cyclic Redundancy Check) checksum of *data*. The + result is an unsigned 32-bit integer. If *value* is present, it is used + as the starting value of the checksum; otherwise, a default value of 0 + is used. Passing in *value* allows computing a running checksum over the concatenation of several inputs. The algorithm is not cryptographically strong, and should not be used for authentication or digital signatures. Since the algorithm is designed for use as a checksum algorithm, it is not suitable for use as a general hash algorithm. - Always returns an unsigned 32-bit integer. - - .. note:: - + .. versionchanged:: 3.0 + Always returns an unsigned value. To generate the same numeric value across all Python versions and - platforms, use ``crc32(data) & 0xffffffff``. If you are only using - the checksum in packed binary format this is not necessary as the - return value is the correct 32-bit binary representation - regardless of sign. + platforms, use ``crc32(data) & 0xffffffff``. .. function:: decompress(data[, wbits[, bufsize]]) Decompresses the bytes in *data*, returning a bytes object containing the - uncompressed data. The *wbits* parameter controls the size of the window - buffer, and is discussed further below. + uncompressed data. The *wbits* parameter depends on + the format of *data*, and is discussed further below. If *bufsize* is given, it is used as the initial size of the output buffer. Raises the :exc:`error` exception if any error occurs. - The absolute value of *wbits* is the base two logarithm of the size of the - history buffer (the "window size") used when compressing data. Its absolute - value should be between 8 and 15 for the most recent versions of the zlib - library, larger values resulting in better compression at the expense of greater - memory usage. When decompressing a stream, *wbits* must not be smaller + .. _decompress-wbits: + + The *wbits* parameter controls the size of the history buffer + (or "window size"), and what header and trailer format is expected. + It is similar to the parameter for :func:`compressobj`, but accepts + more ranges of values: + + * +8 to +15: The base-two logarithm of the window size. The input + must include a zlib header and trailer. + + * 0: Automatically determine the window size from the zlib header. + Only supported since zlib 1.2.3.5. + + * −8 to −15: Uses the absolute value of *wbits* as the window size + logarithm. The input must be a raw stream with no header or trailer. + + * +24 to +31 = 16 + (8 to 15): Uses the low 4 bits of the value as + the window size logarithm. The input must include a gzip header and + trailer. + + * +40 to +47 = 32 + (8 to 15): Uses the low 4 bits of the value as + the window size logarithm, and automatically accepts either + the zlib or gzip format. + + When decompressing a stream, the window size must not be smaller than the size originally used to compress the stream; using a too-small - value will result in an exception. The default value is therefore the - highest value, 15. When *wbits* is negative, the standard - :program:`gzip` header is suppressed. + value may result in an :exc:`error` exception. The default *wbits* value + is 15, which corresponds to the largest window size and requires a zlib + header and trailer to be included. *bufsize* is the initial size of the buffer used to hold decompressed data. If more space is required, the buffer size will be increased as needed, so you @@ -145,7 +173,9 @@ The available exception and functions in this module are: Returns a decompression object, to be used for decompressing data streams that won't fit into memory at once. - The *wbits* parameter controls the size of the window buffer. + The *wbits* parameter controls the size of the history buffer (or the + "window size"), and what header and trailer format is expected. It has + the same meaning as `described for decompress() <#decompress-wbits>`__. The *zdict* parameter specifies a predefined compression dictionary. If provided, this must be the same dictionary as was used by the compressor that diff --git a/Doc/license.rst b/Doc/license.rst index 8a613b21f3..8843116a14 100644 --- a/Doc/license.rst +++ b/Doc/license.rst @@ -11,12 +11,12 @@ History of the software ======================= Python was created in the early 1990s by Guido van Rossum at Stichting -Mathematisch Centrum (CWI, see http://www.cwi.nl/) in the Netherlands as a +Mathematisch Centrum (CWI, see https://www.cwi.nl/) in the Netherlands as a successor of a language called ABC. Guido remains Python's principal author, although it includes many contributions from others. In 1995, Guido continued his work on Python at the Corporation for National -Research Initiatives (CNRI, see http://www.cnri.reston.va.us/) in Reston, +Research Initiatives (CNRI, see https://www.cnri.reston.va.us/) in Reston, Virginia where he released several versions of the software. In May 2000, Guido and the Python core development team moved to BeOpen.com to @@ -27,7 +27,7 @@ https://www.python.org/psf/) was formed, a non-profit organization created specifically to own Python-related Intellectual Property. Zope Corporation is a sponsoring member of the PSF. -All Python releases are Open Source (see http://opensource.org/ for the Open +All Python releases are Open Source (see https://opensource.org/ for the Open Source Definition). Historically, most, but not all, Python releases have also been GPL-compatible; the table below summarizes the various releases. @@ -73,181 +73,189 @@ Terms and conditions for accessing or otherwise using Python ============================================================ -.. centered:: PSF LICENSE AGREEMENT FOR PYTHON |release| - -#. This LICENSE AGREEMENT is between the Python Software Foundation ("PSF"), and - the Individual or Organization ("Licensee") accessing and otherwise using Python - |release| software in source or binary form and its associated documentation. - -#. Subject to the terms and conditions of this License Agreement, PSF hereby - grants Licensee a nonexclusive, royalty-free, world-wide license to reproduce, - analyze, test, perform and/or display publicly, prepare derivative works, - distribute, and otherwise use Python |release| alone or in any derivative - version, provided, however, that PSF's License Agreement and PSF's notice of - copyright, i.e., "Copyright © 2001-2016 Python Software Foundation; All Rights - Reserved" are retained in Python |release| alone or in any derivative version - prepared by Licensee. - -#. In the event Licensee prepares a derivative work that is based on or - incorporates Python |release| or any part thereof, and wants to make the - derivative work available to others as provided herein, then Licensee hereby - agrees to include in any such work a brief summary of the changes made to Python - |release|. - -#. PSF is making Python |release| available to Licensee on an "AS IS" basis. - PSF MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR IMPLIED. BY WAY OF - EXAMPLE, BUT NOT LIMITATION, PSF MAKES NO AND DISCLAIMS ANY REPRESENTATION OR - WARRANTY OF MERCHANTABILITY OR FITNESS FOR ANY PARTICULAR PURPOSE OR THAT THE - USE OF PYTHON |release| WILL NOT INFRINGE ANY THIRD PARTY RIGHTS. - -#. PSF SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF PYTHON |release| - FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS AS A RESULT OF - MODIFYING, DISTRIBUTING, OR OTHERWISE USING PYTHON |release|, OR ANY DERIVATIVE - THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. - -#. This License Agreement will automatically terminate upon a material breach of - its terms and conditions. - -#. Nothing in this License Agreement shall be deemed to create any relationship - of agency, partnership, or joint venture between PSF and Licensee. This License - Agreement does not grant permission to use PSF trademarks or trade name in a - trademark sense to endorse or promote products or services of Licensee, or any - third party. - -#. By copying, installing or otherwise using Python |release|, Licensee agrees - to be bound by the terms and conditions of this License Agreement. - - -.. centered:: BEOPEN.COM LICENSE AGREEMENT FOR PYTHON 2.0 - - -.. centered:: BEOPEN PYTHON OPEN SOURCE LICENSE AGREEMENT VERSION 1 - -#. This LICENSE AGREEMENT is between BeOpen.com ("BeOpen"), having an office at - 160 Saratoga Avenue, Santa Clara, CA 95051, and the Individual or Organization - ("Licensee") accessing and otherwise using this software in source or binary - form and its associated documentation ("the Software"). - -#. Subject to the terms and conditions of this BeOpen Python License Agreement, - BeOpen hereby grants Licensee a non-exclusive, royalty-free, world-wide license - to reproduce, analyze, test, perform and/or display publicly, prepare derivative - works, distribute, and otherwise use the Software alone or in any derivative - version, provided, however, that the BeOpen Python License is retained in the - Software, alone or in any derivative version prepared by Licensee. - -#. BeOpen is making the Software available to Licensee on an "AS IS" basis. - BEOPEN MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR IMPLIED. BY WAY OF - EXAMPLE, BUT NOT LIMITATION, BEOPEN MAKES NO AND DISCLAIMS ANY REPRESENTATION OR - WARRANTY OF MERCHANTABILITY OR FITNESS FOR ANY PARTICULAR PURPOSE OR THAT THE - USE OF THE SOFTWARE WILL NOT INFRINGE ANY THIRD PARTY RIGHTS. - -#. BEOPEN SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF THE SOFTWARE FOR - ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS AS A RESULT OF USING, - MODIFYING OR DISTRIBUTING THE SOFTWARE, OR ANY DERIVATIVE THEREOF, EVEN IF - ADVISED OF THE POSSIBILITY THEREOF. - -#. This License Agreement will automatically terminate upon a material breach of - its terms and conditions. - -#. This License Agreement shall be governed by and interpreted in all respects - by the law of the State of California, excluding conflict of law provisions. - Nothing in this License Agreement shall be deemed to create any relationship of - agency, partnership, or joint venture between BeOpen and Licensee. This License - Agreement does not grant permission to use BeOpen trademarks or trade names in a - trademark sense to endorse or promote products or services of Licensee, or any - third party. As an exception, the "BeOpen Python" logos available at - http://www.pythonlabs.com/logos.html may be used according to the permissions - granted on that web page. - -#. By copying, installing or otherwise using the software, Licensee agrees to be - bound by the terms and conditions of this License Agreement. - - -.. centered:: CNRI LICENSE AGREEMENT FOR PYTHON 1.6.1 - -#. This LICENSE AGREEMENT is between the Corporation for National Research - Initiatives, having an office at 1895 Preston White Drive, Reston, VA 20191 - ("CNRI"), and the Individual or Organization ("Licensee") accessing and - otherwise using Python 1.6.1 software in source or binary form and its - associated documentation. - -#. Subject to the terms and conditions of this License Agreement, CNRI hereby - grants Licensee a nonexclusive, royalty-free, world-wide license to reproduce, - analyze, test, perform and/or display publicly, prepare derivative works, - distribute, and otherwise use Python 1.6.1 alone or in any derivative version, - provided, however, that CNRI's License Agreement and CNRI's notice of copyright, - i.e., "Copyright © 1995-2001 Corporation for National Research Initiatives; All - Rights Reserved" are retained in Python 1.6.1 alone or in any derivative version - prepared by Licensee. Alternately, in lieu of CNRI's License Agreement, - Licensee may substitute the following text (omitting the quotes): "Python 1.6.1 - is made available subject to the terms and conditions in CNRI's License - Agreement. This Agreement together with Python 1.6.1 may be located on the - Internet using the following unique, persistent identifier (known as a handle): - 1895.22/1013. This Agreement may also be obtained from a proxy server on the - Internet using the following URL: http://hdl.handle.net/1895.22/1013." - -#. In the event Licensee prepares a derivative work that is based on or - incorporates Python 1.6.1 or any part thereof, and wants to make the derivative - work available to others as provided herein, then Licensee hereby agrees to - include in any such work a brief summary of the changes made to Python 1.6.1. - -#. CNRI is making Python 1.6.1 available to Licensee on an "AS IS" basis. CNRI - MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR IMPLIED. BY WAY OF EXAMPLE, - BUT NOT LIMITATION, CNRI MAKES NO AND DISCLAIMS ANY REPRESENTATION OR WARRANTY - OF MERCHANTABILITY OR FITNESS FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF - PYTHON 1.6.1 WILL NOT INFRINGE ANY THIRD PARTY RIGHTS. - -#. CNRI SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF PYTHON 1.6.1 FOR - ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS AS A RESULT OF - MODIFYING, DISTRIBUTING, OR OTHERWISE USING PYTHON 1.6.1, OR ANY DERIVATIVE - THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. - -#. This License Agreement will automatically terminate upon a material breach of - its terms and conditions. - -#. This License Agreement shall be governed by the federal intellectual property - law of the United States, including without limitation the federal copyright - law, and, to the extent such U.S. federal law does not apply, by the law of the - Commonwealth of Virginia, excluding Virginia's conflict of law provisions. - Notwithstanding the foregoing, with regard to derivative works based on Python - 1.6.1 that incorporate non-separable material that was previously distributed - under the GNU General Public License (GPL), the law of the Commonwealth of - Virginia shall govern this License Agreement only as to issues arising under or - with respect to Paragraphs 4, 5, and 7 of this License Agreement. Nothing in - this License Agreement shall be deemed to create any relationship of agency, - partnership, or joint venture between CNRI and Licensee. This License Agreement - does not grant permission to use CNRI trademarks or trade name in a trademark - sense to endorse or promote products or services of Licensee, or any third - party. - -#. By clicking on the "ACCEPT" button where indicated, or by copying, installing - or otherwise using Python 1.6.1, Licensee agrees to be bound by the terms and - conditions of this License Agreement. - - -.. centered:: ACCEPT - - -.. centered:: CWI LICENSE AGREEMENT FOR PYTHON 0.9.0 THROUGH 1.2 - -Copyright © 1991 - 1995, Stichting Mathematisch Centrum Amsterdam, The -Netherlands. All rights reserved. - -Permission to use, copy, modify, and distribute this software and its -documentation for any purpose and without fee is hereby granted, provided that -the above copyright notice appear in all copies and that both that copyright -notice and this permission notice appear in supporting documentation, and that -the name of Stichting Mathematisch Centrum or CWI not be used in advertising or -publicity pertaining to distribution of the software without specific, written -prior permission. - -STICHTING MATHEMATISCH CENTRUM DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS -SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS, IN NO -EVENT SHALL STICHTING MATHEMATISCH CENTRUM BE LIABLE FOR ANY SPECIAL, INDIRECT -OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, -DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS -ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS -SOFTWARE. +PSF LICENSE AGREEMENT FOR PYTHON |release| +------------------------------------------ + +.. parsed-literal:: + + 1. This LICENSE AGREEMENT is between the Python Software Foundation ("PSF"), and + the Individual or Organization ("Licensee") accessing and otherwise using Python + |release| software in source or binary form and its associated documentation. + + 2. Subject to the terms and conditions of this License Agreement, PSF hereby + grants Licensee a nonexclusive, royalty-free, world-wide license to reproduce, + analyze, test, perform and/or display publicly, prepare derivative works, + distribute, and otherwise use Python |release| alone or in any derivative + version, provided, however, that PSF's License Agreement and PSF's notice of + copyright, i.e., "Copyright © 2001-2016 Python Software Foundation; All Rights + Reserved" are retained in Python |release| alone or in any derivative version + prepared by Licensee. + + 3. In the event Licensee prepares a derivative work that is based on or + incorporates Python |release| or any part thereof, and wants to make the + derivative work available to others as provided herein, then Licensee hereby + agrees to include in any such work a brief summary of the changes made to Python + |release|. + + 4. PSF is making Python |release| available to Licensee on an "AS IS" basis. + PSF MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR IMPLIED. BY WAY OF + EXAMPLE, BUT NOT LIMITATION, PSF MAKES NO AND DISCLAIMS ANY REPRESENTATION OR + WARRANTY OF MERCHANTABILITY OR FITNESS FOR ANY PARTICULAR PURPOSE OR THAT THE + USE OF PYTHON |release| WILL NOT INFRINGE ANY THIRD PARTY RIGHTS. + + 5. PSF SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF PYTHON |release| + FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS AS A RESULT OF + MODIFYING, DISTRIBUTING, OR OTHERWISE USING PYTHON |release|, OR ANY DERIVATIVE + THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. + + 6. This License Agreement will automatically terminate upon a material breach of + its terms and conditions. + + 7. Nothing in this License Agreement shall be deemed to create any relationship + of agency, partnership, or joint venture between PSF and Licensee. This License + Agreement does not grant permission to use PSF trademarks or trade name in a + trademark sense to endorse or promote products or services of Licensee, or any + third party. + + 8. By copying, installing or otherwise using Python |release|, Licensee agrees + to be bound by the terms and conditions of this License Agreement. + + +BEOPEN.COM LICENSE AGREEMENT FOR PYTHON 2.0 +------------------------------------------- + +BEOPEN PYTHON OPEN SOURCE LICENSE AGREEMENT VERSION 1 + +.. parsed-literal:: + + 1. This LICENSE AGREEMENT is between BeOpen.com ("BeOpen"), having an office at + 160 Saratoga Avenue, Santa Clara, CA 95051, and the Individual or Organization + ("Licensee") accessing and otherwise using this software in source or binary + form and its associated documentation ("the Software"). + + 2. Subject to the terms and conditions of this BeOpen Python License Agreement, + BeOpen hereby grants Licensee a non-exclusive, royalty-free, world-wide license + to reproduce, analyze, test, perform and/or display publicly, prepare derivative + works, distribute, and otherwise use the Software alone or in any derivative + version, provided, however, that the BeOpen Python License is retained in the + Software, alone or in any derivative version prepared by Licensee. + + 3. BeOpen is making the Software available to Licensee on an "AS IS" basis. + BEOPEN MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR IMPLIED. BY WAY OF + EXAMPLE, BUT NOT LIMITATION, BEOPEN MAKES NO AND DISCLAIMS ANY REPRESENTATION OR + WARRANTY OF MERCHANTABILITY OR FITNESS FOR ANY PARTICULAR PURPOSE OR THAT THE + USE OF THE SOFTWARE WILL NOT INFRINGE ANY THIRD PARTY RIGHTS. + + 4. BEOPEN SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF THE SOFTWARE FOR + ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS AS A RESULT OF USING, + MODIFYING OR DISTRIBUTING THE SOFTWARE, OR ANY DERIVATIVE THEREOF, EVEN IF + ADVISED OF THE POSSIBILITY THEREOF. + + 5. This License Agreement will automatically terminate upon a material breach of + its terms and conditions. + + 6. This License Agreement shall be governed by and interpreted in all respects + by the law of the State of California, excluding conflict of law provisions. + Nothing in this License Agreement shall be deemed to create any relationship of + agency, partnership, or joint venture between BeOpen and Licensee. This License + Agreement does not grant permission to use BeOpen trademarks or trade names in a + trademark sense to endorse or promote products or services of Licensee, or any + third party. As an exception, the "BeOpen Python" logos available at + http://www.pythonlabs.com/logos.html may be used according to the permissions + granted on that web page. + + 7. By copying, installing or otherwise using the software, Licensee agrees to be + bound by the terms and conditions of this License Agreement. + + +CNRI LICENSE AGREEMENT FOR PYTHON 1.6.1 +--------------------------------------- + +.. parsed-literal:: + + 1. This LICENSE AGREEMENT is between the Corporation for National Research + Initiatives, having an office at 1895 Preston White Drive, Reston, VA 20191 + ("CNRI"), and the Individual or Organization ("Licensee") accessing and + otherwise using Python 1.6.1 software in source or binary form and its + associated documentation. + + 2. Subject to the terms and conditions of this License Agreement, CNRI hereby + grants Licensee a nonexclusive, royalty-free, world-wide license to reproduce, + analyze, test, perform and/or display publicly, prepare derivative works, + distribute, and otherwise use Python 1.6.1 alone or in any derivative version, + provided, however, that CNRI's License Agreement and CNRI's notice of copyright, + i.e., "Copyright © 1995-2001 Corporation for National Research Initiatives; All + Rights Reserved" are retained in Python 1.6.1 alone or in any derivative version + prepared by Licensee. Alternately, in lieu of CNRI's License Agreement, + Licensee may substitute the following text (omitting the quotes): "Python 1.6.1 + is made available subject to the terms and conditions in CNRI's License + Agreement. This Agreement together with Python 1.6.1 may be located on the + Internet using the following unique, persistent identifier (known as a handle): + 1895.22/1013. This Agreement may also be obtained from a proxy server on the + Internet using the following URL: http://hdl.handle.net/1895.22/1013." + + 3. In the event Licensee prepares a derivative work that is based on or + incorporates Python 1.6.1 or any part thereof, and wants to make the derivative + work available to others as provided herein, then Licensee hereby agrees to + include in any such work a brief summary of the changes made to Python 1.6.1. + + 4. CNRI is making Python 1.6.1 available to Licensee on an "AS IS" basis. CNRI + MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR IMPLIED. BY WAY OF EXAMPLE, + BUT NOT LIMITATION, CNRI MAKES NO AND DISCLAIMS ANY REPRESENTATION OR WARRANTY + OF MERCHANTABILITY OR FITNESS FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF + PYTHON 1.6.1 WILL NOT INFRINGE ANY THIRD PARTY RIGHTS. + + 5. CNRI SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF PYTHON 1.6.1 FOR + ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS AS A RESULT OF + MODIFYING, DISTRIBUTING, OR OTHERWISE USING PYTHON 1.6.1, OR ANY DERIVATIVE + THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. + + 6. This License Agreement will automatically terminate upon a material breach of + its terms and conditions. + + 7. This License Agreement shall be governed by the federal intellectual property + law of the United States, including without limitation the federal copyright + law, and, to the extent such U.S. federal law does not apply, by the law of the + Commonwealth of Virginia, excluding Virginia's conflict of law provisions. + Notwithstanding the foregoing, with regard to derivative works based on Python + 1.6.1 that incorporate non-separable material that was previously distributed + under the GNU General Public License (GPL), the law of the Commonwealth of + Virginia shall govern this License Agreement only as to issues arising under or + with respect to Paragraphs 4, 5, and 7 of this License Agreement. Nothing in + this License Agreement shall be deemed to create any relationship of agency, + partnership, or joint venture between CNRI and Licensee. This License Agreement + does not grant permission to use CNRI trademarks or trade name in a trademark + sense to endorse or promote products or services of Licensee, or any third + party. + + 8. By clicking on the "ACCEPT" button where indicated, or by copying, installing + or otherwise using Python 1.6.1, Licensee agrees to be bound by the terms and + conditions of this License Agreement. + + +CWI LICENSE AGREEMENT FOR PYTHON 0.9.0 THROUGH 1.2 +-------------------------------------------------- + +.. parsed-literal:: + + Copyright © 1991 - 1995, Stichting Mathematisch Centrum Amsterdam, The + Netherlands. All rights reserved. + + Permission to use, copy, modify, and distribute this software and its + documentation for any purpose and without fee is hereby granted, provided that + the above copyright notice appear in all copies and that both that copyright + notice and this permission notice appear in supporting documentation, and that + the name of Stichting Mathematisch Centrum or CWI not be used in advertising or + publicity pertaining to distribution of the software without specific, written + prior permission. + + STICHTING MATHEMATISCH CENTRUM DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS + SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS, IN NO + EVENT SHALL STICHTING MATHEMATISCH CENTRUM BE LIABLE FOR ANY SPECIAL, INDIRECT + OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, + DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS + ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS + SOFTWARE. Licenses and Acknowledgements for Incorporated Software @@ -329,14 +337,14 @@ Project, http://www.wide.ad.jp/. :: without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE PROJECT AND CONTRIBUTORS ``AS IS'' AND - GAI_ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE PROJECT OR CONTRIBUTORS BE LIABLE - FOR GAI_ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) - HOWEVER CAUSED AND ON GAI_ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT - LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN GAI_ANY WAY + HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. @@ -946,5 +954,3 @@ library unless the build is configured ``--with-system-libmpdec``:: LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - - diff --git a/Doc/make.bat b/Doc/make.bat index 251f822abf..5ab80850a3 100644 --- a/Doc/make.bat +++ b/Doc/make.bat @@ -8,9 +8,23 @@ set this=%~n0 if "%SPHINXBUILD%" EQU "" set SPHINXBUILD=sphinx-build
if "%PYTHON%" EQU "" set PYTHON=py
-if DEFINED ProgramFiles(x86) set _PRGMFLS=%ProgramFiles(x86)%
-if NOT DEFINED ProgramFiles(x86) set _PRGMFLS=%ProgramFiles%
-if "%HTMLHELP%" EQU "" set HTMLHELP=%_PRGMFLS%\HTML Help Workshop\hhc.exe
+if "%1" NEQ "htmlhelp" goto :skiphhcsearch
+if exist "%HTMLHELP%" goto :skiphhcsearch
+
+rem Search for HHC in likely places
+set HTMLHELP=
+where hhc /q && set HTMLHELP=hhc && goto :skiphhcsearch
+where /R ..\externals hhc > "%TEMP%\hhc.loc" 2> nul && set /P HTMLHELP= < "%TEMP%\hhc.loc" & del "%TEMP%\hhc.loc"
+if not exist "%HTMLHELP%" where /R "%ProgramFiles(x86)%" hhc > "%TEMP%\hhc.loc" 2> nul && set /P HTMLHELP= < "%TEMP%\hhc.loc" & del "%TEMP%\hhc.loc"
+if not exist "%HTMLHELP%" where /R "%ProgramFiles%" hhc > "%TEMP%\hhc.loc" 2> nul && set /P HTMLHELP= < "%TEMP%\hhc.loc" & del "%TEMP%\hhc.loc"
+if not exist "%HTMLHELP%" (
+ echo.
+ echo.The HTML Help Workshop was not found. Set the HTMLHELP variable
+ echo.to the path to hhc.exe or download and install it from
+ echo.http://msdn.microsoft.com/en-us/library/ms669985
+ exit /B 1
+)
+:skiphhcsearch
if "%DISTVERSION%" EQU "" for /f "usebackq" %%v in (`%PYTHON% tools/extensions/patchlevel.py`) do set DISTVERSION=%%v
@@ -36,7 +50,8 @@ if errorlevel 9009 ( echo.
echo.If you don't have Sphinx installed, grab it from
echo.http://sphinx-doc.org/
- goto end
+ popd
+ exit /B 1
)
rem Targets that do require sphinx-build and have their own label
@@ -76,15 +91,6 @@ if NOT "%PAPER%" == "" ( cmd /C %SPHINXBUILD% %SPHINXOPTS% -b%1 -dbuild\doctrees . %BUILDDIR%\%*
if "%1" EQU "htmlhelp" (
- if not exist "%HTMLHELP%" (
- echo.
- echo.The HTML Help Workshop was not found. Set the HTMLHELP variable
- echo.to the path to hhc.exe or download and install it from
- echo.http://msdn.microsoft.com/en-us/library/ms669985
- rem Set errorlevel to 1 and exit
- cmd /C exit /b 1
- goto end
- )
cmd /C "%HTMLHELP%" build\htmlhelp\python%DISTVERSION:.=%.hhp
rem hhc.exe seems to always exit with code 1, reset to 0 for less than 2
if not errorlevel 2 cmd /C exit /b 0
diff --git a/Doc/reference/compound_stmts.rst b/Doc/reference/compound_stmts.rst index 5dd17eb030..88b94eaa95 100644 --- a/Doc/reference/compound_stmts.rst +++ b/Doc/reference/compound_stmts.rst @@ -51,6 +51,9 @@ Summarizing: : | `with_stmt` : | `funcdef` : | `classdef` + : | `async_with_stmt` + : | `async_for_stmt` + : | `async_funcdef` suite: `stmt_list` NEWLINE | NEWLINE INDENT `statement`+ DEDENT statement: `stmt_list` NEWLINE | `compound_stmt` stmt_list: `simple_stmt` (";" `simple_stmt`)* [";"] @@ -436,7 +439,7 @@ is equivalent to :: .. seealso:: - :pep:`0343` - The "with" statement + :pep:`343` - The "with" statement The specification, background, and examples for the Python :keyword:`with` statement. @@ -466,7 +469,7 @@ A function definition defines a user-defined function object (see section .. productionlist:: funcdef: [`decorators`] "def" `funcname` "(" [`parameter_list`] ")" ["->" `expression`] ":" `suite` decorators: `decorator`+ - decorator: "@" `dotted_name` ["(" [`parameter_list` [","]] ")"] NEWLINE + decorator: "@" `dotted_name` ["(" [`argument_list` [","]] ")"] NEWLINE dotted_name: `identifier` ("." `identifier`)* parameter_list: (`defparameter` ",")* : | "*" [`parameter`] ("," `defparameter`)* ["," "**" `parameter`] @@ -500,11 +503,13 @@ are applied in nested fashion. For example, the following code :: @f2 def func(): pass -is equivalent to :: +is roughly equivalent to :: def func(): pass func = f1(arg)(f2(func)) +except that the original function is not temporarily bound to the name ``func``. + .. index:: triple: default; parameter; value single: argument; function definition @@ -601,7 +606,7 @@ A class definition defines a class object (see section :ref:`types`): .. productionlist:: classdef: [`decorators`] "class" `classname` [`inheritance`] ":" `suite` - inheritance: "(" [`parameter_list`] ")" + inheritance: "(" [`argument_list`] ")" classname: `identifier` A class definition is an executable statement. The inheritance list usually @@ -635,14 +640,13 @@ Classes can also be decorated: just like when decorating functions, :: @f2 class Foo: pass -is equivalent to :: +is roughly equivalent to :: class Foo: pass Foo = f1(arg)(f2(Foo)) The evaluation rules for the decorator expressions are the same as for function -decorators. The result must be a class object, which is then bound to the class -name. +decorators. The result is then bound to the class name. **Programmer's note:** Variables defined in the class definition are class attributes; they are shared by instances. Instance attributes can be set in a @@ -660,6 +664,130 @@ can be used to create instance variables with different implementation details. :pep:`3129` - Class Decorators +Coroutines +========== + +.. versionadded:: 3.5 + +.. index:: statement: async def +.. _`async def`: + +Coroutine function definition +----------------------------- + +.. productionlist:: + async_funcdef: [`decorators`] "async" "def" `funcname` "(" [`parameter_list`] ")" ["->" `expression`] ":" `suite` + +.. index:: + keyword: async + keyword: await + +Execution of Python coroutines can be suspended and resumed at many points +(see :term:`coroutine`). In the body of a coroutine, any ``await`` and +``async`` identifiers become reserved keywords; :keyword:`await` expressions, +:keyword:`async for` and :keyword:`async with` can only be used in +coroutine bodies. + +Functions defined with ``async def`` syntax are always coroutine functions, +even if they do not contain ``await`` or ``async`` keywords. + +It is a :exc:`SyntaxError` to use :keyword:`yield` expressions in +``async def`` coroutines. + +An example of a coroutine function:: + + async def func(param1, param2): + do_stuff() + await some_coroutine() + + +.. index:: statement: async for +.. _`async for`: + +The :keyword:`async for` statement +---------------------------------- + +.. productionlist:: + async_for_stmt: "async" `for_stmt` + +An :term:`asynchronous iterable` is able to call asynchronous code in its +*iter* implementation, and :term:`asynchronous iterator` can call asynchronous +code in its *next* method. + +The ``async for`` statement allows convenient iteration over asynchronous +iterators. + +The following code:: + + async for TARGET in ITER: + BLOCK + else: + BLOCK2 + +Is semantically equivalent to:: + + iter = (ITER) + iter = type(iter).__aiter__(iter) + running = True + while running: + try: + TARGET = await type(iter).__anext__(iter) + except StopAsyncIteration: + running = False + else: + BLOCK + else: + BLOCK2 + +See also :meth:`__aiter__` and :meth:`__anext__` for details. + +It is a :exc:`SyntaxError` to use ``async for`` statement outside of an +:keyword:`async def` function. + + +.. index:: statement: async with +.. _`async with`: + +The :keyword:`async with` statement +----------------------------------- + +.. productionlist:: + async_with_stmt: "async" `with_stmt` + +An :term:`asynchronous context manager` is a :term:`context manager` that is +able to suspend execution in its *enter* and *exit* methods. + +The following code:: + + async with EXPR as VAR: + BLOCK + +Is semantically equivalent to:: + + mgr = (EXPR) + aexit = type(mgr).__aexit__ + aenter = type(mgr).__aenter__(mgr) + exc = True + + VAR = await aenter + try: + BLOCK + except: + if not await aexit(mgr, *sys.exc_info()): + raise + else: + await aexit(mgr, None, None, None) + +See also :meth:`__aenter__` and :meth:`__aexit__` for details. + +It is a :exc:`SyntaxError` to use ``async with`` statement outside of an +:keyword:`async def` function. + +.. seealso:: + + :pep:`492` - Coroutines with async and await syntax + + .. rubric:: Footnotes .. [#] The exception is propagated to the invocation stack unless diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst index 1560ad0d32..f97eb08dd6 100644 --- a/Doc/reference/datamodel.rst +++ b/Doc/reference/datamodel.rst @@ -454,6 +454,19 @@ Callable types .. tabularcolumns:: |l|L|l| + .. index:: + single: __doc__ (function attribute) + single: __name__ (function attribute) + single: __module__ (function attribute) + single: __dict__ (function attribute) + single: __defaults__ (function attribute) + single: __closure__ (function attribute) + single: __code__ (function attribute) + single: __globals__ (function attribute) + single: __annotations__ (function attribute) + single: __kwdefaults__ (function attribute) + pair: global; namespace + +-------------------------+-------------------------------+-----------+ | Attribute | Meaning | | +=========================+===============================+===========+ @@ -462,10 +475,11 @@ Callable types | | unavailable; not inherited by | | | | subclasses | | +-------------------------+-------------------------------+-----------+ - | :attr:`__name__` | The function's name | Writable | + | :attr:`~definition.\ | The function's name | Writable | + | __name__` | | | +-------------------------+-------------------------------+-----------+ - | :attr:`__qualname__` | The function's | Writable | - | | :term:`qualified name` | | + | :attr:`~definition.\ | The function's | Writable | + | __qualname__` | :term:`qualified name` | | | | | | | | .. versionadded:: 3.3 | | +-------------------------+-------------------------------+-----------+ @@ -489,7 +503,7 @@ Callable types | | module in which the function | | | | was defined. | | +-------------------------+-------------------------------+-----------+ - | :attr:`__dict__` | The namespace supporting | Writable | + | :attr:`~object.__dict__`| The namespace supporting | Writable | | | arbitrary function | | | | attributes. | | +-------------------------+-------------------------------+-----------+ @@ -519,19 +533,6 @@ Callable types Additional information about a function's definition can be retrieved from its code object; see the description of internal types below. - .. index:: - single: __doc__ (function attribute) - single: __name__ (function attribute) - single: __module__ (function attribute) - single: __dict__ (function attribute) - single: __defaults__ (function attribute) - single: __closure__ (function attribute) - single: __code__ (function attribute) - single: __globals__ (function attribute) - single: __annotations__ (function attribute) - single: __kwdefaults__ (function attribute) - pair: global; namespace - Instance methods .. index:: object: method @@ -550,7 +551,7 @@ Callable types Special read-only attributes: :attr:`__self__` is the class instance object, :attr:`__func__` is the function object; :attr:`__doc__` is the method's - documentation (same as ``__func__.__doc__``); :attr:`__name__` is the + documentation (same as ``__func__.__doc__``); :attr:`~definition.__name__` is the method name (same as ``__func__.__name__``); :attr:`__module__` is the name of the module the method was defined in, or ``None`` if unavailable. @@ -616,6 +617,16 @@ Callable types exception is raised and the iterator will have reached the end of the set of values to be returned. + Coroutine functions + .. index:: + single: coroutine; function + + A function or method which is defined using :keyword:`async def` is called + a :dfn:`coroutine function`. Such a function, when called, returns a + :term:`coroutine` object. It may contain :keyword:`await` expressions, + as well as :keyword:`async with` and :keyword:`async for` statements. See + also the :ref:`coroutine-objects` section. + Built-in functions .. index:: object: built-in function @@ -627,7 +638,7 @@ Callable types standard built-in module). The number and type of the arguments are determined by the C function. Special read-only attributes: :attr:`__doc__` is the function's documentation string, or ``None`` if - unavailable; :attr:`__name__` is the function's name; :attr:`__self__` is + unavailable; :attr:`~definition.__name__` is the function's name; :attr:`__self__` is set to ``None`` (but see the next item); :attr:`__module__` is the name of the module the function was defined in or ``None`` if unavailable. @@ -677,7 +688,7 @@ Modules .. index:: single: __dict__ (module attribute) - Special read-only attribute: :attr:`__dict__` is the module's namespace as a + Special read-only attribute: :attr:`~object.__dict__` is the module's namespace as a dictionary object. .. impl-detail:: @@ -733,7 +744,7 @@ Custom classes method object, it is transformed into the object wrapped by the static method object. See section :ref:`descriptors` for another way in which attributes retrieved from a class may differ from those actually contained in its - :attr:`__dict__`. + :attr:`~object.__dict__`. .. index:: triple: class; attribute; assignment @@ -751,8 +762,8 @@ Custom classes single: __bases__ (class attribute) single: __doc__ (class attribute) - Special attributes: :attr:`__name__` is the class name; :attr:`__module__` is - the module name in which the class was defined; :attr:`__dict__` is the + Special attributes: :attr:`~definition.__name__` is the class name; :attr:`__module__` is + the module name in which the class was defined; :attr:`~object.__dict__` is the dictionary containing the class's namespace; :attr:`~class.__bases__` is a tuple (possibly empty or a singleton) containing the base classes, in the order of their occurrence in the base class list; :attr:`__doc__` is the @@ -775,7 +786,7 @@ Class instances class method objects are also transformed; see above under "Classes". See section :ref:`descriptors` for another way in which attributes of a class retrieved via its instances may differ from the objects actually stored in - the class's :attr:`__dict__`. If no class attribute is found, and the + the class's :attr:`~object.__dict__`. If no class attribute is found, and the object's class has a :meth:`__getattr__` method, that is called to satisfy the lookup. @@ -836,11 +847,9 @@ Internal types definitions may change with future versions of the interpreter, but they are mentioned here for completeness. - Code objects - .. index:: - single: bytecode - object: code + .. index:: bytecode, object; code, code object + Code objects Code objects represent *byte-compiled* executable Python code, or :term:`bytecode`. The difference between a code object and a function object is that the function object contains an explicit reference to the function's globals (the module in @@ -1458,7 +1467,7 @@ method (a so-called *descriptor* class) appears in an *owner* class (the descriptor must be in either the owner's class dictionary or in the class dictionary for one of its parents). In the examples below, "the attribute" refers to the attribute whose name is the key of the property in the owner -class' :attr:`__dict__`. +class' :attr:`~object.__dict__`. .. method:: object.__get__(self, instance, owner) @@ -1724,6 +1733,11 @@ After the class object is created, it is passed to the class decorators included in the class definition (if any) and the resulting object is bound in the local namespace as the defined class. +When a new class is created by ``type.__new__``, the object provided as the +namespace parameter is copied to a standard Python dictionary and the original +object is discarded. The new copy becomes the :attr:`~object.__dict__` attribute +of the class object. + .. seealso:: :pep:`3135` - New super @@ -1743,11 +1757,11 @@ to remember the order that class variables are defined:: class OrderedClass(type): - @classmethod - def __prepare__(metacls, name, bases, **kwds): + @classmethod + def __prepare__(metacls, name, bases, **kwds): return collections.OrderedDict() - def __new__(cls, name, bases, namespace, **kwds): + def __new__(cls, name, bases, namespace, **kwds): result = type.__new__(cls, name, bases, dict(namespace)) result.members = tuple(namespace) return result @@ -1990,6 +2004,7 @@ left undefined. .. method:: object.__add__(self, other) object.__sub__(self, other) object.__mul__(self, other) + object.__matmul__(self, other) object.__truediv__(self, other) object.__floordiv__(self, other) object.__mod__(self, other) @@ -2006,15 +2021,16 @@ left undefined. builtin: pow builtin: pow - These methods are called to implement the binary arithmetic operations (``+``, - ``-``, ``*``, ``/``, ``//``, ``%``, :func:`divmod`, :func:`pow`, ``**``, ``<<``, - ``>>``, ``&``, ``^``, ``|``). For instance, to evaluate the expression - ``x + y``, where *x* is an instance of a class that has an :meth:`__add__` - method, ``x.__add__(y)`` is called. The :meth:`__divmod__` method should be the - equivalent to using :meth:`__floordiv__` and :meth:`__mod__`; it should not be - related to :meth:`__truediv__`. Note that :meth:`__pow__` should be defined - to accept an optional third argument if the ternary version of the built-in - :func:`pow` function is to be supported. + These methods are called to implement the binary arithmetic operations + (``+``, ``-``, ``*``, ``@``, ``/``, ``//``, ``%``, :func:`divmod`, + :func:`pow`, ``**``, ``<<``, ``>>``, ``&``, ``^``, ``|``). For instance, to + evaluate the expression ``x + y``, where *x* is an instance of a class that + has an :meth:`__add__` method, ``x.__add__(y)`` is called. The + :meth:`__divmod__` method should be the equivalent to using + :meth:`__floordiv__` and :meth:`__mod__`; it should not be related to + :meth:`__truediv__`. Note that :meth:`__pow__` should be defined to accept + an optional third argument if the ternary version of the built-in :func:`pow` + function is to be supported. If one of those methods does not support the operation with the supplied arguments, it should return ``NotImplemented``. @@ -2023,6 +2039,7 @@ left undefined. .. method:: object.__radd__(self, other) object.__rsub__(self, other) object.__rmul__(self, other) + object.__rmatmul__(self, other) object.__rtruediv__(self, other) object.__rfloordiv__(self, other) object.__rmod__(self, other) @@ -2038,14 +2055,14 @@ left undefined. builtin: divmod builtin: pow - These methods are called to implement the binary arithmetic operations (``+``, - ``-``, ``*``, ``/``, ``//``, ``%``, :func:`divmod`, :func:`pow`, ``**``, - ``<<``, ``>>``, ``&``, ``^``, ``|``) with reflected (swapped) operands. - These functions are only called if the left operand does not support the - corresponding operation and the operands are of different types. [#]_ For - instance, to evaluate the expression ``x - y``, where *y* is an instance of - a class that has an :meth:`__rsub__` method, ``y.__rsub__(x)`` is called if - ``x.__sub__(y)`` returns *NotImplemented*. + These methods are called to implement the binary arithmetic operations + (``+``, ``-``, ``*``, ``@``, ``/``, ``//``, ``%``, :func:`divmod`, + :func:`pow`, ``**``, ``<<``, ``>>``, ``&``, ``^``, ``|``) with reflected + (swapped) operands. These functions are only called if the left operand does + not support the corresponding operation and the operands are of different + types. [#]_ For instance, to evaluate the expression ``x - y``, where *y* is + an instance of a class that has an :meth:`__rsub__` method, ``y.__rsub__(x)`` + is called if ``x.__sub__(y)`` returns *NotImplemented*. .. index:: builtin: pow @@ -2063,6 +2080,7 @@ left undefined. .. method:: object.__iadd__(self, other) object.__isub__(self, other) object.__imul__(self, other) + object.__imatmul__(self, other) object.__itruediv__(self, other) object.__ifloordiv__(self, other) object.__imod__(self, other) @@ -2074,17 +2092,17 @@ left undefined. object.__ior__(self, other) These methods are called to implement the augmented arithmetic assignments - (``+=``, ``-=``, ``*=``, ``/=``, ``//=``, ``%=``, ``**=``, ``<<=``, ``>>=``, - ``&=``, ``^=``, ``|=``). These methods should attempt to do the operation - in-place (modifying *self*) and return the result (which could be, but does - not have to be, *self*). If a specific method is not defined, the augmented - assignment falls back to the normal methods. For instance, if *x* is an - instance of a class with an :meth:`__iadd__` method, ``x += y`` is equivalent - to ``x = x.__iadd__(y)`` . Otherwise, ``x.__add__(y)`` and ``y.__radd__(x)`` - are considered, as with the evaluation of ``x + y``. In certain situations, - augmented assignment can result in unexpected errors (see - :ref:`faq-augmented-assignment-tuple-error`), but this behavior is in - fact part of the data model. + (``+=``, ``-=``, ``*=``, ``@=``, ``/=``, ``//=``, ``%=``, ``**=``, ``<<=``, + ``>>=``, ``&=``, ``^=``, ``|=``). These methods should attempt to do the + operation in-place (modifying *self*) and return the result (which could be, + but does not have to be, *self*). If a specific method is not defined, the + augmented assignment falls back to the normal methods. For instance, if *x* + is an instance of a class with an :meth:`__iadd__` method, ``x += y`` is + equivalent to ``x = x.__iadd__(y)`` . Otherwise, ``x.__add__(y)`` and + ``y.__radd__(x)`` are considered, as with the evaluation of ``x + y``. In + certain situations, augmented assignment can result in unexpected errors (see + :ref:`faq-augmented-assignment-tuple-error`), but this behavior is in fact + part of the data model. .. method:: object.__neg__(self) @@ -2174,7 +2192,7 @@ For more information on context managers, see :ref:`typecontextmanager`. .. seealso:: - :pep:`0343` - The "with" statement + :pep:`343` - The "with" statement The specification, background, and examples for the Python :keyword:`with` statement. @@ -2254,6 +2272,203 @@ special methods (the special method *must* be set on the class object itself in order to be consistently invoked by the interpreter). +.. index:: + single: coroutine + +Coroutines +========== + + +Awaitable Objects +----------------- + +An :term:`awaitable` object generally implements an :meth:`__await__` method. +:term:`Coroutine` objects returned from :keyword:`async def` functions +are awaitable. + +.. note:: + + The :term:`generator iterator` objects returned from generators + decorated with :func:`types.coroutine` or :func:`asyncio.coroutine` + are also awaitable, but they do not implement :meth:`__await__`. + +.. method:: object.__await__(self) + + Must return an :term:`iterator`. Should be used to implement + :term:`awaitable` objects. For instance, :class:`asyncio.Future` implements + this method to be compatible with the :keyword:`await` expression. + +.. versionadded:: 3.5 + +.. seealso:: :pep:`492` for additional information about awaitable objects. + + +.. _coroutine-objects: + +Coroutine Objects +----------------- + +:term:`Coroutine` objects are :term:`awaitable` objects. +A coroutine's execution can be controlled by calling :meth:`__await__` and +iterating over the result. When the coroutine has finished executing and +returns, the iterator raises :exc:`StopIteration`, and the exception's +:attr:`~StopIteration.value` attribute holds the return value. If the +coroutine raises an exception, it is propagated by the iterator. Coroutines +should not directly raise unhandled :exc:`StopIteration` exceptions. + +Coroutines also have the methods listed below, which are analogous to +those of generators (see :ref:`generator-methods`). However, unlike +generators, coroutines do not directly support iteration. + +.. versionchanged:: 3.5.2 + It is a :exc:`RuntimeError` to await on a coroutine more than once. + + +.. method:: coroutine.send(value) + + Starts or resumes execution of the coroutine. If *value* is ``None``, + this is equivalent to advancing the iterator returned by + :meth:`__await__`. If *value* is not ``None``, this method delegates + to the :meth:`~generator.send` method of the iterator that caused + the coroutine to suspend. The result (return value, + :exc:`StopIteration`, or other exception) is the same as when + iterating over the :meth:`__await__` return value, described above. + +.. method:: coroutine.throw(type[, value[, traceback]]) + + Raises the specified exception in the coroutine. This method delegates + to the :meth:`~generator.throw` method of the iterator that caused + the coroutine to suspend, if it has such a method. Otherwise, + the exception is raised at the suspension point. The result + (return value, :exc:`StopIteration`, or other exception) is the same as + when iterating over the :meth:`__await__` return value, described + above. If the exception is not caught in the coroutine, it propagates + back to the caller. + +.. method:: coroutine.close() + + Causes the coroutine to clean itself up and exit. If the coroutine + is suspended, this method first delegates to the :meth:`~generator.close` + method of the iterator that caused the coroutine to suspend, if it + has such a method. Then it raises :exc:`GeneratorExit` at the + suspension point, causing the coroutine to immediately clean itself up. + Finally, the coroutine is marked as having finished executing, even if + it was never started. + + Coroutine objects are automatically closed using the above process when + they are about to be destroyed. + +.. _async-iterators: + +Asynchronous Iterators +---------------------- + +An *asynchronous iterable* is able to call asynchronous code in its +``__aiter__`` implementation, and an *asynchronous iterator* can call +asynchronous code in its ``__anext__`` method. + +Asynchronous iterators can be used in an :keyword:`async for` statement. + +.. method:: object.__aiter__(self) + + Must return an *asynchronous iterator* object. + +.. method:: object.__anext__(self) + + Must return an *awaitable* resulting in a next value of the iterator. Should + raise a :exc:`StopAsyncIteration` error when the iteration is over. + +An example of an asynchronous iterable object:: + + class Reader: + async def readline(self): + ... + + def __aiter__(self): + return self + + async def __anext__(self): + val = await self.readline() + if val == b'': + raise StopAsyncIteration + return val + +.. versionadded:: 3.5 + +.. note:: + + .. versionchanged:: 3.5.2 + Starting with CPython 3.5.2, ``__aiter__`` can directly return + :term:`asynchronous iterators <asynchronous iterator>`. Returning + an :term:`awaitable` object will result in a + :exc:`PendingDeprecationWarning`. + + The recommended way of writing backwards compatible code in + CPython 3.5.x is to continue returning awaitables from + ``__aiter__``. If you want to avoid the PendingDeprecationWarning + and keep the code backwards compatible, the following decorator + can be used:: + + import functools + import sys + + if sys.version_info < (3, 5, 2): + def aiter_compat(func): + @functools.wraps(func) + async def wrapper(self): + return func(self) + return wrapper + else: + def aiter_compat(func): + return func + + Example:: + + class AsyncIterator: + + @aiter_compat + def __aiter__(self): + return self + + async def __anext__(self): + ... + + Starting with CPython 3.6, the :exc:`PendingDeprecationWarning` + will be replaced with the :exc:`DeprecationWarning`. + In CPython 3.7, returning an awaitable from ``__aiter__`` will + result in a :exc:`RuntimeError`. + + +Asynchronous Context Managers +----------------------------- + +An *asynchronous context manager* is a *context manager* that is able to +suspend execution in its ``__aenter__`` and ``__aexit__`` methods. + +Asynchronous context managers can be used in an :keyword:`async with` statement. + +.. method:: object.__aenter__(self) + + This method is semantically similar to the :meth:`__enter__`, with only + difference that it must return an *awaitable*. + +.. method:: object.__aexit__(self, exc_type, exc_value, traceback) + + This method is semantically similar to the :meth:`__exit__`, with only + difference that it must return an *awaitable*. + +An example of an asynchronous context manager class:: + + class AsyncContextManager: + async def __aenter__(self): + await log('entering context') + + async def __aexit__(self, exc_type, exc, tb): + await log('exiting context') + +.. versionadded:: 3.5 + + .. rubric:: Footnotes .. [#] It *is* possible in some cases to change an object's type, under certain diff --git a/Doc/reference/expressions.rst b/Doc/reference/expressions.rst index b6b2b006fa..eafd70a835 100644 --- a/Doc/reference/expressions.rst +++ b/Doc/reference/expressions.rst @@ -133,7 +133,7 @@ Parenthesized forms A parenthesized form is an optional expression list enclosed in parentheses: .. productionlist:: - parenth_form: "(" [`expression_list`] ")" + parenth_form: "(" [`starred_expression`] ")" A parenthesized expression list yields whatever that expression list yields: if the list contains at least one comma, it yields a tuple; otherwise, it yields @@ -202,7 +202,7 @@ A list display is a possibly empty series of expressions enclosed in square brackets: .. productionlist:: - list_display: "[" [`expression_list` | `comprehension`] "]" + list_display: "[" [`starred_list` | `comprehension`] "]" A list display yields a new list object, the contents being specified by either a list of expressions or a comprehension. When a comma-separated list of @@ -223,7 +223,7 @@ A set display is denoted by curly braces and distinguishable from dictionary displays by the lack of colons separating keys and values: .. productionlist:: - set_display: "{" (`expression_list` | `comprehension`) "}" + set_display: "{" (`starred_list` | `comprehension`) "}" A set display yields a new mutable set object, the contents being specified by either a sequence of expressions or a comprehension. When a comma-separated @@ -250,7 +250,7 @@ curly braces: .. productionlist:: dict_display: "{" [`key_datum_list` | `dict_comprehension`] "}" key_datum_list: `key_datum` ("," `key_datum`)* [","] - key_datum: `expression` ":" `expression` + key_datum: `expression` ":" `expression` | "**" `or_expr` dict_comprehension: `expression` ":" `expression` `comp_for` A dictionary display yields a new dictionary object. @@ -261,6 +261,16 @@ used as a key into the dictionary to store the corresponding datum. This means that you can specify the same key multiple times in the key/datum list, and the final dictionary's value for that key will be the last one given. +.. index:: unpacking; dictionary, **; in dictionary displays + +A double asterisk ``**`` denotes :dfn:`dictionary unpacking`. +Its operand must be a :term:`mapping`. Each mapping item is added +to the new dictionary. Later values replace values already set by +earlier key/datum pairs and earlier dictionary unpackings. + +.. versionadded:: 3.5 + Unpacking into dictionary displays, originally proposed by :pep:`448`. + A dict comprehension, in contrast to list and set comprehensions, needs two expressions separated with a colon followed by the usual "for" and "if" clauses. When the comprehension is run, the resulting key and value elements are inserted @@ -378,18 +388,19 @@ on the right hand side of an assignment statement. .. seealso:: - :pep:`0255` - Simple Generators + :pep:`255` - Simple Generators The proposal for adding generators and the :keyword:`yield` statement to Python. - :pep:`0342` - Coroutines via Enhanced Generators + :pep:`342` - Coroutines via Enhanced Generators The proposal to enhance the API and syntax of generators, making them usable as simple coroutines. - :pep:`0380` - Syntax for Delegating to a Subgenerator + :pep:`380` - Syntax for Delegating to a Subgenerator The proposal to introduce the :token:`yield_from` syntax, making delegation to sub-generators easy. .. index:: object: generator +.. _generator-methods: Generator-iterator methods ^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -443,12 +454,12 @@ is already executing raises a :exc:`ValueError` exception. .. method:: generator.close() Raises a :exc:`GeneratorExit` at the point where the generator function was - paused. If the generator function then raises :exc:`StopIteration` (by - exiting normally, or due to already being closed) or :exc:`GeneratorExit` (by - not catching the exception), close returns to its caller. If the generator - yields a value, a :exc:`RuntimeError` is raised. If the generator raises any - other exception, it is propagated to the caller. :meth:`close` does nothing - if the generator has already exited due to an exception or normal exit. + paused. If the generator function then exits gracefully, is already closed, + or raises :exc:`GeneratorExit` (by not catching the exception), close + returns to its caller. If the generator yields a value, a + :exc:`RuntimeError` is raised. If the generator raises any other exception, + it is propagated to the caller. :meth:`close` does nothing if the generator + has already exited due to an exception or normal exit. .. index:: single: yield; examples @@ -648,15 +659,15 @@ series of :term:`arguments <argument>`: .. productionlist:: call: `primary` "(" [`argument_list` [","] | `comprehension`] ")" - argument_list: `positional_arguments` ["," `keyword_arguments`] - : ["," "*" `expression`] ["," `keyword_arguments`] - : ["," "**" `expression`] - : | `keyword_arguments` ["," "*" `expression`] - : ["," `keyword_arguments`] ["," "**" `expression`] - : | "*" `expression` ["," `keyword_arguments`] ["," "**" `expression`] - : | "**" `expression` - positional_arguments: `expression` ("," `expression`)* - keyword_arguments: `keyword_item` ("," `keyword_item`)* + argument_list: `positional_arguments` ["," `starred_and_keywords`] + : ["," `keywords_arguments`] + : | `starred_and_keywords` ["," `keywords_arguments`] + : | `keywords_arguments` + positional_arguments: ["*"] `expression` ("," ["*"] `expression`)* + starred_and_keywords: ("*" `expression` | `keyword_item`) + : ("," "*" `expression` | "," `keyword_item`)* + keywords_arguments: (`keyword_item` | "**" `expression`) + : ("," `keyword_item` | "**" `expression`)* keyword_item: `identifier` "=" `expression` An optional trailing comma may be present after the positional and keyword arguments @@ -714,20 +725,21 @@ there were no excess keyword arguments. .. index:: single: *; in function calls + single: unpacking; in function calls If the syntax ``*expression`` appears in the function call, ``expression`` must -evaluate to an iterable. Elements from this iterable are treated as if they -were additional positional arguments; if there are positional arguments -*x1*, ..., *xN*, and ``expression`` evaluates to a sequence *y1*, ..., *yM*, -this is equivalent to a call with M+N positional arguments *x1*, ..., *xN*, -*y1*, ..., *yM*. +evaluate to an :term:`iterable`. Elements from these iterables are +treated as if they were additional positional arguments. For the call +``f(x1, x2, *y, x3, x4)``, if *y* evaluates to a sequence *y1*, ..., *yM*, +this is equivalent to a call with M+4 positional arguments *x1*, *x2*, +*y1*, ..., *yM*, *x3*, *x4*. A consequence of this is that although the ``*expression`` syntax may appear -*after* some keyword arguments, it is processed *before* the keyword arguments -(and the ``**expression`` argument, if any -- see below). So:: +*after* explicit keyword arguments, it is processed *before* the +keyword arguments (and any ``**expression`` arguments -- see below). So:: >>> def f(a, b): - ... print(a, b) + ... print(a, b) ... >>> f(b=1, *(2,)) 2 1 @@ -745,13 +757,20 @@ used in the same call, so in practice this confusion does not arise. single: **; in function calls If the syntax ``**expression`` appears in the function call, ``expression`` must -evaluate to a mapping, the contents of which are treated as additional keyword -arguments. In the case of a keyword appearing in both ``expression`` and as an -explicit keyword argument, a :exc:`TypeError` exception is raised. +evaluate to a :term:`mapping`, the contents of which are treated as +additional keyword arguments. If a keyword is already present +(as an explicit keyword argument, or from another unpacking), +a :exc:`TypeError` exception is raised. Formal parameters using the syntax ``*identifier`` or ``**identifier`` cannot be used as positional argument slots or as keyword argument names. +.. versionchanged:: 3.5 + Function calls accept any number of ``*`` and ``**`` unpackings, + positional arguments may follow iterable unpackings (``*``), + and keyword arguments may follow dictionary unpackings (``**``). + Originally proposed by :pep:`448`. + A call always returns some value, possibly ``None``, unless it raises an exception. How this value is computed depends on the type of the callable object. @@ -811,6 +830,20 @@ a class instance: if that method was called. +.. _await: + +Await expression +================ + +Suspend the execution of :term:`coroutine` on an :term:`awaitable` object. +Can only be used inside a :term:`coroutine function`. + +.. productionlist:: + await_expr: "await" `primary` + +.. versionadded:: 3.5 + + .. _power: The power operator @@ -820,7 +853,7 @@ The power operator binds more tightly than unary operators on its left; it binds less tightly than unary operators on its right. The syntax is: .. productionlist:: - power: `primary` ["**" `u_expr`] + power: ( `await_expr` | `primary` ) ["**" `u_expr`] Thus, in an unparenthesized sequence of power and unary operators, the operators are evaluated from right to left (this does not constrain the evaluation order @@ -891,8 +924,9 @@ from the power operator, there are only two levels, one for multiplicative operators and one for additive operators: .. productionlist:: - m_expr: `u_expr` | `m_expr` "*" `u_expr` | `m_expr` "//" `u_expr` | `m_expr` "/" `u_expr` - : | `m_expr` "%" `u_expr` + m_expr: `u_expr` | `m_expr` "*" `u_expr` | `m_expr` "@" `m_expr` | + : `m_expr` "//" `u_expr`| `m_expr` "/" `u_expr` | + : `m_expr` "%" `u_expr` a_expr: `m_expr` | `a_expr` "+" `m_expr` | `a_expr` "-" `m_expr` .. index:: single: multiplication @@ -903,6 +937,13 @@ the other must be a sequence. In the former case, the numbers are converted to a common type and then multiplied together. In the latter case, sequence repetition is performed; a negative repetition factor yields an empty sequence. +.. index:: single: matrix multiplication + +The ``@`` (at) operator is intended to be used for matrix multiplication. No +builtin Python types implement this operator. + +.. versionadded:: 3.5 + .. index:: exception: ZeroDivisionError single: division @@ -1365,7 +1406,9 @@ Lambdas Lambda expressions (sometimes called lambda forms) are used to create anonymous functions. The expression ``lambda arguments: expression`` yields a function -object. The unnamed object behaves like a function object defined with :: +object. The unnamed object behaves like a function object defined with: + +.. code-block:: none def <lambda>(arguments): return expression @@ -1384,13 +1427,29 @@ Expression lists .. productionlist:: expression_list: `expression` ( "," `expression` )* [","] + starred_list: `starred_item` ( "," `starred_item` )* [","] + starred_expression: `expression` | ( `starred_item` "," )* [`starred_item`] + starred_item: `expression` | "*" `or_expr` .. index:: object: tuple -An expression list containing at least one comma yields a tuple. The length of +Except when part of a list or set display, an expression list +containing at least one comma yields a tuple. The length of the tuple is the number of expressions in the list. The expressions are evaluated from left to right. +.. index:: + pair: iterable; unpacking + single: *; in expression lists + +An asterisk ``*`` denotes :dfn:`iterable unpacking`. Its operand must be +an :term:`iterable`. The iterable is expanded into a sequence of items, +which are included in the new tuple, list, or set, at the site of +the unpacking. + +.. versionadded:: 3.5 + Iterable unpacking in expression lists, originally proposed by :pep:`448`. + .. index:: pair: trailing; comma The trailing comma is required only to create a single tuple (a.k.a. a @@ -1466,13 +1525,16 @@ precedence and have a left-to-right chaining feature as described in the +-----------------------------------------------+-------------------------------------+ | ``+``, ``-`` | Addition and subtraction | +-----------------------------------------------+-------------------------------------+ -| ``*``, ``/``, ``//``, ``%`` | Multiplication, division, remainder | -| | [#]_ | +| ``*``, ``@``, ``/``, ``//``, ``%`` | Multiplication, matrix | +| | multiplication division, | +| | remainder [#]_ | +-----------------------------------------------+-------------------------------------+ | ``+x``, ``-x``, ``~x`` | Positive, negative, bitwise NOT | +-----------------------------------------------+-------------------------------------+ | ``**`` | Exponentiation [#]_ | +-----------------------------------------------+-------------------------------------+ +| ``await`` ``x`` | Await expression | ++-----------------------------------------------+-------------------------------------+ | ``x[index]``, ``x[index:index]``, | Subscription, slicing, | | ``x(arguments...)``, ``x.attribute`` | call, attribute reference | +-----------------------------------------------+-------------------------------------+ diff --git a/Doc/reference/import.rst b/Doc/reference/import.rst index 16d28d8c73..2144c1fa35 100644 --- a/Doc/reference/import.rst +++ b/Doc/reference/import.rst @@ -29,11 +29,10 @@ such as the importing of parent packages, and the updating of various caches a name binding operation. When calling :func:`__import__` as part of an import statement, the -import system first checks the module global namespace for a function by -that name. If it is not found, then the standard builtin :func:`__import__` -is called. Other mechanisms for invoking the import system (such as -:func:`importlib.import_module`) do not perform this check and will always -use the standard import system. +standard builtin :func:`__import__` is called. Other mechanisms for +invoking the import system (such as :func:`importlib.import_module`) may +choose to subvert :func:`__import__` and use its own solution to +implement import semantics. When a module is first imported, Python searches for the module and if found, it creates a module object [#fnmo]_, initializing it. If the named module @@ -339,6 +338,7 @@ of what happens during the loading portion of import:: module = None if spec.loader is not None and hasattr(spec.loader, 'create_module'): + # It is assumed 'exec_module' will also be defined on the loader. module = spec.loader.create_module(spec) if module is None: module = ModuleType(spec.name) @@ -427,7 +427,7 @@ Module loaders may opt in to creating the module object during loading by implementing a :meth:`~importlib.abc.Loader.create_module` method. It takes one argument, the module spec, and returns the new module object to use during loading. ``create_module()`` does not need to set any attributes -on the module object. If the loader does not define ``create_module()``, the +on the module object. If the method returns ``None``, the import machinery will create the new module itself. .. versionadded:: 3.4 @@ -459,7 +459,13 @@ import machinery will create the new module itself. * If loading fails, the loader must remove any modules it has inserted into :data:`sys.modules`, but it must remove **only** the failing - module, and only if the loader itself has loaded it explicitly. + module(s), and only if the loader itself has loaded the module(s) + explicitly. + +.. versionchanged:: 3.5 + A :exc:`DeprecationWarning` is raised when ``exec_module()`` is defined but + ``create_module()`` is not. Starting in Python 3.6 it will be an error to not + define ``create_module()`` on a loader attached to a ModuleSpec. Submodules ---------- @@ -674,7 +680,7 @@ path entry finder that knows how to handle that particular kind of path. The default set of path entry finders implement all the semantics for finding modules on the file system, handling special file types such as Python source -code (``.py`` files), Python byte code (``.pyc`` and ``.pyo`` files) and +code (``.py`` files), Python byte code (``.pyc`` files) and shared libraries (e.g. ``.so`` files). When supported by the :mod:`zipimport` module in the standard library, the default path entry finders also handle loading all of these file types (other than shared libraries) from zipfiles. @@ -788,6 +794,15 @@ hook` callables on :data:`sys.path_hooks`, then the following protocol is used to ask the finder for a module spec, which is then used when loading the module. +The current working directory -- denoted by an empty string -- is handled +slightly differently from other entries on :data:`sys.path`. First, if the +current working directory is found to not exist, no value is stored in +:data:`sys.path_importer_cache`. Second, the value for the current working +directory is looked up fresh for each module lookup. Third, the path used for +:data:`sys.path_importer_cache` and returned by +:meth:`importlib.machinery.PathFinder.find_spec` will be the actual current +working directory and not the empty string. + Path entry finder protocol -------------------------- diff --git a/Doc/reference/introduction.rst b/Doc/reference/introduction.rst index 5633ae3eb1..bb7e3906db 100644 --- a/Doc/reference/introduction.rst +++ b/Doc/reference/introduction.rst @@ -60,7 +60,7 @@ Python for .NET This implementation actually uses the CPython implementation, but is a managed .NET application and makes .NET libraries available. It was created by Brian Lloyd. For more information, see the `Python for .NET home page - <http://pythonnet.sourceforge.net>`_. + <https://pythonnet.github.io/>`_. IronPython An alternate Python for .NET. Unlike Python.NET, this is a complete Python diff --git a/Doc/reference/lexical_analysis.rst b/Doc/reference/lexical_analysis.rst index a1265cfad1..37f25f1def 100644 --- a/Doc/reference/lexical_analysis.rst +++ b/Doc/reference/lexical_analysis.rst @@ -313,7 +313,7 @@ The Unicode category codes mentioned above stand for: * *Nd* - decimal numbers * *Pc* - connector punctuations * *Other_ID_Start* - explicit list of characters in `PropList.txt - <http://www.unicode.org/Public/6.3.0/ucd/PropList.txt>`_ to support backwards + <http://www.unicode.org/Public/8.0.0/ucd/PropList.txt>`_ to support backwards compatibility * *Other_ID_Continue* - likewise @@ -322,7 +322,7 @@ of identifiers is based on NFKC. A non-normative HTML file listing all valid identifier characters for Unicode 4.1 can be found at -http://www.dcl.hpi.uni-potsdam.de/home/loewis/table-3131.html. +https://www.dcl.hpi.uni-potsdam.de/home/loewis/table-3131.html. .. _keywords: @@ -538,8 +538,7 @@ Notes: Support for name aliases [#]_ has been added. (5) - Individual code units which form parts of a surrogate pair can be encoded using - this escape sequence. Exactly four hex digits are required. + Exactly four hex digits are required. (6) Any Unicode character can be encoded this way. Exactly eight hex digits @@ -690,9 +689,12 @@ Operators .. index:: single: operators -The following tokens are operators:: +The following tokens are operators: - + - * ** / // % +.. code-block:: none + + + + - * ** / // % @ << >> & | ^ ~ < > <= >= == != @@ -704,11 +706,13 @@ Delimiters .. index:: single: delimiters -The following tokens serve as delimiters in the grammar:: +The following tokens serve as delimiters in the grammar: + +.. code-block:: none ( ) [ ] { } , : . ; @ = -> - += -= *= /= //= %= + += -= *= /= //= %= @= &= |= ^= >>= <<= **= The period can also occur in floating-point and imaginary literals. A sequence @@ -717,16 +721,20 @@ of the list, the augmented assignment operators, serve lexically as delimiters, but also perform an operation. The following printing ASCII characters have special meaning as part of other -tokens or are otherwise significant to the lexical analyzer:: +tokens or are otherwise significant to the lexical analyzer: + +.. code-block:: none ' " # \ The following printing ASCII characters are not used in Python. Their -occurrence outside string literals and comments is an unconditional error:: +occurrence outside string literals and comments is an unconditional error: + +.. code-block:: none $ ? ` .. rubric:: Footnotes -.. [#] http://www.unicode.org/Public/6.3.0/ucd/NameAliases.txt +.. [#] http://www.unicode.org/Public/8.0.0/ucd/NameAliases.txt diff --git a/Doc/reference/simple_stmts.rst b/Doc/reference/simple_stmts.rst index 8946b4f354..d403c4d546 100644 --- a/Doc/reference/simple_stmts.rst +++ b/Doc/reference/simple_stmts.rst @@ -45,7 +45,7 @@ expression statements are allowed and occasionally useful. The syntax for an expression statement is: .. productionlist:: - expression_stmt: `expression_list` + expression_stmt: `starred_expression` An expression statement evaluates the expression list (which may be a single expression). @@ -81,11 +81,11 @@ Assignment statements are used to (re)bind names to values and to modify attributes or items of mutable objects: .. productionlist:: - assignment_stmt: (`target_list` "=")+ (`expression_list` | `yield_expression`) + assignment_stmt: (`target_list` "=")+ (`starred_expression` | `yield_expression`) target_list: `target` ("," `target`)* [","] target: `identifier` : | "(" `target_list` ")" - : | "[" `target_list` "]" + : | "[" [`target_list`] "]" : | `attributeref` : | `subscription` : | `slicing` @@ -115,21 +115,25 @@ given with the definition of the object types (see section :ref:`types`). Assignment of an object to a target list, optionally enclosed in parentheses or square brackets, is recursively defined as follows. -* If the target list is a single target: The object is assigned to that target. +* If the target list is empty: The object must also be an empty iterable. -* If the target list is a comma-separated list of targets: The object must be an - iterable with the same number of items as there are targets in the target list, - and the items are assigned, from left to right, to the corresponding targets. +* If the target list is a single target in parentheses: The object is assigned + to that target. + +* If the target list is a comma-separated list of targets, or a single target + in square brackets: The object must be an iterable with the same number of + items as there are targets in the target list, and the items are assigned, + from left to right, to the corresponding targets. * If the target list contains one target prefixed with an asterisk, called a - "starred" target: The object must be a sequence with at least as many items + "starred" target: The object must be an iterable with at least as many items as there are targets in the target list, minus one. The first items of the - sequence are assigned, from left to right, to the targets before the starred - target. The final items of the sequence are assigned to the targets after - the starred target. A list of the remaining items in the sequence is then + iterable are assigned, from left to right, to the targets before the starred + target. The final items of the iterable are assigned to the targets after + the starred target. A list of the remaining items in the iterable is then assigned to the starred target (the list can be empty). - * Else: The object must be a sequence with the same number of items as there + * Else: The object must be an iterable with the same number of items as there are targets in the target list, and the items are assigned, from left to right, to the corresponding targets. @@ -150,11 +154,6 @@ Assignment of an object to a single target is recursively defined as follows. count for the object previously bound to the name to reach zero, causing the object to be deallocated and its destructor (if it has one) to be called. -* If the target is a target list enclosed in parentheses or in square brackets: - The object must be an iterable with the same number of items as there are - targets in the target list, and its items are assigned, from left to right, - to the corresponding targets. - .. index:: pair: attribute; assignment * If the target is an attribute reference: The primary expression in the @@ -237,7 +236,7 @@ Assignment of an object to a single target is recursively defined as follows. phase, causing less detailed error messages. Although the definition of assignment implies that overlaps between the -left-hand side and the right-hand side are 'simultanenous' (for example ``a, b = +left-hand side and the right-hand side are 'simultaneous' (for example ``a, b = b, a`` swaps two variables), overlaps *within* the collection of assigned-to variables occur left-to-right, sometimes resulting in confusion. For instance, the following program prints ``[0, 2]``:: @@ -281,7 +280,7 @@ operation and an assignment statement: .. productionlist:: augmented_assignment_stmt: `augtarget` `augop` (`expression_list` | `yield_expression`) augtarget: `identifier` | `attributeref` | `subscription` | `slicing` - augop: "+=" | "-=" | "*=" | "/=" | "//=" | "%=" | "**=" + augop: "+=" | "-=" | "*=" | "@=" | "/=" | "//=" | "%=" | "**=" : | ">>=" | "<<=" | "&=" | "^=" | "|=" (See section :ref:`primaries` for the syntax definitions of the last three @@ -331,12 +330,12 @@ program: The simple form, ``assert expression``, is equivalent to :: if __debug__: - if not expression: raise AssertionError + if not expression: raise AssertionError The extended form, ``assert expression1, expression2``, is equivalent to :: if __debug__: - if not expression1: raise AssertionError(expression2) + if not expression1: raise AssertionError(expression2) .. index:: single: __debug__ @@ -661,7 +660,7 @@ steps: When the statement contains multiple clauses (separated by commas) the two steps are carried out separately for each clause, just -as though the clauses had been separated out into individiual import +as though the clauses had been separated out into individual import statements. The details of the first step, finding and loading modules are described in @@ -893,7 +892,7 @@ The :keyword:`nonlocal` statement nonlocal_stmt: "nonlocal" `identifier` ("," `identifier`)* .. XXX add when implemented - : ["=" (`target_list` "=")+ expression_list] + : ["=" (`target_list` "=")+ starred_expression] : | "nonlocal" identifier augop expression_list The :keyword:`nonlocal` statement causes the listed identifiers to refer to diff --git a/Doc/tools/extensions/pyspecific.py b/Doc/tools/extensions/pyspecific.py index 64a56655f7..63112830ba 100644 --- a/Doc/tools/extensions/pyspecific.py +++ b/Doc/tools/extensions/pyspecific.py @@ -34,7 +34,7 @@ import suspicious ISSUE_URI = 'https://bugs.python.org/issue%s' -SOURCE_URI = 'https://hg.python.org/cpython/file/3.4/%s' +SOURCE_URI = 'https://hg.python.org/cpython/file/3.5/%s' # monkey-patch reST parser to disable alphabetic and roman enumerated lists from docutils.parsers.rst.states import Body @@ -164,6 +164,19 @@ class PyCoroutineMethod(PyCoroutineMixin, PyClassmember): return PyClassmember.run(self) +class PyAbstractMethod(PyClassmember): + + def handle_signature(self, sig, signode): + ret = super(PyAbstractMethod, self).handle_signature(sig, signode) + signode.insert(0, addnodes.desc_annotation('abstractmethod ', + 'abstractmethod ')) + return ret + + def run(self): + self.name = 'py:method' + return PyClassmember.run(self) + + # Support for documenting version of removal in deprecations class DeprecatedRemoved(Directive): @@ -368,5 +381,6 @@ def setup(app): app.add_directive_to_domain('py', 'decoratormethod', PyDecoratorMethod) app.add_directive_to_domain('py', 'coroutinefunction', PyCoroutineFunction) app.add_directive_to_domain('py', 'coroutinemethod', PyCoroutineMethod) + app.add_directive_to_domain('py', 'abstractmethod', PyAbstractMethod) app.add_directive('miscnews', MiscNews) return {'version': '1.0', 'parallel_read_safe': True} diff --git a/Doc/tools/extensions/suspicious.py b/Doc/tools/extensions/suspicious.py index d3ed849157..0a70e57d2b 100644 --- a/Doc/tools/extensions/suspicious.py +++ b/Doc/tools/extensions/suspicious.py @@ -270,5 +270,5 @@ class SuspiciousVisitor(nodes.GenericNodeVisitor): # ignore comments -- too much false positives. # (although doing this could miss some errors; # there were two sections "commented-out" by mistake - # in the Python docs that would not be catched) + # in the Python docs that would not be caught) raise nodes.SkipNode diff --git a/Doc/tools/pydoctheme/static/pydoctheme.css b/Doc/tools/pydoctheme/static/pydoctheme.css index 50835bb92c..e24043f02d 100644 --- a/Doc/tools/pydoctheme/static/pydoctheme.css +++ b/Doc/tools/pydoctheme/static/pydoctheme.css @@ -176,3 +176,8 @@ div.footer a:hover { .stableabi { color: #229; } + +.highlight { + background: none !important; +} + diff --git a/Doc/tools/rstlint.py b/Doc/tools/rstlint.py index 7dccf72f99..de78041f77 100755 --- a/Doc/tools/rstlint.py +++ b/Doc/tools/rstlint.py @@ -43,7 +43,7 @@ directives = [ ] all_directives = '(' + '|'.join(directives) + ')' -seems_directive_re = re.compile(r'\.\. %s([^a-z:]|:(?!:))' % all_directives) +seems_directive_re = re.compile(r'(?<!\.)\.\. %s([^a-z:]|:(?!:))' % all_directives) default_role_re = re.compile(r'(^| )`\w([^`]*?\w)?`($| )') leaked_markup_re = re.compile(r'[a-z]::\s|`|\.\.\s*\w+:') @@ -83,7 +83,7 @@ def check_suspicious_constructs(fn, lines): """Check for suspicious reST constructs.""" inprod = False for lno, line in enumerate(lines): - if seems_directive_re.match(line): + if seems_directive_re.search(line): yield lno+1, 'comment seems to be intended as a directive' if '.. productionlist::' in line: inprod = True diff --git a/Doc/tools/susp-ignored.csv b/Doc/tools/susp-ignored.csv index 032531d3cb..dba93bf3a7 100644 --- a/Doc/tools/susp-ignored.csv +++ b/Doc/tools/susp-ignored.csv @@ -82,7 +82,7 @@ howto/pyporting,,::,Programming Language :: Python :: 2 howto/pyporting,,::,Programming Language :: Python :: 3 howto/regex,,::, howto/regex,,:foo,(?:foo) -howto/urllib2,,:example,"for example ""joe@password:example.com""" +howto/urllib2,,:password,"for example ""joe:password@example.com""" library/audioop,,:ipos,"# factor = audioop.findfactor(in_test[ipos*2:ipos*2+len(out_test)]," library/bisect,32,:hi,all(val >= x for val in a[i:hi]) library/bisect,42,:hi,all(val > x for val in a[i:hi]) @@ -121,6 +121,8 @@ library/ipaddress,,:db8,>>> ipaddress.IPv6Address('2001:db8::1000') library/ipaddress,,::,>>> ipaddress.IPv6Address('2001:db8::1000') library/ipaddress,,:db8,IPv6Address('2001:db8::1000') library/ipaddress,,::,IPv6Address('2001:db8::1000') +library/ipaddress,,:db8,">>> ipaddress.ip_address(""2001:db8::1"").reverse_pointer" +library/ipaddress,,::,">>> ipaddress.ip_address(""2001:db8::1"").reverse_pointer" library/ipaddress,,::,"""::abc:7:def""" library/ipaddress,,:def,"""::abc:7:def""" library/ipaddress,,::,::FFFF/96 @@ -138,8 +140,6 @@ library/itertools,,:stop,elements from seq[start:stop:step] library/logging.handlers,,:port,host:port library/mmap,,:i2,obj[i1:i2] library/multiprocessing,,`,# Add more tasks using `put()` -library/multiprocessing,,`,">>> l._callmethod('__getitem__', (20,)) # equiv to `l[20]`" -library/multiprocessing,,`,">>> l._callmethod('__getitem__', (slice(2, 7),)) # equiv to `l[2:7]`" library/multiprocessing,,:queue,">>> QueueManager.register('get_queue', callable=lambda:queue)" library/multiprocessing,,`,# register the Foo class; make `f()` and `g()` accessible via proxy library/multiprocessing,,`,# register the Foo class; make `g()` and `_h()` accessible via proxy @@ -176,8 +176,6 @@ library/ssl,,:MyState,State or Province Name (full name) [Some-State]:MyState library/ssl,,:ops,Email Address []:ops@myserver.mygroup.myorganization.com library/ssl,,:Some,"Locality Name (eg, city) []:Some City" library/ssl,,:US,Country Name (2 letter code) [AU]:US -library/stdtypes,,::,>>> a[::-1].tolist() -library/stdtypes,,::,>>> a[::2].tolist() library/stdtypes,,:end,s[start:end] library/stdtypes,,::,>>> hash(v[::-2]) == hash(b'abcefg'[::-2]) library/stdtypes,,:len,s[len(s):len(s)] @@ -200,6 +198,7 @@ library/unittest,,:first,"self.assertEqual(cm.output, ['INFO:foo:first message', library/unittest,,:foo,'ERROR:foo.bar:second message']) library/unittest,,:second,'ERROR:foo.bar:second message']) library/urllib.request,,:close,Connection:close +library/urllib.request,,:port,:port library/urllib.request,,:lang,"xmlns=""http://www.w3.org/1999/xhtml"" xml:lang=""en"" lang=""en"">\n\n<head>\n" library/urllib.request,,:password,"""joe:password@python.org""" library/uuid,,:uuid,urn:uuid:12345678-1234-5678-1234-567812345678 @@ -207,16 +206,6 @@ library/venv,,:param,":param nodist: If True, setuptools and pip are not install library/venv,,:param,":param progress: If setuptools or pip are installed, the progress of the" library/venv,,:param,":param nopip: If True, pip is not installed into the created" library/venv,,:param,:param context: The information for the environment creation request -library/xml.etree.elementtree,,:sometag,prefix:sometag -library/xml.etree.elementtree,,:fictional,"<actors xmlns:fictional=""http://characters.example.com""" -library/xml.etree.elementtree,,:character,<fictional:character>Lancelot</fictional:character> -library/xml.etree.elementtree,,:character,<fictional:character>Archie Leach</fictional:character> -library/xml.etree.elementtree,,:character,<fictional:character>Sir Robin</fictional:character> -library/xml.etree.elementtree,,:character,<fictional:character>Gunther</fictional:character> -library/xml.etree.elementtree,,:character,<fictional:character>Commander Clement</fictional:character> -library/xml.etree.elementtree,,:actor,"for actor in root.findall('real_person:actor', ns):" -library/xml.etree.elementtree,,:name,"name = actor.find('real_person:name', ns)" -library/xml.etree.elementtree,,:character,"for char in actor.findall('role:character', ns):" library/xmlrpc.client,,:pass,http://user:pass@host:port/path library/xmlrpc.client,,:pass,user:pass library/xmlrpc.client,,:port,http://user:pass@host:port/path @@ -243,14 +232,13 @@ tutorial/stdlib2,,:start,extra = data[start:start+extra_size] tutorial/stdlib2,,:start,"fields = struct.unpack('<IIIHH', data[start:start+16])" tutorial/stdlib2,,:start,filename = data[start:start+filenamesize] tutorial/stdlib2,,:Warning,WARNING:root:Warning:config file server.conf not found -tutorial/venv,,:c7b9645a6f35,"Python 3.4.3+ (3.4:c7b9645a6f35+, May 22 2015, 09:31:25)" using/cmdline,,:category,action:message:category:module:line using/cmdline,,:errorhandler,:errorhandler using/cmdline,,:line,action:message:category:module:line using/cmdline,,:line,file:line: category: message using/cmdline,,:message,action:message:category:module:line using/cmdline,,:module,action:message:category:module:line -using/unix,,:Packaging,http://en.opensuse.org/Portal:Packaging +using/unix,,:Packaging,https://en.opensuse.org/Portal:Packaging whatsnew/2.0,418,:len, whatsnew/2.3,,::, whatsnew/2.3,,:config, @@ -264,6 +252,11 @@ whatsnew/2.4,,:System, whatsnew/2.5,,:memory,:memory: whatsnew/2.5,,:step,[start:stop:step] whatsnew/2.5,,:stop,[start:stop:step] +whatsnew/2.7,,::,"ParseResult(scheme='http', netloc='[1080::8:800:200C:417A]'," +whatsnew/2.7,,::,>>> urlparse.urlparse('http://[1080::8:800:200C:417A]/foo') +whatsnew/2.7,,:Sunday,'2009:4:Sunday' +whatsnew/2.7,,:Cookie,"export PYTHONWARNINGS=all,error:::Cookie:0" +whatsnew/2.7,,::,"export PYTHONWARNINGS=all,error:::Cookie:0" whatsnew/3.2,,:affe,"netloc='[dead:beef:cafe:5417:affe:8FA3:deaf:feed]'," whatsnew/3.2,,:affe,>>> urllib.parse.urlparse('http://[dead:beef:cafe:5417:affe:8FA3:deaf:feed]/foo/') whatsnew/3.2,,:beef,"netloc='[dead:beef:cafe:5417:affe:8FA3:deaf:feed]'," @@ -279,14 +272,27 @@ whatsnew/3.2,,:feed,>>> urllib.parse.urlparse('http://[dead:beef:cafe:5417:affe: whatsnew/3.2,,:gz,">>> with tarfile.open(name='myarchive.tar.gz', mode='w:gz') as tf:" whatsnew/3.2,,:location,zope9-location = ${zope9:location} whatsnew/3.2,,:prefix,zope-conf = ${custom:prefix}/etc/zope.conf -whatsnew/changelog,,:platform,:platform: whatsnew/changelog,,:gz,": TarFile opened with external fileobj and ""w:gz"" mode didn't" -whatsnew/changelog,,:PythonCmd,"With Tk < 8.5 _tkinter.c:PythonCmd() raised UnicodeDecodeError, caused" -whatsnew/changelog,,::,": Fix FTP tests for IPv6, bind to ""::1"" instead of ""localhost""." whatsnew/changelog,,::,": Use ""127.0.0.1"" or ""::1"" instead of ""localhost"" as much as" -whatsnew/changelog,,:password,user:password -whatsnew/2.7,780,:Sunday,'2009:4:Sunday' -whatsnew/2.7,907,::,"export PYTHONWARNINGS=all,error:::Cookie:0" -whatsnew/2.7,907,:Cookie,"export PYTHONWARNINGS=all,error:::Cookie:0" -whatsnew/2.7,1657,::,>>> urlparse.urlparse('http://[1080::8:800:200C:417A]/foo') -whatsnew/2.7,1657,::,"ParseResult(scheme='http', netloc='[1080::8:800:200C:417A]'," +library/tarfile,149,:xz,'x:xz' +library/xml.etree.elementtree,290,:sometag,prefix:sometag +library/xml.etree.elementtree,301,:fictional,"<actors xmlns:fictional=""http://characters.example.com""" +library/xml.etree.elementtree,301,:character,<fictional:character>Lancelot</fictional:character> +library/xml.etree.elementtree,301,:character,<fictional:character>Archie Leach</fictional:character> +library/xml.etree.elementtree,301,:character,<fictional:character>Sir Robin</fictional:character> +library/xml.etree.elementtree,301,:character,<fictional:character>Gunther</fictional:character> +library/xml.etree.elementtree,301,:character,<fictional:character>Commander Clement</fictional:character> +library/xml.etree.elementtree,,:actor,"for actor in root.findall('real_person:actor', ns):" +library/xml.etree.elementtree,,:name,"name = actor.find('real_person:name', ns)" +library/xml.etree.elementtree,,:character,"for char in actor.findall('role:character', ns):" +library/zipapp,31,:main,"$ python -m zipapp myapp -m ""myapp:main""" +library/zipapp,82,:fn,"argument should have the form ""pkg.mod:fn"", where ""pkg.mod"" is a" +library/zipapp,155,:callable,"""pkg.module:callable"" and the archive will be run by importing" +library/stdtypes,,::,>>> m[::2].tolist() +library/sys,,`,# ``wrapper`` creates a ``wrap(coro)`` coroutine: +tutorial/venv,77,:c7b9645a6f35,"Python 3.4.3+ (3.4:c7b9645a6f35+, May 22 2015, 09:31:25)" +whatsnew/3.5,,:root,'WARNING:root:warning\n' +whatsnew/3.5,,:warning,'WARNING:root:warning\n' +whatsnew/3.5,,::,>>> addr6 = ipaddress.IPv6Address('::1') +whatsnew/3.5,,:root,ERROR:root:exception +whatsnew/3.5,,:exception,ERROR:root:exception diff --git a/Doc/tools/templates/indexcontent.html b/Doc/tools/templates/indexcontent.html index 969099a006..1076c1f51b 100644 --- a/Doc/tools/templates/indexcontent.html +++ b/Doc/tools/templates/indexcontent.html @@ -1,59 +1,59 @@ {% extends "defindex.html" %} {% block tables %} - <p><strong>Parts of the documentation:</strong></p> + <p><strong>{% trans %}Parts of the documentation:{% endtrans %}</strong></p> <table class="contentstable" align="center"><tr> <td width="50%"> - <p class="biglink"><a class="biglink" href="{{ pathto("whatsnew/" + version) }}">What's new in Python {{ version }}?</a><br/> - <span class="linkdescr">or <a href="{{ pathto("whatsnew/index") }}">all "What's new" documents</a> since 2.0</span></p> - <p class="biglink"><a class="biglink" href="{{ pathto("tutorial/index") }}">Tutorial</a><br/> - <span class="linkdescr">start here</span></p> - <p class="biglink"><a class="biglink" href="{{ pathto("library/index") }}">Library Reference</a><br/> - <span class="linkdescr">keep this under your pillow</span></p> - <p class="biglink"><a class="biglink" href="{{ pathto("reference/index") }}">Language Reference</a><br/> - <span class="linkdescr">describes syntax and language elements</span></p> - <p class="biglink"><a class="biglink" href="{{ pathto("using/index") }}">Python Setup and Usage</a><br/> - <span class="linkdescr">how to use Python on different platforms</span></p> - <p class="biglink"><a class="biglink" href="{{ pathto("howto/index") }}">Python HOWTOs</a><br/> - <span class="linkdescr">in-depth documents on specific topics</span></p> + <p class="biglink"><a class="biglink" href="{{ pathto("whatsnew/" + version) }}">{% trans %}What's new in Python {{ version }}?{% endtrans %}</a><br/> + <span class="linkdescr"> {% trans whatsnew_index=pathto("whatsnew/index") %}or <a href="{{ whatsnew_index }}">all "What's new" documents</a> since 2.0{% endtrans %}</span></p> + <p class="biglink"><a class="biglink" href="{{ pathto("tutorial/index") }}">{% trans %}Tutorial{% endtrans %}</a><br/> + <span class="linkdescr">{% trans %}start here{% endtrans %}</span></p> + <p class="biglink"><a class="biglink" href="{{ pathto("library/index") }}">{% trans %}Library Reference{% endtrans %}</a><br/> + <span class="linkdescr">{% trans %}keep this under your pillow{% endtrans %}</span></p> + <p class="biglink"><a class="biglink" href="{{ pathto("reference/index") }}">{% trans %}Language Reference{% endtrans %}</a><br/> + <span class="linkdescr">{% trans %}describes syntax and language elements{% endtrans %}</span></p> + <p class="biglink"><a class="biglink" href="{{ pathto("using/index") }}">{% trans %}Python Setup and Usage{% endtrans %}</a><br/> + <span class="linkdescr">{% trans %}how to use Python on different platforms{% endtrans %}</span></p> + <p class="biglink"><a class="biglink" href="{{ pathto("howto/index") }}">{% trans %}Python HOWTOs{% endtrans %}</a><br/> + <span class="linkdescr">{% trans %}in-depth documents on specific topics{% endtrans %}</span></p> </td><td width="50%"> - <p class="biglink"><a class="biglink" href="{{ pathto("installing/index") }}">Installing Python Modules</a><br/> - <span class="linkdescr">installing from the Python Package Index & other sources</span></p> - <p class="biglink"><a class="biglink" href="{{ pathto("distributing/index") }}">Distributing Python Modules</a><br/> - <span class="linkdescr">publishing modules for installation by others</span></p> - <p class="biglink"><a class="biglink" href="{{ pathto("extending/index") }}">Extending and Embedding</a><br/> - <span class="linkdescr">tutorial for C/C++ programmers</span></p> - <p class="biglink"><a class="biglink" href="{{ pathto("c-api/index") }}">Python/C API</a><br/> - <span class="linkdescr">reference for C/C++ programmers</span></p> - <p class="biglink"><a class="biglink" href="{{ pathto("faq/index") }}">FAQs</a><br/> - <span class="linkdescr">frequently asked questions (with answers!)</span></p> + <p class="biglink"><a class="biglink" href="{{ pathto("installing/index") }}">{% trans %}Installing Python Modules{% endtrans %}</a><br/> + <span class="linkdescr">{% trans %}installing from the Python Package Index & other sources{% endtrans %}</span></p> + <p class="biglink"><a class="biglink" href="{{ pathto("distributing/index") }}">{% trans %}Distributing Python Modules{% endtrans %}</a><br/> + <span class="linkdescr">{% trans %}publishing modules for installation by others{% endtrans %}</span></p> + <p class="biglink"><a class="biglink" href="{{ pathto("extending/index") }}">{% trans %}Extending and Embedding{% endtrans %}</a><br/> + <span class="linkdescr">{% trans %}tutorial for C/C++ programmers{% endtrans %}</span></p> + <p class="biglink"><a class="biglink" href="{{ pathto("c-api/index") }}">{% trans %}Python/C API{% endtrans %}</a><br/> + <span class="linkdescr">{% trans %}reference for C/C++ programmers{% endtrans %}</span></p> + <p class="biglink"><a class="biglink" href="{{ pathto("faq/index") }}">{% trans %}FAQs{% endtrans %}</a><br/> + <span class="linkdescr">{% trans %}frequently asked questions (with answers!){% endtrans %}</span></p> </td></tr> </table> - <p><strong>Indices and tables:</strong></p> + <p><strong>{% trans %}Indices and tables:{% endtrans %}</strong></p> <table class="contentstable" align="center"><tr> <td width="50%"> - <p class="biglink"><a class="biglink" href="{{ pathto("py-modindex") }}">Global Module Index</a><br/> - <span class="linkdescr">quick access to all modules</span></p> - <p class="biglink"><a class="biglink" href="{{ pathto("genindex") }}">General Index</a><br/> - <span class="linkdescr">all functions, classes, terms</span></p> - <p class="biglink"><a class="biglink" href="{{ pathto("glossary") }}">Glossary</a><br/> - <span class="linkdescr">the most important terms explained</span></p> + <p class="biglink"><a class="biglink" href="{{ pathto("py-modindex") }}">{% trans %}Global Module Index{% endtrans %}</a><br/> + <span class="linkdescr">{% trans %}quick access to all modules{% endtrans %}</span></p> + <p class="biglink"><a class="biglink" href="{{ pathto("genindex") }}">{% trans %}General Index{% endtrans %}</a><br/> + <span class="linkdescr">{% trans %}all functions, classes, terms{% endtrans %}</span></p> + <p class="biglink"><a class="biglink" href="{{ pathto("glossary") }}">{% trans %}Glossary{% endtrans %}</a><br/> + <span class="linkdescr">{% trans %}the most important terms explained{% endtrans %}</span></p> </td><td width="50%"> - <p class="biglink"><a class="biglink" href="{{ pathto("search") }}">Search page</a><br/> - <span class="linkdescr">search this documentation</span></p> - <p class="biglink"><a class="biglink" href="{{ pathto("contents") }}">Complete Table of Contents</a><br/> - <span class="linkdescr">lists all sections and subsections</span></p> + <p class="biglink"><a class="biglink" href="{{ pathto("search") }}">{% trans %}Search page{% endtrans %}</a><br/> + <span class="linkdescr">{% trans %}search this documentation{% endtrans %}</span></p> + <p class="biglink"><a class="biglink" href="{{ pathto("contents") }}">{% trans %}Complete Table of Contents{% endtrans %}</a><br/> + <span class="linkdescr">{% trans %}lists all sections and subsections{% endtrans %}</span></p> </td></tr> </table> - <p><strong>Meta information:</strong></p> + <p><strong>{% trans %}Meta information:{% endtrans %}</strong></p> <table class="contentstable" align="center"><tr> <td width="50%"> - <p class="biglink"><a class="biglink" href="{{ pathto("bugs") }}">Reporting bugs</a></p> - <p class="biglink"><a class="biglink" href="{{ pathto("about") }}">About the documentation</a></p> + <p class="biglink"><a class="biglink" href="{{ pathto("bugs") }}">{% trans %}Reporting bugs{% endtrans %}</a></p> + <p class="biglink"><a class="biglink" href="{{ pathto("about") }}">{% trans %}About the documentation{% endtrans %}</a></p> </td><td width="50%"> - <p class="biglink"><a class="biglink" href="{{ pathto("license") }}">History and License of Python</a></p> - <p class="biglink"><a class="biglink" href="{{ pathto("copyright") }}">Copyright</a></p> + <p class="biglink"><a class="biglink" href="{{ pathto("license") }}">{% trans %}History and License of Python{% endtrans %}</a></p> + <p class="biglink"><a class="biglink" href="{{ pathto("copyright") }}">{% trans %}Copyright{% endtrans %}</a></p> </td></tr> </table> {% endblock %} diff --git a/Doc/tools/templates/indexsidebar.html b/Doc/tools/templates/indexsidebar.html index abdf070772..bb449ef1cd 100644 --- a/Doc/tools/templates/indexsidebar.html +++ b/Doc/tools/templates/indexsidebar.html @@ -1,18 +1,17 @@ -<h3>Download</h3> -<p><a href="{{ pathto('download') }}">Download these documents</a></p> -<h3>Docs for other versions</h3> +<h3>{% trans %}Download{% endtrans %}</h3> +<p><a href="{{ pathto('download') }}">{% trans %}Download these documents{% endtrans %}</a></p> +<h3>{% trans %}Docs for other versions{% endtrans %}</h3> <ul> - <li><a href="https://docs.python.org/2.7/">Python 2.7 (stable)</a></li> - <li><a href="https://docs.python.org/3.3/">Python 3.3 (stable)</a></li> - <li><a href="https://docs.python.org/3.5/">Python 3.5 (in development)</a></li> - <li><a href="https://www.python.org/doc/versions/">Old versions</a></li> + <li><a href="https://docs.python.org/2.7/">{% trans %}Python 2.7 (stable){% endtrans %}</a></li> + <li><a href="https://docs.python.org/3.4/">{% trans %}Python 3.4 (stable){% endtrans %}</a></li> + <li><a href="https://www.python.org/doc/versions/">{% trans %}Old versions{% endtrans %}</a></li> </ul> -<h3>Other resources</h3> +<h3>{% trans %}Other resources{% endtrans %}</h3> <ul> {# XXX: many of these should probably be merged in the main docs #} - <li><a href="https://www.python.org/dev/peps/">PEP Index</a></li> - <li><a href="https://wiki.python.org/moin/BeginnersGuide">Beginner's Guide</a></li> - <li><a href="https://wiki.python.org/moin/PythonBooks">Book List</a></li> - <li><a href="https://www.python.org/doc/av/">Audio/Visual Talks</a></li> + <li><a href="https://www.python.org/dev/peps/">{% trans %}PEP Index{% endtrans %}</a></li> + <li><a href="https://wiki.python.org/moin/BeginnersGuide">{% trans %}Beginner's Guide{% endtrans %}</a></li> + <li><a href="https://wiki.python.org/moin/PythonBooks">{% trans %}Book List{% endtrans %}</a></li> + <li><a href="https://www.python.org/doc/av/">{% trans %}Audio/Visual Talks{% endtrans %}</a></li> </ul> diff --git a/Doc/tools/templates/layout.html b/Doc/tools/templates/layout.html index 8ae6e23f80..1887b85b60 100644 --- a/Doc/tools/templates/layout.html +++ b/Doc/tools/templates/layout.html @@ -6,7 +6,7 @@ <li> {%- if versionswitcher is defined %} <span class="version_switcher_placeholder">{{ release }}</span> - <a href="{{ pathto('index') }}">Documentation</a>{{ reldelim1 }} + <a href="{{ pathto('index') }}">{% trans %}Documentation {% endtrans %}</a>{{ reldelim1 }} {%- else %} <a href="{{ pathto('index') }}">{{ shorttitle }}</a>{{ reldelim1 }} {%- endif %} @@ -79,24 +79,24 @@ {% endblock %} {% block footer %} <div class="footer"> - © <a href="{{ pathto('copyright') }}">Copyright</a> {{ copyright|e }}. + © <a href="{{ pathto('copyright') }}">{% trans %}Copyright{% endtrans %}</a> {{ copyright|e }}. <br /> - The Python Software Foundation is a non-profit corporation. - <a href="https://www.python.org/psf/donations/">Please donate.</a> + {% trans %}The Python Software Foundation is a non-profit corporation.{% endtrans %} + <a href="https://www.python.org/psf/donations/">{% trans %}Please donate.{% endtrans %}</a> <br /> - Last updated on {{ last_updated|e }}. - <a href="{{ pathto('bugs') }}">Found a bug</a>? + {% trans last_updated=last_updated|e %}Last updated on {{ last_updated }}.{% endtrans %} + {% trans pathto_bugs=pathto('bugs') %}<a href="{{ pathto_bugs }}">Found a bug</a>?{% endtrans %} <br /> - Created using <a href="http://sphinx.pocoo.org/">Sphinx</a> {{ sphinx_version|e }}. + {% trans sphinx_version=sphinx_version|e %}Created using <a href="http://sphinx.pocoo.org/">Sphinx</a> {{ sphinx_version }}.{% endtrans %} </div> {% endblock %} {% block sidebarsourcelink %} {%- if show_source and has_source and sourcename %} <h3>{{ _('This Page') }}</h3> <ul class="this-page-menu"> - <li><a href="{{ pathto('bugs') }}">Report a Bug</a></li> + <li><a href="{{ pathto('bugs') }}">{% trans %}Report a Bug{% endtrans %}</a></li> <li><a href="{{ pathto('_sources/' + sourcename, true)|e }}" - rel="nofollow">Show Source</a></li> + rel="nofollow">{% trans %}Show Source{% endtrans %}</a></li> </ul> {%- endif %} {% endblock %} diff --git a/Doc/tutorial/appendix.rst b/Doc/tutorial/appendix.rst index ce50c1e8c3..ffd16aa97a 100644 --- a/Doc/tutorial/appendix.rst +++ b/Doc/tutorial/appendix.rst @@ -40,7 +40,7 @@ Executable Python Scripts On BSD'ish Unix systems, Python scripts can be made directly executable, like shell scripts, by putting the line :: - #!/usr/bin/env python3.4 + #!/usr/bin/env python3.5 (assuming that the interpreter is on the user's :envvar:`PATH`) at the beginning of the script and giving the file an executable mode. The ``#!`` must be the @@ -92,7 +92,7 @@ in the script:: filename = os.environ.get('PYTHONSTARTUP') if filename and os.path.isfile(filename): with open(filename) as fobj: - startup_file = fobj.read() + startup_file = fobj.read() exec(startup_file) @@ -107,7 +107,7 @@ of your user site-packages directory. Start Python and run this code:: >>> import site >>> site.getusersitepackages() - '/home/user/.local/lib/python3.4/site-packages' + '/home/user/.local/lib/python3.5/site-packages' Now you can create a file named :file:`usercustomize.py` in that directory and put anything you want in it. It will affect every invocation of Python, unless diff --git a/Doc/tutorial/classes.rst b/Doc/tutorial/classes.rst index 7e014eff38..03b77e0775 100644 --- a/Doc/tutorial/classes.rst +++ b/Doc/tutorial/classes.rst @@ -120,7 +120,7 @@ are directly accessible: If a name is declared global, then all references and assignments go directly to the middle scope containing the module's global names. To rebind variables found outside of the innermost scope, the :keyword:`nonlocal` statement can be -used; if not declared nonlocal, those variable are read-only (an attempt to +used; if not declared nonlocal, those variables are read-only (an attempt to write to such a variable will simply create a *new* local variable in the innermost scope, leaving the identically named outer variable unchanged). @@ -162,12 +162,15 @@ binding:: def scope_test(): def do_local(): spam = "local spam" + def do_nonlocal(): nonlocal spam spam = "nonlocal spam" + def do_global(): global spam spam = "global spam" + spam = "test spam" do_local() print("After local assignment:", spam) @@ -260,6 +263,7 @@ definition looked like this:: class MyClass: """A simple example class""" i = 12345 + def f(self): return 'hello world' @@ -508,8 +512,10 @@ variable in the class is also ok. For example:: class C: f = f1 + def g(self): return 'hello world' + h = g Now ``f``, ``g`` and ``h`` are all attributes of class :class:`C` that refer to @@ -523,8 +529,10 @@ argument:: class Bag: def __init__(self): self.data = [] + def add(self, x): self.data.append(x) + def addtwice(self, x): self.add(x) self.add(x) @@ -713,7 +721,7 @@ will do nicely:: class Employee: pass - john = Employee() # Create an empty employee record + john = Employee() # Create an empty employee record # Fill the fields of the record john.name = 'John Doe' @@ -839,8 +847,10 @@ defines :meth:`__next__`, then :meth:`__iter__` can just return ``self``:: def __init__(self, data): self.data = data self.index = len(data) + def __iter__(self): return self + def __next__(self): if self.index == 0: raise StopIteration @@ -941,8 +951,8 @@ Examples:: .. rubric:: Footnotes .. [#] Except for one thing. Module objects have a secret read-only attribute called - :attr:`__dict__` which returns the dictionary used to implement the module's - namespace; the name :attr:`__dict__` is an attribute but not a global name. + :attr:`~object.__dict__` which returns the dictionary used to implement the module's + namespace; the name :attr:`~object.__dict__` is an attribute but not a global name. Obviously, using this violates the abstraction of namespace implementation, and should be restricted to things like post-mortem debuggers. diff --git a/Doc/tutorial/controlflow.rst b/Doc/tutorial/controlflow.rst index 813c8285f8..12989b2a53 100644 --- a/Doc/tutorial/controlflow.rst +++ b/Doc/tutorial/controlflow.rst @@ -312,7 +312,7 @@ You can see it if you really want to using :func:`print`:: It is simple to write a function that returns a list of the numbers of the Fibonacci series, instead of printing it:: - >>> def fib2(n): # return Fibonacci series up to n + >>> def fib2(n): # return Fibonacci series up to n ... """Return a list containing the Fibonacci series up to n.""" ... result = [] ... a, b = 0, 1 @@ -361,7 +361,7 @@ The most useful form is to specify a default value for one or more arguments. This creates a function that can be called with fewer arguments than it is defined to allow. For example:: - def ask_ok(prompt, retries=4, complaint='Yes or no, please!'): + def ask_ok(prompt, retries=4, reminder='Please try again!'): while True: ok = input(prompt) if ok in ('y', 'ye', 'yes'): @@ -370,8 +370,8 @@ defined to allow. For example:: return False retries = retries - 1 if retries < 0: - raise OSError('uncooperative user') - print(complaint) + raise ValueError('invalid user response') + print(reminder) This function can be called in several ways: @@ -501,7 +501,9 @@ It could be called like this:: client="John Cleese", sketch="Cheese Shop Sketch") -and of course it would print:: +and of course it would print: + +.. code-block:: none -- Do you have any Limburger ? -- I'm sorry, we're all out of Limburger @@ -540,7 +542,7 @@ parameter are 'keyword-only' arguments, meaning that they can only be used as keywords rather than positional arguments. :: >>> def concat(*args, sep="/"): - ... return sep.join(args) + ... return sep.join(args) ... >>> concat("earth", "mars", "venus") 'earth/mars/venus' diff --git a/Doc/tutorial/datastructures.rst b/Doc/tutorial/datastructures.rst index 1ea299fbed..b39bdf4dfb 100644 --- a/Doc/tutorial/datastructures.rst +++ b/Doc/tutorial/datastructures.rst @@ -73,10 +73,11 @@ objects: Return the number of times *x* appears in the list. -.. method:: list.sort() +.. method:: list.sort(key=None, reverse=False) :noindex: - Sort the items of the list in place. + Sort the items of the list in place (the arguments can be used for sort + customization, see :func:`sorted` for their explanation). .. method:: list.reverse() @@ -396,7 +397,7 @@ objects, such as lists. Though tuples may seem similar to lists, they are often used in different situations and for different purposes. -Tuples are :term:`immutable`, and usually contain an heterogeneous sequence of +Tuples are :term:`immutable`, and usually contain a heterogeneous sequence of elements that are accessed via unpacking (see later in this section) or indexing (or even by attribute in the case of :func:`namedtuples <collections.namedtuple>`). Lists are :term:`mutable`, and their elements are usually homogeneous and are @@ -611,18 +612,18 @@ returns a new sorted list while leaving the source unaltered. :: orange pear -To change a sequence you are iterating over while inside the loop (for -example to duplicate certain items), it is recommended that you first make -a copy. Looping over a sequence does not implicitly make a copy. The slice -notation makes this especially convenient:: +It is sometimes tempting to change a list while you are looping over it; +however, it is often simpler and safer to create a new list instead. :: - >>> words = ['cat', 'window', 'defenestrate'] - >>> for w in words[:]: # Loop over a slice copy of the entire list. - ... if len(w) > 6: - ... words.insert(0, w) + >>> import math + >>> raw_data = [56.2, float('NaN'), 51.7, 55.3, 52.5, float('NaN'), 47.8] + >>> filtered_data = [] + >>> for value in raw_data: + ... if not math.isnan(value): + ... filtered_data.append(value) ... - >>> words - ['defenestrate', 'cat', 'window', 'defenestrate'] + >>> filtered_data + [56.2, 51.7, 55.3, 52.5, 47.8] .. _tut-conditions: diff --git a/Doc/tutorial/errors.rst b/Doc/tutorial/errors.rst index 351ee52b4e..4195c7e892 100644 --- a/Doc/tutorial/errors.rst +++ b/Doc/tutorial/errors.rst @@ -170,15 +170,15 @@ reference ``.args``. One may also instantiate an exception first before raising it and add any attributes to it as desired. :: >>> try: - ... raise Exception('spam', 'eggs') + ... raise Exception('spam', 'eggs') ... except Exception as inst: - ... print(type(inst)) # the exception instance - ... print(inst.args) # arguments stored in .args - ... print(inst) # __str__ allows args to be printed directly, - ... # but may be overridden in exception subclasses - ... x, y = inst.args # unpack args - ... print('x =', x) - ... print('y =', y) + ... print(type(inst)) # the exception instance + ... print(inst.args) # arguments stored in .args + ... print(inst) # __str__ allows args to be printed directly, + ... # but may be overridden in exception subclasses + ... x, y = inst.args # unpack args + ... print('x =', x) + ... print('y =', y) ... <class 'Exception'> ('spam', 'eggs') diff --git a/Doc/tutorial/floatingpoint.rst b/Doc/tutorial/floatingpoint.rst index 9c3c1435c3..d440e53518 100644 --- a/Doc/tutorial/floatingpoint.rst +++ b/Doc/tutorial/floatingpoint.rst @@ -155,7 +155,7 @@ which implements arithmetic based on rational numbers (so the numbers like If you are a heavy user of floating point operations you should take a look at the Numerical Python package and many other packages for mathematical and -statistical operations supplied by the SciPy project. See <http://scipy.org>. +statistical operations supplied by the SciPy project. See <https://scipy.org>. Python provides tools that may help on those rare occasions when you really *do* want to know the exact value of a float. The diff --git a/Doc/tutorial/inputoutput.rst b/Doc/tutorial/inputoutput.rst index c157f22e6c..dd9c7cd731 100644 --- a/Doc/tutorial/inputoutput.rst +++ b/Doc/tutorial/inputoutput.rst @@ -152,11 +152,11 @@ Positional and keyword arguments can be arbitrarily combined:: ``'!a'`` (apply :func:`ascii`), ``'!s'`` (apply :func:`str`) and ``'!r'`` (apply :func:`repr`) can be used to convert the value before it is formatted:: - >>> import math - >>> print('The value of PI is approximately {}.'.format(math.pi)) - The value of PI is approximately 3.14159265359. - >>> print('The value of PI is approximately {!r}.'.format(math.pi)) - The value of PI is approximately 3.141592653589793. + >>> contents = 'eels' + >>> print('My hovercraft is full of {}.'.format(contents)) + My hovercraft is full of eels. + >>> print('My hovercraft is full of {!r}.'.format(contents)) + My hovercraft is full of 'eels'. An optional ``':'`` and format specifier can follow the field name. This allows greater control over how the value is formatted. The following example @@ -271,10 +271,11 @@ The rest of the examples in this section will assume that a file object called ``f`` has already been created. To read a file's contents, call ``f.read(size)``, which reads some quantity of -data and returns it as a string or bytes object. *size* is an optional numeric -argument. When *size* is omitted or negative, the entire contents of the file -will be read and returned; it's your problem if the file is twice as large as -your machine's memory. Otherwise, at most *size* bytes are read and returned. +data and returns it as a string (in text mode) or bytes object (in binary mode). +*size* is an optional numeric argument. When *size* is omitted or negative, the +entire contents of the file will be read and returned; it's your problem if the +file is twice as large as your machine's memory. Otherwise, at most *size* bytes +are read and returned. If the end of the file has been reached, ``f.read()`` will return an empty string (``''``). :: @@ -315,11 +316,11 @@ the number of characters written. :: >>> f.write('This is a test\n') 15 -To write something other than a string, it needs to be converted to a string -first:: +Other types of objects need to be converted -- either to a string (in text mode) +or a bytes object (in binary mode) -- before writing them:: >>> value = ('the answer', 42) - >>> s = str(value) + >>> s = str(value) # convert the tuple to string >>> f.write(s) 18 @@ -337,11 +338,11 @@ beginning of the file as the reference point. :: >>> f = open('workfile', 'rb+') >>> f.write(b'0123456789abcdef') 16 - >>> f.seek(5) # Go to the 6th byte in the file + >>> f.seek(5) # Go to the 6th byte in the file 5 >>> f.read(1) b'5' - >>> f.seek(-3, 2) # Go to the 3rd byte before the end + >>> f.seek(-3, 2) # Go to the 3rd byte before the end 13 >>> f.read(1) b'd' diff --git a/Doc/tutorial/interactive.rst b/Doc/tutorial/interactive.rst index abf30f020d..d73cfeb34f 100644 --- a/Doc/tutorial/interactive.rst +++ b/Doc/tutorial/interactive.rst @@ -49,6 +49,6 @@ into other applications. Another similar enhanced interactive environment is bpython_. -.. _GNU Readline: http://tiswww.case.edu/php/chet/readline/rltop.html -.. _IPython: http://ipython.scipy.org/ +.. _GNU Readline: https://tiswww.case.edu/php/chet/readline/rltop.html +.. _IPython: https://ipython.org/ .. _bpython: http://www.bpython-interpreter.org/ diff --git a/Doc/tutorial/interpreter.rst b/Doc/tutorial/interpreter.rst index dd6b1a9f38..e8d8e2b56a 100644 --- a/Doc/tutorial/interpreter.rst +++ b/Doc/tutorial/interpreter.rst @@ -10,13 +10,13 @@ Using the Python Interpreter Invoking the Interpreter ======================== -The Python interpreter is usually installed as :file:`/usr/local/bin/python3.4` +The Python interpreter is usually installed as :file:`/usr/local/bin/python3.5` on those machines where it is available; putting :file:`/usr/local/bin` in your Unix shell's search path makes it possible to start it by typing the command: .. code-block:: text - python3.4 + python3.5 to the shell. [#]_ Since the choice of the directory where the interpreter lives is an installation option, other places are possible; check with your local @@ -24,11 +24,11 @@ Python guru or system administrator. (E.g., :file:`/usr/local/python` is a popular alternative location.) On Windows machines, the Python installation is usually placed in -:file:`C:\\Python34`, though you can change this when you're running the +:file:`C:\\Python35`, though you can change this when you're running the installer. To add this directory to your path, you can type the following command into the command prompt in a DOS box:: - set path=%path%;C:\python34 + set path=%path%;C:\python35 Typing an end-of-file character (:kbd:`Control-D` on Unix, :kbd:`Control-Z` on Windows) at the primary prompt causes the interpreter to exit with a zero exit @@ -94,10 +94,12 @@ mode*. In this mode it prompts for the next command with the *primary prompt*, usually three greater-than signs (``>>>``); for continuation lines it prompts with the *secondary prompt*, by default three dots (``...``). The interpreter prints a welcome message stating its version number and a copyright notice -before printing the first prompt:: +before printing the first prompt: - $ python3.4 - Python 3.4 (default, Mar 16 2014, 09:25:04) +.. code-block:: shell-session + + $ python3.5 + Python 3.5 (default, Sep 16 2015, 09:25:04) [GCC 4.8.2] on linux Type "help", "copyright", "credits" or "license" for more information. >>> diff --git a/Doc/tutorial/introduction.rst b/Doc/tutorial/introduction.rst index c5c1343843..214032917b 100644 --- a/Doc/tutorial/introduction.rst +++ b/Doc/tutorial/introduction.rst @@ -232,7 +232,7 @@ If you want to concatenate variables or a variable and a literal, use ``+``:: This feature is particularly useful when you want to break long strings:: >>> text = ('Put several strings within parentheses ' - 'to have them joined together.') + ... 'to have them joined together.') >>> text 'Put several strings within parentheses to have them joined together.' @@ -276,11 +276,11 @@ makes sure that ``s[:i] + s[i:]`` is always equal to ``s``:: Slice indices have useful defaults; an omitted first index defaults to zero, an omitted second index defaults to the size of the string being sliced. :: - >>> word[:2] # character from the beginning to position 2 (excluded) + >>> word[:2] # character from the beginning to position 2 (excluded) 'Py' - >>> word[4:] # characters from position 4 (included) to the end + >>> word[4:] # characters from position 4 (included) to the end 'on' - >>> word[-2:] # characters from the second-last (included) to the end + >>> word[-2:] # characters from the second-last (included) to the end 'on' One way to remember how slices work is to think of the indices as pointing @@ -352,9 +352,8 @@ The built-in function :func:`len` returns the length of a string:: Strings support a large number of methods for basic transformations and searching. - :ref:`string-formatting` - Information about string formatting with :meth:`str.format` is described - here. + :ref:`formatstrings` + Information about string formatting with :meth:`str.format`. :ref:`old-string-formatting` The old formatting operations invoked when strings and Unicode strings are diff --git a/Doc/tutorial/modules.rst b/Doc/tutorial/modules.rst index fd361aed69..35db305d9e 100644 --- a/Doc/tutorial/modules.rst +++ b/Doc/tutorial/modules.rst @@ -34,7 +34,7 @@ called :file:`fibo.py` in the current directory with the following contents:: a, b = b, a+b print() - def fib2(n): # return Fibonacci series up to n + def fib2(n): # return Fibonacci series up to n result = [] a, b = 0, 1 while b < n: @@ -117,7 +117,8 @@ use it to save typing in interactive sessions. For efficiency reasons, each module is only imported once per interpreter session. Therefore, if you change your modules, you must restart the interpreter -- or, if it's just one module you want to test interactively, - use :func:`imp.reload`, e.g. ``import imp; imp.reload(modulename)``. + use :func:`importlib.reload`, e.g. ``import importlib; + importlib.reload(modulename)``. .. _tut-modulesasscripts: @@ -139,7 +140,9 @@ the end of your module:: you can make the file usable as a script as well as an importable module, because the code that parses the command line only runs if the module is -executed as the "main" file:: +executed as the "main" file: + +.. code-block:: shell-session $ python fibo.py 50 1 1 2 3 5 8 13 21 34 @@ -216,15 +219,15 @@ Some tips for experts: statements, the ``-OO`` switch removes both assert statements and __doc__ strings. Since some programs may rely on having these available, you should only use this option if you know what you're doing. "Optimized" modules have - a .pyo rather than a .pyc suffix and are usually smaller. Future releases may + an ``opt-`` tag and are usually smaller. Future releases may change the effects of optimization. -* A program doesn't run any faster when it is read from a ``.pyc`` or ``.pyo`` +* A program doesn't run any faster when it is read from a ``.pyc`` file than when it is read from a ``.py`` file; the only thing that's faster - about ``.pyc`` or ``.pyo`` files is the speed with which they are loaded. + about ``.pyc`` files is the speed with which they are loaded. -* The module :mod:`compileall` can create .pyc files (or .pyo files when - :option:`-O` is used) for all modules in a directory. +* The module :mod:`compileall` can create .pyc files for all modules in a + directory. * There is more detail on this process, including a flow chart of the decisions, in PEP 3147. @@ -548,4 +551,3 @@ modules found in a package. .. [#] In fact function definitions are also 'statements' that are 'executed'; the execution of a module-level function definition enters the function name in the module's global symbol table. - diff --git a/Doc/tutorial/stdlib.rst b/Doc/tutorial/stdlib.rst index 72d51de1c0..52ffdbe063 100644 --- a/Doc/tutorial/stdlib.rst +++ b/Doc/tutorial/stdlib.rst @@ -15,7 +15,7 @@ operating system:: >>> import os >>> os.getcwd() # Return the current working directory - 'C:\\Python34' + 'C:\\Python35' >>> os.chdir('/server/accesslogs') # Change current working directory >>> os.system('mkdir today') # Run the command mkdir in the system shell 0 @@ -152,7 +152,7 @@ The :mod:`statistics` module calculates basic statistical properties >>> statistics.variance(data) 1.3720238095238095 -The SciPy project <http://scipy.org> has many other modules for numerical +The SciPy project <https://scipy.org> has many other modules for numerical computations. .. _tut-internet-access: @@ -301,7 +301,7 @@ file:: with self.assertRaises(TypeError): average(20, 30, 70) - unittest.main() # Calling from the command line invokes all tests + unittest.main() # Calling from the command line invokes all tests .. _tut-batteries-included: diff --git a/Doc/tutorial/stdlib2.rst b/Doc/tutorial/stdlib2.rst index c0197ea5ef..3714384eb6 100644 --- a/Doc/tutorial/stdlib2.rst +++ b/Doc/tutorial/stdlib2.rst @@ -18,7 +18,7 @@ abbreviated displays of large or deeply nested containers:: >>> import reprlib >>> reprlib.repr(set('supercalifragilisticexpialidocious')) - "set(['a', 'c', 'd', 'e', 'f', 'g', ...])" + "{'a', 'c', 'd', 'e', 'f', 'g', ...}" The :mod:`pprint` module offers more sophisticated control over printing both built-in and user defined objects in a way that is readable by the interpreter. @@ -180,6 +180,7 @@ tasks in background while the main program continues to run:: threading.Thread.__init__(self) self.infile = infile self.outfile = outfile + def run(self): f = zipfile.ZipFile(self.outfile, 'w', zipfile.ZIP_DEFLATED) f.write(self.infile) @@ -277,7 +278,7 @@ applications include caching objects that are expensive to create:: Traceback (most recent call last): File "<stdin>", line 1, in <module> d['primary'] # entry was automatically removed - File "C:/python34/lib/weakref.py", line 46, in __getitem__ + File "C:/python35/lib/weakref.py", line 46, in __getitem__ o = self.data[key]() KeyError: 'primary' diff --git a/Doc/tutorial/whatnow.rst b/Doc/tutorial/whatnow.rst index 479542c172..1ea22ae5cd 100644 --- a/Doc/tutorial/whatnow.rst +++ b/Doc/tutorial/whatnow.rst @@ -43,7 +43,7 @@ More Python resources: for download. Once you begin releasing code, you can register it here so that others can find it. -* http://code.activestate.com/recipes/langs/python/: The Python Cookbook is a +* https://code.activestate.com/recipes/langs/python/: The Python Cookbook is a sizable collection of code examples, larger modules, and useful scripts. Particularly notable contributions are collected in a book also titled Python Cookbook (O'Reilly & Associates, ISBN 0-596-00797-3.) @@ -51,7 +51,7 @@ More Python resources: * http://www.pyvideo.org collects links to Python-related videos from conferences and user-group meetings. -* http://scipy.org: The Scientific Python project includes modules for fast +* https://scipy.org: The Scientific Python project includes modules for fast array computations and manipulations plus a host of packages for such things as linear algebra, Fourier transforms, non-linear solvers, random number distributions, statistical analysis and the like. diff --git a/Doc/using/cmdline.rst b/Doc/using/cmdline.rst index a46349bcde..ec744a351d 100644 --- a/Doc/using/cmdline.rst +++ b/Doc/using/cmdline.rst @@ -77,7 +77,7 @@ source. the :mod:`__main__` module. Since the argument is a *module* name, you must not give a file extension - (``.py``). The ``module-name`` should be a valid Python module name, but + (``.py``). The module name should be a valid absolute Python module name, but the implementation may not always enforce this (e.g. it may allow you to use a name that includes a hyphen). @@ -190,13 +190,16 @@ Miscellaneous options .. cmdoption:: -b - Issue a warning when comparing str and bytes. Issue an error when the + Issue a warning when comparing :class:`bytes` or :class:`bytearray` with + :class:`str` or :class:`bytes` with :class:`int`. Issue an error when the option is given twice (:option:`-bb`). + .. versionchanged:: 3.5 + Affects comparisons of :class:`bytes` with :class:`int`. .. cmdoption:: -B - If given, Python won't try to write ``.pyc`` or ``.pyo`` files on the + If given, Python won't try to write ``.pyc`` files on the import of source modules. See also :envvar:`PYTHONDONTWRITEBYTECODE`. @@ -395,7 +398,7 @@ Miscellaneous options tracing with a traceback limit of *NFRAME* frames. See the :func:`tracemalloc.start` for more information. - It also allows to pass arbitrary values and retrieve them through the + It also allows passing arbitrary values and retrieving them through the :data:`sys._xoptions` dictionary. .. versionchanged:: 3.2 diff --git a/Doc/using/mac.rst b/Doc/using/mac.rst index ef091e5275..05c91bba59 100644 --- a/Doc/using/mac.rst +++ b/Doc/using/mac.rst @@ -25,7 +25,7 @@ there. What you get after installing is a number of things: -* A :file:`MacPython 3.4` folder in your :file:`Applications` folder. In here +* A :file:`MacPython 3.5` folder in your :file:`Applications` folder. In here you find IDLE, the development environment that is a standard part of official Python distributions; PythonLauncher, which handles double-clicking Python scripts from the Finder; and the "Build Applet" tool, which allows you to @@ -65,7 +65,7 @@ number of standard Unix command line editors, :program:`vim` and :program:`emacs` among them. If you want a more Mac-like editor, :program:`BBEdit` or :program:`TextWrangler` from Bare Bones Software (see http://www.barebones.com/products/bbedit/index.html) are good choices, as is -:program:`TextMate` (see http://macromates.com/). Other editors include +:program:`TextMate` (see https://macromates.com/). Other editors include :program:`Gvim` (http://macvim.org) and :program:`Aquamacs` (http://aquamacs.org/). @@ -93,7 +93,7 @@ aware of: programs that talk to the Aqua window manager (in other words, anything that has a GUI) need to be run in a special way. Use :program:`pythonw` instead of :program:`python` to start such scripts. -With Python 3.4, you can use either :program:`python` or :program:`pythonw`. +With Python 3.5, you can use either :program:`python` or :program:`pythonw`. Configuration @@ -144,22 +144,22 @@ the foundation of most modern Mac development. Information on PyObjC is available from https://pythonhosted.org/pyobjc/. The standard Python GUI toolkit is :mod:`tkinter`, based on the cross-platform -Tk toolkit (http://www.tcl.tk). An Aqua-native version of Tk is bundled with OS +Tk toolkit (https://www.tcl.tk). An Aqua-native version of Tk is bundled with OS X by Apple, and the latest version can be downloaded and installed from -http://www.activestate.com; it can also be built from source. +https://www.activestate.com; it can also be built from source. *wxPython* is another popular cross-platform GUI toolkit that runs natively on Mac OS X. Packages and documentation are available from http://www.wxpython.org. *PyQt* is another popular cross-platform GUI toolkit that runs natively on Mac OS X. More information can be found at -http://www.riverbankcomputing.co.uk/software/pyqt/intro. +https://riverbankcomputing.com/software/pyqt/intro. Distributing Python Applications on the Mac =========================================== -The "Build Applet" tool that is placed in the MacPython 3.4 folder is fine for +The "Build Applet" tool that is placed in the MacPython 3.5 folder is fine for packaging small Python scripts on your own machine to run as a standard Mac application. This tool, however, is not robust enough to distribute Python applications to other users. diff --git a/Doc/using/unix.rst b/Doc/using/unix.rst index 5da1f23346..4449d4f4d4 100644 --- a/Doc/using/unix.rst +++ b/Doc/using/unix.rst @@ -26,11 +26,11 @@ following links: .. seealso:: - http://www.debian.org/doc/manuals/maint-guide/first.en.html + https://www.debian.org/doc/manuals/maint-guide/first.en.html for Debian users - http://en.opensuse.org/Portal:Packaging + https://en.opensuse.org/Portal:Packaging for OpenSuse users - http://docs.fedoraproject.org/en-US/Fedora_Draft_Documentation/0.1/html/RPM_Guide/ch-creating-rpms.html + https://docs.fedoraproject.org/en-US/Fedora_Draft_Documentation/0.1/html/RPM_Guide/ch-creating-rpms.html for Fedora users http://www.slackbook.org/html/package-management-making-packages.html for Slackware users @@ -55,7 +55,7 @@ On FreeBSD and OpenBSD On OpenSolaris -------------- -You can get Python from `OpenCSW <http://www.opencsw.org/>`_. Various versions +You can get Python from `OpenCSW <https://www.opencsw.org/>`_. Various versions of Python are available and can be installed with e.g. ``pkgutil -i python27``. @@ -65,7 +65,7 @@ Building Python =============== If you want to compile CPython yourself, first thing you should do is get the -`source <https://www.python.org/download/source/>`_. You can download either the +`source <https://www.python.org/downloads/source/>`_. You can download either the latest release's source or just grab a fresh `clone <https://docs.python.org/devguide/setup.html#getting-the-source-code>`_. (If you want to contribute patches, you will need a clone.) @@ -139,10 +139,10 @@ Vim and Emacs are excellent editors which support Python very well. For more information on how to code in Python in these editors, look at: * http://www.vim.org/scripts/script.php?script_id=790 -* http://sourceforge.net/projects/python-mode +* https://sourceforge.net/projects/python-mode Geany is an excellent IDE with support for a lot of languages. For more -information, read: http://www.geany.org/ +information, read: https://www.geany.org/ Komodo edit is another extremely good IDE. It also has support for a lot of -languages. For more information, read http://komodoide.com/. +languages. For more information, read https://komodoide.com/. diff --git a/Doc/using/venv-create.inc b/Doc/using/venv-create.inc index 45bdd5a46d..7ad3008826 100644 --- a/Doc/using/venv-create.inc +++ b/Doc/using/venv-create.inc @@ -14,23 +14,24 @@ subdirectory (on Windows, this is ``Lib\site-packages``). .. seealso:: `Python Packaging User Guide: Creating and using virtual environments - <https://packaging.python.org/en/latest/installing.html#virtual-environments>`__ + <https://packaging.python.org/en/latest/installing/#creating-virtual-environments>`__ .. highlight:: none On Windows, you may have to invoke the ``pyvenv`` script as follows, if you don't have the relevant PATH and PATHEXT settings:: - c:\Temp>c:\Python34\python c:\Python34\Tools\Scripts\pyvenv.py myenv + c:\Temp>c:\Python35\python c:\Python35\Tools\Scripts\pyvenv.py myenv or equivalently:: - c:\Temp>c:\Python34\python -m venv myenv + c:\Temp>c:\Python35\python -m venv myenv The command, if run with ``-h``, will show the available options:: - usage: venv [-h] [--system-site-packages] [--symlinks] [--clear] - [--upgrade] [--without-pip] ENV_DIR [ENV_DIR ...] + usage: venv [-h] [--system-site-packages] [--symlinks | --copies] [--clear] + [--upgrade] [--without-pip] + ENV_DIR [ENV_DIR ...] Creates virtual Python environments in one or more target directories. @@ -39,15 +40,14 @@ The command, if run with ``-h``, will show the available options:: optional arguments: -h, --help show this help message and exit - --system-site-packages Give access to the global site-packages dir to the - virtual environment. + --system-site-packages Give the virtual environment access to the system + site-packages dir. --symlinks Try to use symlinks rather than copies, when symlinks are not the default for the platform. --copies Try to use copies rather than symlinks, even when symlinks are the default for the platform. - --clear Delete the environment directory if it already exists. - If not specified and the directory exists, an error is - raised. + --clear Delete the contents of the environment directory if it + already exists, before environment creation. --upgrade Upgrade the environment directory to use this version of Python, assuming Python has been upgraded in-place. --without-pip Skips installing or upgrading pip in the virtual @@ -89,9 +89,9 @@ venv's binary directory. The invocation of the script is platform-specific: +-------------+-----------------+-----------------------------------------+ | | csh/tcsh | $ source <venv>/bin/activate.csh | +-------------+-----------------+-----------------------------------------+ -| Windows | cmd.exe | C:\> <venv>/Scripts/activate.bat | +| Windows | cmd.exe | C:\\> <venv>\\Scripts\\activate.bat | +-------------+-----------------+-----------------------------------------+ -| | PowerShell | PS C:\> <venv>/Scripts/Activate.ps1 | +| | PowerShell | PS C:\\> <venv>\\Scripts\\Activate.ps1 | +-------------+-----------------+-----------------------------------------+ You don't specifically *need* to activate an environment; activation just diff --git a/Doc/using/win_installer.png b/Doc/using/win_installer.png Binary files differnew file mode 100644 index 0000000000..00c88a830f --- /dev/null +++ b/Doc/using/win_installer.png diff --git a/Doc/using/windows.rst b/Doc/using/windows.rst index c05f72a853..d010a61be4 100644 --- a/Doc/using/windows.rst +++ b/Doc/using/windows.rst @@ -7,40 +7,256 @@ ************************* .. sectionauthor:: Robert Lehmann <lehmannro@gmail.com> +.. sectionauthor:: Steve Dower <steve.dower@microsoft.com> This document aims to give an overview of Windows-specific behaviour you should know about when using Python on Microsoft Windows. -.. XXX (ncoghlan) - - This looks rather stale to me... - - Installing Python ================= -Unlike most Unix systems and services, Windows does not require Python natively -and thus does not pre-install a version of Python. However, the CPython team +Unlike most Unix systems and services, Windows does not include a system +supported installation of Python. To make Python available, the CPython team has compiled Windows installers (MSI packages) with every `release -<https://www.python.org/download/releases/>`_ for many years. +<https://www.python.org/download/releases/>`_ for many years. These installers +are primarily intended to add a per-user installation of Python, with the +core interpreter and library being used by a single user. The installer is also +able to install for all users of a single machine, and a separate ZIP file is +available for application-local distributions. + +Supported Versions +------------------ + +As specified in :pep:`11`, a Python release only supports a Windows platform +while Microsoft considers the platform under extended support. This means that +Python 3.5 supports Windows Vista and newer. If you require Windows XP support +then please install Python 3.4. + +Installation Steps +------------------ + +Four Python 3.5 installers are available for download - two each for the 32-bit +and 64-bit versions of the interpreter. The *web installer* is a small initial +download, and it will automatically download the required components as +necessary. The *offline installer* includes the components necessary for a +default installation and only requires an internet connection for optional +features. See :ref:`install-layout-option` for other ways to avoid downloading +during installation. + +After starting the installer, one of two options may be selected: + +.. image:: win_installer.png + +If you select "Install Now": + +* You will *not* need to be an administrator (unless a system update for the + C Runtime Library is required or you install the :ref:`launcher` for all + users) +* Python will be installed into your user directory +* The :ref:`launcher` will be installed according to the option at the bottom + of the first page +* The standard library, test suite, launcher and pip will be installed +* If selected, the install directory will be added to your :envvar:`PATH` +* Shortcuts will only be visible for the current user + +Selecting "Customize installation" will allow you to select the features to +install, the installation location and other options or post-install actions. +To install debugging symbols or binaries, you will need to use this option. + +To perform an all-users installation, you should select "Customize +installation". In this case: + +* You may be required to provide administrative credentials or approval +* Python will be installed into the Program Files directory +* The :ref:`launcher` will be installed into the Windows directory +* Optional features may be selected during installation +* The standard library can be pre-compiled to bytecode +* If selected, the install directory will be added to the system :envvar:`PATH` +* Shortcuts are available for all users + +.. _install-quiet-option: + +Installing Without UI +--------------------- + +All of the options available in the installer UI can also be specified from the +command line, allowing scripted installers to replicate an installation on many +machines without user interaction. These options may also be set without +suppressing the UI in order to change some of the defaults. + +To completely hide the installer UI and install Python silently, pass the +``/quiet`` option. To skip past the user interaction but still display +progress and errors, pass the ``/passive`` option. The ``/uninstall`` +option may be passed to immediately begin removing Python - no prompt will be +displayed. + +All other options are passed as ``name=value``, where the value is usually +``0`` to disable a feature, ``1`` to enable a feature, or a path. The full list +of available options is shown below. + ++---------------------------+--------------------------------------+--------------------------+ +| Name | Description | Default | ++===========================+======================================+==========================+ +| InstallAllUsers | Perform a system-wide installation. | 0 | ++---------------------------+--------------------------------------+--------------------------+ +| TargetDir | The installation directory | Selected based on | +| | | InstallAllUsers | ++---------------------------+--------------------------------------+--------------------------+ +| DefaultAllUsersTargetDir | The default installation directory | :file:`%ProgramFiles%\\\ | +| | for all-user installs | Python X.Y` or :file:`\ | +| | | %ProgramFiles(x86)%\\\ | +| | | Python X.Y` | ++---------------------------+--------------------------------------+--------------------------+ +| DefaultJustForMeTargetDir | The default install directory for | :file:`%LocalAppData%\\\ | +| | just-for-me installs | Programs\\PythonXY` or | +| | | :file:`%LocalAppData%\\\ | +| | | Programs\\PythonXY-32` | ++---------------------------+--------------------------------------+--------------------------+ +| DefaultCustomTargetDir | The default custom install directory | (empty) | +| | displayed in the UI | | ++---------------------------+--------------------------------------+--------------------------+ +| AssociateFiles | Create file associations if the | 1 | +| | launcher is also installed. | | ++---------------------------+--------------------------------------+--------------------------+ +| CompileAll | Compile all ``.py`` files to | 0 | +| | ``.pyc``. | | ++---------------------------+--------------------------------------+--------------------------+ +| PrependPath | Add install and Scripts directories | 0 | +| | tho :envvar:`PATH` and ``.PY`` to | | +| | :envvar:`PATHEXT` | | ++---------------------------+--------------------------------------+--------------------------+ +| Shortcuts | Create shortcuts for the interpreter,| 1 | +| | documentation and IDLE if installed. | | ++---------------------------+--------------------------------------+--------------------------+ +| Include_doc | Install Python manual | 1 | ++---------------------------+--------------------------------------+--------------------------+ +| Include_debug | Install debug binaries | 0 | ++---------------------------+--------------------------------------+--------------------------+ +| Include_dev | Install developer headers and | 1 | +| | libraries | | ++---------------------------+--------------------------------------+--------------------------+ +| Include_exe | Install :file:`python.exe` and | 1 | +| | related files | | ++---------------------------+--------------------------------------+--------------------------+ +| Include_launcher | Install :ref:`launcher`. | 1 | ++---------------------------+--------------------------------------+--------------------------+ +| InstallLauncherAllUsers | Installs :ref:`launcher` for all | 1 | +| | users. | | ++---------------------------+--------------------------------------+--------------------------+ +| Include_lib | Install standard library and | 1 | +| | extension modules | | ++---------------------------+--------------------------------------+--------------------------+ +| Include_pip | Install bundled pip and setuptools | 1 | ++---------------------------+--------------------------------------+--------------------------+ +| Include_symbols | Install debugging symbols (`*`.pdb) | 0 | ++---------------------------+--------------------------------------+--------------------------+ +| Include_tcltk | Install Tcl/Tk support and IDLE | 1 | ++---------------------------+--------------------------------------+--------------------------+ +| Include_test | Install standard library test suite | 1 | ++---------------------------+--------------------------------------+--------------------------+ +| Include_tools | Install utility scripts | 1 | ++---------------------------+--------------------------------------+--------------------------+ +| LauncherOnly | Only installs the launcher. This | 0 | +| | will override most other options. | | ++---------------------------+--------------------------------------+--------------------------+ +| SimpleInstall | Disable most install UI | 0 | ++---------------------------+--------------------------------------+--------------------------+ +| SimpleInstallDescription | A custom message to display when the | (empty) | +| | simplified install UI is used. | | ++---------------------------+--------------------------------------+--------------------------+ + +For example, to silently install a default, system-wide Python installation, +you could use the following command (from an elevated command prompt):: + + python-3.5.0.exe /quiet InstallAllUsers=1 PrependPath=1 Include_test=0 + +To allow users to easily install a personal copy of Python without the test +suite, you could provide a shortcut with the following command. This will +display a simplified initial page and disallow customization:: + + python-3.5.0.exe InstallAllUsers=0 Include_launcher=0 Include_test=0 + SimpleInstall=1 SimpleInstallDescription="Just for me, no test suite." + +(Note that omitting the launcher also omits file associations, and is only +recommended for per-user installs when there is also a system-wide installation +that included the launcher.) + +The options listed above can also be provided in a file named ``unattend.xml`` +alongside the executable. This file specifies a list of options and values. +When a value is provided as an attribute, it will be converted to a number if +possible. Values provided as element text are always left as strings. This +example file sets the same options and the previous example:: + + <Options> + <Option Name="InstallAllUsers" Value="no" /> + <Option Name="Include_launcher" Value="0" /> + <Option Name="Include_test" Value="no" /> + <Option Name="SimpleInstall" Value="yes" /> + <Option Name="SimpleInstallDescription">Just for me, no test suite</Option> + </Options> + +.. _install-layout-option: + +Installing Without Downloading +------------------------------ + +As some features of Python are not included in the initial installer download, +selecting those features may require an internet connection. To avoid this +need, all possible components may be downloaded on-demand to create a complete +*layout* that will no longer require an internet connection regardless of the +selected features. Note that this download may be bigger than required, but +where a large number of installations are going to be performed it is very +useful to have a locally cached copy. + +Execute the following command from Command Prompt to download all possible +required files. Remember to substitute ``python-3.5.0.exe`` for the actual +name of your installer, and to create layouts in their own directories to +avoid collisions between files with the same name. + +:: + + python-3.5.0.exe /layout [optional target directory] + +You may also specify the ``/quiet`` option to hide the progress display. + +Modifying an install +-------------------- + +Once Python has been installed, you can add or remove features through the +Programs and Features tool that is part of Windows. Select the Python entry and +choose "Uninstall/Change" to open the installer in maintenance mode. + +"Modify" allows you to add or remove features by modifying the checkboxes - +unchanged checkboxes will not install or remove anything. Some options cannot be +changed in this mode, such as the install directory; to modify these, you will +need to remove and then reinstall Python completely. + +"Repair" will verify all the files that should be installed using the current +settings and replace any that have been removed or modified. + +"Uninstall" will remove Python entirely, with the exception of the +:ref:`launcher`, which has its own entry in Programs and Features. + +Other Platforms +--------------- With ongoing development of Python, some platforms that used to be supported earlier are no longer supported (due to the lack of users or developers). Check :pep:`11` for details on all unsupported platforms. * `Windows CE <http://pythonce.sourceforge.net/>`_ is still supported. -* The `Cygwin <http://cygwin.com/>`_ installer offers to install the Python +* The `Cygwin <https://cygwin.com/>`_ installer offers to install the Python interpreter as well (cf. `Cygwin package source <ftp://ftp.uni-erlangen.de/pub/pc/gnuwin32/cygwin/mirrors/cygnus/ release/python>`_, `Maintainer releases <http://www.tishler.net/jason/software/python/>`_) -See `Python for Windows <https://www.python.org/download/windows/>`_ +See `Python for Windows <https://www.python.org/downloads/windows/>`_ for detailed information about platforms with pre-compiled installers. .. seealso:: - `Python on XP <http://www.richarddooling.com/index.php/2006/03/14/python-on-xp-7-minutes-to-hello-world/>`_ + `Python on XP <http://dooling.com/index.php/2006/03/14/python-on-xp-7-minutes-to-hello-world/>`_ "7 Minutes to "Hello World!"" by Richard Dooling, 2006 @@ -50,9 +266,9 @@ for detailed information about platforms with pre-compiled installers. by Mark Pilgrim, 2004, ISBN 1-59059-356-1 - `For Windows users <http://www.swaroopch.com/notes/python/#install_windows>`_ + `For Windows users <http://python.swaroopch.com/installation.html#installation-on-windows>`_ in "Installing Python" - in "`A Byte of Python <http://www.swaroopch.com/notes/python/>`_" + in "`A Byte of Python <http://python.swaroopch.com/>`_" by Swaroop C H, 2003 @@ -63,22 +279,34 @@ Besides the standard CPython distribution, there are modified packages including additional functionality. The following is a list of popular versions and their key features: -`ActivePython <http://www.activestate.com/activepython/>`_ +`ActivePython <https://www.activestate.com/activepython/>`_ Installer with multi-platform compatibility, documentation, PyWin32 -`Enthought Python Distribution <https://www.enthought.com/products/epd/>`_ - Popular modules (such as PyWin32) with their respective documentation, tool - suite for building extensible Python applications +`Anaconda <https://www.continuum.io/downloads/>`_ + Popular scientific modules (such as numpy, scipy and pandas) and the + ``conda`` package manager. + +`Canopy <https://www.enthought.com/products/canopy/>`_ + A "comprehensive Python analysis environment" with editors and other + development tools. + +`WinPython <https://winpython.github.io/>`_ + Windows-specific distribution with prebuilt scientific packages and + tools for building packages. -Notice that these packages are likely to install *older* versions of Python. +Note that these packages may not include the latest versions of Python or +other libraries, and are not maintained or supported by the core Python team. Configuring Python ================== -In order to run Python flawlessly, you might have to change certain environment -settings in Windows. +To run Python conveniently from a command prompt, you might consider changing +some default environment variables in Windows. While the installer provides an +option to configure the PATH and PATHEXT variables for you, this is only +reliable for a single, system-wide installation. If you regularly use multiple +versions of Python, consider using the :ref:`launcher`. .. _setting-envvars: @@ -86,163 +314,86 @@ settings in Windows. Excursus: Setting environment variables --------------------------------------- -Windows has a built-in dialog for changing environment variables (following -guide applies to XP classical view): Right-click the icon for your machine -(usually located on your Desktop and called "My Computer") and choose -:menuselection:`Properties` there. Then, open the :guilabel:`Advanced` tab -and click the :guilabel:`Environment Variables` button. +Windows allows environment variables to be configured permanently at both the +User level and the System level, or temporarily in a command prompt. -In short, your path is: +To temporarily set environment variables, open Command Prompt and use the +:command:`set` command:: - :menuselection:`My Computer - --> Properties - --> Advanced - --> Environment Variables` + C:\>set PATH=C:\Program Files\Python 3.5;%PATH% + C:\>set PYTHONPATH=%PYTHONPATH%;C:\My_python_lib + C:\>python +These changes will apply to any further commands executed in that console, and +will be inherited by any applications started from the console. + +Including the variable name within percent signs will expand to the existing +value, allowing you to add your new value at either the start or the end. +Modifying :envvar:`PATH` by adding the directory containing +:program:`python.exe` to the start is a common way to ensure the correct version +of Python is launched. + +To permanently modify the default environment variables, click Start and search +for 'edit environment variables', or open System properties, :guilabel:`Advanced +system settings` and click the :guilabel:`Environment Variables` button. In this dialog, you can add or modify User and System variables. To change System variables, you need non-restricted access to your machine (i.e. Administrator rights). -Another way of adding variables to your environment is using the :command:`set` -command:: - - set PYTHONPATH=%PYTHONPATH%;C:\My_python_lib - -To make this setting permanent, you could add the corresponding command line to -your :file:`autoexec.bat`. :program:`msconfig` is a graphical interface to this -file. - -Viewing environment variables can also be done more straight-forward: The -command prompt will expand strings wrapped into percent signs automatically:: +.. note:: - echo %PATH% + Windows will concatenate User variables *after* System variables, which may + cause unexpected results when modifying :envvar:`PATH`. -Consult :command:`set /?` for details on this behaviour. + The :envvar:`PYTHONPATH` variable is used by all versions of Python 2 and + Python 3, so you should not permanently configure this variable unless it + only includes code that is compatible with all of your installed Python + versions. .. seealso:: - http://support.microsoft.com/kb/100843 + https://support.microsoft.com/kb/100843 Environment variables in Windows NT - http://support.microsoft.com/kb/310519 + https://technet.microsoft.com/en-us/library/cc754250.aspx + The SET command, for temporarily modifying environment variables + + https://technet.microsoft.com/en-us/library/cc755104.aspx + The SETX command, for permanently modifying environment variables + + https://support.microsoft.com/kb/310519 How To Manage Environment Variables in Windows XP - http://www.chem.gla.ac.uk/~louis/software/faq/q1.html + https://www.chem.gla.ac.uk/~louis/software/faq/q1.html Setting Environment variables, Louis J. Farrugia - .. _windows-path-mod: Finding the Python executable ----------------------------- -.. versionchanged:: 3.3 +.. versionchanged:: 3.5 Besides using the automatically created start menu entry for the Python -interpreter, you might want to start Python in the command prompt. As of -Python 3.3, the installer has an option to set that up for you. - -At the "Customize Python 3.3" screen, an option called -"Add python.exe to search path" can be enabled to have the installer place -your installation into the :envvar:`%PATH%`. This allows you to type -:command:`python` to run the interpreter. Thus, you can also execute your +interpreter, you might want to start Python in the command prompt. The +installer for Python 3.5 and later has an option to set that up for you. + +On the first page of the installer, an option labelled "Add Python 3.5 to +PATH" can be selected to have the installer add the install location into the +:envvar:`PATH`. The location of the :file:`Scripts\\` folder is also added. +This allows you to type :command:`python` to run the interpreter, and +:command:`pip` or . Thus, you can also execute your scripts with command line options, see :ref:`using-on-cmdline` documentation. If you don't enable this option at install time, you can always re-run the -installer to choose it. - -The alternative is manually modifying the :envvar:`%PATH%` using the -directions in :ref:`setting-envvars`. You need to set your :envvar:`%PATH%` -environment variable to include the directory of your Python distribution, -delimited by a semicolon from other entries. An example variable could look -like this (assuming the first two entries are Windows' default):: - - C:\WINDOWS\system32;C:\WINDOWS;C:\Python33 - - -Finding modules ---------------- - -Python usually stores its library (and thereby your site-packages folder) in the -installation directory. So, if you had installed Python to -:file:`C:\\Python\\`, the default library would reside in -:file:`C:\\Python\\Lib\\` and third-party modules should be stored in -:file:`C:\\Python\\Lib\\site-packages\\`. - -This is how :data:`sys.path` is populated on Windows: - -* An empty entry is added at the start, which corresponds to the current - directory. - -* If the environment variable :envvar:`PYTHONPATH` exists, as described in - :ref:`using-on-envvars`, its entries are added next. Note that on Windows, - paths in this variable must be separated by semicolons, to distinguish them - from the colon used in drive identifiers (``C:\`` etc.). - -* Additional "application paths" can be added in the registry as subkeys of - :samp:`\\SOFTWARE\\Python\\PythonCore\\{version}\\PythonPath` under both the - ``HKEY_CURRENT_USER`` and ``HKEY_LOCAL_MACHINE`` hives. Subkeys which have - semicolon-delimited path strings as their default value will cause each path - to be added to :data:`sys.path`. (Note that all known installers only use - HKLM, so HKCU is typically empty.) - -* If the environment variable :envvar:`PYTHONHOME` is set, it is assumed as - "Python Home". Otherwise, the path of the main Python executable is used to - locate a "landmark file" (``Lib\os.py``) to deduce the "Python Home". If a - Python home is found, the relevant sub-directories added to :data:`sys.path` - (``Lib``, ``plat-win``, etc) are based on that folder. Otherwise, the core - Python path is constructed from the PythonPath stored in the registry. - -* If the Python Home cannot be located, no :envvar:`PYTHONPATH` is specified in - the environment, and no registry entries can be found, a default path with - relative entries is used (e.g. ``.\Lib;.\plat-win``, etc). - -The end result of all this is: - -* When running :file:`python.exe`, or any other .exe in the main Python - directory (either an installed version, or directly from the PCbuild - directory), the core path is deduced, and the core paths in the registry are - ignored. Other "application paths" in the registry are always read. - -* When Python is hosted in another .exe (different directory, embedded via COM, - etc), the "Python Home" will not be deduced, so the core path from the - registry is used. Other "application paths" in the registry are always read. - -* If Python can't find its home and there is no registry (eg, frozen .exe, some - very strange installation setup) you get a path with some default, but - relative, paths. - - -Executing scripts ------------------ - -As of Python 3.3, Python includes a launcher which facilitates running Python -scripts. See :ref:`launcher` for more information. - -Executing scripts without the Python launcher ---------------------------------------------- - -Without the Python launcher installed, Python scripts (files with the extension -``.py``) will be executed by :program:`python.exe` by default. This executable -opens a terminal, which stays open even if the program uses a GUI. If you do -not want this to happen, use the extension ``.pyw`` which will cause the script -to be executed by :program:`pythonw.exe` by default (both executables are -located in the top-level of your Python installation directory). This -suppresses the terminal window on startup. - -You can also make all ``.py`` scripts execute with :program:`pythonw.exe`, -setting this through the usual facilities, for example (might require -administrative rights): - -#. Launch a command prompt. -#. Associate the correct file group with ``.py`` scripts:: - - assoc .py=Python.File - -#. Redirect all Python files to the new executable:: - - ftype Python.File=C:\Path\to\pythonw.exe "%1" %* +installer, select Modify, and enable it. Alternatively, you can manually +modify the :envvar:`PATH` using the directions in :ref:`setting-envvars`. You +need to set your :envvar:`PATH` environment variable to include the directory +of your Python installation, delimited by a semicolon from other entries. An +example variable could look like this (assuming the first two entries already +existed):: + C:\WINDOWS\system32;C:\WINDOWS;C:\Program Files\Python 3.5 .. _launcher: @@ -251,21 +402,26 @@ Python Launcher for Windows .. versionadded:: 3.3 -The Python launcher for Windows is a utility which aids in the location and -execution of different Python versions. It allows scripts (or the +The Python launcher for Windows is a utility which aids in locating and +executing of different Python versions. It allows scripts (or the command-line) to indicate a preference for a specific Python version, and will locate and execute that version. +Unlike the :envvar:`PATH` variable, the launcher will correctly select the most +appropriate version of Python. It will prefer per-user installations over +system-wide ones, and orders by language version rather than using the most +recently installed version. + Getting started --------------- From the command-line ^^^^^^^^^^^^^^^^^^^^^ -You should ensure the launcher is on your PATH - depending on how it was -installed it may already be there, but check just in case it is not. - -From a command-prompt, execute the following command: +System-wide installations of Python 3.3 and later will put the launcher on your +:envvar:`PATH`. The launcher is compatible with all available versions of +Python, so it does not matter which version is installed. To check that the +launcher is available, execute the following command in Command Prompt: :: @@ -291,6 +447,28 @@ If you have a Python 3.x installed, try the command: You should find the latest version of Python 3.x starts. +If you see the following error, you do not have the launcher installed: + +:: + + 'py' is not recognized as an internal or external command, + operable program or batch file. + +Per-user installations of Python do not add the launcher to :envvar:`PATH` +unless the option was selected on installation. + +Virtual environments +^^^^^^^^^^^^^^^^^^^^ + +.. versionadded:: 3.5 + +If the launcher is run with no explicit Python version specification, and a +virtual environment (created with the standard library :mod:`venv` module or +the external ``virtualenv`` tool) active, the launcher will run the virtual +environment's interpreter rather than the global one. To run the global +interpreter, either deactivate the virtual environment, or explicitly specify +the global Python version. + From a script ^^^^^^^^^^^^^ @@ -326,7 +504,7 @@ From file associations ^^^^^^^^^^^^^^^^^^^^^^ The launcher should have been associated with Python files (i.e. ``.py``, -``.pyw``, ``.pyc``, ``.pyo`` files) when it was installed. This means that +``.pyw``, ``.pyc`` files) when it was installed. This means that when you double-click on one of these files from Windows explorer the launcher will be used, and therefore you can use the same facilities described above to have the script specify the version which should be used. @@ -365,6 +543,16 @@ be used by the launcher without modification. If you are writing a new script on Windows which you hope will be useful on Unix, you should use one of the shebang lines starting with ``/usr``. +Any of the above virtual commands can be suffixed with an explicit version +(either just the major version, or the major and minor version) - for example +``/usr/bin/python2.7`` - which will cause that specific version to be located +and used. + +The ``/usr/bin/env`` form of shebang line has one further special property. +Before looking for installed Python interpreters, this form will search the +executable :envvar:`PATH` for a Python executable. This corresponds to the +behaviour of the Unix ``env`` program, which performs a :envvar:`PATH` search. + Arguments in shebang lines -------------------------- @@ -383,17 +571,16 @@ Customization Customization via INI files ^^^^^^^^^^^^^^^^^^^^^^^^^^^ - Two .ini files will be searched by the launcher - ``py.ini`` in the - current user's "application data" directory (i.e. the directory returned - by calling the Windows function SHGetFolderPath with CSIDL_LOCAL_APPDATA) - and ``py.ini`` in the same directory as the launcher. The same .ini - files are used for both the 'console' version of the launcher (i.e. - py.exe) and for the 'windows' version (i.e. pyw.exe) +Two .ini files will be searched by the launcher - ``py.ini`` in the current +user's "application data" directory (i.e. the directory returned by calling the +Windows function SHGetFolderPath with CSIDL_LOCAL_APPDATA) and ``py.ini`` in the +same directory as the launcher. The same .ini files are used for both the +'console' version of the launcher (i.e. py.exe) and for the 'windows' version +(i.e. pyw.exe) - Customization specified in the "application directory" will have - precedence over the one next to the executable, so a user, who may not - have write access to the .ini file next to the launcher, can override - commands in that global .ini file) +Customization specified in the "application directory" will have precedence over +the one next to the executable, so a user, who may not have write access to the +.ini file next to the launcher, can override commands in that global .ini file) Customizing default Python versions ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -488,6 +675,99 @@ particular version was chosen and the exact command-line used to execute the target Python. + +.. finding_modules: + +Finding modules +=============== + +Python usually stores its library (and thereby your site-packages folder) in the +installation directory. So, if you had installed Python to +:file:`C:\\Python\\`, the default library would reside in +:file:`C:\\Python\\Lib\\` and third-party modules should be stored in +:file:`C:\\Python\\Lib\\site-packages\\`. + +This is how :data:`sys.path` is populated on Windows: + +* An empty entry is added at the start, which corresponds to the current + directory. + +* If the environment variable :envvar:`PYTHONPATH` exists, as described in + :ref:`using-on-envvars`, its entries are added next. Note that on Windows, + paths in this variable must be separated by semicolons, to distinguish them + from the colon used in drive identifiers (``C:\`` etc.). + +* Additional "application paths" can be added in the registry as subkeys of + :samp:`\\SOFTWARE\\Python\\PythonCore\\{version}\\PythonPath` under both the + ``HKEY_CURRENT_USER`` and ``HKEY_LOCAL_MACHINE`` hives. Subkeys which have + semicolon-delimited path strings as their default value will cause each path + to be added to :data:`sys.path`. (Note that all known installers only use + HKLM, so HKCU is typically empty.) + +* If the environment variable :envvar:`PYTHONHOME` is set, it is assumed as + "Python Home". Otherwise, the path of the main Python executable is used to + locate a "landmark file" (``Lib\os.py``) to deduce the "Python Home". If a + Python home is found, the relevant sub-directories added to :data:`sys.path` + (``Lib``, ``plat-win``, etc) are based on that folder. Otherwise, the core + Python path is constructed from the PythonPath stored in the registry. + +* If the Python Home cannot be located, no :envvar:`PYTHONPATH` is specified in + the environment, and no registry entries can be found, a default path with + relative entries is used (e.g. ``.\Lib;.\plat-win``, etc). + +If a ``pyvenv.cfg`` file is found alongside the main executable or in the +directory one level above the executable, the following variations apply: + +* If ``home`` is an absolute path and :envvar:`PYTHONHOME` is not set, this + path is used instead of the path to the main executable when deducing the + home location. + +* If ``applocal`` is set to true, the ``home`` property or the main executable + is always used as the home path, and all environment variables or registry + values affecting the path are ignored. The landmark file is not checked. + +The end result of all this is: + +* When running :file:`python.exe`, or any other .exe in the main Python + directory (either an installed version, or directly from the PCbuild + directory), the core path is deduced, and the core paths in the registry are + ignored. Other "application paths" in the registry are always read. + +* When Python is hosted in another .exe (different directory, embedded via COM, + etc), the "Python Home" will not be deduced, so the core path from the + registry is used. Other "application paths" in the registry are always read. + +* If Python can't find its home and there are no registry value (frozen .exe, + some very strange installation setup) you get a path with some default, but + relative, paths. + +For those who want to bundle Python into their application or distribution, the +following advice will prevent conflicts with other installations: + +* Include a ``pyvenv.cfg`` file alongside your executable containing + ``applocal = true``. This will ensure that your own directory will be used to + resolve paths even if you have included the standard library in a ZIP file. + It will also ignore user site-packages and other paths listed in the + registry. + +* If you are loading :file:`python3.dll` or :file:`python35.dll` in your own + executable, explicitly call :c:func:`Py_SetPath` or (at least) + :c:func:`Py_SetProgramName` before :c:func:`Py_Initialize`. + +* Clear and/or overwrite :envvar:`PYTHONPATH` and set :envvar:`PYTHONHOME` + before launching :file:`python.exe` from your application. + +* If you cannot use the previous suggestions (for example, you are a + distribution that allows people to run :file:`python.exe` directly), ensure + that the landmark file (:file:`Lib\\os.py`) exists in your install directory. + (Note that it will not be detected inside a ZIP file.) + +These will ensure that the files in a system-wide installation will not take +precedence over the copy of the standard library bundled with your application. +Otherwise, your users may experience problems using your application. Note that +the first suggestion is the best, as the other may still be susceptible to +non-standard paths in the registry and user site-packages. + Additional modules ================== @@ -498,22 +778,21 @@ and external, and snippets exist to use these features. The Windows-specific standard modules are documented in :ref:`mswin-specific-services`. - PyWin32 ------- -The `PyWin32 <http://python.net/crew/mhammond/win32/>`_ module by Mark Hammond +The `PyWin32 <https://pypi.python.org/pypi/pywin32>`_ module by Mark Hammond is a collection of modules for advanced Windows-specific support. This includes utilities for: -* `Component Object Model <http://www.microsoft.com/com/>`_ (COM) +* `Component Object Model <https://www.microsoft.com/com/>`_ (COM) * Win32 API calls * Registry * Event log -* `Microsoft Foundation Classes <http://msdn.microsoft.com/en-us/library/fe1cf721%28VS.80%29.aspx>`_ (MFC) +* `Microsoft Foundation Classes <https://msdn.microsoft.com/en-us/library/fe1cf721%28VS.80%29.aspx>`_ (MFC) user interfaces -`PythonWin <http://web.archive.org/web/20060524042422/ +`PythonWin <https://web.archive.org/web/20060524042422/ https://www.python.org/windows/pythonwin/>`_ is a sample MFC application shipped with PyWin32. It is an embeddable IDE with a built-in debugger. @@ -552,25 +831,13 @@ Compiling Python on Windows =========================== If you want to compile CPython yourself, first thing you should do is get the -`source <https://www.python.org/download/source/>`_. You can download either the +`source <https://www.python.org/downloads/source/>`_. You can download either the latest release's source or just grab a fresh `checkout <https://docs.python.org/devguide/setup.html#getting-the-source-code>`_. The source tree contains a build solution and project files for Microsoft -Visual C++, which is the compiler used to build the official Python releases. -View the :file:`readme.txt` in their respective directories: - -+--------------------+--------------+-----------------------+ -| Directory | MSVC version | Visual Studio version | -+====================+==============+=======================+ -| :file:`PC/VS9.0/` | 9.0 | 2008 | -+--------------------+--------------+-----------------------+ -| :file:`PCbuild/` | 10.0 | 2010 | -+--------------------+--------------+-----------------------+ - -Note that any build directories within the :file:`PC` directory are not -necessarily fully supported. The :file:`PCbuild` directory contains the files -for the compiler used to build the official release. +Visual Studio 2015, which is the compiler used to build the official Python +releases. These files are in the :file:`PCbuild` directory. Check :file:`PCbuild/readme.txt` for general information on the build process. @@ -588,6 +855,83 @@ For extension modules, consult :ref:`building-on-windows`. by Trent Apted et al, 2007 +Embedded Distribution +===================== + +.. versionadded:: 3.5 + +The embedded distribution is a ZIP file containing a minimal Python environment. +It is intended for acting as part of another application, rather than being +directly accessed by end-users. + +When extracted, the embedded distribution is (almost) fully isolated from the +user's system, including environment variables, system registry settings, and +installed packages. The standard library is included as pre-compiled and +optimized ``.pyc`` files in a ZIP, and ``python3.dll``, ``python35.dll``, +``python.exe`` and ``pythonw.exe`` are all provided. Tcl/tk (including all +dependants, such as Idle), pip and the Python documentation are not included. + +.. note:: + + The embedded distribution does not include the `Microsoft C Runtime + <https://www.microsoft.com/en-us/download/details.aspx?id=48145>`_ and it is + the responsibility of the application installer to provide this. The + runtime may have already been installed on a user's system previously or + automatically via Windows Update, and can be detected by finding + ``ucrtbase.dll`` in the system directory. + +Third-party packages should be installed by the application installer alongside +the embedded distribution. Using pip to manage dependencies as for a regular +Python installation is not supported with this distribution, though with some +care it may be possible to include and use pip for automatic updates. In +general, third-party packages should be treated as part of the application +("vendoring") so that the developer can ensure compatibility with newer +versions before providing updates to users. + +The two recommended use cases for this distribution are described below. + +Python Application +------------------ + +An application written in Python does not necessarily require users to be aware +of that fact. The embedded distribution may be used in this case to include a +private version of Python in an install package. Depending on how transparent it +should be (or conversely, how professional it should appear), there are two +options. + +Using a specialized executable as a launcher requires some coding, but provides +the most transparent experience for users. With a customized launcher, there are +no obvious indications that the program is running on Python: icons can be +customized, company and version information can be specified, and file +associations behave properly. In most cases, a custom launcher should simply be +able to call ``Py_Main`` with a hard-coded command line. + +The simpler approach is to provide a batch file or generated shortcut that +directly calls the ``python.exe`` or ``pythonw.exe`` with the required +command-line arguments. In this case, the application will appear to be Python +and not its actual name, and users may have trouble distinguishing it from other +running Python processes or file associations. + +With the latter approach, packages should be installed as directories alongside +the Python executable to ensure they are available on the path. With the +specialized launcher, packages can be located in other locations as there is an +opportunity to specify the search path before launching the application. + +Embedding Python +---------------- + +Applications written in native code often require some form of scripting +language, and the embedded Python distribution can be used for this purpose. In +general, the majority of the application is in native code, and some part will +either invoke ``python.exe`` or directly use ``python3.dll``. For either case, +extracting the embedded distribution to a subdirectory of the application +installation is sufficient to provide a loadable Python interpreter. + +As with the application use, packages can be installed to any location as there +is an opportunity to specify search paths before initializing the interpreter. +Otherwise, there is no fundamental differences between using the embedded +distribution and a regular installation. + Other resources =============== @@ -603,5 +947,3 @@ Other resources :pep:`397` - Python launcher for Windows The proposal for the launcher to be included in the Python distribution. - - diff --git a/Doc/whatsnew/2.0.rst b/Doc/whatsnew/2.0.rst index 10bb29ee8a..4d49af1475 100644 --- a/Doc/whatsnew/2.0.rst +++ b/Doc/whatsnew/2.0.rst @@ -61,7 +61,7 @@ how Python is developed: in May 2000 the Python developers began using the tools made available by SourceForge for storing source code, tracking bug reports, and managing the queue of patch submissions. To report bugs or submit patches for Python 2.0, use the bug tracking and patch manager tools available from -Python's project page, located at http://sourceforge.net/projects/python/. +Python's project page, located at https://sourceforge.net/projects/python/. The most important of the services now hosted at SourceForge is the Python CVS tree, the version-controlled repository containing the source code for Python. @@ -130,7 +130,7 @@ Guidelines": Read the rest of PEP 1 for the details of the PEP editorial process, style, and format. PEPs are kept in the Python CVS tree on SourceForge, though they're not part of the Python 2.0 distribution, and are also available in HTML form from -https://www.python.org/peps/. As of September 2000, there are 25 PEPS, ranging +https://www.python.org/dev/peps/. As of September 2000, there are 25 PEPS, ranging from PEP 201, "Lockstep Iteration", to PEP 225, "Elementwise/Objectwise Operators". @@ -337,7 +337,7 @@ comprehension below is a syntax error, while the second one is correct:: [ (x,y) for x in seq1 for y in seq2] The idea of list comprehensions originally comes from the functional programming -language Haskell (http://www.haskell.org). Greg Ewing argued most effectively +language Haskell (https://www.haskell.org). Greg Ewing argued most effectively for adding them to Python and wrote the initial list comprehension patch, which was then discussed for a seemingly endless time on the python-dev mailing list and kept up-to-date by Skip Montanaro. @@ -506,7 +506,7 @@ arguments and/or a dictionary of keyword arguments. In Python 1.5 and earlier, you'd use the :func:`apply` built-in function: ``apply(f, args, kw)`` calls the function :func:`f` with the argument tuple *args* and the keyword arguments in the dictionary *kw*. :func:`apply` is the same in 2.0, but thanks to a patch -from Greg Ewing, ``f(*args, **kw)`` as a shorter and clearer way to achieve the +from Greg Ewing, ``f(*args, **kw)`` is a shorter and clearer way to achieve the same effect. This syntax is symmetrical with the syntax for defining functions:: diff --git a/Doc/whatsnew/2.1.rst b/Doc/whatsnew/2.1.rst index 6de5bf57b3..06366b8ba0 100644 --- a/Doc/whatsnew/2.1.rst +++ b/Doc/whatsnew/2.1.rst @@ -442,8 +442,8 @@ Python syntax:: f.grammar = "A ::= B (C D)*" The dictionary containing attributes can be accessed as the function's -:attr:`__dict__`. Unlike the :attr:`__dict__` attribute of class instances, in -functions you can actually assign a new dictionary to :attr:`__dict__`, though +:attr:`~object.__dict__`. Unlike the :attr:`~object.__dict__` attribute of class instances, in +functions you can actually assign a new dictionary to :attr:`~object.__dict__`, though the new value is restricted to a regular Python dictionary; you *can't* be tricky and set it to a :class:`UserDict` instance, or any other random object that behaves like a mapping. @@ -562,7 +562,7 @@ You can start creating packages containing :file:`PKG-INFO` even if you're not using Python 2.1, since a new release of the Distutils will be made for users of earlier Python versions. Version 1.0.2 of the Distutils includes the changes described in PEP 241, as well as various bugfixes and enhancements. It will be -available from the Distutils SIG at https://www.python.org/sigs/distutils-sig/. +available from the Distutils SIG at https://www.python.org/community/sigs/current/distutils-sig/. .. seealso:: @@ -731,7 +731,7 @@ of the more notable changes are: ... For a fuller discussion of the line I/O changes, see the python-dev summary for - January 1-15, 2001 at https://www.python.org/dev/summary/2001-01-1/. + January 1-15, 2001 at https://mail.python.org/pipermail/python-dev/2001-January/. * A new method, :meth:`popitem`, was added to dictionaries to enable destructively iterating through the contents of a dictionary; this can be faster diff --git a/Doc/whatsnew/2.2.rst b/Doc/whatsnew/2.2.rst index f3c4a91346..4e8d7fa18a 100644 --- a/Doc/whatsnew/2.2.rst +++ b/Doc/whatsnew/2.2.rst @@ -24,8 +24,8 @@ up irregularities and dark corners of the language design. This article doesn't attempt to provide a complete specification of the new features, but instead provides a convenient overview. For full details, you should refer to the documentation for Python 2.2, such as the `Python Library -Reference <https://www.python.org/doc/2.2/lib/lib.html>`_ and the `Python -Reference Manual <https://www.python.org/doc/2.2/ref/ref.html>`_. If you want to +Reference <https://docs.python.org/2.2/lib/lib.html>`_ and the `Python +Reference Manual <https://docs.python.org/2.2/ref/ref.html>`_. If you want to understand the complete implementation and design rationale for a change, refer to the PEP for a particular new feature. @@ -157,7 +157,7 @@ attributes and methods were supported by an object. There were some informal conventions, such as defining :attr:`__members__` and :attr:`__methods__` attributes that were lists of names, but often the author of an extension type or a class wouldn't bother to define them. You could fall back on inspecting -the :attr:`__dict__` of an object, but when class inheritance or an arbitrary +the :attr:`~object.__dict__` of an object, but when class inheritance or an arbitrary :meth:`__getattr__` hook were in use this could still be inaccurate. The one big idea underlying the new class model is that an API for describing @@ -169,7 +169,7 @@ possible, as well as more exotic constructs. Attribute descriptors are objects that live inside class objects, and have a few attributes of their own: -* :attr:`__name__` is the attribute's name. +* :attr:`~definition.__name__` is the attribute's name. * :attr:`__doc__` is the attribute's docstring. @@ -329,7 +329,7 @@ However, Python 2.2's support for :dfn:`properties` will often be a simpler way to trap attribute references. Writing a :meth:`__getattr__` method is complicated because to avoid recursion you can't use regular attribute accesses inside them, and instead have to mess around with the contents of -:attr:`__dict__`. :meth:`__getattr__` methods also end up being called by Python +:attr:`~object.__dict__`. :meth:`__getattr__` methods also end up being called by Python when it checks for other methods such as :meth:`__repr__` or :meth:`__coerce__`, and so have to be written with this in mind. Finally, calling a function on every attribute access results in a sizable performance loss. @@ -357,15 +357,15 @@ write:: That is certainly clearer and easier to write than a pair of :meth:`__getattr__`/:meth:`__setattr__` methods that check for the :attr:`size` attribute and handle it specially while retrieving all other attributes from the -instance's :attr:`__dict__`. Accesses to :attr:`size` are also the only ones +instance's :attr:`~object.__dict__`. Accesses to :attr:`size` are also the only ones which have to perform the work of calling a function, so references to other attributes run at their usual speed. Finally, it's possible to constrain the list of attributes that can be -referenced on an object using the new :attr:`__slots__` class attribute. Python +referenced on an object using the new :attr:`~object.__slots__` class attribute. Python objects are usually very dynamic; at any time it's possible to define a new attribute on an instance by just doing ``obj.new_attr=1``. A new-style class -can define a class attribute named :attr:`__slots__` to limit the legal +can define a class attribute named :attr:`~object.__slots__` to limit the legal attributes to a particular set of names. An example will make this clear:: >>> class C(object): @@ -383,7 +383,7 @@ attributes to a particular set of names. An example will make this clear:: AttributeError: 'C' object has no attribute 'newattr' Note how you get an :exc:`AttributeError` on the attempt to assign to an -attribute not listed in :attr:`__slots__`. +attribute not listed in :attr:`~object.__slots__`. .. _sect-rellinks: @@ -395,7 +395,7 @@ This section has just been a quick overview of the new features, giving enough of an explanation to start you programming, but many details have been simplified or ignored. Where should you go to get a more complete picture? -https://www.python.org/2.2/descrintro.html is a lengthy tutorial introduction to +https://docs.python.org/dev/howto/descriptor.html is a lengthy tutorial introduction to the descriptor features, written by Guido van Rossum. If my description has whetted your appetite, go read this tutorial next, because it goes into much more detail about the new features while still remaining quite easy to read. @@ -632,10 +632,10 @@ queen threatens another) and the Knight's Tour (a route that takes a knight to every square of an $NxN$ chessboard without visiting any square twice). The idea of generators comes from other programming languages, especially Icon -(http://www.cs.arizona.edu/icon/), where the idea of generators is central. In +(https://www.cs.arizona.edu/icon/), where the idea of generators is central. In Icon, every expression and function call behaves like a generator. One example from "An Overview of the Icon Programming Language" at -http://www.cs.arizona.edu/icon/docs/ipd266.htm gives an idea of what this looks +https://www.cs.arizona.edu/icon/docs/ipd266.htm gives an idea of what this looks like:: sentence := "Store it in the neighboring harbor" @@ -720,7 +720,7 @@ possible types of the operands. (The controversy is over whether this is *really* a design flaw, and whether it's worth breaking existing code to fix this. It's caused endless discussions -on python-dev, and in July 2001 erupted into an storm of acidly sarcastic +on python-dev, and in July 2001 erupted into a storm of acidly sarcastic postings on :newsgroup:`comp.lang.python`. I won't argue for either side here and will stick to describing what's implemented in 2.2. Read :pep:`238` for a summary of arguments and counter-arguments.) @@ -758,7 +758,7 @@ Here are the changes 2.2 introduces: operators. * Python 2.2 supports some command-line arguments for testing whether code will - works with the changed division semantics. Running python with :option:`-Q + work with the changed division semantics. Running python with :option:`-Q warn` will cause a warning to be issued whenever division is applied to two integers. You can use this to find code that's affected by the change and fix it. By default, Python 2.2 will simply perform classic division without a diff --git a/Doc/whatsnew/2.3.rst b/Doc/whatsnew/2.3.rst index 9d99074f22..ebdae69519 100644 --- a/Doc/whatsnew/2.3.rst +++ b/Doc/whatsnew/2.3.rst @@ -218,10 +218,10 @@ queen threatens another) and the Knight's Tour (a route that takes a knight to every square of an $NxN$ chessboard without visiting any square twice). The idea of generators comes from other programming languages, especially Icon -(http://www.cs.arizona.edu/icon/), where the idea of generators is central. In +(https://www.cs.arizona.edu/icon/), where the idea of generators is central. In Icon, every expression and function call behaves like a generator. One example from "An Overview of the Icon Programming Language" at -http://www.cs.arizona.edu/icon/docs/ipd266.htm gives an idea of what this looks +https://www.cs.arizona.edu/icon/docs/ipd266.htm gives an idea of what this looks like:: sentence := "Store it in the neighboring harbor" @@ -291,7 +291,9 @@ PEP 273: Importing Modules from ZIP Archives The new :mod:`zipimport` module adds support for importing modules from a ZIP- format archive. You don't need to import the module explicitly; it will be automatically imported if a ZIP archive's filename is added to ``sys.path``. -For example:: +For example: + +.. code-block:: shell-session amk@nyman:~/src/python$ unzip -l /tmp/example.zip Archive: /tmp/example.zip @@ -1057,7 +1059,7 @@ Here are all of the changes that Python 2.3 makes to the core Python language. * A new warning, :exc:`PendingDeprecationWarning` was added to indicate features which are in the process of being deprecated. The warning will *not* be printed by default. To check for use of features that will be deprecated in the future, - supply :option:`-Walways::PendingDeprecationWarning::` on the command line or + supply :option:`-Walways::PendingDeprecationWarning:: <-W>` on the command line or use :func:`warnings.filterwarnings`. * The process of deprecating string-based exceptions, as in ``raise "Error @@ -1080,9 +1082,9 @@ Here are all of the changes that Python 2.3 makes to the core Python language. hierarchy. Classic classes are unaffected by this change. Python 2.2 originally used a topological sort of a class's ancestors, but 2.3 now uses the C3 algorithm as described in the paper `"A Monotonic Superclass Linearization - for Dylan" <http://www.webcom.com/haahr/dylan/linearization-oopsla96.html>`_. To + for Dylan" <http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.19.3910>`_. To understand the motivation for this change, read Michele Simionato's article - `"Python 2.3 Method Resolution Order" <https://www.python.org/2.3/mro.html>`_, or + `"Python 2.3 Method Resolution Order" <http://www.phyast.pitt.edu/~micheles/mro.html>`_, or read the thread on python-dev starting with the message at https://mail.python.org/pipermail/python-dev/2002-October/029035.html. Samuele Pedroni first pointed out the problem and also implemented the fix by coding the @@ -1111,10 +1113,10 @@ Here are all of the changes that Python 2.3 makes to the core Python language. <type '_socket.socket'> * One of the noted incompatibilities between old- and new-style classes has been - removed: you can now assign to the :attr:`__name__` and :attr:`__bases__` + removed: you can now assign to the :attr:`~definition.__name__` and :attr:`~class.__bases__` attributes of new-style classes. There are some restrictions on what can be - assigned to :attr:`__bases__` along the lines of those relating to assigning to - an instance's :attr:`__class__` attribute. + assigned to :attr:`~class.__bases__` along the lines of those relating to assigning to + an instance's :attr:`~instance.__class__` attribute. .. ====================================================================== @@ -1306,7 +1308,7 @@ complete list of changes, or look through the CVS logs for all the details. partially sorted order such that, for every index *k*, ``heap[k] <= heap[2*k+1]`` and ``heap[k] <= heap[2*k+2]``. This makes it quick to remove the smallest item, and inserting a new item while maintaining the heap property is - O(lg n). (See http://www.nist.gov/dads/HTML/priorityque.html for more + O(lg n). (See https://xlinux.nist.gov/dads//HTML/priorityque.html for more information about the priority queue data structure.) The :mod:`heapq` module provides :func:`heappush` and :func:`heappop` functions @@ -1734,7 +1736,7 @@ The optparse Module The :mod:`getopt` module provides simple parsing of command-line arguments. The new :mod:`optparse` module (originally named Optik) provides more elaborate command-line parsing that follows the Unix conventions, automatically creates -the output for :option:`--help`, and can perform different actions for different +the output for :option:`!--help`, and can perform different actions for different options. You start by creating an instance of :class:`OptionParser` and telling it what @@ -1761,7 +1763,9 @@ This returns an object containing all of the option values, and a list of strings containing the remaining arguments. Invoking the script with the various arguments now works as you'd expect it to. -Note that the length argument is automatically converted to an integer. :: +Note that the length argument is automatically converted to an integer. + +.. code-block:: shell-session $ ./python opt.py -i data arg1 <Values at 0x400cad4c: {'input': 'data', 'length': None}> @@ -1771,7 +1775,9 @@ Note that the length argument is automatically converted to an integer. :: [] $ -The help message is automatically generated for you:: +The help message is automatically generated for you: + +.. code-block:: shell-session $ ./python opt.py --help usage: opt.py [options] @@ -1858,7 +1864,7 @@ and bundle it with the source of your extension. .. seealso:: - https://svn.python.org/view/python/trunk/Objects/obmalloc.c + https://hg.python.org/cpython/file/default/Objects/obmalloc.c For the full details of the pymalloc implementation, see the comments at the top of the file :file:`Objects/obmalloc.c` in the Python source code. The above link points to the file within the python.org SVN browser. @@ -1920,7 +1926,7 @@ Changes to Python's build process and to the C API include: * If you dynamically allocate type objects in your extension, you should be aware of a change in the rules relating to the :attr:`__module__` and - :attr:`__name__` attributes. In summary, you will want to ensure the type's + :attr:`~definition.__name__` attributes. In summary, you will want to ensure the type's dictionary contains a ``'__module__'`` key; making the module name the part of the type name leading up to the final period will no longer have the desired effect. For more detail, read the API reference documentation or the source. @@ -1949,7 +1955,7 @@ The RPM spec files, found in the :file:`Misc/RPM/` directory in the Python source distribution, were updated for 2.3. (Contributed by Sean Reifschneider.) Other new platforms now supported by Python include AtheOS -(http://www.atheos.cx/), GNU/Hurd, and OpenVMS. +(http://atheos.cx/), GNU/Hurd, and OpenVMS. .. ====================================================================== @@ -1973,7 +1979,7 @@ Some of the more notable changes are: the Python program as part of its execution. * The :file:`regrtest.py` script now provides a way to allow "all resources - except *foo*." A resource name passed to the :option:`-u` option can now be + except *foo*." A resource name passed to the :option:`!-u` option can now be prefixed with a hyphen (``'-'``) to mean "remove this resource." For example, the option '``-uall,-bsddb``' could be used to enable the use of all resources except ``bsddb``. @@ -2078,4 +2084,3 @@ Michael Hudson, Chris Lambert, Detlef Lannert, Martin von Löwis, Andrew MacIntyre, Lalo Martins, Chad Netzer, Gustavo Niemeyer, Neal Norwitz, Hans Nowak, Chris Reedy, Francesco Ricciardi, Vinay Sajip, Neil Schemenauer, Roman Suzi, Jason Tishler, Just van Rossum. - diff --git a/Doc/whatsnew/2.4.rst b/Doc/whatsnew/2.4.rst index 569e5e925f..5fb52fee9c 100644 --- a/Doc/whatsnew/2.4.rst +++ b/Doc/whatsnew/2.4.rst @@ -337,7 +337,7 @@ returned. wrote patches implementing function decorators, but the one that was actually checked in was patch #979728, written by Mark Russell. - https://www.python.org/moin/PythonDecoratorLibrary + https://wiki.python.org/moin/PythonDecoratorLibrary This Wiki page contains several examples of decorators. .. ====================================================================== @@ -687,7 +687,7 @@ includes a quick-start tutorial and a reference. The article uses Fortran code to illustrate many of the problems that floating- point inaccuracy can cause. - http://www2.hursley.ibm.com/decimal/ + http://speleotrove.com/decimal/ A description of a decimal-based representation. This representation is being proposed as a standard, and underlies the new Python decimal type. Much of this material was written by Mike Cowlishaw, designer of the Rexx language. @@ -756,7 +756,7 @@ API that perform ASCII-only conversions, ignoring the locale setting: :c:type:`double` to an ASCII string. The code for these functions came from the GLib library -(http://library.gnome.org/devel/glib/stable/), whose developers kindly +(https://developer.gnome.org/glib/stable/), whose developers kindly relicensed the relevant functions and donated them to the Python Software Foundation. The :mod:`locale` module can now change the numeric locale, letting extensions such as GTK+ produce the correct results. @@ -1425,7 +1425,9 @@ specifying the :const:`doctest.REPORT_UDIFF` (unified diffs), print word Running the above function's tests with :const:`doctest.REPORT_UDIFF` specified, -you get the following output:: +you get the following output: + +.. code-block:: none ********************************************************************** File "t.py", line 15, in g @@ -1481,10 +1483,10 @@ Some of the changes to Python's build process and to the C API are: * Python can now be built with additional profiling for the interpreter itself, intended as an aid to people developing the Python core. Providing - :option:`----enable-profiling` to the :program:`configure` script will let you + :option:`--enable-profiling` to the :program:`configure` script will let you profile the interpreter with :program:`gprof`, and providing the - :option:`----with-tsc` switch enables profiling using the Pentium's Time-Stamp- - Counter register. Note that the :option:`----with-tsc` switch is slightly + :option:`--with-tsc` switch enables profiling using the Pentium's Time-Stamp- + Counter register. Note that the :option:`--with-tsc` switch is slightly misnamed, because the profiling feature also works on the PowerPC platform, though that processor architecture doesn't call that register "the TSC register". (Contributed by Jeremy Hylton.) diff --git a/Doc/whatsnew/2.5.rst b/Doc/whatsnew/2.5.rst index cb92e08d6f..093189e8ee 100644 --- a/Doc/whatsnew/2.5.rst +++ b/Doc/whatsnew/2.5.rst @@ -330,7 +330,7 @@ statement, only the ``from ... import`` form. :pep:`328` - Imports: Multi-Line and Absolute/Relative PEP written by Aahz; implemented by Thomas Wouters. - http://codespeak.net/py/current/doc/index.html + https://pylib.readthedocs.org/ The py library by Holger Krekel, which contains the :mod:`py.std` package. .. ====================================================================== @@ -547,7 +547,7 @@ exhausted. Earlier versions of these features were proposed in :pep:`288` by Raymond Hettinger and :pep:`325` by Samuele Pedroni. - http://en.wikipedia.org/wiki/Coroutine + https://en.wikipedia.org/wiki/Coroutine The Wikipedia entry for coroutines. http://www.sidhe.org/~dan/blog/archives/000178.html @@ -1095,7 +1095,7 @@ Here are all of the changes that Python 2.5 makes to the core Python language. log all the paths searched. In Python 2.5, a new :exc:`ImportWarning` warning is triggered when an import would have picked up a directory as a package but no :file:`__init__.py` was found. This warning is silently ignored by default; - provide the :option:`-Wd` option when running the Python executable to display + provide the :option:`-Wd <-W>` option when running the Python executable to display the warning message. (Implemented by Thomas Wouters.) * The list of base classes in a class definition can now be empty. As an @@ -1126,7 +1126,7 @@ or ``exit()`` will now exit the interpreter as they expect. (Implemented by Georg Brandl.) The Python executable now accepts the standard long options :option:`--help` -and :option:`--version`; on Windows, it also accepts the :option:`/?` option +and :option:`--version`; on Windows, it also accepts the :option:`/? <-?>` option for displaying a help message. (Implemented by Georg Brandl.) .. ====================================================================== @@ -1528,7 +1528,7 @@ complete list of changes, or look through the SVN logs for all the details. * The :mod:`socket` module now supports :const:`AF_NETLINK` sockets on Linux, thanks to a patch from Philippe Biondi. Netlink sockets are a Linux-specific mechanism for communications between a user-space process and kernel code; an - introductory article about them is at http://www.linuxjournal.com/article/7356. + introductory article about them is at https://www.linuxjournal.com/article/7356. In Python code, netlink addresses are represented as a tuple of 2 integers, ``(pid, group_mask)``. @@ -1640,7 +1640,7 @@ complete list of changes, or look through the SVN logs for all the details. * The :mod:`webbrowser` module received a number of enhancements. It's now usable as a script with ``python -m webbrowser``, taking a URL as the argument; there are a number of switches to control the behaviour (:option:`-n` for a new - browser window, :option:`-t` for a new tab). New module-level functions, + browser window, :option:`!-t` for a new tab). New module-level functions, :func:`open_new` and :func:`open_new_tab`, were added to support this. The module's :func:`open` function supports an additional feature, an *autoraise* parameter that signals whether to raise the open window when possible. A number @@ -2013,7 +2013,7 @@ This example uses the iterator form:: >>> For more information about the SQL dialect supported by SQLite, see -http://www.sqlite.org. +https://www.sqlite.org. .. seealso:: @@ -2021,7 +2021,7 @@ http://www.sqlite.org. http://www.pysqlite.org The pysqlite web page. - http://www.sqlite.org + https://www.sqlite.org The SQLite web page; the documentation describes the syntax and the available data types for the supported SQL dialect. @@ -2088,7 +2088,7 @@ Changes to Python's build process and to the C API include: provided the results of their examination of the Python source code. The analysis found about 60 bugs that were quickly fixed. Many of the bugs were refcounting problems, often occurring in error-handling code. See - http://scan.coverity.com for the statistics. + https://scan.coverity.com for the statistics. * The largest change to the C API came from :pep:`353`, which modifies the interpreter to use a :c:type:`Py_ssize_t` type definition instead of diff --git a/Doc/whatsnew/2.6.rst b/Doc/whatsnew/2.6.rst index e763265269..4ab16563e8 100644 --- a/Doc/whatsnew/2.6.rst +++ b/Doc/whatsnew/2.6.rst @@ -153,10 +153,10 @@ The infrastructure committee of the Python Software Foundation therefore posted a call for issue trackers, asking volunteers to set up different products and import some of the bugs and patches from SourceForge. Four different trackers were examined: `Jira -<http://www.atlassian.com/software/jira/>`__, -`Launchpad <http://www.launchpad.net>`__, +<https://www.atlassian.com/software/jira/>`__, +`Launchpad <https://launchpad.net/>`__, `Roundup <http://roundup.sourceforge.net/>`__, and -`Trac <http://trac.edgewall.org/>`__. +`Trac <https://trac.edgewall.org/>`__. The committee eventually settled on Jira and Roundup as the two candidates. Jira is a commercial product that offers no-cost hosted instances to free-software projects; Roundup @@ -217,7 +217,7 @@ the time required to finish the job. During the 2.6 development cycle, Georg Brandl put a lot of effort into building a new toolchain for processing the documentation. The resulting package is called Sphinx, and is available from -http://sphinx.pocoo.org/. +http://sphinx-doc.org/. Sphinx concentrates on HTML output, producing attractively styled and modern HTML; printed output is still supported through conversion to @@ -1431,7 +1431,7 @@ one, :func:`math.trunc`, that's been backported to Python 2.6. :pep:`3141` - A Type Hierarchy for Numbers PEP written by Jeffrey Yasskin. - `Scheme's numerical tower <http://www.gnu.org/software/guile/manual/html_node/Numerical-Tower.html#Numerical-Tower>`__, from the Guile manual. + `Scheme's numerical tower <https://www.gnu.org/software/guile/manual/html_node/Numerical-Tower.html#Numerical-Tower>`__, from the Guile manual. `Scheme's number datatypes <http://schemers.org/Documents/Standards/R5RS/HTML/r5rs-Z-H-9.html#%_sec_6.2>`__ from the R5RS Scheme specification. @@ -1796,7 +1796,7 @@ changes, or look through the Subversion logs for all the details. * The :mod:`bsddb` module also has a new maintainer, Jesús Cea Avión, and the package is now available as a standalone package. The web page for the package is `www.jcea.es/programacion/pybsddb.htm - <http://www.jcea.es/programacion/pybsddb.htm>`__. + <https://www.jcea.es/programacion/pybsddb.htm>`__. The plan is to remove the package from the standard library in Python 3.0, because its pace of releases is much more frequent than Python's. @@ -1926,7 +1926,7 @@ changes, or look through the Subversion logs for all the details. the left to six places. (Contributed by Skip Montanaro; :issue:`1158`.) * The :mod:`decimal` module was updated to version 1.66 of - `the General Decimal Specification <http://www2.hursley.ibm.com/decimal/decarith.html>`__. New features + `the General Decimal Specification <http://speleotrove.com/decimal/decarith.html>`__. New features include some methods for some basic mathematical functions such as :meth:`exp` and :meth:`log10`:: @@ -2889,7 +2889,7 @@ Improved SSL Support Bill Janssen made extensive improvements to Python 2.6's support for the Secure Sockets Layer by adding a new module, :mod:`ssl`, that's -built atop the `OpenSSL <http://www.openssl.org/>`__ library. +built atop the `OpenSSL <https://www.openssl.org/>`__ library. This new module provides more control over the protocol negotiated, the X.509 certificates used, and has better support for writing SSL servers (as opposed to clients) in Python. The existing SSL support @@ -3154,7 +3154,7 @@ Port-Specific Changes: Mac OS X :func:`macostools.touched` function to be removed because it depended on the :mod:`macfs` module. (:issue:`1490190`) -* Many other Mac OS modules have been deprecated and will removed in +* Many other Mac OS modules have been deprecated and will be removed in Python 3.0: :mod:`_builtinSuites`, :mod:`aepack`, diff --git a/Doc/whatsnew/2.7.rst b/Doc/whatsnew/2.7.rst index 3966cb4323..86f0197070 100644 --- a/Doc/whatsnew/2.7.rst +++ b/Doc/whatsnew/2.7.rst @@ -613,6 +613,9 @@ PEP 3137: The memoryview Object The :class:`memoryview` object provides a view of another object's memory content that matches the :class:`bytes` type's interface. +.. doctest:: + :options: +SKIP + >>> import string >>> m = memoryview(string.letters) >>> m @@ -628,6 +631,9 @@ memory content that matches the :class:`bytes` type's interface. The content of the view can be converted to a string of bytes or a list of integers: +.. doctest:: + :options: +SKIP + >>> m2.tobytes() 'abcdefghijklmnopqrstuvwxyz' >>> m2.tolist() @@ -637,6 +643,9 @@ a list of integers: :class:`memoryview` objects allow modifying the underlying object if it's a mutable object. +.. doctest:: + :options: +SKIP + >>> m2[0] = 75 Traceback (most recent call last): File "<stdin>", line 1, in <module> @@ -671,6 +680,9 @@ Some smaller changes made to the core Python language are: ``{}`` continues to represent an empty dictionary; use ``set()`` for an empty set. + .. doctest:: + :options: +SKIP + >>> {1, 2, 3, 4, 5} set([1, 2, 3, 4, 5]) >>> set() # empty set @@ -684,6 +696,9 @@ Some smaller changes made to the core Python language are: 3.x, generalizing list/generator comprehensions to use the literal syntax for sets and dictionaries. + .. doctest:: + :options: +SKIP + >>> {x: x*x for x in range(6)} {0: 0, 1: 1, 2: 4, 3: 9, 4: 16, 5: 25} >>> {('a'*x) for x in range(6)} @@ -1029,7 +1044,7 @@ changes, or look through the Subversion logs for all the details. * Updated module: the :mod:`bsddb` module has been updated from 4.7.2devel9 to version 4.8.4 of - `the pybsddb package <http://www.jcea.es/programacion/pybsddb.htm>`__. + `the pybsddb package <https://www.jcea.es/programacion/pybsddb.htm>`__. The new version features better Python 3.x compatibility, various bug fixes, and adds several new BerkeleyDB flags and methods. (Updated by Jesús Cea Avión; :issue:`8156`. The pybsddb @@ -1052,7 +1067,7 @@ changes, or look through the Subversion logs for all the details. >>> for letter in 'here is a sample of english text': ... c[letter] += 1 ... - >>> c + >>> c # doctest: +SKIP Counter({' ': 6, 'e': 5, 's': 3, 'a': 2, 'i': 2, 'h': 2, 'l': 2, 't': 2, 'g': 1, 'f': 1, 'm': 1, 'o': 1, 'n': 1, 'p': 1, 'r': 1, 'x': 1}) @@ -1157,7 +1172,7 @@ changes, or look through the Subversion logs for all the details. * The :mod:`ctypes` module now always converts ``None`` to a C NULL pointer for arguments declared as pointers. (Changed by Thomas Heller; :issue:`4606`.) The underlying `libffi library - <http://sourceware.org/libffi/>`__ has been updated to version + <https://sourceware.org/libffi/>`__ has been updated to version 3.0.9, containing various fixes for different platforms. (Updated by Matthias Klose; :issue:`8142`.) @@ -1513,7 +1528,7 @@ changes, or look through the Subversion logs for all the details. (Contributed by Kristján Valur Jónsson; :issue:`6192` and :issue:`6267`.) * Updated module: the :mod:`sqlite3` module has been updated to - version 2.6.0 of the `pysqlite package <http://code.google.com/p/pysqlite/>`__. Version 2.6.0 includes a number of bugfixes, and adds + version 2.6.0 of the `pysqlite package <https://github.com/ghaering/pysqlite>`__. Version 2.6.0 includes a number of bugfixes, and adds the ability to load SQLite extensions from shared libraries. Call the ``enable_load_extension(True)`` method to enable extensions, and then call :meth:`~sqlite3.Connection.load_extension` to load a particular shared library. @@ -1530,7 +1545,7 @@ changes, or look through the Subversion logs for all the details. *ciphers* argument that's a string listing the encryption algorithms to be allowed; the format of the string is described `in the OpenSSL documentation - <http://www.openssl.org/docs/apps/ciphers.html#CIPHER_LIST_FORMAT>`__. + <https://www.openssl.org/docs/apps/ciphers.html#CIPHER-LIST-FORMAT>`__. (Added by Antoine Pitrou; :issue:`8322`.) Another change makes the extension load all of OpenSSL's ciphers and @@ -1638,12 +1653,18 @@ changes, or look through the Subversion logs for all the details. worked around the old behaviour. For example, Python 2.6.4 or 2.5 will return the following: + .. doctest:: + :options: +SKIP + >>> import urlparse >>> urlparse.urlsplit('invented://host/filename?query') ('invented', '', '//host/filename?query', '', '') Python 2.7 (and Python 2.6.5) will return: + .. doctest:: + :options: +SKIP + >>> import urlparse >>> urlparse.urlsplit('invented://host/filename?query') ('invented', 'host', '/filename?query', '', '') @@ -1652,7 +1673,10 @@ changes, or look through the Subversion logs for all the details. returns a named tuple instead of a standard tuple.) The :mod:`urlparse` module also supports IPv6 literal addresses as defined by - :rfc:`2732` (contributed by Senthil Kumaran; :issue:`2987`). :: + :rfc:`2732` (contributed by Senthil Kumaran; :issue:`2987`). + + .. doctest:: + :options: +SKIP >>> urlparse.urlparse('http://[1080::8:800:200C:417A]/foo') ParseResult(scheme='http', netloc='[1080::8:800:200C:417A]', @@ -1783,7 +1807,7 @@ on being added to Tcl/Tck release 8.5. To learn more, read the :mod:`ttk` module documentation. You may also wish to read the Tcl/Tk manual page describing the Ttk theme engine, available at -http://www.tcl.tk/man/tcl8.5/TkCmd/ttk_intro.htm. Some +https://www.tcl.tk/man/tcl8.5/TkCmd/ttk_intro.htm. Some screenshots of the Python/Ttk code in use are at http://code.google.com/p/python-ttk/wiki/Screenshots. @@ -1820,12 +1844,12 @@ Consult the :mod:`unittest` module documentation for more details. The :func:`~unittest.main` function supports some other new options: -* :option:`-b` or :option:`--buffer` will buffer the standard output +* :option:`-b <unittest -b>` or :option:`--buffer` will buffer the standard output and standard error streams during each test. If the test passes, any resulting output will be discarded; on failure, the buffered output will be displayed. -* :option:`-c` or :option:`--catch` will cause the control-C interrupt +* :option:`-c <unittest -c>` or :option:`--catch` will cause the control-C interrupt to be handled more gracefully. Instead of interrupting the test process immediately, the currently running test will be completed and then the partial results up to the interruption will be reported. @@ -1839,7 +1863,7 @@ The :func:`~unittest.main` function supports some other new options: :func:`~unittest.removeHandler` decorator that can be used to mark tests that should have the control-C handling disabled. -* :option:`-f` or :option:`--failfast` makes +* :option:`-f <unittest -f>` or :option:`--failfast` makes test execution stop immediately when a test fails instead of continuing to execute further tests. (Suggested by Cliff Dyer and implemented by Michael Foord; :issue:`8074`.) @@ -2079,7 +2103,7 @@ Changes to Python's build process and to the C API include: * The latest release of the GNU Debugger, GDB 7, can be `scripted using Python - <http://sourceware.org/gdb/current/onlinedocs/gdb/Python.html>`__. + <https://sourceware.org/gdb/current/onlinedocs/gdb/Python.html>`__. When you begin debugging an executable program P, GDB will look for a file named ``P-gdb.py`` and automatically read it. Dave Malcolm contributed a :file:`python-gdb.py` that adds a number of @@ -2149,7 +2173,7 @@ Changes to Python's build process and to the C API include: with *updatepath* set to false. Security issue reported as `CVE-2008-5983 - <http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2008-5983>`_; + <https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2008-5983>`_; discussed in :issue:`5753`, and fixed by Antoine Pitrou. * New macros: the Python header files now define the following macros: @@ -2266,7 +2290,9 @@ There is an existing data type already used for this, written in pure Python could cause a segmentation fault by taking a :c:type:`PyCObject` from module A and somehow substituting it for the :c:type:`PyCObject` in module B. Capsules know their own name, -and getting the pointer requires providing the name:: +and getting the pointer requires providing the name: + +.. code-block:: c void *vtable; @@ -2381,7 +2407,7 @@ Other Changes and Fixes takes an integer specifying how many tests run in parallel. This allows reducing the total runtime on multi-core machines. This option is compatible with several other options, including the - :option:`-R` switch which is known to produce long runtimes. + :option:`!-R` switch which is known to produce long runtimes. (Added by Antoine Pitrou, :issue:`6152`.) This can also be used with a new :option:`-F` switch that runs selected tests in a loop until they fail. (Added by Antoine Pitrou; :issue:`7312`.) @@ -2475,12 +2501,18 @@ In the standard library: worked around the old behaviour. For example, Python 2.6.4 or 2.5 will return the following: + .. doctest:: + :options: +SKIP + >>> import urlparse >>> urlparse.urlsplit('invented://host/filename?query') ('invented', '', '//host/filename?query', '', '') Python 2.7 (and Python 2.6.5) will return: + .. doctest:: + :options: +SKIP + >>> import urlparse >>> urlparse.urlsplit('invented://host/filename?query') ('invented', 'host', '/filename?query', '', '') @@ -2586,4 +2618,3 @@ The author would like to thank the following people for offering suggestions, corrections and assistance with various drafts of this article: Nick Coghlan, Philip Jenvey, Ryan Lovett, R. David Murray, Hugh Secker-Walker. - diff --git a/Doc/whatsnew/3.0.rst b/Doc/whatsnew/3.0.rst index 99411305a5..a3d3fad661 100644 --- a/Doc/whatsnew/3.0.rst +++ b/Doc/whatsnew/3.0.rst @@ -117,7 +117,9 @@ You can also customize the separator between items, e.g.:: print("There are <", 2**32, "> possibilities!", sep="") -which produces:: +which produces: + +.. code-block:: none There are <4294967296> possibilities! @@ -204,11 +206,11 @@ Python 3.0 has simplified the rules for ordering comparisons: Integers -------- -* :pep:`0237`: Essentially, :class:`long` renamed to :class:`int`. +* :pep:`237`: Essentially, :class:`long` renamed to :class:`int`. That is, there is only one built-in integral type, named :class:`int`; but it behaves mostly like the old :class:`long` type. -* :pep:`0238`: An expression like ``1/2`` returns a float. Use +* :pep:`238`: An expression like ``1/2`` returns a float. Use ``1//2`` to get the truncating behavior. (The latter syntax has existed for years, at least since Python 2.2.) @@ -384,7 +386,7 @@ New Syntax * Dictionary comprehensions: ``{k: v for k, v in stuff}`` means the same thing as ``dict(stuff)`` but is more flexible. (This is - :pep:`0274` vindicated. :-) + :pep:`274` vindicated. :-) * Set literals, e.g. ``{1, 2}``. Note that ``{}`` is an empty dictionary; use ``set()`` for an empty set. Set comprehensions are @@ -469,7 +471,7 @@ Removed Syntax * The only acceptable syntax for relative imports is :samp:`from .[{module}] import {name}`. All :keyword:`import` forms not starting with ``.`` are - interpreted as absolute imports. (:pep:`0328`) + interpreted as absolute imports. (:pep:`328`) * Classic classes are gone. @@ -555,9 +557,9 @@ review: * Many old modules were removed. Some, like :mod:`gopherlib` (no longer used) and :mod:`md5` (replaced by :mod:`hashlib`), were - already deprecated by :pep:`0004`. Others were removed as a result + already deprecated by :pep:`4`. Others were removed as a result of the removal of support for various platforms such as Irix, BeOS - and Mac OS 9 (see :pep:`0011`). Some modules were also selected for + and Mac OS 9 (see :pep:`11`). Some modules were also selected for removal in Python 3.0 due to lack of use or because a better replacement exists. See :pep:`3108` for an exhaustive list. @@ -565,10 +567,10 @@ review: core standard library has proved over time to be a particular burden for the core developers due to testing instability and Berkeley DB's release schedule. However, the package is alive and well, - externally maintained at http://www.jcea.es/programacion/pybsddb.htm. + externally maintained at https://www.jcea.es/programacion/pybsddb.htm. * Some modules were renamed because their old name disobeyed - :pep:`0008`, or for various other reasons. Here's the list: + :pep:`8`, or for various other reasons. Here's the list: ======================= ======================= Old Name New Name @@ -685,7 +687,7 @@ Changes To Exceptions The APIs for raising and catching exception have been cleaned up and new powerful features added: -* :pep:`0352`: All exceptions must be derived (directly or indirectly) +* :pep:`352`: All exceptions must be derived (directly or indirectly) from :exc:`BaseException`. This is the root of the exception hierarchy. This is not new as a recommendation, but the *requirement* to inherit from :exc:`BaseException` is new. (Python @@ -783,8 +785,8 @@ Operators And Special Methods :attr:`func_closure`, :attr:`func_code`, :attr:`func_defaults`, :attr:`func_dict`, :attr:`func_doc`, :attr:`func_globals`, :attr:`func_name` were renamed to :attr:`__closure__`, - :attr:`__code__`, :attr:`__defaults__`, :attr:`__dict__`, - :attr:`__doc__`, :attr:`__globals__`, :attr:`__name__`, + :attr:`__code__`, :attr:`__defaults__`, :attr:`~object.__dict__`, + :attr:`__doc__`, :attr:`__globals__`, :attr:`~definition.__name__`, respectively. * :meth:`__nonzero__` is now :meth:`__bool__`. diff --git a/Doc/whatsnew/3.2.rst b/Doc/whatsnew/3.2.rst index 582250450d..66c2aec241 100644 --- a/Doc/whatsnew/3.2.rst +++ b/Doc/whatsnew/3.2.rst @@ -116,7 +116,7 @@ or more positional arguments is present, and making a required option:: Example of calling the parser on a command string:: - >>> cmd = 'deploy sneezy.example.com sleepy.example.com -u skycaptain' + >>> cmd = 'deploy sneezy.example.com sleepy.example.com -u skycaptain' >>> result = parser.parse_args(cmd.split()) >>> result.action 'deploy' @@ -160,6 +160,8 @@ each with their own argument patterns and help displays:: parser_m.add_argument('-c', '--course', type=int, required=True) parser_m.add_argument('-s', '--speed', type=int, default=0) +.. code-block:: shell-session + $ ./helm.py --help # top level help (launch and move) $ ./helm.py launch --help # help for launch options $ ./helm.py launch --missiles # set missiles=True and torpedos=False @@ -212,7 +214,8 @@ loaded and called with code like this:: >>> import json, logging.config >>> with open('conf.json') as f: - conf = json.load(f) + ... conf = json.load(f) + ... >>> logging.config.dictConfig(conf) >>> logging.info("Transaction completed normally") INFO : root : Transaction completed normally @@ -460,15 +463,15 @@ Some smaller changes made to the core Python language are: 'The testing project status is green as of February 15, 2011' >>> class LowerCasedDict(dict): - def __getitem__(self, key): - return dict.__getitem__(self, key.lower()) + ... def __getitem__(self, key): + ... return dict.__getitem__(self, key.lower()) >>> lcd = LowerCasedDict(part='widgets', quantity=10) >>> 'There are {QUANTITY} {Part} in stock'.format_map(lcd) 'There are 10 widgets in stock' >>> class PlaceholderDict(dict): - def __missing__(self, key): - return '<{}>'.format(key) + ... def __missing__(self, key): + ... return '<{}>'.format(key) >>> 'Hello {name}, welcome to {location}'.format_map(PlaceholderDict()) 'Hello <name>, welcome to <location>' @@ -477,7 +480,9 @@ Some smaller changes made to the core Python language are: * The interpreter can now be started with a quiet option, ``-q``, to prevent the copyright and version information from being displayed in the interactive - mode. The option can be introspected using the :attr:`sys.flags` attribute:: + mode. The option can be introspected using the :attr:`sys.flags` attribute: + + .. code-block:: shell-session $ python -q >>> sys.flags @@ -496,10 +501,10 @@ Some smaller changes made to the core Python language are: exceptions pass through:: >>> class A: - @property - def f(self): - return 1 // 0 - + ... @property + ... def f(self): + ... return 1 // 0 + ... >>> a = A() >>> hasattr(a, 'f') Traceback (most recent call last): @@ -537,7 +542,7 @@ Some smaller changes made to the core Python language are: def outer(x): def inner(): - return x + return x inner() del x @@ -547,12 +552,12 @@ Some smaller changes made to the core Python language are: def f(): def print_error(): - print(e) + print(e) try: - something + something except Exception as e: - print_error() - # implicit "del e" here + print_error() + # implicit "del e" here (See :issue:`4617`.) @@ -572,7 +577,9 @@ Some smaller changes made to the core Python language are: by Benjamin Peterson in :issue:`8413`.) * Warnings are now easier to control using the :envvar:`PYTHONWARNINGS` - environment variable as an alternative to using ``-W`` at the command line:: + environment variable as an alternative to using ``-W`` at the command line: + + .. code-block:: shell-session $ export PYTHONWARNINGS='ignore::RuntimeWarning::,once::UnicodeWarning::' @@ -594,7 +601,9 @@ Some smaller changes made to the core Python language are: object ensures it closes the underlying operating system resource (usually, a file descriptor), the delay in deallocating the object could produce various issues, especially under Windows. Here is an example - of enabling the warning from the command line:: + of enabling the warning from the command line: + + .. code-block:: shell-session $ python -q -Wdefault >>> f = open("foo", "wb") @@ -769,8 +778,8 @@ functools (Contributed by Raymond Hettinger and incorporating design ideas from Jim Baker, Miki Tebeka, and Nick Coghlan; see `recipe 498245 - <http://code.activestate.com/recipes/498245>`_\, `recipe 577479 - <http://code.activestate.com/recipes/577479>`_\, :issue:`10586`, and + <https://code.activestate.com/recipes/498245>`_\, `recipe 577479 + <https://code.activestate.com/recipes/577479>`_\, :issue:`10586`, and :issue:`10593`.) * The :func:`functools.wraps` decorator now adds a :attr:`__wrapped__` attribute @@ -799,6 +808,7 @@ functools def __eq__(self, other): return ((self.lastname.lower(), self.firstname.lower()) == (other.lastname.lower(), other.firstname.lower())) + def __lt__(self, other): return ((self.lastname.lower(), self.firstname.lower()) < (other.lastname.lower(), other.firstname.lower())) @@ -845,9 +855,9 @@ collections * The :class:`collections.Counter` class now has two forms of in-place subtraction, the existing *-=* operator for `saturating subtraction - <http://en.wikipedia.org/wiki/Saturation_arithmetic>`_ and the new + <https://en.wikipedia.org/wiki/Saturation_arithmetic>`_ and the new :meth:`~collections.Counter.subtract` method for regular subtraction. The - former is suitable for `multisets <http://en.wikipedia.org/wiki/Multiset>`_ + former is suitable for `multisets <https://en.wikipedia.org/wiki/Multiset>`_ which only have positive counts, and the latter is more suitable for use cases that allow negative counts: @@ -906,7 +916,7 @@ with multiple preconditions does not run until all of the predecessor tasks are complete. Barriers can work with an arbitrary number of threads. This is a generalization -of a `Rendezvous <http://en.wikipedia.org/wiki/Synchronous_rendezvous>`_ which +of a `Rendezvous <https://en.wikipedia.org/wiki/Synchronous_rendezvous>`_ which is defined for only two threads. Implemented as a two-phase cyclic barrier, :class:`~threading.Barrier` objects @@ -942,7 +952,7 @@ released and a :exc:`~threading.BrokenBarrierError` exception is raised:: def get_votes(site): ballots = conduct_election(site) try: - all_polls_closed.wait(timeout = midnight - time.now()) + all_polls_closed.wait(timeout=midnight - time.now()) except BrokenBarrierError: lockbox = seal_ballots(ballots) queue.put(lockbox) @@ -955,7 +965,7 @@ sites do not finish before midnight, the barrier times-out and the ballots are sealed and deposited in a queue for later handling. See `Barrier Synchronization Patterns -<http://parlab.eecs.berkeley.edu/wiki/_media/patterns/paraplop_g1_3.pdf>`_ for +<https://parlab.eecs.berkeley.edu/wiki/_media/patterns/paraplop_g1_3.pdf>`_ for more examples of how barriers can be used in parallel computing. Also, there is a simple but thorough explanation of barriers in `The Little Book of Semaphores <http://greenteapress.com/semaphores/downey08semaphores.pdf>`_, *section 3.6*. @@ -1043,7 +1053,7 @@ of nearly equal quantities: 0.013765762467652909 The :func:`~math.erf` function computes a probability integral or `Gaussian -error function <http://en.wikipedia.org/wiki/Error_function>`_. The +error function <https://en.wikipedia.org/wiki/Error_function>`_. The complementary error function, :func:`~math.erfc`, is ``1 - erf(x)``: >>> erf(1.0/sqrt(2.0)) # portion of normal distribution within 1 standard deviation @@ -1054,7 +1064,7 @@ complementary error function, :func:`~math.erfc`, is ``1 - erf(x)``: 1.0 The :func:`~math.gamma` function is a continuous extension of the factorial -function. See http://en.wikipedia.org/wiki/Gamma_function for details. Because +function. See https://en.wikipedia.org/wiki/Gamma_function for details. Because the function is related to factorials, it grows large even for small values of *x*, so there is also a :func:`~math.lgamma` function for computing the natural logarithm of the gamma function: @@ -1097,16 +1107,16 @@ for slice notation are well-suited to in-place editing:: >>> REC_LEN, LOC_START, LOC_LEN = 34, 7, 11 >>> def change_location(buffer, record_number, location): - start = record_number * REC_LEN + LOC_START - buffer[start: start+LOC_LEN] = location + ... start = record_number * REC_LEN + LOC_START + ... buffer[start: start+LOC_LEN] = location >>> import io >>> byte_stream = io.BytesIO( - b'G3805 storeroom Main chassis ' - b'X7899 shipping Reserve cog ' - b'L6988 receiving Primary sprocket' - ) + ... b'G3805 storeroom Main chassis ' + ... b'X7899 shipping Reserve cog ' + ... b'L6988 receiving Primary sprocket' + ... ) >>> buffer = byte_stream.getbuffer() >>> change_location(buffer, 1, b'warehouse ') >>> change_location(buffer, 0, b'showroom ') @@ -1131,10 +1141,10 @@ decorator, :func:`~reprlib.recursive_repr`, for detecting recursive calls to :meth:`__repr__` and substituting a placeholder string instead:: >>> class MyList(list): - @recursive_repr() - def __repr__(self): - return '<' + '|'.join(map(repr, self)) + '>' - + ... @recursive_repr() + ... def __repr__(self): + ... return '<' + '|'.join(map(repr, self)) + '>' + ... >>> m = MyList('abc') >>> m.append(m) >>> m.append('x') @@ -1197,8 +1207,8 @@ the field names:: >>> w.writeheader() "name","dept" >>> w.writerows([ - {'name': 'tom', 'dept': 'accounting'}, - {'name': 'susan', 'dept': 'Salesl'}]) + ... {'name': 'tom', 'dept': 'accounting'}, + ... {'name': 'susan', 'dept': 'Salesl'}]) "tom","accounting" "susan","sales" @@ -1423,14 +1433,14 @@ function can return *None*:: >>> import tarfile, glob >>> def myfilter(tarinfo): - if tarinfo.isfile(): # only save real files - tarinfo.uname = 'monty' # redact the user name - return tarinfo + ... if tarinfo.isfile(): # only save real files + ... tarinfo.uname = 'monty' # redact the user name + ... return tarinfo >>> with tarfile.open(name='myarchive.tar.gz', mode='w:gz') as tf: - for filename in glob.glob('*.txt'): - tf.add(filename, filter=myfilter) - tf.list() + ... for filename in glob.glob('*.txt'): + ... tf.add(filename, filter=myfilter) + ... tf.list() -rw-r--r-- monty/501 902 2011-01-26 17:59:11 annotations.txt -rw-r--r-- monty/501 123 2011-01-26 17:59:11 general_questions.txt -rw-r--r-- monty/501 3514 2011-01-26 17:59:11 prion.txt @@ -1536,26 +1546,26 @@ step is non-destructive (the original files are left unchanged). >>> import shutil, pprint - >>> os.chdir('mydata') # change to the source directory + >>> os.chdir('mydata') # change to the source directory >>> f = shutil.make_archive('/var/backup/mydata', - 'zip') # archive the current directory - >>> f # show the name of archive + ... 'zip') # archive the current directory + >>> f # show the name of archive '/var/backup/mydata.zip' - >>> os.chdir('tmp') # change to an unpacking + >>> os.chdir('tmp') # change to an unpacking >>> shutil.unpack_archive('/var/backup/mydata.zip') # recover the data - >>> pprint.pprint(shutil.get_archive_formats()) # display known formats + >>> pprint.pprint(shutil.get_archive_formats()) # display known formats [('bztar', "bzip2'ed tar-file"), ('gztar', "gzip'ed tar-file"), ('tar', 'uncompressed tar file'), ('zip', 'ZIP file')] - >>> shutil.register_archive_format( # register a new archive format - name = 'xz', - function = xz.compress, # callable archiving function - extra_args = [('level', 8)], # arguments to the function - description = 'xz compression' - ) + >>> shutil.register_archive_format( # register a new archive format + ... name='xz', + ... function=xz.compress, # callable archiving function + ... extra_args=[('level', 8)], # arguments to the function + ... description='xz compression' + ... ) (Contributed by Tarek Ziadé.) @@ -1618,7 +1628,7 @@ for secure (encrypted, authenticated) internet connections: * The :func:`ssl.wrap_socket` constructor function now takes a *ciphers* argument. The *ciphers* string lists the allowed encryption algorithms using the format described in the `OpenSSL documentation - <http://www.openssl.org/docs/apps/ciphers.html#CIPHER_LIST_FORMAT>`__. + <https://www.openssl.org/docs/apps/ciphers.html#CIPHER-LIST-FORMAT>`__. * When linked against recent versions of OpenSSL, the :mod:`ssl` module now supports the Server Name Indication extension to the TLS protocol, allowing @@ -1718,7 +1728,9 @@ names. test discovery can find tests within packages, locating any test importable from the top-level directory. The top-level directory can be specified with the `-t` option, a pattern for matching files with ``-p``, and a directory to - start discovery with ``-s``:: + start discovery with ``-s``: + + .. code-block:: shell-session $ python -m unittest discover -s my_proj_dir -p _test.py @@ -1854,7 +1866,7 @@ inspect >>> from inspect import getgeneratorstate >>> def gen(): - yield 'demo' + ... yield 'demo' >>> g = gen() >>> getgeneratorstate(g) 'GEN_CREATED' @@ -1874,11 +1886,11 @@ inspect change state while it is searching:: >>> class A: - @property - def f(self): - print('Running') - return 10 - + ... @property + ... def f(self): + ... print('Running') + ... return 10 + ... >>> a = A() >>> getattr(a, 'f') Running @@ -1893,7 +1905,9 @@ pydoc The :mod:`pydoc` module now provides a much-improved Web server interface, as well as a new command-line option ``-b`` to automatically open a browser window -to display that server:: +to display that server: + +.. code-block:: shell-session $ pydoc3.2 -b @@ -1996,7 +2010,9 @@ details of a given Python installation. '/Users/raymondhettinger/Library/Python/3.2/lib/python/site-packages' Conveniently, some of site's functionality is accessible directly from the -command-line:: +command-line: + +.. code-block:: shell-session $ python -m site --user-base /Users/raymondhettinger/.local @@ -2029,7 +2045,9 @@ seven named schemes used by :mod:`distutils`. Those include *posix_prefix*, * :func:`~sysconfig.get_config_vars` returns a dictionary of platform specific variables. -There is also a convenient command-line interface:: +There is also a convenient command-line interface: + +.. code-block:: doscon C:\Python32>python -m sysconfig Platform: "win32" @@ -2102,19 +2120,19 @@ Config parsers gained a new API based on the mapping protocol:: >>> parser = ConfigParser() >>> parser.read_string(""" - [DEFAULT] - location = upper left - visible = yes - editable = no - color = blue - - [main] - title = Main Menu - color = green - - [options] - title = Options - """) + ... [DEFAULT] + ... location = upper left + ... visible = yes + ... editable = no + ... color = blue + ... + ... [main] + ... title = Main Menu + ... color = green + ... + ... [options] + ... title = Options + ... """) >>> parser['main']['color'] 'green' >>> parser['main']['editable'] @@ -2138,24 +2156,24 @@ handler :class:`~configparser.ExtendedInterpolation`:: >>> parser = ConfigParser(interpolation=ExtendedInterpolation()) >>> parser.read_dict({'buildout': {'directory': '/home/ambv/zope9'}, - 'custom': {'prefix': '/usr/local'}}) + ... 'custom': {'prefix': '/usr/local'}}) >>> parser.read_string(""" - [buildout] - parts = - zope9 - instance - find-links = - ${buildout:directory}/downloads/dist - - [zope9] - recipe = plone.recipe.zope9install - location = /opt/zope - - [instance] - recipe = plone.recipe.zope9instance - zope9-location = ${zope9:location} - zope-conf = ${custom:prefix}/etc/zope.conf - """) + ... [buildout] + ... parts = + ... zope9 + ... instance + ... find-links = + ... ${buildout:directory}/downloads/dist + ... + ... [zope9] + ... recipe = plone.recipe.zope9install + ... location = /opt/zope + ... + ... [instance] + ... recipe = plone.recipe.zope9instance + ... zope9-location = ${zope9:location} + ... zope-conf = ${custom:prefix}/etc/zope.conf + ... """) >>> parser['buildout']['find-links'] '\n/home/ambv/zope9/downloads/dist' >>> parser['instance']['zope-conf'] @@ -2180,7 +2198,7 @@ urllib.parse A number of usability improvements were made for the :mod:`urllib.parse` module. The :func:`~urllib.parse.urlparse` function now supports `IPv6 -<http://en.wikipedia.org/wiki/IPv6>`_ addresses as described in :rfc:`2732`: +<https://en.wikipedia.org/wiki/IPv6>`_ addresses as described in :rfc:`2732`: >>> import urllib.parse >>> urllib.parse.urlparse('http://[dead:beef:cafe:5417:affe:8FA3:deaf:feed]/foo/') @@ -2207,9 +2225,9 @@ string, then the *safe*, *encoding*, and *error* parameters are sent to :func:`~urllib.parse.quote_plus` for encoding:: >>> urllib.parse.urlencode([ - ('type', 'telenovela'), - ('name', '¿Dónde Está Elisa?')], - encoding='latin-1') + ... ('type', 'telenovela'), + ... ('name', '¿Dónde Está Elisa?')], + ... encoding='latin-1') 'type=telenovela&name=%BFD%F3nde+Est%E1+Elisa%3F' As detailed in :ref:`parsing-ascii-encoded-bytes`, all the :mod:`urllib.parse` @@ -2263,7 +2281,9 @@ turtledemo The demonstration code for the :mod:`turtle` module was moved from the *Demo* directory to main library. It includes over a dozen sample scripts with lively displays. Being on :attr:`sys.path`, it can now be run directly -from the command-line:: +from the command-line: + +.. code-block:: shell-session $ python -m turtledemo @@ -2328,7 +2348,7 @@ A number of small performance enhancements have been added: (Contributed by Alexandre Vassalotti, Antoine Pitrou and the Unladen Swallow team in :issue:`9410` and :issue:`3873`.) -* The `Timsort algorithm <http://en.wikipedia.org/wiki/Timsort>`_ used in +* The `Timsort algorithm <https://en.wikipedia.org/wiki/Timsort>`_ used in :meth:`list.sort` and :func:`sorted` now runs faster and uses less memory when called with a :term:`key function`. Previously, every element of a list was wrapped with a temporary object that remembered the key value @@ -2380,7 +2400,7 @@ Unicode Python has been updated to `Unicode 6.0.0 <http://unicode.org/versions/Unicode6.0.0/>`_. The update to the standard adds -over 2,000 new characters including `emoji <http://en.wikipedia.org/wiki/Emoji>`_ +over 2,000 new characters including `emoji <https://en.wikipedia.org/wiki/Emoji>`_ symbols which are important for mobile phones. In addition, the updated standard has altered the character properties for two @@ -2432,7 +2452,7 @@ The documentation continues to be improved. **Source code** :source:`Lib/functools.py`. (Contributed by Raymond Hettinger; see - `rationale <http://rhettinger.wordpress.com/2011/01/28/open-your-source-more/>`_.) + `rationale <https://rhettinger.wordpress.com/2011/01/28/open-your-source-more/>`_.) * The docs now contain more examples and recipes. In particular, :mod:`re` module has an extensive section, :ref:`re-examples`. Likewise, the @@ -2468,7 +2488,7 @@ Code Repository =============== In addition to the existing Subversion code repository at http://svn.python.org -there is now a `Mercurial <http://mercurial.selenic.com/>`_ repository at +there is now a `Mercurial <https://www.mercurial-scm.org/>`_ repository at https://hg.python.org/\ . After the 3.2 release, there are plans to switch to Mercurial as the primary @@ -2478,7 +2498,7 @@ members of the community to create and share external changesets. See To learn to use the new version control system, see the `tutorial by Joel Spolsky <http://hginit.com>`_ or the `Guide to Mercurial Workflows -<http://mercurial.selenic.com/guide>`_. +<https://www.mercurial-scm.org/guide>`_. Build and C API Changes @@ -2559,7 +2579,7 @@ Also, there were a number of updates to the Mac OS X build, see :source:`Mac/BuildScript/README.txt` for details. For users running a 32/64-bit build, there is a known problem with the default Tcl/Tk on Mac OS X 10.6. Accordingly, we recommend installing an updated alternative such as -`ActiveState Tcl/Tk 8.5.9 <http://www.activestate.com/activetcl/downloads>`_\. +`ActiveState Tcl/Tk 8.5.9 <https://www.activestate.com/activetcl/downloads>`_\. See https://www.python.org/download/mac/tcltk/ for additional details. Porting to Python 3.2 @@ -2664,7 +2684,7 @@ require changes to your code: * The :class:`xml.etree.ElementTree` class now raises an :exc:`xml.etree.ElementTree.ParseError` when a parse fails. Previously it - raised a :exc:`xml.parsers.expat.ExpatError`. + raised an :exc:`xml.parsers.expat.ExpatError`. * The new, longer :func:`str` value on floats may break doctests which rely on the old output format. @@ -2699,4 +2719,3 @@ require changes to your code: * Due to the new :term:`GIL` implementation, :c:func:`PyEval_InitThreads()` cannot be called before :c:func:`Py_Initialize()` anymore. - diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst index 094eff87db..3e48c82b0e 100644 --- a/Doc/whatsnew/3.3.rst +++ b/Doc/whatsnew/3.3.rst @@ -440,15 +440,15 @@ return a final value to the outer generator:: ... >>> tallies = [] >>> acc = gather_tallies(tallies) - >>> next(acc) # Ensure the accumulator is ready to accept values + >>> next(acc) # Ensure the accumulator is ready to accept values >>> for i in range(4): ... acc.send(i) ... - >>> acc.send(None) # Finish the first tally + >>> acc.send(None) # Finish the first tally >>> for i in range(5): ... acc.send(i) ... - >>> acc.send(None) # Finish the second tally + >>> acc.send(None) # Finish the second tally >>> tallies [6, 10] @@ -871,7 +871,9 @@ signal. Call :func:`faulthandler.enable` to install fault handlers for the :envvar:`PYTHONFAULTHANDLER` environment variable or by using :option:`-X` ``faulthandler`` command line option. -Example of a segmentation fault on Linux: :: +Example of a segmentation fault on Linux: + +.. code-block:: shell-session $ python -q -X faulthandler >>> import ctypes @@ -999,7 +1001,6 @@ byte of an invalid byte sequence. For example, ``b'\xff\n'.decode('gb2312', Incremental CJK codec encoders are no longer reset at each call to their encode() methods. For example:: - $ ./python -q >>> import codecs >>> encoder = codecs.getincrementalencoder('hz')('strict') >>> b''.join(encoder.encode(x) for x in '\u52ff\u65bd\u65bc\u4eba\u3002 Bye.') @@ -1528,7 +1529,7 @@ by Petri Lehtinen in :issue:`12021`.) multiprocessing --------------- -The new :func:`multiprocessing.connection.wait` function allows to poll +The new :func:`multiprocessing.connection.wait` function allows polling multiple objects (such as connections, sockets and pipes) with a timeout. (Contributed by Richard Oudkerk in :issue:`12328`.) @@ -1715,8 +1716,8 @@ pickle ------ :class:`pickle.Pickler` objects now have an optional -:attr:`~pickle.Pickler.dispatch_table` attribute allowing to set per-pickler -reduction functions. +:attr:`~pickle.Pickler.dispatch_table` attribute allowing per-pickler +reduction functions to be set. (Contributed by Richard Oudkerk in :issue:`14166`.) @@ -1884,13 +1885,13 @@ socket Heiko Wundram) * The :class:`~socket.socket` class now supports the PF_CAN protocol family - (http://en.wikipedia.org/wiki/Socketcan), on Linux - (http://lwn.net/Articles/253425). + (https://en.wikipedia.org/wiki/Socketcan), on Linux + (https://lwn.net/Articles/253425). (Contributed by Matthias Fuchs, updated by Tiago Gonçalves in :issue:`10141`.) * The :class:`~socket.socket` class now supports the PF_RDS protocol family - (http://en.wikipedia.org/wiki/Reliable_Datagram_Sockets and + (https://en.wikipedia.org/wiki/Reliable_Datagram_Sockets and https://oss.oracle.com/projects/rds/). * The :class:`~socket.socket` class now supports the ``PF_SYSTEM`` protocol diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst index 7d9bc0d8af..1e5c9d1fbd 100644 --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -144,7 +144,7 @@ Security improvements: all of the parent's inheritable handles, only the necessary ones. * A new :func:`hashlib.pbkdf2_hmac` function provides the `PKCS#5 password-based key derivation function 2 - <http://en.wikipedia.org/wiki/PBKDF2>`_. + <https://en.wikipedia.org/wiki/PBKDF2>`_. * :ref:`TLSv1.1 and TLSv1.2 support <whatsnew-tls-11-12>` for :mod:`ssl`. * :ref:`Retrieving certificates from the Windows system cert store support <whatsnew34-win-cert-store>` for :mod:`ssl`. @@ -746,7 +746,7 @@ optional *current_offset*), and the resulting object can be iterated to produce method, equivalent to calling :mod:`~dis.dis` on the constructor argument, but returned as a multi-line string:: - >>> bytecode = dis.Bytecode(lambda x: x +1, current_offset=3) + >>> bytecode = dis.Bytecode(lambda x: x + 1, current_offset=3) >>> for instr in bytecode: ... print('{} ({})'.format(instr.opname, instr.opcode)) LOAD_FAST (124) @@ -902,7 +902,7 @@ hashlib A new :func:`hashlib.pbkdf2_hmac` function provides the `PKCS#5 password-based key derivation function 2 -<http://en.wikipedia.org/wiki/PBKDF2>`_. (Contributed by Christian +<https://en.wikipedia.org/wiki/PBKDF2>`_. (Contributed by Christian Heimes in :issue:`18582`.) The :attr:`~hashlib.hash.name` attribute of :mod:`hashlib` hash objects is now @@ -1322,7 +1322,7 @@ kernel version of 2.6.36 or later and glibc of 2.13 or later, provides the ability to query or set the resource limits for processes other than the one making the call. (Contributed by Christian Heimes in :issue:`16595`.) -On Linux kernel version 2.6.36 or later, there are there are also some new +On Linux kernel version 2.6.36 or later, there are also some new Linux specific constants: :attr:`~resource.RLIMIT_MSGQUEUE`, :attr:`~resource.RLIMIT_NICE`, :attr:`~resource.RLIMIT_RTPRIO`, :attr:`~resource.RLIMIT_RTTIME`, and :attr:`~resource.RLIMIT_SIGPENDING`. @@ -1410,7 +1410,7 @@ sqlite3 A new boolean parameter to the :func:`~sqlite3.connect` function, *uri*, can be used to indicate that the *database* parameter is a ``uri`` (see the `SQLite -URI documentation <http://www.sqlite.org/uri.html>`_). (Contributed by poq in +URI documentation <https://www.sqlite.org/uri.html>`_). (Contributed by poq in :issue:`13773`.) @@ -1457,7 +1457,7 @@ s), as well as a :meth:`~ssl.SSLContext.get_ca_certs` method that returns a list of the loaded ``CA`` certificates. (Contributed by Christian Heimes in :issue:`18147`.) -If OpenSSL 0.9.8 or later is available, :class:`~ssl.SSLContext` has an new +If OpenSSL 0.9.8 or later is available, :class:`~ssl.SSLContext` has a new attribute :attr:`~ssl.SSLContext.verify_flags` that can be used to control the certificate verification process by setting it to some combination of the new constants :data:`~ssl.VERIFY_DEFAULT`, :data:`~ssl.VERIFY_CRL_CHECK_LEAF`, @@ -1917,8 +1917,8 @@ Other Build and C API Changes :issue:`18596`.) * The Windows build now uses `Address Space Layout Randomization - <http://en.wikipedia.org/wiki/ASLR>`_ and `Data Execution Prevention - <http://en.wikipedia.org/wiki/Data_Execution_Prevention>`_. (Contributed by + <https://en.wikipedia.org/wiki/Address_space_layout_randomization>`_ and `Data Execution Prevention + <https://en.wikipedia.org/wiki/Data_Execution_Prevention>`_. (Contributed by Christian Heimes in :issue:`16632`.) * New function :c:func:`PyObject_LengthHint` is the C API equivalent diff --git a/Doc/whatsnew/3.5.rst b/Doc/whatsnew/3.5.rst new file mode 100644 index 0000000000..339dbb6d3d --- /dev/null +++ b/Doc/whatsnew/3.5.rst @@ -0,0 +1,2537 @@ +**************************** + What's New In Python 3.5 +**************************** + +:Editors: Elvis Pranskevichus <elvis@magic.io>, Yury Selivanov <yury@magic.io> + +.. Rules for maintenance: + + * Anyone can add text to this document. Do not spend very much time + on the wording of your changes, because your text will probably + get rewritten to some degree. + + * The maintainer will go through Misc/NEWS periodically and add + changes; it's therefore more important to add your changes to + Misc/NEWS than to this file. + + * This is not a complete list of every single change; completeness + is the purpose of Misc/NEWS. Some changes I consider too small + or esoteric to include. If such a change is added to the text, + I'll just remove it. (This is another reason you shouldn't spend + too much time on writing your addition.) + + * If you want to draw your new text to the attention of the + maintainer, add 'XXX' to the beginning of the paragraph or + section. + + * It's OK to just add a fragmentary note about a change. For + example: "XXX Describe the transmogrify() function added to the + socket module." The maintainer will research the change and + write the necessary text. + + * You can comment out your additions if you like, but it's not + necessary (especially when a final release is some months away). + + * Credit the author of a patch or bugfix. Just the name is + sufficient; the e-mail address isn't necessary. + + * It's helpful to add the bug/patch number as a comment: + + XXX Describe the transmogrify() function added to the socket + module. + (Contributed by P.Y. Developer in :issue:`12345`.) + + This saves the maintainer the effort of going through the Mercurial log + when researching a change. + +This article explains the new features in Python 3.5, compared to 3.4. +Python 3.5 was released on September 13, 2015. See the +`changelog <https://docs.python.org/3.5/whatsnew/changelog.html>`_ for a full +list of changes. + +.. seealso:: + + :pep:`478` - Python 3.5 Release Schedule + + +Summary -- Release highlights +============================= + +New syntax features: + +* :ref:`PEP 492 <whatsnew-pep-492>`, coroutines with async and await syntax. +* :ref:`PEP 465 <whatsnew-pep-465>`, a new matrix multiplication operator: ``a @ b``. +* :ref:`PEP 448 <whatsnew-pep-448>`, additional unpacking generalizations. + + +New library modules: + +* :mod:`typing`: :ref:`PEP 484 -- Type Hints <whatsnew-pep-484>`. +* :mod:`zipapp`: :ref:`PEP 441 Improving Python ZIP Application Support + <whatsnew-zipapp>`. + + +New built-in features: + +* ``bytes % args``, ``bytearray % args``: :ref:`PEP 461 <whatsnew-pep-461>` -- + Adding ``%`` formatting to bytes and bytearray. + +* New :meth:`bytes.hex`, :meth:`bytearray.hex` and :meth:`memoryview.hex` + methods. (Contributed by Arnon Yaari in :issue:`9951`.) + +* :class:`memoryview` now supports tuple indexing (including multi-dimensional). + (Contributed by Antoine Pitrou in :issue:`23632`.) + +* Generators have a new ``gi_yieldfrom`` attribute, which returns the + object being iterated by ``yield from`` expressions. (Contributed + by Benno Leslie and Yury Selivanov in :issue:`24450`.) + +* A new :exc:`RecursionError` exception is now raised when maximum + recursion depth is reached. (Contributed by Georg Brandl + in :issue:`19235`.) + + +CPython implementation improvements: + +* When the ``LC_TYPE`` locale is the POSIX locale (``C`` locale), + :py:data:`sys.stdin` and :py:data:`sys.stdout` now use the + ``surrogateescape`` error handler, instead of the ``strict`` error handler. + (Contributed by Victor Stinner in :issue:`19977`.) + +* ``.pyo`` files are no longer used and have been replaced by a more flexible + scheme that includes the optimization level explicitly in ``.pyc`` name. + (See :ref:`PEP 488 overview <whatsnew-pep-488>`.) + +* Builtin and extension modules are now initialized in a multi-phase process, + which is similar to how Python modules are loaded. + (See :ref:`PEP 489 overview <whatsnew-pep-489>`.) + + +Significant improvements in the standard library: + +* :class:`collections.OrderedDict` is now + :ref:`implemented in C <whatsnew-ordereddict>`, which makes it + 4 to 100 times faster. + +* The :mod:`ssl` module gained + :ref:`support for Memory BIO <whatsnew-sslmemorybio>`, which decouples SSL + protocol handling from network IO. + +* The new :func:`os.scandir` function provides a + :ref:`better and significantly faster way <whatsnew-pep-471>` + of directory traversal. + +* :func:`functools.lru_cache` has been mostly + :ref:`reimplemented in C <whatsnew-lrucache>`, yielding much better + performance. + +* The new :func:`subprocess.run` function provides a + :ref:`streamlined way to run subprocesses <whatsnew-subprocess>`. + +* The :mod:`traceback` module has been significantly + :ref:`enhanced <whatsnew-traceback>` for improved + performance and developer convenience. + + +Security improvements: + +* SSLv3 is now disabled throughout the standard library. + It can still be enabled by instantiating a :class:`ssl.SSLContext` + manually. (See :issue:`22638` for more details; this change was + backported to CPython 3.4 and 2.7.) + +* HTTP cookie parsing is now stricter, in order to protect + against potential injection attacks. (Contributed by Antoine Pitrou + in :issue:`22796`.) + + +Windows improvements: + +* A new installer for Windows has replaced the old MSI. + See :ref:`using-on-windows` for more information. + +* Windows builds now use Microsoft Visual C++ 14.0, and extension modules + should use the same. + + +Please read on for a comprehensive list of user-facing changes, including many +other smaller improvements, CPython optimizations, deprecations, and potential +porting issues. + + +New Features +============ + +.. _whatsnew-pep-492: + +PEP 492 - Coroutines with async and await syntax +------------------------------------------------ + +:pep:`492` greatly improves support for asynchronous programming in Python +by adding :term:`awaitable objects <awaitable>`, +:term:`coroutine functions <coroutine function>`, +:term:`asynchronous iteration <asynchronous iterable>`, +and :term:`asynchronous context managers <asynchronous context manager>`. + +Coroutine functions are declared using the new :keyword:`async def` syntax:: + + >>> async def coro(): + ... return 'spam' + +Inside a coroutine function, the new :keyword:`await` expression can be used +to suspend coroutine execution until the result is available. Any object +can be *awaited*, as long as it implements the :term:`awaitable` protocol by +defining the :meth:`__await__` method. + +PEP 492 also adds :keyword:`async for` statement for convenient iteration +over asynchronous iterables. + +An example of a rudimentary HTTP client written using the new syntax:: + + import asyncio + + async def http_get(domain): + reader, writer = await asyncio.open_connection(domain, 80) + + writer.write(b'\r\n'.join([ + b'GET / HTTP/1.1', + b'Host: %b' % domain.encode('latin-1'), + b'Connection: close', + b'', b'' + ])) + + async for line in reader: + print('>>>', line) + + writer.close() + + loop = asyncio.get_event_loop() + try: + loop.run_until_complete(http_get('example.com')) + finally: + loop.close() + + +Similarly to asynchronous iteration, there is a new syntax for asynchronous +context managers. The following script:: + + import asyncio + + async def coro(name, lock): + print('coro {}: waiting for lock'.format(name)) + async with lock: + print('coro {}: holding the lock'.format(name)) + await asyncio.sleep(1) + print('coro {}: releasing the lock'.format(name)) + + loop = asyncio.get_event_loop() + lock = asyncio.Lock() + coros = asyncio.gather(coro(1, lock), coro(2, lock)) + try: + loop.run_until_complete(coros) + finally: + loop.close() + +will output:: + + coro 2: waiting for lock + coro 2: holding the lock + coro 1: waiting for lock + coro 2: releasing the lock + coro 1: holding the lock + coro 1: releasing the lock + +Note that both :keyword:`async for` and :keyword:`async with` can only +be used inside a coroutine function declared with :keyword:`async def`. + +Coroutine functions are intended to be run inside a compatible event loop, +such as the :ref:`asyncio loop <asyncio-event-loop>`. + + +.. note:: + + .. versionchanged:: 3.5.2 + Starting with CPython 3.5.2, ``__aiter__`` can directly return + :term:`asynchronous iterators <asynchronous iterator>`. Returning + an :term:`awaitable` object will result in a + :exc:`PendingDeprecationWarning`. + + See more details in the :ref:`async-iterators` documentation + section. + + +.. seealso:: + + :pep:`492` -- Coroutines with async and await syntax + PEP written and implemented by Yury Selivanov. + + +.. _whatsnew-pep-465: + +PEP 465 - A dedicated infix operator for matrix multiplication +-------------------------------------------------------------- + +:pep:`465` adds the ``@`` infix operator for matrix multiplication. +Currently, no builtin Python types implement the new operator, however, it +can be implemented by defining :meth:`__matmul__`, :meth:`__rmatmul__`, +and :meth:`__imatmul__` for regular, reflected, and in-place matrix +multiplication. The semantics of these methods is similar to that of +methods defining other infix arithmetic operators. + +Matrix multiplication is a notably common operation in many fields of +mathematics, science, engineering, and the addition of ``@`` allows writing +cleaner code:: + + S = (H @ beta - r).T @ inv(H @ V @ H.T) @ (H @ beta - r) + +instead of:: + + S = dot((dot(H, beta) - r).T, + dot(inv(dot(dot(H, V), H.T)), dot(H, beta) - r)) + +NumPy 1.10 has support for the new operator:: + + >>> import numpy + + >>> x = numpy.ones(3) + >>> x + array([ 1., 1., 1.]) + + >>> m = numpy.eye(3) + >>> m + array([[ 1., 0., 0.], + [ 0., 1., 0.], + [ 0., 0., 1.]]) + + >>> x @ m + array([ 1., 1., 1.]) + + +.. seealso:: + + :pep:`465` -- A dedicated infix operator for matrix multiplication + PEP written by Nathaniel J. Smith; implemented by Benjamin Peterson. + + +.. _whatsnew-pep-448: + +PEP 448 - Additional Unpacking Generalizations +---------------------------------------------- + +:pep:`448` extends the allowed uses of the ``*`` iterable unpacking +operator and ``**`` dictionary unpacking operator. It is now possible +to use an arbitrary number of unpackings in :ref:`function calls <calls>`:: + + >>> print(*[1], *[2], 3, *[4, 5]) + 1 2 3 4 5 + + >>> def fn(a, b, c, d): + ... print(a, b, c, d) + ... + + >>> fn(**{'a': 1, 'c': 3}, **{'b': 2, 'd': 4}) + 1 2 3 4 + +Similarly, tuple, list, set, and dictionary displays allow multiple +unpackings (see :ref:`exprlists` and :ref:`dict`):: + + >>> *range(4), 4 + (0, 1, 2, 3, 4) + + >>> [*range(4), 4] + [0, 1, 2, 3, 4] + + >>> {*range(4), 4, *(5, 6, 7)} + {0, 1, 2, 3, 4, 5, 6, 7} + + >>> {'x': 1, **{'y': 2}} + {'x': 1, 'y': 2} + +.. seealso:: + + :pep:`448` -- Additional Unpacking Generalizations + PEP written by Joshua Landau; implemented by Neil Girdhar, + Thomas Wouters, and Joshua Landau. + + +.. _whatsnew-pep-461: + +PEP 461 - percent formatting support for bytes and bytearray +------------------------------------------------------------ + +:pep:`461` adds support for the ``%`` +:ref:`interpolation operator <bytes-formatting>` to :class:`bytes` +and :class:`bytearray`. + +While interpolation is usually thought of as a string operation, there are +cases where interpolation on ``bytes`` or ``bytearrays`` makes sense, and the +work needed to make up for this missing functionality detracts from the +overall readability of the code. This issue is particularly important when +dealing with wire format protocols, which are often a mixture of binary +and ASCII compatible text. + +Examples:: + + >>> b'Hello %b!' % b'World' + b'Hello World!' + + >>> b'x=%i y=%f' % (1, 2.5) + b'x=1 y=2.500000' + +Unicode is not allowed for ``%b``, but it is accepted by ``%a`` (equivalent of +``repr(obj).encode('ascii', 'backslashreplace')``):: + + >>> b'Hello %b!' % 'World' + Traceback (most recent call last): + File "<stdin>", line 1, in <module> + TypeError: %b requires bytes, or an object that implements __bytes__, not 'str' + + >>> b'price: %a' % '10€' + b"price: '10\\u20ac'" + +Note that ``%s`` and ``%r`` conversion types, although supported, should +only be used in codebases that need compatibility with Python 2. + +.. seealso:: + + :pep:`461` -- Adding % formatting to bytes and bytearray + PEP written by Ethan Furman; implemented by Neil Schemenauer and + Ethan Furman. + + +.. _whatsnew-pep-484: + +PEP 484 - Type Hints +-------------------- + +Function annotation syntax has been a Python feature since version 3.0 +(:pep:`3107`), however the semantics of annotations has been left undefined. + +Experience has shown that the majority of function annotation +uses were to provide type hints to function parameters and return values. It +became evident that it would be beneficial for Python users, if the +standard library included the base definitions and tools for type annotations. + +:pep:`484` introduces a :term:`provisional module <provisional api>` to +provide these standard definitions and tools, along with some conventions +for situations where annotations are not available. + +For example, here is a simple function whose argument and return type +are declared in the annotations:: + + def greeting(name: str) -> str: + return 'Hello ' + name + +While these annotations are available at runtime through the usual +:attr:`__annotations__` attribute, *no automatic type checking happens at +runtime*. Instead, it is assumed that a separate off-line type checker +(e.g. `mypy <http://mypy-lang.org>`_) will be used for on-demand +source code analysis. + +The type system supports unions, generic types, and a special type +named :class:`~typing.Any` which is consistent with (i.e. assignable to +and from) all types. + +.. seealso:: + + * :mod:`typing` module documentation + * :pep:`484` -- Type Hints + PEP written by Guido van Rossum, Jukka Lehtosalo, and Łukasz Langa; + implemented by Guido van Rossum. + * :pep:`483` -- The Theory of Type Hints + PEP written by Guido van Rossum + + +.. _whatsnew-pep-471: + +PEP 471 - os.scandir() function -- a better and faster directory iterator +------------------------------------------------------------------------- + +:pep:`471` adds a new directory iteration function, :func:`os.scandir`, +to the standard library. Additionally, :func:`os.walk` is now +implemented using ``scandir``, which makes it 3 to 5 times faster +on POSIX systems and 7 to 20 times faster on Windows systems. This is +largely achieved by greatly reducing the number of calls to :func:`os.stat` +required to walk a directory tree. + +Additionally, ``scandir`` returns an iterator, as opposed to returning +a list of file names, which improves memory efficiency when iterating +over very large directories. + +The following example shows a simple use of :func:`os.scandir` to display all +the files (excluding directories) in the given *path* that don't start with +``'.'``. The :meth:`entry.is_file() <os.DirEntry.is_file>` call will generally +not make an additional system call:: + + for entry in os.scandir(path): + if not entry.name.startswith('.') and entry.is_file(): + print(entry.name) + +.. seealso:: + + :pep:`471` -- os.scandir() function -- a better and faster directory iterator + PEP written and implemented by Ben Hoyt with the help of Victor Stinner. + + +.. _whatsnew-pep-475: + +PEP 475: Retry system calls failing with EINTR +---------------------------------------------- + +An :py:data:`errno.EINTR` error code is returned whenever a system call, that +is waiting for I/O, is interrupted by a signal. Previously, Python would +raise :exc:`InterruptedError` in such cases. This meant that, when writing a +Python application, the developer had two choices: + +#. Ignore the ``InterruptedError``. +#. Handle the ``InterruptedError`` and attempt to restart the interrupted + system call at every call site. + +The first option makes an application fail intermittently. +The second option adds a large amount of boilerplate that makes the +code nearly unreadable. Compare:: + + print("Hello World") + +and:: + + while True: + try: + print("Hello World") + break + except InterruptedError: + continue + +:pep:`475` implements automatic retry of system calls on +``EINTR``. This removes the burden of dealing with ``EINTR`` +or :exc:`InterruptedError` in user code in most situations and makes +Python programs, including the standard library, more robust. Note that +the system call is only retried if the signal handler does not raise an +exception. + +Below is a list of functions which are now retried when interrupted +by a signal: + +* :func:`open` and :func:`io.open`; + +* functions of the :mod:`faulthandler` module; + +* :mod:`os` functions: :func:`~os.fchdir`, :func:`~os.fchmod`, + :func:`~os.fchown`, :func:`~os.fdatasync`, :func:`~os.fstat`, + :func:`~os.fstatvfs`, :func:`~os.fsync`, :func:`~os.ftruncate`, + :func:`~os.mkfifo`, :func:`~os.mknod`, :func:`~os.open`, + :func:`~os.posix_fadvise`, :func:`~os.posix_fallocate`, :func:`~os.pread`, + :func:`~os.pwrite`, :func:`~os.read`, :func:`~os.readv`, :func:`~os.sendfile`, + :func:`~os.wait3`, :func:`~os.wait4`, :func:`~os.wait`, + :func:`~os.waitid`, :func:`~os.waitpid`, :func:`~os.write`, + :func:`~os.writev`; + +* special cases: :func:`os.close` and :func:`os.dup2` now ignore + :py:data:`~errno.EINTR` errors; the syscall is not retried (see the PEP + for the rationale); + +* :mod:`select` functions: :func:`devpoll.poll() <select.devpoll.poll>`, + :func:`epoll.poll() <select.epoll.poll>`, + :func:`kqueue.control() <select.kqueue.control>`, + :func:`poll.poll() <select.poll.poll>`, :func:`~select.select`; + +* methods of the :class:`~socket.socket` class: :meth:`~socket.socket.accept`, + :meth:`~socket.socket.connect` (except for non-blocking sockets), + :meth:`~socket.socket.recv`, :meth:`~socket.socket.recvfrom`, + :meth:`~socket.socket.recvmsg`, :meth:`~socket.socket.send`, + :meth:`~socket.socket.sendall`, :meth:`~socket.socket.sendmsg`, + :meth:`~socket.socket.sendto`; + +* :func:`signal.sigtimedwait` and :func:`signal.sigwaitinfo`; + +* :func:`time.sleep`. + +.. seealso:: + + :pep:`475` -- Retry system calls failing with EINTR + PEP and implementation written by Charles-François Natali and + Victor Stinner, with the help of Antoine Pitrou (the French connection). + + +.. _whatsnew-pep-479: + +PEP 479: Change StopIteration handling inside generators +-------------------------------------------------------- + +The interaction of generators and :exc:`StopIteration` in Python 3.4 and +earlier was sometimes surprising, and could conceal obscure bugs. Previously, +``StopIteration`` raised accidentally inside a generator function was +interpreted as the end of the iteration by the loop construct driving the +generator. + +:pep:`479` changes the behavior of generators: when a ``StopIteration`` +exception is raised inside a generator, it is replaced with a +:exc:`RuntimeError` before it exits the generator frame. The main goal of +this change is to ease debugging in the situation where an unguarded +:func:`next` call raises ``StopIteration`` and causes the iteration controlled +by the generator to terminate silently. This is particularly pernicious in +combination with the ``yield from`` construct. + +This is a backwards incompatible change, so to enable the new behavior, +a :term:`__future__` import is necessary:: + + >>> from __future__ import generator_stop + + >>> def gen(): + ... next(iter([])) + ... yield + ... + >>> next(gen()) + Traceback (most recent call last): + File "<stdin>", line 2, in gen + StopIteration + + The above exception was the direct cause of the following exception: + + Traceback (most recent call last): + File "<stdin>", line 1, in <module> + RuntimeError: generator raised StopIteration + +Without a ``__future__`` import, a :exc:`PendingDeprecationWarning` will be +raised whenever a ``StopIteration`` exception is raised inside a generator. + +.. seealso:: + + :pep:`479` -- Change StopIteration handling inside generators + PEP written by Chris Angelico and Guido van Rossum. Implemented by + Chris Angelico, Yury Selivanov and Nick Coghlan. + + +.. _whatsnew-pep-485: + +PEP 485: A function for testing approximate equality +---------------------------------------------------- + +:pep:`485` adds the :func:`math.isclose` and :func:`cmath.isclose` +functions which tell whether two values are approximately equal or +"close" to each other. Whether or not two values are considered +close is determined according to given absolute and relative tolerances. +Relative tolerance is the maximum allowed difference between ``isclose`` +arguments, relative to the larger absolute value:: + + >>> import math + >>> a = 5.0 + >>> b = 4.99998 + >>> math.isclose(a, b, rel_tol=1e-5) + True + >>> math.isclose(a, b, rel_tol=1e-6) + False + +It is also possible to compare two values using absolute tolerance, which +must be a non-negative value:: + + >>> import math + >>> a = 5.0 + >>> b = 4.99998 + >>> math.isclose(a, b, abs_tol=0.00003) + True + >>> math.isclose(a, b, abs_tol=0.00001) + False + +.. seealso:: + + :pep:`485` -- A function for testing approximate equality + PEP written by Christopher Barker; implemented by Chris Barker and + Tal Einat. + + +.. _whatsnew-pep-486: + +PEP 486: Make the Python Launcher aware of virtual environments +--------------------------------------------------------------- + +:pep:`486` makes the Windows launcher (see :pep:`397`) aware of an active +virtual environment. When the default interpreter would be used and the +``VIRTUAL_ENV`` environment variable is set, the interpreter in the virtual +environment will be used. + +.. seealso:: + + :pep:`486` -- Make the Python Launcher aware of virtual environments + PEP written and implemented by Paul Moore. + + +.. _whatsnew-pep-488: + +PEP 488: Elimination of PYO files +--------------------------------- + +:pep:`488` does away with the concept of ``.pyo`` files. This means that +``.pyc`` files represent both unoptimized and optimized bytecode. To prevent the +need to constantly regenerate bytecode files, ``.pyc`` files now have an +optional ``opt-`` tag in their name when the bytecode is optimized. This has the +side-effect of no more bytecode file name clashes when running under either +:option:`-O` or :option:`-OO`. Consequently, bytecode files generated from +:option:`-O`, and :option:`-OO` may now exist simultaneously. +:func:`importlib.util.cache_from_source` has an updated API to help with +this change. + +.. seealso:: + + :pep:`488` -- Elimination of PYO files + PEP written and implemented by Brett Cannon. + + +.. _whatsnew-pep-489: + +PEP 489: Multi-phase extension module initialization +---------------------------------------------------- + +:pep:`489` updates extension module initialization to take advantage of the +two step module loading mechanism introduced by :pep:`451` in Python 3.4. + +This change brings the import semantics of extension modules that opt-in to +using the new mechanism much closer to those of Python source and bytecode +modules, including the ability to use any valid identifier as a module name, +rather than being restricted to ASCII. + +.. seealso:: + + :pep:`489` -- Multi-phase extension module initialization + PEP written by Petr Viktorin, Stefan Behnel, and Nick Coghlan; + implemented by Petr Viktorin. + + +Other Language Changes +====================== + +Some smaller changes made to the core Python language are: + +* Added the ``"namereplace"`` error handlers. The ``"backslashreplace"`` + error handlers now work with decoding and translating. + (Contributed by Serhiy Storchaka in :issue:`19676` and :issue:`22286`.) + +* The :option:`-b` option now affects comparisons of :class:`bytes` with + :class:`int`. (Contributed by Serhiy Storchaka in :issue:`23681`.) + +* New Kazakh ``kz1048`` and Tajik ``koi8_t`` :ref:`codecs <standard-encodings>`. + (Contributed by Serhiy Storchaka in :issue:`22682` and :issue:`22681`.) + +* Property docstrings are now writable. This is especially useful for + :func:`collections.namedtuple` docstrings. + (Contributed by Berker Peksag in :issue:`24064`.) + +* Circular imports involving relative imports are now supported. + (Contributed by Brett Cannon and Antoine Pitrou in :issue:`17636`.) + + +New Modules +=========== + +typing +------ + +The new :mod:`typing` :term:`provisional <provisional api>` module +provides standard definitions and tools for function type annotations. +See :ref:`Type Hints <whatsnew-pep-484>` for more information. + +.. _whatsnew-zipapp: + +zipapp +------ + +The new :mod:`zipapp` module (specified in :pep:`441`) provides an API and +command line tool for creating executable Python Zip Applications, which +were introduced in Python 2.6 in :issue:`1739468`, but which were not well +publicized, either at the time or since. + +With the new module, bundling your application is as simple as putting all +the files, including a ``__main__.py`` file, into a directory ``myapp`` +and running: + +.. code-block:: shell-session + + $ python -m zipapp myapp + $ python myapp.pyz + +The module implementation has been contributed by Paul Moore in +:issue:`23491`. + +.. seealso:: + + :pep:`441` -- Improving Python ZIP Application Support + + +Improved Modules +================ + +argparse +-------- + +The :class:`~argparse.ArgumentParser` class now allows disabling +:ref:`abbreviated usage <prefix-matching>` of long options by setting +:ref:`allow_abbrev` to ``False``. (Contributed by Jonathan Paugh, +Steven Bethard, paul j3 and Daniel Eriksson in :issue:`14910`.) + + +asyncio +------- + +Since the :mod:`asyncio` module is :term:`provisional <provisional api>`, +all changes introduced in Python 3.5 have also been backported to Python 3.4.x. + +Notable changes in the :mod:`asyncio` module since Python 3.4.0: + +* New debugging APIs: :meth:`loop.set_debug() <asyncio.BaseEventLoop.set_debug>` + and :meth:`loop.get_debug() <asyncio.BaseEventLoop.get_debug>` methods. + (Contributed by Victor Stinner.) + +* The proactor event loop now supports SSL. + (Contributed by Antoine Pitrou and Victor Stinner in :issue:`22560`.) + +* A new :meth:`loop.is_closed() <asyncio.BaseEventLoop.is_closed>` method to + check if the event loop is closed. + (Contributed by Victor Stinner in :issue:`21326`.) + +* A new :meth:`loop.create_task() <asyncio.BaseEventLoop.create_task>` + to conveniently create and schedule a new :class:`~asyncio.Task` + for a coroutine. The ``create_task`` method is also used by all + asyncio functions that wrap coroutines into tasks, such as + :func:`asyncio.wait`, :func:`asyncio.gather`, etc. + (Contributed by Victor Stinner.) + +* A new :meth:`transport.get_write_buffer_limits() <asyncio.WriteTransport.get_write_buffer_limits>` + method to inquire for *high-* and *low-* water limits of the flow + control. + (Contributed by Victor Stinner.) + +* The :func:`~asyncio.async` function is deprecated in favor of + :func:`~asyncio.ensure_future`. + (Contributed by Yury Selivanov.) + +* New :meth:`loop.set_task_factory() <asyncio.BaseEventLoop.set_task_factory>` + and :meth:`loop.set_task_factory() <asyncio.BaseEventLoop.get_task_factory>` + methods to customize the task factory that + :meth:`loop.create_task() <asyncio.BaseEventLoop.create_task>` method uses. + (Contributed by Yury Selivanov.) + +* New :meth:`Queue.join() <asyncio.Queue.join>` and + :meth:`Queue.task_done() <asyncio.Queue.task_done>` queue methods. + (Contributed by Victor Stinner.) + +* The ``JoinableQueue`` class was removed, in favor of the + :class:`asyncio.Queue` class. + (Contributed by Victor Stinner.) + +Updates in 3.5.1: + +* The :func:`~asyncio.ensure_future` function and all functions that + use it, such as :meth:`loop.run_until_complete() <asyncio.BaseEventLoop.run_until_complete>`, + now accept all kinds of :term:`awaitable objects <awaitable>`. + (Contributed by Yury Selivanov.) + +* New :func:`~asyncio.run_coroutine_threadsafe` function to submit + coroutines to event loops from other threads. + (Contributed by Vincent Michel.) + +* New :meth:`Transport.is_closing() <asyncio.BaseTransport.is_closing>` + method to check if the transport is closing or closed. + (Contributed by Yury Selivanov.) + +* The :meth:`loop.create_server() <asyncio.BaseEventLoop.create_server>` + method can now accept a list of hosts. + (Contributed by Yann Sionneau.) + +Updates in 3.5.2: + +* New :meth:`loop.create_future() <asyncio.BaseEventLoop.create_future>` + method to create Future objects. This allows alternative event + loop implementations, such as + `uvloop <https://github.com/MagicStack/uvloop>`_, to provide a faster + :class:`asyncio.Future` implementation. + (Contributed by Yury Selivanov.) + +* New :meth:`loop.get_exception_handler() <asyncio.BaseEventLoop.get_exception_handler>` + method to get the current exception handler. + (Contributed by Yury Selivanov.) + +* New :meth:`StreamReader.readuntil() <asyncio.StreamReader.readuntil>` + method to read data from the stream until a separator bytes + sequence appears. + (Contributed by Mark Korenberg.) + +* The :meth:`loop.create_connection() <asyncio.BaseEventLoop.create_connection>` + and :meth:`loop.create_server() <asyncio.BaseEventLoop.create_server>` + methods are optimized to avoid calling the system ``getaddrinfo`` + function if the address is already resolved. + (Contributed by A. Jesse Jiryu Davis.) + +* The :meth:`loop.sock_connect(sock, address) <asyncio.BaseEventLoop.sock_connect>` + no longer requires the *address* to be resolved prior to the call. + (Contributed by A. Jesse Jiryu Davis.) + + +bz2 +--- + +The :meth:`BZ2Decompressor.decompress <bz2.BZ2Decompressor.decompress>` +method now accepts an optional *max_length* argument to limit the maximum +size of decompressed data. (Contributed by Nikolaus Rath in :issue:`15955`.) + + +cgi +--- + +The :class:`~cgi.FieldStorage` class now supports the :term:`context manager` +protocol. (Contributed by Berker Peksag in :issue:`20289`.) + + +cmath +----- + +A new function :func:`~cmath.isclose` provides a way to test for approximate +equality. (Contributed by Chris Barker and Tal Einat in :issue:`24270`.) + + +code +---- + +The :func:`InteractiveInterpreter.showtraceback() <code.InteractiveInterpreter.showtraceback>` +method now prints the full chained traceback, just like the interactive +interpreter. (Contributed by Claudiu Popa in :issue:`17442`.) + + +collections +----------- + +.. _whatsnew-ordereddict: + +The :class:`~collections.OrderedDict` class is now implemented in C, which +makes it 4 to 100 times faster. (Contributed by Eric Snow in :issue:`16991`.) + +:meth:`OrderedDict.items() <collections.OrderedDict.items>`, +:meth:`OrderedDict.keys() <collections.OrderedDict.keys>`, +:meth:`OrderedDict.values() <collections.OrderedDict.values>` views now support +:func:`reversed` iteration. +(Contributed by Serhiy Storchaka in :issue:`19505`.) + +The :class:`~collections.deque` class now defines +:meth:`~collections.deque.index`, :meth:`~collections.deque.insert`, and +:meth:`~collections.deque.copy`, and supports the ``+`` and ``*`` operators. +This allows deques to be recognized as a :class:`~collections.abc.MutableSequence` +and improves their substitutability for lists. +(Contributed by Raymond Hettinger in :issue:`23704`.) + +Docstrings produced by :func:`~collections.namedtuple` can now be updated:: + + Point = namedtuple('Point', ['x', 'y']) + Point.__doc__ += ': Cartesian coodinate' + Point.x.__doc__ = 'abscissa' + Point.y.__doc__ = 'ordinate' + +(Contributed by Berker Peksag in :issue:`24064`.) + +The :class:`~collections.UserString` class now implements the +:meth:`__getnewargs__`, :meth:`__rmod__`, :meth:`~str.casefold`, +:meth:`~str.format_map`, :meth:`~str.isprintable`, and :meth:`~str.maketrans` +methods to match the corresponding methods of :class:`str`. +(Contributed by Joe Jevnik in :issue:`22189`.) + + +collections.abc +--------------- + +The :meth:`Sequence.index() <collections.abc.Sequence.index>` method now +accepts *start* and *stop* arguments to match the corresponding methods +of :class:`tuple`, :class:`list`, etc. +(Contributed by Devin Jeanpierre in :issue:`23086`.) + +A new :class:`~collections.abc.Generator` abstract base class. (Contributed +by Stefan Behnel in :issue:`24018`.) + +New :class:`~collections.abc.Awaitable`, :class:`~collections.abc.Coroutine`, +:class:`~collections.abc.AsyncIterator`, and +:class:`~collections.abc.AsyncIterable` abstract base classes. +(Contributed by Yury Selivanov in :issue:`24184`.) + +For earlier Python versions, a backport of the new ABCs is available in an +external `PyPI package <https://pypi.python.org/pypi/backports_abc>`_. + + +compileall +---------- + +A new :mod:`compileall` option, :samp:`-j {N}`, allows running *N* workers +simultaneously to perform parallel bytecode compilation. +The :func:`~compileall.compile_dir` function has a corresponding ``workers`` +parameter. (Contributed by Claudiu Popa in :issue:`16104`.) + +Another new option, ``-r``, allows controlling the maximum recursion +level for subdirectories. (Contributed by Claudiu Popa in :issue:`19628`.) + +The ``-q`` command line option can now be specified more than once, in +which case all output, including errors, will be suppressed. The corresponding +``quiet`` parameter in :func:`~compileall.compile_dir`, +:func:`~compileall.compile_file`, and :func:`~compileall.compile_path` can now +accept an integer value indicating the level of output suppression. +(Contributed by Thomas Kluyver in :issue:`21338`.) + + +concurrent.futures +------------------ + +The :meth:`Executor.map() <concurrent.futures.Executor.map>` method now accepts a +*chunksize* argument to allow batching of tasks to improve performance when +:meth:`~concurrent.futures.ProcessPoolExecutor` is used. +(Contributed by Dan O'Reilly in :issue:`11271`.) + +The number of workers in the :class:`~concurrent.futures.ThreadPoolExecutor` +constructor is optional now. The default value is 5 times the number of CPUs. +(Contributed by Claudiu Popa in :issue:`21527`.) + + +configparser +------------ + +:mod:`configparser` now provides a way to customize the conversion +of values by specifying a dictionary of converters in the +:class:`~configparser.ConfigParser` constructor, or by defining them +as methods in ``ConfigParser`` subclasses. Converters defined in +a parser instance are inherited by its section proxies. + +Example:: + + >>> import configparser + >>> conv = {} + >>> conv['list'] = lambda v: [e.strip() for e in v.split() if e.strip()] + >>> cfg = configparser.ConfigParser(converters=conv) + >>> cfg.read_string(""" + ... [s] + ... list = a b c d e f g + ... """) + >>> cfg.get('s', 'list') + 'a b c d e f g' + >>> cfg.getlist('s', 'list') + ['a', 'b', 'c', 'd', 'e', 'f', 'g'] + >>> section = cfg['s'] + >>> section.getlist('list') + ['a', 'b', 'c', 'd', 'e', 'f', 'g'] + +(Contributed by Łukasz Langa in :issue:`18159`.) + + +contextlib +---------- + +The new :func:`~contextlib.redirect_stderr` :term:`context manager` (similar to +:func:`~contextlib.redirect_stdout`) makes it easier for utility scripts to +handle inflexible APIs that write their output to :data:`sys.stderr` and +don't provide any options to redirect it:: + + >>> import contextlib, io, logging + >>> f = io.StringIO() + >>> with contextlib.redirect_stderr(f): + ... logging.warning('warning') + ... + >>> f.getvalue() + 'WARNING:root:warning\n' + +(Contributed by Berker Peksag in :issue:`22389`.) + + +csv +--- + +The :meth:`~csv.csvwriter.writerow` method now supports arbitrary iterables, +not just sequences. (Contributed by Serhiy Storchaka in :issue:`23171`.) + + +curses +------ + +The new :func:`~curses.update_lines_cols` function updates the :envvar:`LINES` +and :envvar:`COLS` environment variables. This is useful for detecting +manual screen resizing. (Contributed by Arnon Yaari in :issue:`4254`.) + + +dbm +--- + +:func:`dumb.open <dbm.dumb.open>` always creates a new database when the flag +has the value ``"n"``. (Contributed by Claudiu Popa in :issue:`18039`.) + + +difflib +------- + +The charset of HTML documents generated by +:meth:`HtmlDiff.make_file() <difflib.HtmlDiff.make_file>` +can now be customized by using a new *charset* keyword-only argument. +The default charset of HTML document changed from ``"ISO-8859-1"`` +to ``"utf-8"``. +(Contributed by Berker Peksag in :issue:`2052`.) + +The :func:`~difflib.diff_bytes` function can now compare lists of byte +strings. This fixes a regression from Python 2. +(Contributed by Terry J. Reedy and Greg Ward in :issue:`17445`.) + + +distutils +--------- + +Both the ``build`` and ``build_ext`` commands now accept a ``-j`` option to +enable parallel building of extension modules. +(Contributed by Antoine Pitrou in :issue:`5309`.) + +The :mod:`distutils` module now supports ``xz`` compression, and can be +enabled by passing ``xztar`` as an argument to ``bdist --format``. +(Contributed by Serhiy Storchaka in :issue:`16314`.) + + +doctest +------- + +The :func:`~doctest.DocTestSuite` function returns an empty +:class:`unittest.TestSuite` if *module* contains no docstrings, instead of +raising :exc:`ValueError`. (Contributed by Glenn Jones in :issue:`15916`.) + + +email +----- + +A new policy option :attr:`Policy.mangle_from_ <email.policy.Policy.mangle_from_>` +controls whether or not lines that start with ``"From "`` in email bodies are +prefixed with a ``">"`` character by generators. The default is ``True`` for +:attr:`~email.policy.compat32` and ``False`` for all other policies. +(Contributed by Milan Oberkirch in :issue:`20098`.) + +A new +:meth:`Message.get_content_disposition() <email.message.Message.get_content_disposition>` +method provides easy access to a canonical value for the +:mailheader:`Content-Disposition` header. +(Contributed by Abhilash Raj in :issue:`21083`.) + +A new policy option :attr:`EmailPolicy.utf8 <email.policy.EmailPolicy.utf8>` +can be set to ``True`` to encode email headers using the UTF-8 charset instead +of using encoded words. This allows ``Messages`` to be formatted according to +:rfc:`6532` and used with an SMTP server that supports the :rfc:`6531` +``SMTPUTF8`` extension. (Contributed by R. David Murray in +:issue:`24211`.) + +The :class:`mime.text.MIMEText <email.mime.text.MIMEText>` constructor now +accepts a :class:`charset.Charset <email.charset.Charset>` instance. +(Contributed by Claude Paroz and Berker Peksag in :issue:`16324`.) + + +enum +---- + +The :class:`~enum.Enum` callable has a new parameter *start* to +specify the initial number of enum values if only *names* are provided:: + + >>> Animal = enum.Enum('Animal', 'cat dog', start=10) + >>> Animal.cat + <Animal.cat: 10> + >>> Animal.dog + <Animal.dog: 11> + +(Contributed by Ethan Furman in :issue:`21706`.) + + +faulthandler +------------ + +The :func:`~faulthandler.enable`, :func:`~faulthandler.register`, +:func:`~faulthandler.dump_traceback` and +:func:`~faulthandler.dump_traceback_later` functions now accept file +descriptors in addition to file-like objects. +(Contributed by Wei Wu in :issue:`23566`.) + + +functools +--------- + +.. _whatsnew-lrucache: + +Most of the :func:`~functools.lru_cache` machinery is now implemented in C, making +it significantly faster. (Contributed by Matt Joiner, Alexey Kachayev, and +Serhiy Storchaka in :issue:`14373`.) + + +glob +---- + +The :func:`~glob.iglob` and :func:`~glob.glob` functions now support recursive +search in subdirectories, using the ``"**"`` pattern. +(Contributed by Serhiy Storchaka in :issue:`13968`.) + + +gzip +---- + +The *mode* argument of the :class:`~gzip.GzipFile` constructor now +accepts ``"x"`` to request exclusive creation. +(Contributed by Tim Heaney in :issue:`19222`.) + + +heapq +----- + +Element comparison in :func:`~heapq.merge` can now be customized by +passing a :term:`key function` in a new optional *key* keyword argument, +and a new optional *reverse* keyword argument can be used to reverse element +comparison:: + + >>> import heapq + >>> a = ['9', '777', '55555'] + >>> b = ['88', '6666'] + >>> list(heapq.merge(a, b, key=len)) + ['9', '88', '777', '6666', '55555'] + >>> list(heapq.merge(reversed(a), reversed(b), key=len, reverse=True)) + ['55555', '6666', '777', '88', '9'] + +(Contributed by Raymond Hettinger in :issue:`13742`.) + + +http +---- + +A new :class:`HTTPStatus <http.HTTPStatus>` enum that defines a set of +HTTP status codes, reason phrases and long descriptions written in English. +(Contributed by Demian Brecht in :issue:`21793`.) + + +http.client +----------- + +:meth:`HTTPConnection.getresponse() <http.client.HTTPConnection.getresponse>` +now raises a :exc:`~http.client.RemoteDisconnected` exception when a +remote server connection is closed unexpectedly. Additionally, if a +:exc:`ConnectionError` (of which ``RemoteDisconnected`` +is a subclass) is raised, the client socket is now closed automatically, +and will reconnect on the next request:: + + import http.client + conn = http.client.HTTPConnection('www.python.org') + for retries in range(3): + try: + conn.request('GET', '/') + resp = conn.getresponse() + except http.client.RemoteDisconnected: + pass + +(Contributed by Martin Panter in :issue:`3566`.) + + +idlelib and IDLE +---------------- + +Since idlelib implements the IDLE shell and editor and is not intended for +import by other programs, it gets improvements with every release. See +:file:`Lib/idlelib/NEWS.txt` for a cumulative list of changes since 3.4.0, +as well as changes made in future 3.5.x releases. This file is also available +from the IDLE :menuselection:`Help --> About IDLE` dialog. + + +imaplib +------- + +The :class:`~imaplib.IMAP4` class now supports the :term:`context manager` protocol. +When used in a :keyword:`with` statement, the IMAP4 ``LOGOUT`` +command will be called automatically at the end of the block. +(Contributed by Tarek Ziadé and Serhiy Storchaka in :issue:`4972`.) + +The :mod:`imaplib` module now supports :rfc:`5161` (ENABLE Extension) +and :rfc:`6855` (UTF-8 Support) via the :meth:`IMAP4.enable() <imaplib.IMAP4.enable>` +method. A new :attr:`IMAP4.utf8_enabled <imaplib.IMAP4.utf8_enabled>` +attribute tracks whether or not :rfc:`6855` support is enabled. +(Contributed by Milan Oberkirch, R. David Murray, and Maciej Szulik in +:issue:`21800`.) + +The :mod:`imaplib` module now automatically encodes non-ASCII string usernames +and passwords using UTF-8, as recommended by the RFCs. (Contributed by Milan +Oberkirch in :issue:`21800`.) + + +imghdr +------ + +The :func:`~imghdr.what` function now recognizes the +`OpenEXR <http://www.openexr.com>`_ format +(contributed by Martin Vignali and Claudiu Popa in :issue:`20295`), +and the `WebP <https://en.wikipedia.org/wiki/WebP>`_ format +(contributed by Fabrice Aneche and Claudiu Popa in :issue:`20197`.) + + +importlib +--------- + +The :class:`util.LazyLoader <importlib.util.LazyLoader>` class allows for +lazy loading of modules in applications where startup time is important. +(Contributed by Brett Cannon in :issue:`17621`.) + +The :func:`abc.InspectLoader.source_to_code() <importlib.abc.InspectLoader.source_to_code>` +method is now a static method. This makes it easier to initialize a module +object with code compiled from a string by running +``exec(code, module.__dict__)``. +(Contributed by Brett Cannon in :issue:`21156`.) + +The new :func:`util.module_from_spec() <importlib.util.module_from_spec>` +function is now the preferred way to create a new module. As opposed to +creating a :class:`types.ModuleType` instance directly, this new function +will set the various import-controlled attributes based on the passed-in +spec object. (Contributed by Brett Cannon in :issue:`20383`.) + + +inspect +------- + +Both the :class:`~inspect.Signature` and :class:`~inspect.Parameter` classes are +now picklable and hashable. (Contributed by Yury Selivanov in :issue:`20726` +and :issue:`20334`.) + +A new +:meth:`BoundArguments.apply_defaults() <inspect.BoundArguments.apply_defaults>` +method provides a way to set default values for missing arguments:: + + >>> def foo(a, b='ham', *args): pass + >>> ba = inspect.signature(foo).bind('spam') + >>> ba.apply_defaults() + >>> ba.arguments + OrderedDict([('a', 'spam'), ('b', 'ham'), ('args', ())]) + +(Contributed by Yury Selivanov in :issue:`24190`.) + +A new class method +:meth:`Signature.from_callable() <inspect.Signature.from_callable>` makes +subclassing of :class:`~inspect.Signature` easier. (Contributed +by Yury Selivanov and Eric Snow in :issue:`17373`.) + +The :func:`~inspect.signature` function now accepts a *follow_wrapped* +optional keyword argument, which, when set to ``False``, disables automatic +following of ``__wrapped__`` links. +(Contributed by Yury Selivanov in :issue:`20691`.) + +A set of new functions to inspect +:term:`coroutine functions <coroutine function>` and +:term:`coroutine objects <coroutine>` has been added: +:func:`~inspect.iscoroutine`, :func:`~inspect.iscoroutinefunction`, +:func:`~inspect.isawaitable`, :func:`~inspect.getcoroutinelocals`, +and :func:`~inspect.getcoroutinestate`. +(Contributed by Yury Selivanov in :issue:`24017` and :issue:`24400`.) + +The :func:`~inspect.stack`, :func:`~inspect.trace`, +:func:`~inspect.getouterframes`, and :func:`~inspect.getinnerframes` +functions now return a list of named tuples. +(Contributed by Daniel Shahaf in :issue:`16808`.) + + +io +-- + +A new :meth:`BufferedIOBase.readinto1() <io.BufferedIOBase.readinto1>` +method, that uses at most one call to the underlying raw stream's +:meth:`RawIOBase.read() <io.RawIOBase.read>` or +:meth:`RawIOBase.readinto() <io.RawIOBase.readinto>` methods. +(Contributed by Nikolaus Rath in :issue:`20578`.) + + +ipaddress +--------- + +Both the :class:`~ipaddress.IPv4Network` and :class:`~ipaddress.IPv6Network` classes +now accept an ``(address, netmask)`` tuple argument, so as to easily construct +network objects from existing addresses:: + + >>> import ipaddress + >>> ipaddress.IPv4Network(('127.0.0.0', 8)) + IPv4Network('127.0.0.0/8') + >>> ipaddress.IPv4Network(('127.0.0.0', '255.0.0.0')) + IPv4Network('127.0.0.0/8') + +(Contributed by Peter Moody and Antoine Pitrou in :issue:`16531`.) + +A new :attr:`~ipaddress.IPv4Network.reverse_pointer` attribute for the +:class:`~ipaddress.IPv4Network` and :class:`~ipaddress.IPv6Network` classes +returns the name of the reverse DNS PTR record:: + + >>> import ipaddress + >>> addr = ipaddress.IPv4Address('127.0.0.1') + >>> addr.reverse_pointer + '1.0.0.127.in-addr.arpa' + >>> addr6 = ipaddress.IPv6Address('::1') + >>> addr6.reverse_pointer + '1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa' + +(Contributed by Leon Weber in :issue:`20480`.) + + +json +---- + +The :mod:`json.tool` command line interface now preserves the order of keys in +JSON objects passed in input. The new ``--sort-keys`` option can be used +to sort the keys alphabetically. (Contributed by Berker Peksag +in :issue:`21650`.) + +JSON decoder now raises :exc:`~json.JSONDecodeError` instead of +:exc:`ValueError` to provide better context information about the error. +(Contributed by Serhiy Storchaka in :issue:`19361`.) + + +linecache +--------- + +A new :func:`~linecache.lazycache` function can be used to capture information +about a non-file-based module to permit getting its lines later via +:func:`~linecache.getline`. This avoids doing I/O until a line is actually +needed, without having to carry the module globals around indefinitely. +(Contributed by Robert Collins in :issue:`17911`.) + + +locale +------ + +A new :func:`~locale.delocalize` function can be used to convert a string into +a normalized number string, taking the ``LC_NUMERIC`` settings into account:: + + >>> import locale + >>> locale.setlocale(locale.LC_NUMERIC, 'de_DE.UTF-8') + 'de_DE.UTF-8' + >>> locale.delocalize('1.234,56') + '1234.56' + >>> locale.setlocale(locale.LC_NUMERIC, 'en_US.UTF-8') + 'en_US.UTF-8' + >>> locale.delocalize('1,234.56') + '1234.56' + +(Contributed by Cédric Krier in :issue:`13918`.) + + +logging +------- + +All logging methods (:class:`~logging.Logger` :meth:`~logging.Logger.log`, +:meth:`~logging.Logger.exception`, :meth:`~logging.Logger.critical`, +:meth:`~logging.Logger.debug`, etc.), now accept exception instances +as an *exc_info* argument, in addition to boolean values and exception +tuples:: + + >>> import logging + >>> try: + ... 1/0 + ... except ZeroDivisionError as ex: + ... logging.error('exception', exc_info=ex) + ERROR:root:exception + +(Contributed by Yury Selivanov in :issue:`20537`.) + +The :class:`handlers.HTTPHandler <logging.handlers.HTTPHandler>` class now +accepts an optional :class:`ssl.SSLContext` instance to configure SSL +settings used in an HTTP connection. +(Contributed by Alex Gaynor in :issue:`22788`.) + +The :class:`handlers.QueueListener <logging.handlers.QueueListener>` class now +takes a *respect_handler_level* keyword argument which, if set to ``True``, +will pass messages to handlers taking handler levels into account. +(Contributed by Vinay Sajip.) + + +lzma +---- + +The :meth:`LZMADecompressor.decompress() <lzma.LZMADecompressor.decompress>` +method now accepts an optional *max_length* argument to limit the maximum +size of decompressed data. +(Contributed by Martin Panter in :issue:`15955`.) + + +math +---- + +Two new constants have been added to the :mod:`math` module: :data:`~math.inf` +and :data:`~math.nan`. (Contributed by Mark Dickinson in :issue:`23185`.) + +A new function :func:`~math.isclose` provides a way to test for approximate +equality. (Contributed by Chris Barker and Tal Einat in :issue:`24270`.) + +A new :func:`~math.gcd` function has been added. The :func:`fractions.gcd` +function is now deprecated. (Contributed by Mark Dickinson and Serhiy +Storchaka in :issue:`22486`.) + + +multiprocessing +--------------- + +:func:`sharedctypes.synchronized() <multiprocessing.sharedctypes.synchronized>` +objects now support the :term:`context manager` protocol. +(Contributed by Charles-François Natali in :issue:`21565`.) + + +operator +-------- + +:func:`~operator.attrgetter`, :func:`~operator.itemgetter`, +and :func:`~operator.methodcaller` objects now support pickling. +(Contributed by Josh Rosenberg and Serhiy Storchaka in :issue:`22955`.) + +New :func:`~operator.matmul` and :func:`~operator.imatmul` functions +to perform matrix multiplication. +(Contributed by Benjamin Peterson in :issue:`21176`.) + + +os +-- + +The new :func:`~os.scandir` function returning an iterator of +:class:`~os.DirEntry` objects has been added. If possible, :func:`~os.scandir` +extracts file attributes while scanning a directory, removing the need to +perform subsequent system calls to determine file type or attributes, which may +significantly improve performance. (Contributed by Ben Hoyt with the help +of Victor Stinner in :issue:`22524`.) + +On Windows, a new +:attr:`stat_result.st_file_attributes <os.stat_result.st_file_attributes>` +attribute is now available. It corresponds to the ``dwFileAttributes`` member +of the ``BY_HANDLE_FILE_INFORMATION`` structure returned by +``GetFileInformationByHandle()``. (Contributed by Ben Hoyt in :issue:`21719`.) + +The :func:`~os.urandom` function now uses the ``getrandom()`` syscall on Linux 3.17 +or newer, and ``getentropy()`` on OpenBSD 5.6 and newer, removing the need to +use ``/dev/urandom`` and avoiding failures due to potential file descriptor +exhaustion. (Contributed by Victor Stinner in :issue:`22181`.) + +New :func:`~os.get_blocking` and :func:`~os.set_blocking` functions allow +getting and setting a file descriptor's blocking mode (:data:`~os.O_NONBLOCK`.) +(Contributed by Victor Stinner in :issue:`22054`.) + +The :func:`~os.truncate` and :func:`~os.ftruncate` functions are now supported +on Windows. (Contributed by Steve Dower in :issue:`23668`.) + +There is a new :func:`os.path.commonpath` function returning the longest +common sub-path of each passed pathname. Unlike the +:func:`os.path.commonprefix` function, it always returns a valid +path:: + + >>> os.path.commonprefix(['/usr/lib', '/usr/local/lib']) + '/usr/l' + + >>> os.path.commonpath(['/usr/lib', '/usr/local/lib']) + '/usr' + +(Contributed by Rafik Draoui and Serhiy Storchaka in :issue:`10395`.) + + +pathlib +------- + +The new :meth:`Path.samefile() <pathlib.Path.samefile>` method can be used +to check whether the path points to the same file as another path, which can +be either another :class:`~pathlib.Path` object, or a string:: + + >>> import pathlib + >>> p1 = pathlib.Path('/etc/hosts') + >>> p2 = pathlib.Path('/etc/../etc/hosts') + >>> p1.samefile(p2) + True + +(Contributed by Vajrasky Kok and Antoine Pitrou in :issue:`19775`.) + +The :meth:`Path.mkdir() <pathlib.Path.mkdir>` method now accepts a new optional +*exist_ok* argument to match ``mkdir -p`` and :func:`os.makedirs` +functionality. (Contributed by Berker Peksag in :issue:`21539`.) + +There is a new :meth:`Path.expanduser() <pathlib.Path.expanduser>` method to +expand ``~`` and ``~user`` prefixes. (Contributed by Serhiy Storchaka and +Claudiu Popa in :issue:`19776`.) + +A new :meth:`Path.home() <pathlib.Path.home>` class method can be used to get +a :class:`~pathlib.Path` instance representing the user’s home +directory. +(Contributed by Victor Salgado and Mayank Tripathi in :issue:`19777`.) + +New :meth:`Path.write_text() <pathlib.Path.write_text>`, +:meth:`Path.read_text() <pathlib.Path.read_text>`, +:meth:`Path.write_bytes() <pathlib.Path.write_bytes>`, +:meth:`Path.read_bytes() <pathlib.Path.read_bytes>` methods to simplify +read/write operations on files. + +The following code snippet will create or rewrite existing file +``~/spam42``:: + + >>> import pathlib + >>> p = pathlib.Path('~/spam42') + >>> p.expanduser().write_text('ham') + 3 + +(Contributed by Christopher Welborn in :issue:`20218`.) + + +pickle +------ + +Nested objects, such as unbound methods or nested classes, can now be pickled +using :ref:`pickle protocols <pickle-protocols>` older than protocol version 4. +Protocol version 4 already supports these cases. (Contributed by Serhiy +Storchaka in :issue:`23611`.) + + +poplib +------ + +A new :meth:`POP3.utf8() <poplib.POP3.utf8>` command enables :rfc:`6856` +(Internationalized Email) support, if a POP server supports it. +(Contributed by Milan OberKirch in :issue:`21804`.) + + +re +-- + +References and conditional references to groups with fixed length are now +allowed in lookbehind assertions:: + + >>> import re + >>> pat = re.compile(r'(a|b).(?<=\1)c') + >>> pat.match('aac') + <_sre.SRE_Match object; span=(0, 3), match='aac'> + >>> pat.match('bbc') + <_sre.SRE_Match object; span=(0, 3), match='bbc'> + +(Contributed by Serhiy Storchaka in :issue:`9179`.) + +The number of capturing groups in regular expressions is no longer limited to +100. (Contributed by Serhiy Storchaka in :issue:`22437`.) + +The :func:`~re.sub` and :func:`~re.subn` functions now replace unmatched +groups with empty strings instead of raising an exception. +(Contributed by Serhiy Storchaka in :issue:`1519638`.) + +The :class:`re.error` exceptions have new attributes, +:attr:`~re.error.msg`, :attr:`~re.error.pattern`, +:attr:`~re.error.pos`, :attr:`~re.error.lineno`, +and :attr:`~re.error.colno`, that provide better context +information about the error:: + + >>> re.compile(""" + ... (?x) + ... .++ + ... """) + Traceback (most recent call last): + ... + sre_constants.error: multiple repeat at position 16 (line 3, column 7) + +(Contributed by Serhiy Storchaka in :issue:`22578`.) + + +readline +-------- + +A new :func:`~readline.append_history_file` function can be used to append +the specified number of trailing elements in history to the given file. +(Contributed by Bruno Cauet in :issue:`22940`.) + + +selectors +--------- + +The new :class:`~selectors.DevpollSelector` supports efficient +``/dev/poll`` polling on Solaris. +(Contributed by Giampaolo Rodola' in :issue:`18931`.) + + +shutil +------ + +The :func:`~shutil.move` function now accepts a *copy_function* argument, +allowing, for example, the :func:`~shutil.copy` function to be used instead of +the default :func:`~shutil.copy2` if there is a need to ignore file metadata +when moving. +(Contributed by Claudiu Popa in :issue:`19840`.) + +The :func:`~shutil.make_archive` function now supports the *xztar* format. +(Contributed by Serhiy Storchaka in :issue:`5411`.) + + +signal +------ + +On Windows, the :func:`~signal.set_wakeup_fd` function now also supports +socket handles. (Contributed by Victor Stinner in :issue:`22018`.) + +Various ``SIG*`` constants in the :mod:`signal` module have been converted into +:mod:`Enums <enum>`. This allows meaningful names to be printed +during debugging, instead of integer "magic numbers". +(Contributed by Giampaolo Rodola' in :issue:`21076`.) + + +smtpd +----- + +Both the :class:`~smtpd.SMTPServer` and :class:`~smtpd.SMTPChannel` classes now +accept a *decode_data* keyword argument to determine if the ``DATA`` portion of +the SMTP transaction is decoded using the ``"utf-8"`` codec or is instead +provided to the +:meth:`SMTPServer.process_message() <smtpd.SMTPServer.process_message>` +method as a byte string. The default is ``True`` for backward compatibility +reasons, but will change to ``False`` in Python 3.6. If *decode_data* is set +to ``False``, the ``process_message`` method must be prepared to accept keyword +arguments. +(Contributed by Maciej Szulik in :issue:`19662`.) + +The :class:`~smtpd.SMTPServer` class now advertises the ``8BITMIME`` extension +(:rfc:`6152`) if *decode_data* has been set ``True``. If the client +specifies ``BODY=8BITMIME`` on the ``MAIL`` command, it is passed to +:meth:`SMTPServer.process_message() <smtpd.SMTPServer.process_message>` +via the *mail_options* keyword. +(Contributed by Milan Oberkirch and R. David Murray in :issue:`21795`.) + +The :class:`~smtpd.SMTPServer` class now also supports the ``SMTPUTF8`` +extension (:rfc:`6531`: Internationalized Email). If the client specified +``SMTPUTF8 BODY=8BITMIME`` on the ``MAIL`` command, they are passed to +:meth:`SMTPServer.process_message() <smtpd.SMTPServer.process_message>` +via the *mail_options* keyword. It is the responsibility of the +``process_message`` method to correctly handle the ``SMTPUTF8`` data. +(Contributed by Milan Oberkirch in :issue:`21725`.) + +It is now possible to provide, directly or via name resolution, IPv6 +addresses in the :class:`~smtpd.SMTPServer` constructor, and have it +successfully connect. (Contributed by Milan Oberkirch in :issue:`14758`.) + + +smtplib +------- + +A new :meth:`SMTP.auth() <smtplib.SMTP.auth>` method provides a convenient way to +implement custom authentication mechanisms. (Contributed by Milan +Oberkirch in :issue:`15014`.) + +The :meth:`SMTP.set_debuglevel() <smtplib.SMTP.set_debuglevel>` method now +accepts an additional debuglevel (2), which enables timestamps in debug +messages. (Contributed by Gavin Chappell and Maciej Szulik in :issue:`16914`.) + +Both the :meth:`SMTP.sendmail() <smtplib.SMTP.sendmail>` and +:meth:`SMTP.send_message() <smtplib.SMTP.send_message>` methods now +support :rfc:`6531` (SMTPUTF8). +(Contributed by Milan Oberkirch and R. David Murray in :issue:`22027`.) + + +sndhdr +------ + +The :func:`~sndhdr.what` and :func:`~sndhdr.whathdr` functions now return +a :func:`~collections.namedtuple`. (Contributed by Claudiu Popa in +:issue:`18615`.) + + +socket +------ + +Functions with timeouts now use a monotonic clock, instead of a system clock. +(Contributed by Victor Stinner in :issue:`22043`.) + +A new :meth:`socket.sendfile() <socket.socket.sendfile>` method allows +sending a file over a socket by using the high-performance :func:`os.sendfile` +function on UNIX, resulting in uploads being from 2 to 3 times faster than when +using plain :meth:`socket.send() <socket.socket.send>`. +(Contributed by Giampaolo Rodola' in :issue:`17552`.) + +The :meth:`socket.sendall() <socket.socket.sendall>` method no longer resets the +socket timeout every time bytes are received or sent. The socket timeout is +now the maximum total duration to send all data. +(Contributed by Victor Stinner in :issue:`23853`.) + +The *backlog* argument of the :meth:`socket.listen() <socket.socket.listen>` +method is now optional. By default it is set to +:data:`SOMAXCONN <socket.SOMAXCONN>` or to ``128``, whichever is less. +(Contributed by Charles-François Natali in :issue:`21455`.) + + +ssl +--- + +.. _whatsnew-sslmemorybio: + +Memory BIO Support +~~~~~~~~~~~~~~~~~~ + +(Contributed by Geert Jansen in :issue:`21965`.) + +The new :class:`~ssl.SSLObject` class has been added to provide SSL protocol +support for cases when the network I/O capabilities of :class:`~ssl.SSLSocket` +are not necessary or are suboptimal. ``SSLObject`` represents +an SSL protocol instance, but does not implement any network I/O methods, and +instead provides a memory buffer interface. The new :class:`~ssl.MemoryBIO` +class can be used to pass data between Python and an SSL protocol instance. + +The memory BIO SSL support is primarily intended to be used in frameworks +implementing asynchronous I/O for which :class:`~ssl.SSLSocket`'s readiness +model ("select/poll") is inefficient. + +A new :meth:`SSLContext.wrap_bio() <ssl.SSLContext.wrap_bio>` method can be used +to create a new ``SSLObject`` instance. + + +Application-Layer Protocol Negotiation Support +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +(Contributed by Benjamin Peterson in :issue:`20188`.) + +Where OpenSSL support is present, the :mod:`ssl` module now implements +the *Application-Layer Protocol Negotiation* TLS extension as described +in :rfc:`7301`. + +The new :meth:`SSLContext.set_alpn_protocols() <ssl.SSLContext.set_alpn_protocols>` +can be used to specify which protocols a socket should advertise during +the TLS handshake. + +The new +:meth:`SSLSocket.selected_alpn_protocol() <ssl.SSLSocket.selected_alpn_protocol>` +returns the protocol that was selected during the TLS handshake. +The :data:`~ssl.HAS_ALPN` flag indicates whether ALPN support is present. + + +Other Changes +~~~~~~~~~~~~~ + +There is a new :meth:`SSLSocket.version() <ssl.SSLSocket.version>` method to +query the actual protocol version in use. +(Contributed by Antoine Pitrou in :issue:`20421`.) + +The :class:`~ssl.SSLSocket` class now implements +a :meth:`SSLSocket.sendfile() <ssl.SSLSocket.sendfile>` method. +(Contributed by Giampaolo Rodola' in :issue:`17552`.) + +The :meth:`SSLSocket.send() <ssl.SSLSocket.send>` method now raises either +the :exc:`ssl.SSLWantReadError` or :exc:`ssl.SSLWantWriteError` exception on a +non-blocking socket if the operation would block. Previously, it would return +``0``. (Contributed by Nikolaus Rath in :issue:`20951`.) + +The :func:`~ssl.cert_time_to_seconds` function now interprets the input time +as UTC and not as local time, per :rfc:`5280`. Additionally, the return +value is always an :class:`int`. (Contributed by Akira Li in :issue:`19940`.) + +New :meth:`SSLObject.shared_ciphers() <ssl.SSLObject.shared_ciphers>` and +:meth:`SSLSocket.shared_ciphers() <ssl.SSLSocket.shared_ciphers>` methods return +the list of ciphers sent by the client during the handshake. +(Contributed by Benjamin Peterson in :issue:`23186`.) + +The :meth:`SSLSocket.do_handshake() <ssl.SSLSocket.do_handshake>`, +:meth:`SSLSocket.read() <ssl.SSLSocket.read>`, +:meth:`SSLSocket.shutdown() <ssl.SSLSocket.shutdown>`, and +:meth:`SSLSocket.write() <ssl.SSLSocket.write>` methods of the :class:`~ssl.SSLSocket` +class no longer reset the socket timeout every time bytes are received or sent. +The socket timeout is now the maximum total duration of the method. +(Contributed by Victor Stinner in :issue:`23853`.) + +The :func:`~ssl.match_hostname` function now supports matching of IP addresses. +(Contributed by Antoine Pitrou in :issue:`23239`.) + + +sqlite3 +------- + +The :class:`~sqlite3.Row` class now fully supports the sequence protocol, +in particular :func:`reversed` iteration and slice indexing. +(Contributed by Claudiu Popa in :issue:`10203`; by Lucas Sinclair, +Jessica McKellar, and Serhiy Storchaka in :issue:`13583`.) + + +.. _whatsnew-subprocess: + +subprocess +---------- + +The new :func:`~subprocess.run` function has been added. +It runs the specified command and returns a +:class:`~subprocess.CompletedProcess` object, which describes a finished +process. The new API is more consistent and is the recommended approach +to invoking subprocesses in Python code that does not need to maintain +compatibility with earlier Python versions. +(Contributed by Thomas Kluyver in :issue:`23342`.) + +Examples:: + + >>> subprocess.run(["ls", "-l"]) # doesn't capture output + CompletedProcess(args=['ls', '-l'], returncode=0) + + >>> subprocess.run("exit 1", shell=True, check=True) + Traceback (most recent call last): + ... + subprocess.CalledProcessError: Command 'exit 1' returned non-zero exit status 1 + + >>> subprocess.run(["ls", "-l", "/dev/null"], stdout=subprocess.PIPE) + CompletedProcess(args=['ls', '-l', '/dev/null'], returncode=0, + stdout=b'crw-rw-rw- 1 root root 1, 3 Jan 23 16:23 /dev/null\n') + + +sys +--- + +A new :func:`~sys.set_coroutine_wrapper` function allows setting a global +hook that will be called whenever a :term:`coroutine object <coroutine>` +is created by an :keyword:`async def` function. A corresponding +:func:`~sys.get_coroutine_wrapper` can be used to obtain a currently set +wrapper. Both functions are :term:`provisional <provisional api>`, +and are intended for debugging purposes only. (Contributed by Yury Selivanov +in :issue:`24017`.) + +A new :func:`~sys.is_finalizing` function can be used to check if the Python +interpreter is :term:`shutting down <interpreter shutdown>`. +(Contributed by Antoine Pitrou in :issue:`22696`.) + + +sysconfig +--------- + +The name of the user scripts directory on Windows now includes the first +two components of the Python version. (Contributed by Paul Moore +in :issue:`23437`.) + + +tarfile +------- + +The *mode* argument of the :func:`~tarfile.open` function now accepts ``"x"`` +to request exclusive creation. (Contributed by Berker Peksag in :issue:`21717`.) + +The :meth:`TarFile.extractall() <tarfile.TarFile.extractall>` and +:meth:`TarFile.extract() <tarfile.TarFile.extract>` methods now take a keyword +argument *numeric_only*. If set to ``True``, the extracted files and +directories will be owned by the numeric ``uid`` and ``gid`` from the tarfile. +If set to ``False`` (the default, and the behavior in versions prior to 3.5), +they will be owned by the named user and group in the tarfile. +(Contributed by Michael Vogt and Eric Smith in :issue:`23193`.) + +The :meth:`TarFile.list() <tarfile.TarFile.list>` now accepts an optional +*members* keyword argument that can be set to a subset of the list returned +by :meth:`TarFile.getmembers() <tarfile.TarFile.getmembers>`. +(Contributed by Serhiy Storchaka in :issue:`21549`.) + + +threading +--------- + +Both the :meth:`Lock.acquire() <threading.Lock.acquire>` and +:meth:`RLock.acquire() <threading.RLock.acquire>` methods +now use a monotonic clock for timeout management. +(Contributed by Victor Stinner in :issue:`22043`.) + + +time +---- + +The :func:`~time.monotonic` function is now always available. +(Contributed by Victor Stinner in :issue:`22043`.) + + +timeit +------ + +A new command line option ``-u`` or :samp:`--unit={U}` can be used to specify the time +unit for the timer output. Supported options are ``usec``, ``msec``, +or ``sec``. (Contributed by Julian Gindi in :issue:`18983`.) + +The :func:`~timeit.timeit` function has a new *globals* parameter for +specifying the namespace in which the code will be running. +(Contributed by Ben Roberts in :issue:`2527`.) + + +tkinter +------- + +The :mod:`tkinter._fix` module used for setting up the Tcl/Tk environment +on Windows has been replaced by a private function in the :mod:`_tkinter` +module which makes no permanent changes to environment variables. +(Contributed by Zachary Ware in :issue:`20035`.) + + +.. _whatsnew-traceback: + +traceback +--------- + +New :func:`~traceback.walk_stack` and :func:`~traceback.walk_tb` +functions to conveniently traverse frame and traceback objects. +(Contributed by Robert Collins in :issue:`17911`.) + +New lightweight classes: :class:`~traceback.TracebackException`, +:class:`~traceback.StackSummary`, and :class:`~traceback.FrameSummary`. +(Contributed by Robert Collins in :issue:`17911`.) + +Both the :func:`~traceback.print_tb` and :func:`~traceback.print_stack` functions +now support negative values for the *limit* argument. +(Contributed by Dmitry Kazakov in :issue:`22619`.) + + +types +----- + +A new :func:`~types.coroutine` function to transform +:term:`generator <generator iterator>` and +:class:`generator-like <collections.abc.Generator>` objects into +:term:`awaitables <awaitable>`. +(Contributed by Yury Selivanov in :issue:`24017`.) + +A new type called :class:`~types.CoroutineType`, which is used for +:term:`coroutine` objects created by :keyword:`async def` functions. +(Contributed by Yury Selivanov in :issue:`24400`.) + + +unicodedata +----------- + +The :mod:`unicodedata` module now uses data from `Unicode 8.0.0 +<http://unicode.org/versions/Unicode8.0.0/>`_. + + +unittest +-------- + +The :meth:`TestLoader.loadTestsFromModule() <unittest.TestLoader.loadTestsFromModule>` +method now accepts a keyword-only argument *pattern* which is passed to +``load_tests`` as the third argument. Found packages are now checked for +``load_tests`` regardless of whether their path matches *pattern*, because it +is impossible for a package name to match the default pattern. +(Contributed by Robert Collins and Barry A. Warsaw in :issue:`16662`.) + +Unittest discovery errors now are exposed in the +:data:`TestLoader.errors <unittest.TestLoader.errors>` attribute of the +:class:`~unittest.TestLoader` instance. +(Contributed by Robert Collins in :issue:`19746`.) + +A new command line option ``--locals`` to show local variables in +tracebacks. (Contributed by Robert Collins in :issue:`22936`.) + + +unittest.mock +------------- + +The :class:`~unittest.mock.Mock` class has the following improvements: + +* The class constructor has a new *unsafe* parameter, which causes mock + objects to raise :exc:`AttributeError` on attribute names starting + with ``"assert"``. + (Contributed by Kushal Das in :issue:`21238`.) + +* A new :meth:`Mock.assert_not_called() <unittest.mock.Mock.assert_not_called>` + method to check if the mock object was called. + (Contributed by Kushal Das in :issue:`21262`.) + +The :class:`~unittest.mock.MagicMock` class now supports :meth:`__truediv__`, +:meth:`__divmod__` and :meth:`__matmul__` operators. +(Contributed by Johannes Baiter in :issue:`20968`, and Håkan Lövdahl +in :issue:`23581` and :issue:`23568`.) + +It is no longer necessary to explicitly pass ``create=True`` to the +:func:`~unittest.mock.patch` function when patching builtin names. +(Contributed by Kushal Das in :issue:`17660`.) + + +urllib +------ + +A new +:class:`request.HTTPPasswordMgrWithPriorAuth <urllib.request.HTTPPasswordMgrWithPriorAuth>` +class allows HTTP Basic Authentication credentials to be managed so as to +eliminate unnecessary ``401`` response handling, or to unconditionally send +credentials on the first request in order to communicate with servers that +return a ``404`` response instead of a ``401`` if the ``Authorization`` header +is not sent. (Contributed by Matej Cepl in :issue:`19494` and Akshit Khurana in +:issue:`7159`.) + +A new *quote_via* argument for the +:func:`parse.urlencode() <urllib.parse.urlencode>` +function provides a way to control the encoding of query parts if needed. +(Contributed by Samwyse and Arnon Yaari in :issue:`13866`.) + +The :func:`request.urlopen() <urllib.request.urlopen>` function accepts an +:class:`ssl.SSLContext` object as a *context* argument, which will be used for +the HTTPS connection. (Contributed by Alex Gaynor in :issue:`22366`.) + +The :func:`parse.urljoin() <urllib.parse.urljoin>` was updated to use the +:rfc:`3986` semantics for the resolution of relative URLs, rather than +:rfc:`1808` and :rfc:`2396`. +(Contributed by Demian Brecht and Senthil Kumaran in :issue:`22118`.) + + +wsgiref +------- + +The *headers* argument of the :class:`headers.Headers <wsgiref.headers.Headers>` +class constructor is now optional. +(Contributed by Pablo Torres Navarrete and SilentGhost in :issue:`5800`.) + + +xmlrpc +------ + +The :class:`client.ServerProxy <xmlrpc.client.ServerProxy>` class now supports +the :term:`context manager` protocol. +(Contributed by Claudiu Popa in :issue:`20627`.) + +The :class:`client.ServerProxy <xmlrpc.client.ServerProxy>` constructor now accepts +an optional :class:`ssl.SSLContext` instance. +(Contributed by Alex Gaynor in :issue:`22960`.) + + +xml.sax +------- + +SAX parsers now support a character stream of the +:class:`xmlreader.InputSource <xml.sax.xmlreader.InputSource>` object. +(Contributed by Serhiy Storchaka in :issue:`2175`.) + +:func:`~xml.sax.parseString` now accepts a :class:`str` instance. +(Contributed by Serhiy Storchaka in :issue:`10590`.) + + +zipfile +------- + +ZIP output can now be written to unseekable streams. +(Contributed by Serhiy Storchaka in :issue:`23252`.) + +The *mode* argument of :meth:`ZipFile.open() <zipfile.ZipFile.open>` method now +accepts ``"x"`` to request exclusive creation. +(Contributed by Serhiy Storchaka in :issue:`21717`.) + + +Other module-level changes +========================== + +Many functions in the :mod:`mmap`, :mod:`ossaudiodev`, :mod:`socket`, +:mod:`ssl`, and :mod:`codecs` modules now accept writable +:term:`bytes-like objects <bytes-like object>`. +(Contributed by Serhiy Storchaka in :issue:`23001`.) + + +Optimizations +============= + +The :func:`os.walk` function has been sped up by 3 to 5 times on POSIX systems, +and by 7 to 20 times on Windows. This was done using the new :func:`os.scandir` +function, which exposes file information from the underlying ``readdir`` or +``FindFirstFile``/``FindNextFile`` system calls. (Contributed by +Ben Hoyt with help from Victor Stinner in :issue:`23605`.) + +Construction of ``bytes(int)`` (filled by zero bytes) is faster and uses less +memory for large objects. ``calloc()`` is used instead of ``malloc()`` to +allocate memory for these objects. +(Contributed by Victor Stinner in :issue:`21233`.) + +Some operations on :mod:`ipaddress` :class:`~ipaddress.IPv4Network` and +:class:`~ipaddress.IPv6Network` have been massively sped up, such as +:meth:`~ipaddress.IPv4Network.subnets`, :meth:`~ipaddress.IPv4Network.supernet`, +:func:`~ipaddress.summarize_address_range`, :func:`~ipaddress.collapse_addresses`. +The speed up can range from 3 to 15 times. +(Contributed by Antoine Pitrou, Michel Albert, and Markus in +:issue:`21486`, :issue:`21487`, :issue:`20826`, :issue:`23266`.) + +Pickling of :mod:`ipaddress` objects was optimized to produce significantly +smaller output. (Contributed by Serhiy Storchaka in :issue:`23133`.) + +Many operations on :class:`io.BytesIO` are now 50% to 100% faster. +(Contributed by Serhiy Storchaka in :issue:`15381` and David Wilson in +:issue:`22003`.) + +The :func:`marshal.dumps` function is now faster: 65-85% with versions 3 +and 4, 20-25% with versions 0 to 2 on typical data, and up to 5 times in +best cases. +(Contributed by Serhiy Storchaka in :issue:`20416` and :issue:`23344`.) + +The UTF-32 encoder is now 3 to 7 times faster. +(Contributed by Serhiy Storchaka in :issue:`15027`.) + +Regular expressions are now parsed up to 10% faster. +(Contributed by Serhiy Storchaka in :issue:`19380`.) + +The :func:`json.dumps` function was optimized to run with +``ensure_ascii=False`` as fast as with ``ensure_ascii=True``. +(Contributed by Naoki Inada in :issue:`23206`.) + +The :c:func:`PyObject_IsInstance` and :c:func:`PyObject_IsSubclass` +functions have been sped up in the common case that the second argument +has :class:`type` as its metaclass. +(Contributed Georg Brandl by in :issue:`22540`.) + +Method caching was slightly improved, yielding up to 5% performance +improvement in some benchmarks. +(Contributed by Antoine Pitrou in :issue:`22847`.) + +Objects from the :mod:`random` module now use 50% less memory on 64-bit +builds. (Contributed by Serhiy Storchaka in :issue:`23488`.) + +The :func:`property` getter calls are up to 25% faster. +(Contributed by Joe Jevnik in :issue:`23910`.) + +Instantiation of :class:`fractions.Fraction` is now up to 30% faster. +(Contributed by Stefan Behnel in :issue:`22464`.) + +String methods :meth:`~str.find`, :meth:`~str.rfind`, :meth:`~str.split`, +:meth:`~str.partition` and the :keyword:`in` string operator are now significantly +faster for searching 1-character substrings. +(Contributed by Serhiy Storchaka in :issue:`23573`.) + + +Build and C API Changes +======================= + +New ``calloc`` functions were added: + +* :c:func:`PyMem_RawCalloc`, +* :c:func:`PyMem_Calloc`, +* :c:func:`PyObject_Calloc`, +* :c:func:`_PyObject_GC_Calloc`. + +(Contributed by Victor Stinner in :issue:`21233`.) + +New encoding/decoding helper functions: + +* :c:func:`Py_DecodeLocale` (replaced ``_Py_char2wchar()``), +* :c:func:`Py_EncodeLocale` (replaced ``_Py_wchar2char()``). + +(Contributed by Victor Stinner in :issue:`18395`.) + +A new :c:func:`PyCodec_NameReplaceErrors` function to replace the unicode +encode error with ``\N{...}`` escapes. +(Contributed by Serhiy Storchaka in :issue:`19676`.) + +A new :c:func:`PyErr_FormatV` function similar to :c:func:`PyErr_Format`, +but accepts a ``va_list`` argument. +(Contributed by Antoine Pitrou in :issue:`18711`.) + +A new :c:data:`PyExc_RecursionError` exception. +(Contributed by Georg Brandl in :issue:`19235`.) + +New :c:func:`PyModule_FromDefAndSpec`, :c:func:`PyModule_FromDefAndSpec2`, +and :c:func:`PyModule_ExecDef` functions introduced by :pep:`489` -- +multi-phase extension module initialization. +(Contributed by Petr Viktorin in :issue:`24268`.) + +New :c:func:`PyNumber_MatrixMultiply` and +:c:func:`PyNumber_InPlaceMatrixMultiply` functions to perform matrix +multiplication. +(Contributed by Benjamin Peterson in :issue:`21176`. See also :pep:`465` +for details.) + +The :c:member:`PyTypeObject.tp_finalize` slot is now part of the stable ABI. + +Windows builds now require Microsoft Visual C++ 14.0, which +is available as part of `Visual Studio 2015 <https://www.visualstudio.com/>`_. + +Extension modules now include a platform information tag in their filename on +some platforms (the tag is optional, and CPython will import extensions without +it, although if the tag is present and mismatched, the extension won't be +loaded): + +* On Linux, extension module filenames end with + ``.cpython-<major><minor>m-<architecture>-<os>.pyd``: + + * ``<major>`` is the major number of the Python version; + for Python 3.5 this is ``3``. + + * ``<minor>`` is the minor number of the Python version; + for Python 3.5 this is ``5``. + + * ``<architecture>`` is the hardware architecture the extension module + was built to run on. It's most commonly either ``i386`` for 32-bit Intel + platforms or ``x86_64`` for 64-bit Intel (and AMD) platforms. + + * ``<os>`` is always ``linux-gnu``, except for extensions built to + talk to the 32-bit ABI on 64-bit platforms, in which case it is + ``linux-gnu32`` (and ``<architecture>`` will be ``x86_64``). + +* On Windows, extension module filenames end with + ``<debug>.cp<major><minor>-<platform>.pyd``: + + * ``<major>`` is the major number of the Python version; + for Python 3.5 this is ``3``. + + * ``<minor>`` is the minor number of the Python version; + for Python 3.5 this is ``5``. + + * ``<platform>`` is the platform the extension module was built for, + either ``win32`` for Win32, ``win_amd64`` for Win64, ``win_ia64`` for + Windows Itanium 64, and ``win_arm`` for Windows on ARM. + + * If built in debug mode, ``<debug>`` will be ``_d``, + otherwise it will be blank. + +* On OS X platforms, extension module filenames now end with ``-darwin.so``. + +* On all other platforms, extension module filenames are the same as they were + with Python 3.4. + + +Deprecated +========== + +New Keywords +------------ + +``async`` and ``await`` are not recommended to be used as variable, class, +function or module names. Introduced by :pep:`492` in Python 3.5, they will +become proper keywords in Python 3.7. + + +Deprecated Python Behavior +-------------------------- + +Raising the :exc:`StopIteration` exception inside a generator will now generate a silent +:exc:`PendingDeprecationWarning`, which will become a non-silent deprecation +warning in Python 3.6 and will trigger a :exc:`RuntimeError` in Python 3.7. +See :ref:`PEP 479: Change StopIteration handling inside generators <whatsnew-pep-479>` +for details. + + +Unsupported Operating Systems +----------------------------- + +Windows XP is no longer supported by Microsoft, thus, per :PEP:`11`, CPython +3.5 is no longer officially supported on this OS. + + +Deprecated Python modules, functions and methods +------------------------------------------------ + +The :mod:`formatter` module has now graduated to full deprecation and is still +slated for removal in Python 3.6. + +The :func:`asyncio.async` function is deprecated in favor of +:func:`~asyncio.ensure_future`. + +The :mod:`smtpd` module has in the past always decoded the DATA portion of +email messages using the ``utf-8`` codec. This can now be controlled by the +new *decode_data* keyword to :class:`~smtpd.SMTPServer`. The default value is +``True``, but this default is deprecated. Specify the *decode_data* keyword +with an appropriate value to avoid the deprecation warning. + +Directly assigning values to the :attr:`~http.cookies.Morsel.key`, +:attr:`~http.cookies.Morsel.value` and +:attr:`~http.cookies.Morsel.coded_value` of :class:`http.cookies.Morsel` +objects is deprecated. Use the :meth:`~http.cookies.Morsel.set` method +instead. In addition, the undocumented *LegalChars* parameter of +:meth:`~http.cookies.Morsel.set` is deprecated, and is now ignored. + +Passing a format string as keyword argument *format_string* to the +:meth:`~string.Formatter.format` method of the :class:`string.Formatter` +class has been deprecated. +(Contributed by Serhiy Storchaka in :issue:`23671`.) + +The :func:`platform.dist` and :func:`platform.linux_distribution` functions +are now deprecated. Linux distributions use too many different ways of +describing themselves, so the functionality is left to a package. +(Contributed by Vajrasky Kok and Berker Peksag in :issue:`1322`.) + +The previously undocumented ``from_function`` and ``from_builtin`` methods of +:class:`inspect.Signature` are deprecated. Use the new +:meth:`Signature.from_callable() <inspect.Signature.from_callable>` +method instead. (Contributed by Yury Selivanov in :issue:`24248`.) + +The :func:`inspect.getargspec` function is deprecated and scheduled to be +removed in Python 3.6. (See :issue:`20438` for details.) + +The :mod:`inspect` :func:`~inspect.getfullargspec`, +:func:`~inspect.getargvalues`, :func:`~inspect.getcallargs`, +:func:`~inspect.getargvalues`, :func:`~inspect.formatargspec`, and +:func:`~inspect.formatargvalues` functions are deprecated in favor of +the :func:`inspect.signature` API. +(Contributed by Yury Selivanov in :issue:`20438`.) + +Use of :const:`re.LOCALE` flag with str patterns or :const:`re.ASCII` is now +deprecated. (Contributed by Serhiy Storchaka in :issue:`22407`.) + +Use of unrecognized special sequences consisting of ``'\'`` and an ASCII letter +in regular expression patterns and replacement patterns now raises a +deprecation warning and will be forbidden in Python 3.6. +(Contributed by Serhiy Storchaka in :issue:`23622`.) + +The undocumented and unofficial *use_load_tests* default argument of the +:meth:`unittest.TestLoader.loadTestsFromModule` method now is +deprecated and ignored. +(Contributed by Robert Collins and Barry A. Warsaw in :issue:`16662`.) + + +Removed +======= + +API and Feature Removals +------------------------ + +The following obsolete and previously deprecated APIs and features have been +removed: + +* The ``__version__`` attribute has been dropped from the email package. The + email code hasn't been shipped separately from the stdlib for a long time, + and the ``__version__`` string was not updated in the last few releases. + +* The internal ``Netrc`` class in the :mod:`ftplib` module was deprecated in + 3.4, and has now been removed. + (Contributed by Matt Chaput in :issue:`6623`.) + +* The concept of ``.pyo`` files has been removed. + +* The JoinableQueue class in the provisional :mod:`asyncio` module was + deprecated in 3.4.4 and is now removed. + (Contributed by A. Jesse Jiryu Davis in :issue:`23464`.) + + +Porting to Python 3.5 +===================== + +This section lists previously described changes and other bugfixes +that may require changes to your code. + + +Changes in Python behavior +-------------------------- + +* Due to an oversight, earlier Python versions erroneously accepted the + following syntax:: + + f(1 for x in [1], *args) + f(1 for x in [1], **kwargs) + + Python 3.5 now correctly raises a :exc:`SyntaxError`, as generator + expressions must be put in parentheses if not a sole argument to a function. + + +Changes in the Python API +------------------------- + +* :pep:`475`: System calls are now retried when interrupted by a signal instead + of raising :exc:`InterruptedError` if the Python signal handler does not + raise an exception. + +* Before Python 3.5, a :class:`datetime.time` object was considered to be false + if it represented midnight in UTC. This behavior was considered obscure and + error-prone and has been removed in Python 3.5. See :issue:`13936` for full + details. + +* The :meth:`ssl.SSLSocket.send()` method now raises either + :exc:`ssl.SSLWantReadError` or :exc:`ssl.SSLWantWriteError` + on a non-blocking socket if the operation would block. Previously, + it would return ``0``. (Contributed by Nikolaus Rath in :issue:`20951`.) + +* The ``__name__`` attribute of generators is now set from the function name, + instead of being set from the code name. Use ``gen.gi_code.co_name`` to + retrieve the code name. Generators also have a new ``__qualname__`` + attribute, the qualified name, which is now used for the representation + of a generator (``repr(gen)``). + (Contributed by Victor Stinner in :issue:`21205`.) + +* The deprecated "strict" mode and argument of :class:`~html.parser.HTMLParser`, + :meth:`HTMLParser.error`, and the :exc:`HTMLParserError` exception have been + removed. (Contributed by Ezio Melotti in :issue:`15114`.) + The *convert_charrefs* argument of :class:`~html.parser.HTMLParser` is + now ``True`` by default. (Contributed by Berker Peksag in :issue:`21047`.) + +* Although it is not formally part of the API, it is worth noting for porting + purposes (ie: fixing tests) that error messages that were previously of the + form "'sometype' does not support the buffer protocol" are now of the form "a + :term:`bytes-like object` is required, not 'sometype'". + (Contributed by Ezio Melotti in :issue:`16518`.) + +* If the current directory is set to a directory that no longer exists then + :exc:`FileNotFoundError` will no longer be raised and instead + :meth:`~importlib.machinery.FileFinder.find_spec` will return ``None`` + **without** caching ``None`` in :data:`sys.path_importer_cache`, which is + different than the typical case (:issue:`22834`). + +* HTTP status code and messages from :mod:`http.client` and :mod:`http.server` + were refactored into a common :class:`~http.HTTPStatus` enum. The values in + :mod:`http.client` and :mod:`http.server` remain available for backwards + compatibility. (Contributed by Demian Brecht in :issue:`21793`.) + +* When an import loader defines :meth:`importlib.machinery.Loader.exec_module` + it is now expected to also define + :meth:`~importlib.machinery.Loader.create_module` (raises a + :exc:`DeprecationWarning` now, will be an error in Python 3.6). If the loader + inherits from :class:`importlib.abc.Loader` then there is nothing to do, else + simply define :meth:`~importlib.machinery.Loader.create_module` to return + ``None``. (Contributed by Brett Cannon in :issue:`23014`.) + +* The :func:`re.split` function always ignored empty pattern matches, so the + ``"x*"`` pattern worked the same as ``"x+"``, and the ``"\b"`` pattern never + worked. Now :func:`re.split` raises a warning if the pattern could match + an empty string. For compatibility, use patterns that never match an empty + string (e.g. ``"x+"`` instead of ``"x*"``). Patterns that could only match + an empty string (such as ``"\b"``) now raise an error. + (Contributed by Serhiy Storchaka in :issue:`22818`.) + +* The :class:`http.cookies.Morsel` dict-like interface has been made self + consistent: morsel comparison now takes the :attr:`~http.cookies.Morsel.key` + and :attr:`~http.cookies.Morsel.value` into account, + :meth:`~http.cookies.Morsel.copy` now results in a + :class:`~http.cookies.Morsel` instance rather than a :class:`dict`, and + :meth:`~http.cookies.Morsel.update` will now raise an exception if any of the + keys in the update dictionary are invalid. In addition, the undocumented + *LegalChars* parameter of :func:`~http.cookies.Morsel.set` is deprecated and + is now ignored. (Contributed by Demian Brecht in :issue:`2211`.) + +* :pep:`488` has removed ``.pyo`` files from Python and introduced the optional + ``opt-`` tag in ``.pyc`` file names. The + :func:`importlib.util.cache_from_source` has gained an *optimization* + parameter to help control the ``opt-`` tag. Because of this, the + *debug_override* parameter of the function is now deprecated. `.pyo` files + are also no longer supported as a file argument to the Python interpreter and + thus serve no purpose when distributed on their own (i.e. sourcless code + distribution). Due to the fact that the magic number for bytecode has changed + in Python 3.5, all old `.pyo` files from previous versions of Python are + invalid regardless of this PEP. + +* The :mod:`socket` module now exports the :data:`~socket.CAN_RAW_FD_FRAMES` + constant on linux 3.6 and greater. + +* The :func:`ssl.cert_time_to_seconds` function now interprets the input time + as UTC and not as local time, per :rfc:`5280`. Additionally, the return + value is always an :class:`int`. (Contributed by Akira Li in :issue:`19940`.) + +* The ``pygettext.py`` Tool now uses the standard +NNNN format for timezones in + the POT-Creation-Date header. + +* The :mod:`smtplib` module now uses :data:`sys.stderr` instead of the previous + module-level :data:`stderr` variable for debug output. If your (test) + program depends on patching the module-level variable to capture the debug + output, you will need to update it to capture sys.stderr instead. + +* The :meth:`str.startswith` and :meth:`str.endswith` methods no longer return + ``True`` when finding the empty string and the indexes are completely out of + range. (Contributed by Serhiy Storchaka in :issue:`24284`.) + +* The :func:`inspect.getdoc` function now returns documentation strings + inherited from base classes. Documentation strings no longer need to be + duplicated if the inherited documentation is appropriate. To suppress an + inherited string, an empty string must be specified (or the documentation + may be filled in). This change affects the output of the :mod:`pydoc` + module and the :func:`help` function. + (Contributed by Serhiy Storchaka in :issue:`15582`.) + +* Nested :func:`functools.partial` calls are now flattened. If you were + relying on the previous behavior, you can now either add an attribute to a + :func:`functools.partial` object or you can create a subclass of + :func:`functools.partial`. + (Contributed by Alexander Belopolsky in :issue:`7830`.) + +Changes in the C API +-------------------- + +* The undocumented :c:member:`~PyMemoryViewObject.format` member of the + (non-public) :c:type:`PyMemoryViewObject` structure has been removed. + All extensions relying on the relevant parts in ``memoryobject.h`` + must be rebuilt. + +* The :c:type:`PyMemAllocator` structure was renamed to + :c:type:`PyMemAllocatorEx` and a new ``calloc`` field was added. + +* Removed non-documented macro :c:macro:`PyObject_REPR` which leaked references. + Use format character ``%R`` in :c:func:`PyUnicode_FromFormat`-like functions + to format the :func:`repr` of the object. + (Contributed by Serhiy Storchaka in :issue:`22453`.) + +* Because the lack of the :attr:`__module__` attribute breaks pickling and + introspection, a deprecation warning is now raised for builtin types without + the :attr:`__module__` attribute. This would be an AttributeError in + the future. + (Contributed by Serhiy Storchaka in :issue:`20204`.) + +* As part of the :pep:`492` implementation, the ``tp_reserved`` slot of + :c:type:`PyTypeObject` was replaced with a + :c:member:`tp_as_async` slot. Refer to :ref:`coro-objects` for + new types, structures and functions. + diff --git a/Doc/whatsnew/index.rst b/Doc/whatsnew/index.rst index 29902e407c..edb55022bd 100644 --- a/Doc/whatsnew/index.rst +++ b/Doc/whatsnew/index.rst @@ -11,6 +11,7 @@ anyone wishing to stay up-to-date after a new release. .. toctree:: :maxdepth: 2 + 3.5.rst 3.4.rst 3.3.rst 3.2.rst |
