path: root/doc/release/1.17.0-notes.rst

==========================
NumPy 1.17.0 Release Notes
==========================


Highlights
==========

* Experimental support for overriding numpy functions is now always available,
  see ``__array_function__`` below.

* NumPy's FFT implementation has switched to pocketfft

New functions
=============


Deprecations
============

``np.polynomial`` functions warn when passed ``float`` in place of ``int``
--------------------------------------------------------------------------
Previously functions in this module would accept ``float`` values provided they
were integral (``1.0``, ``2.0``, etc). For consistency with the rest of numpy,
doing so is now deprecated, and in future will raise a ``TypeError``.

Similarly, passing a float like ``0.5`` in place of an integer will now raise a
``TypeError`` instead of the previous ``ValueError``.

Deprecate ``numpy.distutils.exec_command`` and ``numpy.distutils.temp_file_name``
---------------------------------------------------------------------------------
The internal use of these functions has been refactored and there are better
alternatives. Relace ``exec_command`` with `subprocess.Popen` and
``temp_file_name`` with `tempfile.mkstemp`.


Future Changes
==============


Expired deprecations
====================


Compatibility notes
===================

float16 subnormal rounding
--------------------------
Casting from a different floating point precision to float16 used incorrect
rounding in some edge cases. This means in rare cases, subnormal results will
now be rounded up instead of down, changing the last bit (ULP) of the result.

Signed zero when using divmod
-----------------------------
Starting in version 1.12.0, numpy incorrectly returned a negatively signed zero
when using the ``divmod`` and ``floor_divide`` functions when the result was
zero. For example::

   >>> np.zeros(10)//1
   array([-0., -0., -0., -0., -0., -0., -0., -0., -0., -0.])

With this release, the result is correctly returned as a positively signed
zero::

   >>> np.zeros(10)//1
   array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])

Do not lookup ``__buffer__`` attribute in `numpy.frombuffer`
------------------------------------------------------------
Looking up ``__buffer__`` attribute in `numpy.frombuffer` was undocumented and
non-functional. This code was removed. If needed, use
``frombuffer(memoryview(obj), ...)`` instead.

``out``is buffered for memory overlaps in ``np.take``, ``np.choose``, ``np.put``
--------------------------------------------------------------------------------
If the out argument to these functions is provided and has memory overlap with
the other arguments, it is now buffered to avoid order-dependent behavior.

Unpickling while loading requires explicit opt-in
-------------------------------------------------
The functions ``np.load``, and ``np.lib.format.read_array`` take an
`allow_pickle` keyword which now defaults to ``False`` in response to
`CVE-2019-6446 <https://nvd.nist.gov/vuln/detail/CVE-2019-6446>`_.

Potential changes to the random stream
--------------------------------------
Due to bugs in the application of log to random floating point numbers,
the stream may change when sampling from ``np.random.beta``, ``np.random.binomial``,
``np.random.laplace``, ``np.random.logistic``, ``np.random.logseries`` or
``np.random.multinomial`` if a 0 is generated in the underlying MT19937 random stream.
There is a 1 in :math:`10^{53}` chance of this occurring, and so the probability that
the stream changes for any given seed is extremely small. If a 0 is encountered in the
underlying generator, then the incorrect value produced (either ``np.inf``
or ``np.nan``) is now dropped.

C API changes
=============


New Features
============

``np.ufunc.reduce`` and related functions now accept a ``where`` mask
---------------------------------------------------------------------
``np.ufunc.reduce``, ``np.sum``, ``np.prod``, ``np.min``, ``np.max`` all
now accept a ``where`` keyword argument, which can be used to tell which
elements to include in the reduction.  For reductions that do not have an
identity, it is necessary to also pass in an initial value (e.g.,
``initial=np.inf`` for ``np.min``).  For instance, the equivalent of
``nansum`` would be, ``np.sum(a, where=~np.isnan(a))``.

Timsort has replaced mergesort as the stable sorting implementation
-------------------------------------------------------------------
Timsort has been implemented and is now used in place of mergesort. Due to the
need to maintain backward compatibility, the sorting ``kind`` options ``"stable"``
and ``"mergesort"`` have been made aliases of each other with the actual sort
implementation used a function of the array type. Timsort features improved
performace on already or nearly sorted data and performs like mergesort on
random data.  The algorithm is stable and requires O(n/2) working space.  For
details of the algorithm, refer to
`CPython listsort.txt <https://github.com/python/cpython/blob/3.7/Objects/listsort.txt>`_.

In addition, for very small dtypes, radix sort is used instead of timsort. In
general, we attempt to use the fastest possible implementation.

``np.unpackbits`` now accepts a ``count`` parameter
---------------------------------------------------
``count`` allows subsetting the number of bits that will be unpacked up-front,
rather than reshaping and subsetting later, making the ``packbits`` operation
invertible, and the unpacking less wasteful. Counts larger than the number of
available bits add zero padding. Negative counts trim bits off the end instead
of counting from the beginning. None counts implement the existing behavior of
unpacking everything.

``np.linalg.svd`` and ``np.linalg.pinv`` can be faster on hermitian inputs
--------------------------------------------------------------------------
These functions now accept a ``hermitian`` argument, matching the one added
to ``np.linalg.matrix_rank`` in 1.14.0.

divmod operation is now supported for two ``timedelta64`` operands
------------------------------------------------------------------
The divmod operator now handles two ``np.timedelta64`` operands, with
type signature mm->qm.

New mode "empty" for ``np.pad``
-------------------------------
This mode pads an array to a desired shape without initializing the new
entries.


``np.empty_like`` and related functions now accept a ``shape`` argument
-----------------------------------------------------------------------
``np.empty_like``, ``np.full_like``, ``np.ones_like`` and ``np.zeros_like`` now
accept a ``shape`` keyword argument, which can be used to create a new array
as the prototype, overriding its shape as well. This is particularly useful
when combined with the ``__array_function__`` protocol, allowing the creation
of new arbitrary-shape arrays from NumPy-like libraries when such an array
is used as the prototype.

Floating point scalars implement ``as_integer_ratio`` to match the builtin float
--------------------------------------------------------------------------------
This returns a (numerator, denominator) pair, which can be used to construct a
`fractions.Fraction`.

`numpy.packbits` and `numpy.unpackbits` accept an ``order`` keyword
-------------------------------------------------------------------
The ``order`` keyword defaults to ``big``, and will order the **bits**
accordingly. For ``'big'`` 3 will become ``[0, 0, 0, 0, 0, 0, 1, 1]``, and
``[1, 1, 0, 0, 0, 0, 0, 0]`` for ``little``

Improvements
============

Array comparison assertions include maximum differences
-------------------------------------------------------
Error messages from array comparison tests such as
`np.testing.assert_allclose` now include "max absolute difference" and
"max relative difference," in addition to the previous "mismatch" percentage.
This information makes it easier to update absolute and relative error
tolerances.

Replacement of the `fftpack`-based FFT module by the `pocketfft` library
------------------------------------------------------------------------
Both implementations have the same ancestor (Fortran77 `FFTPACK` by Paul N.
Swarztrauber), but `pocketfft` contains additional modifications which
improve both accuracy and performance in some circumstances. For FFT lengths
containing large prime factors, `pocketfft` uses Bluestein's algorithm, which
maintains `O(N log N)` run time complexity instead of deteriorating towards
`O(N*N)` for prime lengths. Also, accuracy for real-valued FFTs with near-prime
lengths has improved and is on par with complex-valued FFTs.

Performance improvements for integer sorts
------------------------------------------

``sort``, ``argsort``, ``ndarray.sort`` and ``ndarray.argsort`` now use radix
sort as the default stable sort for integers and booleans. This is faster than
the old default, mergesort, in the vast majority of cases.


Further improvements to ``ctypes`` support in ``np.ctypeslib``
--------------------------------------------------------------
A new `numpy.ctypeslib.as_ctypes_type` function has been added, which can be
used to converts a `dtype` into a best-guess `ctypes` type. Thanks to this
new function, `numpy.ctypeslib.as_ctypes` now supports a much wider range of
array types, including structures, booleans, and integers of non-native
endianness.

`numpy.errstate` is now also function decorator
-----------------------------------------------

Currently, if you have a function like::

    def foo():
        pass

and you want to wrap the whole thing in `errstate`, you have to rewrite it like so::

    def foo():
        with np.errstate(...):
            pass

but with this change, you can do::

    @np.errstate(...)
    def foo():
        pass

thereby saving a level of indentation

`numpy.exp and numpy.log` speed up for float32 implementation
-------------------------------------------------------------
float32 implementation of numpy.exp and numpy.log now benefit from AVX2/AVX512
instruction set which are detected during runtime. numpy.exp has a max ulp
error of 2.52 and numpy.log has a max ulp error or 3.83.

Improve performance of ``np.pad``
---------------------------------
The performance of the function has been improved for most cases by filling in
a preallocated array with the desired padded shape instead of using
concatenation.

``np.interp`` handles infinities more robustly
----------------------------------------------
In some cases where ``np.interp`` would previously return ``np.nan``, it now
returns an appropriate infinity.

Specialized ``np.isnan``, ``np.isinf``, and ``np.isfinite`` ufuncs for bool and int types
-----------------------------------------------------------------------------------------
The boolean and integer types are incapable of storing ``np.nan`` and
``np.inf`` values, which allows us to provide specialized ufuncs that are up to
250x faster than the current approach.

``np.isfinite`` supports ``datetime64`` and ``timedelta64`` types
-----------------------------------------------------------------
Previously, `np.isfinite` used to raise a ``TypeError`` on being used on these
two types.

New keywords added to ``np.nan_to_num``
---------------------------------------
``np.nan_to_num`` now accepts keywords ``nan``, ``posinf`` and ``neginf``
allowing the user to define the value to replace the ``nan``, positive and
negative ``np.inf`` values respectively.

MemoryErrors caused by allocated overly large arrays are more descriptive
-------------------------------------------------------------------------
Often the cause of a MemoryError is incorrect broadcasting, which results in a
very large and incorrect shape. The message of the error now includes this
shape to help diagnose the cause of failure.

`floor`, `ceil`, and `trunc` now respect builtin magic methods
--------------------------------------------------------------
These ufuncs now call the ``__floor__``, ``__ceil__``, and ``__trunc__``
methods when called on object arrays, making them compatible with
`decimal.Decimal` and `fractions.Fraction` objects.

`quantile` now works on `fraction.Fraction` and `decimal.Decimal` objects
-------------------------------------------------------------------------
In general, this handles object arrays more gracefully, and avoids floating-
point operations if exact arithmetic types are used.


Changes
=======

``median`` and ``percentile`` family of functions no longer warn about ``nan``
------------------------------------------------------------------------------
`numpy.median`, `numpy.percentile`, and `numpy.quantile` used to emit a
``RuntimeWarning`` when encountering an `numpy.nan`. Since they return the
``nan`` value, the warning is redundant and has been removed.

``timedelta64 % 0`` behavior adjusted to return ``NaT``
-------------------------------------------------------
The modulus operation with two ``np.timedelta64`` operands now returns
``NaT`` in the case of division by zero, rather than returning zero

NumPy functions now always support overrides with ``__array_function__``
------------------------------------------------------------------------
NumPy now always checks the ``__array_function__`` method to implement overrides
of NumPy functions on non-NumPy arrays, as described in `NEP 18`_. The feature
was available for testing with NumPy 1.16 if appropriate environment variables
are set, but is now always enabled.

`numpy.lib.recfunctions.structured_to_unstructured` does not squeeze single-field views
---------------------------------------------------------------------------------------
Previously ``structured_to_unstructured(arr[['a']])`` would produce a squeezed
result inconsistent with ``structured_to_unstructured(arr[['a', b']])``. This
was accidental. The old behavior can be retained with
``structured_to_unstructured(arr[['a']]).squeeze(axis=-1)`` or far more simply,
``arr['a']``.

``__array_interface__`` offset now works as documented
------------------------------------------------------
The interface may use an ``offset`` value that was mistakenly ignored.

.. _`NEP 18` : http://www.numpy.org/neps/nep-0018-array-function-protocol.html