diff options
Diffstat (limited to 'docs/src/userguide')
25 files changed, 3437 insertions, 897 deletions
diff --git a/docs/src/userguide/buffer.rst b/docs/src/userguide/buffer.rst index 08661a184..3687cf2fd 100644 --- a/docs/src/userguide/buffer.rst +++ b/docs/src/userguide/buffer.rst @@ -3,6 +3,10 @@ Implementing the buffer protocol ================================ +.. include:: + ../two-syntax-variants-used + + Cython objects can expose memory buffers to Python code by implementing the "buffer protocol". This chapter shows how to implement the protocol @@ -16,7 +20,15 @@ The following Cython/C++ code implements a matrix of floats, where the number of columns is fixed at construction time but rows can be added dynamically. -.. literalinclude:: ../../examples/userguide/buffer/matrix.pyx +.. tabs:: + + .. group-tab:: Pure Python + + .. literalinclude:: ../../examples/userguide/buffer/matrix.py + + .. group-tab:: Cython + + .. literalinclude:: ../../examples/userguide/buffer/matrix.pyx There are no methods to do anything productive with the matrices' contents. We could implement custom ``__getitem__``, ``__setitem__``, etc. for this, @@ -27,7 +39,15 @@ Implementing the buffer protocol requires adding two methods, ``__getbuffer__`` and ``__releasebuffer__``, which Cython handles specially. -.. literalinclude:: ../../examples/userguide/buffer/matrix_with_buffer.pyx +.. tabs:: + + .. group-tab:: Pure Python + + .. literalinclude:: ../../examples/userguide/buffer/matrix_with_buffer.py + + .. group-tab:: Cython + + .. literalinclude:: ../../examples/userguide/buffer/matrix_with_buffer.pyx The method ``Matrix.__getbuffer__`` fills a descriptor structure, called a ``Py_buffer``, that is defined by the Python C-API. @@ -75,7 +95,15 @@ This is where ``__releasebuffer__`` comes in. We can add a reference count to each matrix, and lock it for mutation whenever a view exists. -.. literalinclude:: ../../examples/userguide/buffer/view_count.pyx +.. tabs:: + + .. group-tab:: Pure Python + + .. literalinclude:: ../../examples/userguide/buffer/view_count.py + + .. group-tab:: Cython + + .. literalinclude:: ../../examples/userguide/buffer/view_count.pyx Flags ----- diff --git a/docs/src/userguide/compute_typed_html.jpg b/docs/src/userguide/compute_typed_html.jpg Binary files differindex f8607bdd8..a1e006573 100644 --- a/docs/src/userguide/compute_typed_html.jpg +++ b/docs/src/userguide/compute_typed_html.jpg diff --git a/docs/src/userguide/cpow_table.csv b/docs/src/userguide/cpow_table.csv new file mode 100644 index 000000000..a781e77c2 --- /dev/null +++ b/docs/src/userguide/cpow_table.csv @@ -0,0 +1,6 @@ +Type of ``a``,Type of ``b``,``cpow==True``,``cpow==False`` +C integer,Negative integer compile-time constant,Return type is C double,Return type is C double (special case) +C integer,C integer (known to be >= 0 at compile time),Return type is integer,Return type is integer +C integer,C integer (may be negative),Return type is integer,"Return type is C double (note that Python would dynamically pick ``int`` or ``float`` here, while Cython doesn’t)" +C floating point,C integer,Return type is floating point,Return type is floating point +C floating point (or C integer),C floating point,"Return type is floating point, result is NaN if the result would be complex",Either a C real or complex number at cost of some speed diff --git a/docs/src/userguide/debugging.rst b/docs/src/userguide/debugging.rst index 06af18bbf..a33ff8dd8 100644 --- a/docs/src/userguide/debugging.rst +++ b/docs/src/userguide/debugging.rst @@ -21,19 +21,20 @@ source, and then running:: make sudo make install +Installing the Cython debugger can be quite tricky. `This installation script and example code <https://gitlab.com/volkerweissmann/cygdb_installation>`_ might be useful. + The debugger will need debug information that the Cython compiler can export. This can be achieved from within the setup script by passing ``gdb_debug=True`` to ``cythonize()``:: - from distutils.core import setup - from distutils.extension import Extension + from setuptools import Extension, setup extensions = [Extension('source', ['source.pyx'])] setup(..., ext_modules=cythonize(extensions, gdb_debug=True)) For development it's often helpful to pass the ``--inplace`` flag to -the ``setup.py`` script, which makes distutils build your project +the ``setup.py`` script, which makes setuptools build your project "in place", i.e., not in a separate `build` directory. When invoking Cython from the command line directly you can have it write diff --git a/docs/src/userguide/early_binding_for_speed.rst b/docs/src/userguide/early_binding_for_speed.rst index 9bb8cf724..4a442d973 100644 --- a/docs/src/userguide/early_binding_for_speed.rst +++ b/docs/src/userguide/early_binding_for_speed.rst @@ -6,6 +6,9 @@ Early Binding for Speed ************************** +.. include:: + ../two-syntax-variants-used + As a dynamic language, Python encourages a programming style of considering classes and objects in terms of their methods and attributes, more than where they fit into the class hierarchy. @@ -22,7 +25,15 @@ use of 'early binding' programming techniques. For example, consider the following (silly) code example: -.. literalinclude:: ../../examples/userguide/early_binding_for_speed/rectangle.pyx +.. tabs:: + + .. group-tab:: Pure Python + + .. literalinclude:: ../../examples/userguide/early_binding_for_speed/rectangle.py + + .. group-tab:: Cython + + .. literalinclude:: ../../examples/userguide/early_binding_for_speed/rectangle.pyx In the :func:`rectArea` method, the call to :meth:`rect.area` and the :meth:`.area` method contain a lot of Python overhead. @@ -30,7 +41,15 @@ In the :func:`rectArea` method, the call to :meth:`rect.area` and the However, in Cython, it is possible to eliminate a lot of this overhead in cases where calls occur within Cython code. For example: -.. literalinclude:: ../../examples/userguide/early_binding_for_speed/rectangle_cdef.pyx +.. tabs:: + + .. group-tab:: Pure Python + + .. literalinclude:: ../../examples/userguide/early_binding_for_speed/rectangle_cdef.py + + .. group-tab:: Cython + + .. literalinclude:: ../../examples/userguide/early_binding_for_speed/rectangle_cdef.pyx Here, in the Rectangle extension class, we have defined two different area calculation methods, the efficient :meth:`_area` C method, and the @@ -46,10 +65,18 @@ dual-access methods - methods that can be efficiently called at C level, but can also be accessed from pure Python code at the cost of the Python access overheads. Consider this code: -.. literalinclude:: ../../examples/userguide/early_binding_for_speed/rectangle_cpdef.pyx +.. tabs:: + + .. group-tab:: Pure Python + + .. literalinclude:: ../../examples/userguide/early_binding_for_speed/rectangle_cpdef.py + + .. group-tab:: Cython + + .. literalinclude:: ../../examples/userguide/early_binding_for_speed/rectangle_cpdef.pyx -Here, we just have a single area method, declared as :keyword:`cpdef` to make it -efficiently callable as a C function, but still accessible from pure Python +Here, we just have a single area method, declared as :keyword:`cpdef` or with ``@ccall`` decorator +to make it efficiently callable as a C function, but still accessible from pure Python (or late-binding Cython) code. If within Cython code, we have a variable already 'early-bound' (ie, declared diff --git a/docs/src/userguide/extension_types.rst b/docs/src/userguide/extension_types.rst index 6ad953ac9..42d77c378 100644 --- a/docs/src/userguide/extension_types.rst +++ b/docs/src/userguide/extension_types.rst @@ -9,20 +9,56 @@ Extension Types Introduction ============== +.. include:: + ../two-syntax-variants-used + As well as creating normal user-defined classes with the Python class statement, Cython also lets you create new built-in Python types, known as -extension types. You define an extension type using the :keyword:`cdef` class -statement. Here's an example: +:term:`extension types<Extension type>`. You define an extension type using the :keyword:`cdef` class +statement or decorating the class with the ``@cclass`` decorator. Here's an example: + +.. tabs:: + + .. group-tab:: Pure Python + + .. literalinclude:: ../../examples/userguide/extension_types/shrubbery.py -.. literalinclude:: ../../examples/userguide/extension_types/shrubbery.pyx + .. group-tab:: Cython + + .. literalinclude:: ../../examples/userguide/extension_types/shrubbery.pyx As you can see, a Cython extension type definition looks a lot like a Python -class definition. Within it, you use the def statement to define methods that +class definition. Within it, you use the :keyword:`def` statement to define methods that can be called from Python code. You can even define many of the special methods such as :meth:`__init__` as you would in Python. -The main difference is that you can use the :keyword:`cdef` statement to define -attributes. The attributes may be Python objects (either generic or of a +The main difference is that you can define attributes using + +* the :keyword:`cdef` statement, +* the :func:`cython.declare()` function or +* the annotation of an attribute name. + +.. tabs:: + + .. group-tab:: Pure Python + + .. code-block:: python + + @cython.cclass + class Shrubbery: + width = declare(cython.int) + height: cython.int + + .. group-tab:: Cython + + .. code-block:: cython + + cdef class Shrubbery: + + cdef int width + cdef int height + +The attributes may be Python objects (either generic or of a particular extension type), or they may be of any C data type. So you can use extension types to wrap arbitrary C data structures and provide a Python-like interface to them. @@ -50,7 +86,15 @@ not Python access, which means that they are not accessible from Python code. To make them accessible from Python code, you need to declare them as :keyword:`public` or :keyword:`readonly`. For example: -.. literalinclude:: ../../examples/userguide/extension_types/python_access.pyx +.. tabs:: + + .. group-tab:: Pure Python + + .. literalinclude:: ../../examples/userguide/extension_types/python_access.py + + .. group-tab:: Cython + + .. literalinclude:: ../../examples/userguide/extension_types/python_access.pyx makes the width and height attributes readable and writable from Python code, and the depth attribute readable but not writable. @@ -74,15 +118,32 @@ Dynamic Attributes It is not possible to add attributes to an extension type at runtime by default. You have two ways of avoiding this limitation, both add an overhead when -a method is called from Python code. Especially when calling ``cpdef`` methods. +a method is called from Python code. Especially when calling hybrid methods declared +with :keyword:`cpdef` in .pyx files or with the ``@ccall`` decorator. + +The first approach is to create a Python subclass: + +.. tabs:: + + .. group-tab:: Pure Python -The first approach is to create a Python subclass.: + .. literalinclude:: ../../examples/userguide/extension_types/extendable_animal.py -.. literalinclude:: ../../examples/userguide/extension_types/extendable_animal.pyx + .. group-tab:: Cython -Declaring a ``__dict__`` attribute is the second way of enabling dynamic attributes.: + .. literalinclude:: ../../examples/userguide/extension_types/extendable_animal.pyx -.. literalinclude:: ../../examples/userguide/extension_types/dict_animal.pyx +Declaring a ``__dict__`` attribute is the second way of enabling dynamic attributes: + +.. tabs:: + + .. group-tab:: Pure Python + + .. literalinclude:: ../../examples/userguide/extension_types/dict_animal.py + + .. group-tab:: Cython + + .. literalinclude:: ../../examples/userguide/extension_types/dict_animal.pyx Type declarations =================== @@ -93,10 +154,24 @@ generic Python object. It knows this already in the case of the ``self`` parameter of the methods of that type, but in other cases you will have to use a type declaration. -For example, in the following function:: +For example, in the following function: - cdef widen_shrubbery(sh, extra_width): # BAD - sh.width = sh.width + extra_width +.. tabs:: + + .. group-tab:: Pure Python + + .. code-block:: python + + @cython.cfunc + def widen_shrubbery(sh, extra_width): # BAD + sh.width = sh.width + extra_width + + .. group-tab:: Cython + + .. code-block:: cython + + cdef widen_shrubbery(sh, extra_width): # BAD + sh.width = sh.width + extra_width because the ``sh`` parameter hasn't been given a type, the width attribute will be accessed by a Python attribute lookup. If the attribute has been @@ -107,18 +182,35 @@ will be very inefficient. If the attribute is private, it will not work at all The solution is to declare ``sh`` as being of type :class:`Shrubbery`, as follows: -.. literalinclude:: ../../examples/userguide/extension_types/widen_shrubbery.pyx +.. tabs:: + + .. group-tab:: Pure Python + + .. literalinclude:: ../../examples/userguide/extension_types/widen_shrubbery.py + + .. group-tab:: Cython + + .. literalinclude:: ../../examples/userguide/extension_types/widen_shrubbery.pyx Now the Cython compiler knows that ``sh`` has a C attribute called :attr:`width` and will generate code to access it directly and efficiently. The same consideration applies to local variables, for example: -.. literalinclude:: ../../examples/userguide/extension_types/shrubbery_2.pyx +.. tabs:: + + .. group-tab:: Pure Python + + .. literalinclude:: ../../examples/userguide/extension_types/shrubbery_2.py + + .. group-tab:: Cython + + .. literalinclude:: ../../examples/userguide/extension_types/shrubbery_2.pyx .. note:: - We here ``cimport`` the class :class:`Shrubbery`, and this is necessary - to declare the type at compile time. To be able to ``cimport`` an extension type, + Here, we *cimport* the class :class:`Shrubbery` (using the :keyword:`cimport` statement + or importing from special ``cython.cimports`` package), and this is necessary + to declare the type at compile time. To be able to cimport an extension type, we split the class definition into two parts, one in a definition file and the other in the corresponding implementation file. You should read :ref:`sharing_extension_types` to learn to do that. @@ -128,24 +220,61 @@ Type Testing and Casting ------------------------ Suppose I have a method :meth:`quest` which returns an object of type :class:`Shrubbery`. -To access it's width I could write:: +To access its width I could write: + +.. tabs:: + + .. group-tab:: Pure Python - cdef Shrubbery sh = quest() - print(sh.width) + .. code-block:: python + + sh: Shrubbery = quest() + print(sh.width) + + .. group-tab:: Cython + + .. code-block:: cython + + cdef Shrubbery sh = quest() + print(sh.width) which requires the use of a local variable and performs a type test on assignment. If you *know* the return value of :meth:`quest` will be of type :class:`Shrubbery` -you can use a cast to write:: +you can use a cast to write: + +.. tabs:: + + .. group-tab:: Pure Python + + .. code-block:: python - print( (<Shrubbery>quest()).width ) + print( cython.cast(Shrubbery, quest()).width ) + + .. group-tab:: Cython + + .. code-block:: cython + + print( (<Shrubbery>quest()).width ) This may be dangerous if :meth:`quest()` is not actually a :class:`Shrubbery`, as it will try to access width as a C struct member which may not exist. At the C level, rather than raising an :class:`AttributeError`, either an nonsensical result will be returned (interpreting whatever data is at that address as an int) or a segfault -may result from trying to access invalid memory. Instead, one can write:: +may result from trying to access invalid memory. Instead, one can write: + +.. tabs:: + + .. group-tab:: Pure Python + + .. code-block:: python + + print( cython.cast(Shrubbery, quest(), typecheck=True).width ) - print( (<Shrubbery?>quest()).width ) + .. group-tab:: Cython + + .. code-block:: cython + + print( (<Shrubbery?>quest()).width ) which performs a type check (possibly raising a :class:`TypeError`) before making the cast and allowing the code to proceed. @@ -155,14 +284,18 @@ For known builtin or extension types, Cython translates these into a fast and safe type check that ignores changes to the object's ``__class__`` attribute etc., so that after a successful :meth:`isinstance` test, code can rely on the expected C structure of the -extension type and its :keyword:`cdef` attributes and methods. +extension type and its C-level attributes (stored in the object’s C struct) and +:keyword:`cdef`/``@cfunc`` methods. .. _extension_types_and_none: Extension types and None ========================= -When you declare a parameter or C variable as being of an extension type, +Cython handles ``None`` values differently in C-like type declarations and when Python annotations are used. + +In :keyword:`cdef` declarations and C-like function argument declarations (``func(list x)``), +when you declare an argument or C variable as having an extension or Python builtin type, Cython will allow it to take on the value ``None`` as well as values of its declared type. This is analogous to the way a C pointer can take on the value ``NULL``, and you need to exercise the same caution because of it. There is no @@ -172,24 +305,24 @@ of an extension type (as in the widen_shrubbery function above), it's up to you to make sure the reference you're using is not ``None`` -- in the interests of efficiency, Cython does not check this. -You need to be particularly careful when exposing Python functions which take -extension types as arguments. If we wanted to make :func:`widen_shrubbery` a -Python function, for example, if we simply wrote:: +With the C-like declaration syntax, you need to be particularly careful when +exposing Python functions which take extension types as arguments:: def widen_shrubbery(Shrubbery sh, extra_width): # This is sh.width = sh.width + extra_width # dangerous! -then users of our module could crash it by passing ``None`` for the ``sh`` +The users of our module could crash it by passing ``None`` for the ``sh`` parameter. -One way to fix this would be:: +As in Python, whenever it is unclear whether a variable can be ``None``, +but the code requires a non-None value, an explicit check can help:: def widen_shrubbery(Shrubbery sh, extra_width): if sh is None: raise TypeError sh.width = sh.width + extra_width -but since this is anticipated to be such a frequent requirement, Cython +but since this is anticipated to be such a frequent requirement, Cython language provides a more convenient way. Parameters of a Python function declared as an extension type can have a ``not None`` clause:: @@ -199,18 +332,41 @@ extension type can have a ``not None`` clause:: Now the function will automatically check that ``sh`` is ``not None`` along with checking that it has the right type. +When annotations are used, the behaviour follows the Python typing semantics of +`PEP-484 <https://www.python.org/dev/peps/pep-0484/>`_ instead. +The value ``None`` is not allowed when a variable is annotated only with its plain type:: + + def widen_shrubbery(sh: Shrubbery, extra_width): # TypeError is raised + sh.width = sh.width + extra_width # when sh is None + +To also allow ``None``, ``typing.Optional[ ]`` must be used explicitly. +For function arguments, this is also automatically allowed when they have a +default argument of `None``, e.g. ``func(x: list = None)`` does not require ``typing.Optional``:: + + import typing + def widen_shrubbery(sh: typing.Optional[Shrubbery], extra_width): + if sh is None: + # We want to raise a custom exception in case of a None value. + raise ValueError + sh.width = sh.width + extra_width + +The upside of using annotations here is that they are safe by default because +you need to explicitly allow ``None`` values for them. + + .. note:: - ``not None`` clause can only be used in Python functions (defined with - :keyword:`def`) and not C functions (defined with :keyword:`cdef`). If - you need to check whether a parameter to a C function is None, you will + The ``not None`` and ``typing.Optional`` can only be used in Python functions (defined with + :keyword:`def` and without ``@cython.cfunc`` decorator) and not C functions + (defined with :keyword:`cdef` or decorated using ``@cython.cfunc``). If + you need to check whether a parameter to a C function is ``None``, you will need to do it yourself. .. note:: Some more things: - * The self parameter of a method of an extension type is guaranteed never to + * The ``self`` parameter of a method of an extension type is guaranteed never to be ``None``. * When comparing a value with ``None``, keep in mind that, if ``x`` is a Python object, ``x is None`` and ``x is not None`` are very efficient because they @@ -232,23 +388,49 @@ extension types. Properties ============ -You can declare properties in an extension class using the same syntax as in ordinary Python code:: +You can declare properties in an extension class using the same syntax as in ordinary Python code: - cdef class Spam: +.. tabs:: - @property - def cheese(self): - # This is called when the property is read. - ... + .. group-tab:: Pure Python - @cheese.setter - def cheese(self, value): - # This is called when the property is written. - ... + .. code-block:: python - @cheese.deleter - def cheese(self): - # This is called when the property is deleted. + @cython.cclass + class Spam: + @property + def cheese(self): + # This is called when the property is read. + ... + + @cheese.setter + def cheese(self, value): + # This is called when the property is written. + ... + + @cheese.deleter + def cheese(self): + # This is called when the property is deleted. + + .. group-tab:: Cython + + .. code-block:: cython + + cdef class Spam: + + @property + def cheese(self): + # This is called when the property is read. + ... + + @cheese.setter + def cheese(self, value): + # This is called when the property is written. + ... + + @cheese.deleter + def cheese(self): + # This is called when the property is deleted. There is also a special (deprecated) legacy syntax for defining properties in an extension class:: @@ -277,72 +459,127 @@ corresponding operation is attempted. Here's a complete example. It defines a property which adds to a list each time it is written to, returns the list when it is read, and empties the list -when it is deleted.:: +when it is deleted: - # cheesy.pyx - cdef class CheeseShop: +.. tabs:: - cdef object cheeses + .. group-tab:: Pure Python - def __cinit__(self): - self.cheeses = [] + .. literalinclude:: ../../examples/userguide/extension_types/cheesy.py - @property - def cheese(self): - return "We don't have: %s" % self.cheeses + .. group-tab:: Cython - @cheese.setter - def cheese(self, value): - self.cheeses.append(value) + .. literalinclude:: ../../examples/userguide/extension_types/cheesy.pyx - @cheese.deleter - def cheese(self): - del self.cheeses[:] +.. code-block:: text - # Test input - from cheesy import CheeseShop + # Test output + We don't have: [] + We don't have: ['camembert'] + We don't have: ['camembert', 'cheddar'] + We don't have: [] - shop = CheeseShop() - print(shop.cheese) - shop.cheese = "camembert" - print(shop.cheese) +C methods +========= - shop.cheese = "cheddar" - print(shop.cheese) +Extension types can have C methods as well as Python methods. Like C +functions, C methods are declared using - del shop.cheese - print(shop.cheese) +* :keyword:`cdef` instead of :keyword:`def` or ``@cfunc`` decorator for *C methods*, or +* :keyword:`cpdef` instead of :keyword:`def` or ``@ccall`` decorator for *hybrid methods*. -.. sourcecode:: text +C methods are "virtual", and may be overridden in derived extension types. +In addition, :keyword:`cpdef`/``@ccall`` methods can even be overridden by Python +methods when called as C method. This adds a little to their calling overhead +compared to a :keyword:`cdef`/``@cfunc`` method: + +.. tabs:: + + .. group-tab:: Pure Python + + .. literalinclude:: ../../examples/userguide/extension_types/pets.py + + .. group-tab:: Cython + + .. literalinclude:: ../../examples/userguide/extension_types/pets.pyx + +.. code-block:: text + + # Output + p1: + This parrot is resting. + p2: + This parrot is resting. + Lovely plumage! + +The above example also illustrates that a C method can call an inherited C +method using the usual Python technique, i.e.:: + + Parrot.describe(self) + +:keyword:`cdef`/``@ccall`` methods can be declared static by using the ``@staticmethod`` decorator. +This can be especially useful for constructing classes that take non-Python compatible types: + +.. tabs:: + + .. group-tab:: Pure Python + + .. literalinclude:: ../../examples/userguide/extension_types/owned_pointer.py + + .. group-tab:: Cython + + .. literalinclude:: ../../examples/userguide/extension_types/owned_pointer.pyx + +.. note:: + + Cython currently does not support decorating :keyword:`cdef`/``@ccall`` methods with + the ``@classmethod`` decorator. - # Test output - We don't have: [] - We don't have: ['camembert'] - We don't have: ['camembert', 'cheddar'] - We don't have: [] .. _subclassing: Subclassing ============= -An extension type may inherit from a built-in type or another extension type:: +If an extension type inherits from other types, the first base class must be +a built-in type or another extension type: - cdef class Parrot: - ... +.. tabs:: - cdef class Norwegian(Parrot): - ... + .. group-tab:: Pure Python + + .. code-block:: python + + @cython.cclass + class Parrot: + ... + + @cython.cclass + class Norwegian(Parrot): + ... + + .. group-tab:: Cython + + .. code-block:: cython + + cdef class Parrot: + ... + + + cdef class Norwegian(Parrot): + ... A complete definition of the base type must be available to Cython, so if the base type is a built-in type, it must have been previously declared as an extern extension type. If the base type is defined in another Cython module, it must either be declared as an extern extension type or imported using the -:keyword:`cimport` statement. +:keyword:`cimport` statement or importing from the special ``cython.cimports`` package. -An extension type can only have one base class (no multiple inheritance). +Multiple inheritance is supported, however the second and subsequent base +classes must be an ordinary Python class (not an extension type or a built-in +type). Cython extension types can also be subclassed in Python. A Python class can inherit from multiple extension types provided that the usual Python rules for @@ -351,84 +588,54 @@ must be compatible). There is a way to prevent extension types from being subtyped in Python. This is done via the ``final`` directive, -usually set on an extension type using a decorator:: - - cimport cython +usually set on an extension type or C method using a decorator: - @cython.final - cdef class Parrot: - def done(self): pass +.. tabs:: -Trying to create a Python subclass from this type will raise a -:class:`TypeError` at runtime. Cython will also prevent subtyping a -final type inside of the same module, i.e. creating an extension type -that uses a final type as its base type will fail at compile time. -Note, however, that this restriction does not currently propagate to -other extension modules, so even final extension types can still be -subtyped at the C level by foreign code. + .. group-tab:: Pure Python + .. code-block:: python -C methods -========= + import cython -Extension types can have C methods as well as Python methods. Like C -functions, C methods are declared using :keyword:`cdef` or :keyword:`cpdef` instead of -:keyword:`def`. C methods are "virtual", and may be overridden in derived -extension types. In addition, :keyword:`cpdef` methods can even be overridden by python -methods when called as C method. This adds a little to their calling overhead -compared to a :keyword:`cdef` method:: - - # pets.pyx - cdef class Parrot: + @cython.final + @cython.cclass + class Parrot: + def describe(self): pass - cdef void describe(self): - print("This parrot is resting.") + @cython.cclass + class Lizard: - cdef class Norwegian(Parrot): + @cython.final + @cython.cfunc + def done(self): pass - cdef void describe(self): - Parrot.describe(self) - print("Lovely plumage!") + .. group-tab:: Cython + .. code-block:: cython - cdef Parrot p1, p2 - p1 = Parrot() - p2 = Norwegian() - print("p1:") - p1.describe() - print("p2:") - p2.describe() + cimport cython -.. sourcecode:: text + @cython.final + cdef class Parrot: + def describe(self): pass - # Output - p1: - This parrot is resting. - p2: - This parrot is resting. - Lovely plumage! -The above example also illustrates that a C method can call an inherited C -method using the usual Python technique, i.e.:: - Parrot.describe(self) + cdef class Lizard: -`cdef` methods can be declared static by using the @staticmethod decorator. -This can be especially useful for constructing classes that take non-Python -compatible types.:: - cdef class OwnedPointer: - cdef void* ptr + @cython.final + cdef done(self): pass - def __dealloc__(self): - if self.ptr is not NULL: - free(self.ptr) +Trying to create a Python subclass from a final type or overriding a final method will raise +a :class:`TypeError` at runtime. Cython will also prevent subtyping a +final type or overriding a final method inside of the same module, i.e. creating +an extension type that uses a final type as its base type will fail at compile time. +Note, however, that this restriction does not currently propagate to +other extension modules, so Cython is unable to prevent final extension types +from being subtyped at the C level by foreign code. - @staticmethod - cdef create(void* ptr): - p = OwnedPointer() - p.ptr = ptr - return p .. _forward_declaring_extension_types: @@ -457,43 +664,41 @@ Fast instantiation Cython provides two ways to speed up the instantiation of extension types. The first one is a direct call to the ``__new__()`` special static method, as known from Python. For an extension type ``Penguin``, you could use -the following code:: +the following code: + +.. tabs:: - cdef class Penguin: - cdef object food + .. group-tab:: Pure Python - def __cinit__(self, food): - self.food = food + .. literalinclude:: ../../examples/userguide/extension_types/penguin.py - def __init__(self, food): - print("eating!") + .. group-tab:: Cython - normal_penguin = Penguin('fish') - fast_penguin = Penguin.__new__(Penguin, 'wheat') # note: not calling __init__() ! + .. literalinclude:: ../../examples/userguide/extension_types/penguin.pyx Note that the path through ``__new__()`` will *not* call the type's ``__init__()`` method (again, as known from Python). Thus, in the example above, the first instantiation will print ``eating!``, but the second will not. This is only one of the reasons why the ``__cinit__()`` method is -safer and preferable over the normal ``__init__()`` method for extension -types. +safer than the normal ``__init__()`` method for initialising extension types +and bringing them into a correct and safe state. +See the :ref:`Initialisation Methods Section <initialisation_methods>` about +the differences. The second performance improvement applies to types that are often created and deleted in a row, so that they can benefit from a freelist. Cython provides the decorator ``@cython.freelist(N)`` for this, which creates a -statically sized freelist of ``N`` instances for a given type. Example:: +statically sized freelist of ``N`` instances for a given type. Example: + +.. tabs:: + + .. group-tab:: Pure Python - cimport cython + .. literalinclude:: ../../examples/userguide/extension_types/penguin2.py - @cython.freelist(8) - cdef class Penguin: - cdef object food - def __cinit__(self, food): - self.food = food + .. group-tab:: Cython - penguin = Penguin('fish 1') - penguin = None - penguin = Penguin('fish 2') # does not need to allocate memory! + .. literalinclude:: ../../examples/userguide/extension_types/penguin2.pyx .. _existing-pointers-instantiation: @@ -504,63 +709,17 @@ It is quite common to want to instantiate an extension class from an existing (pointer to a) data structure, often as returned by external C/C++ functions. As extension classes can only accept Python objects as arguments in their -contructors, this necessitates the use of factory functions. For example, :: - - from libc.stdlib cimport malloc, free - - # Example C struct - ctypedef struct my_c_struct: - int a - int b - - - cdef class WrapperClass: - """A wrapper class for a C/C++ data structure""" - cdef my_c_struct *_ptr - cdef bint ptr_owner - - def __cinit__(self): - self.ptr_owner = False - - def __dealloc__(self): - # De-allocate if not null and flag is set - if self._ptr is not NULL and self.ptr_owner is True: - free(self._ptr) - self._ptr = NULL - - # Extension class properties - @property - def a(self): - return self._ptr.a if self._ptr is not NULL else None - - @property - def b(self): - return self._ptr.b if self._ptr is not NULL else None - - @staticmethod - cdef WrapperClass from_ptr(my_c_struct *_ptr, bint owner=False): - """Factory function to create WrapperClass objects from - given my_c_struct pointer. - - Setting ``owner`` flag to ``True`` causes - the extension type to ``free`` the structure pointed to by ``_ptr`` - when the wrapper object is deallocated.""" - # Call to __new__ bypasses __init__ constructor - cdef WrapperClass wrapper = WrapperClass.__new__(WrapperClass) - wrapper._ptr = _ptr - wrapper.ptr_owner = owner - return wrapper - - @staticmethod - cdef WrapperClass new_struct(): - """Factory function to create WrapperClass objects with - newly allocated my_c_struct""" - cdef my_c_struct *_ptr = <my_c_struct *>malloc(sizeof(my_c_struct)) - if _ptr is NULL: - raise MemoryError - _ptr.a = 0 - _ptr.b = 0 - return WrapperClass.from_ptr(_ptr, owner=True) +constructors, this necessitates the use of factory functions or factory methods. For example: + +.. tabs:: + + .. group-tab:: Pure Python + + .. literalinclude:: ../../examples/userguide/extension_types/wrapper_class.py + + .. group-tab:: Cython + + .. literalinclude:: ../../examples/userguide/extension_types/wrapper_class.pyx To then create a ``WrapperClass`` object from an existing ``my_c_struct`` @@ -602,19 +761,139 @@ Making extension types weak-referenceable By default, extension types do not support having weak references made to them. You can enable weak referencing by declaring a C attribute of type -object called :attr:`__weakref__`. For example,:: +object called :attr:`__weakref__`. For example: + +.. tabs:: + + .. group-tab:: Pure Python + + .. code-block:: python + + @cython.cclass + class ExplodingAnimal: + """This animal will self-destruct when it is + no longer strongly referenced.""" + + __weakref__: object - cdef class ExplodingAnimal: - """This animal will self-destruct when it is - no longer strongly referenced.""" + .. group-tab:: Cython - cdef object __weakref__ + .. code-block:: cython + cdef class ExplodingAnimal: + """This animal will self-destruct when it is + no longer strongly referenced.""" -Controlling cyclic garbage collection in CPython -================================================ + cdef object __weakref__ + + +Controlling deallocation and garbage collection in CPython +========================================================== + +.. NOTE:: + + This section only applies to the usual CPython implementation + of Python. Other implementations like PyPy work differently. + +.. _dealloc_intro: + +Introduction +------------ -By default each extension type will support the cyclic garbage collector of +First of all, it is good to understand that there are two ways to +trigger deallocation of Python objects in CPython: +CPython uses reference counting for all objects and any object with a +reference count of zero is immediately deallocated. This is the most +common way of deallocating an object. For example, consider :: + + >>> x = "foo" + >>> x = "bar" + +After executing the second line, the string ``"foo"`` is no longer referenced, +so it is deallocated. This is done using the ``tp_dealloc`` slot, which can be +customized in Cython by implementing ``__dealloc__``. + +The second mechanism is the cyclic garbage collector. +This is meant to resolve cyclic reference cycles such as :: + + >>> class Object: + ... pass + >>> def make_cycle(): + ... x = Object() + ... y = [x] + ... x.attr = y + +When calling ``make_cycle``, a reference cycle is created since ``x`` +references ``y`` and vice versa. Even though neither ``x`` or ``y`` +are accessible after ``make_cycle`` returns, both have a reference count +of 1, so they are not immediately deallocated. At regular times, the garbage +collector runs, which will notice the reference cycle +(using the ``tp_traverse`` slot) and break it. +Breaking a reference cycle means taking an object in the cycle +and removing all references from it to other Python objects (we call this +*clearing* an object). Clearing is almost the same as deallocating, except +that the actual object is not yet freed. For ``x`` in the example above, +the attributes of ``x`` would be removed from ``x``. + +Note that it suffices to clear just one object in the reference cycle, +since there is no longer a cycle after clearing one object. Once the cycle +is broken, the usual refcount-based deallocation will actually remove the +objects from memory. Clearing is implemented in the ``tp_clear`` slot. +As we just explained, it is sufficient that one object in the cycle +implements ``tp_clear``. + +.. _trashcan: + +Enabling the deallocation trashcan +---------------------------------- + +In CPython, it is possible to create deeply recursive objects. For example:: + + >>> L = None + >>> for i in range(2**20): + ... L = [L] + +Now imagine that we delete the final ``L``. Then ``L`` deallocates +``L[0]``, which deallocates ``L[0][0]`` and so on until we reach a +recursion depth of ``2**20``. This deallocation is done in C and such +a deep recursion will likely overflow the C call stack, crashing Python. + +CPython invented a mechanism for this called the *trashcan*. It limits the +recursion depth of deallocations by delaying some deallocations. + +By default, Cython extension types do not use the trashcan but it can be +enabled by setting the ``trashcan`` directive to ``True``. For example: + +.. tabs:: + + .. group-tab:: Pure Python + + .. code-block:: python + + import cython + @cython.trashcan(True) + @cython.cclass + class Object: + __dict__: dict + + .. group-tab:: Cython + + .. code-block:: cython + + cimport cython + @cython.trashcan(True) + cdef class Object: + cdef dict __dict__ + +Trashcan usage is inherited by subclasses +(unless explicitly disabled by ``@cython.trashcan(False)``). +Some builtin types like ``list`` use the trashcan, so subclasses of it +use the trashcan by default. + +Disabling cycle breaking (``tp_clear``) +--------------------------------------- + +By default, each extension type will support the cyclic garbage collector of CPython. If any Python objects can be referenced, Cython will automatically generate the ``tp_traverse`` and ``tp_clear`` slots. This is usually what you want. @@ -622,48 +901,90 @@ want. There is at least one reason why this might not be what you want: If you need to cleanup some external resources in the ``__dealloc__`` special function and your object happened to be in a reference cycle, the garbage collector may -have triggered a call to ``tp_clear`` to drop references. This is the way that -reference cycles are broken so that the garbage can actually be reclaimed. - -In that case any object references have vanished by the time when -``__dealloc__`` is called. Now your cleanup code lost access to the objects it -has to clean up. In that case you can disable the cycle breaker ``tp_clear`` -by using the ``no_gc_clear`` decorator :: - - @cython.no_gc_clear - cdef class DBCursor: - cdef DBConnection conn - cdef DBAPI_Cursor *raw_cursor - # ... - def __dealloc__(self): - DBAPI_close_cursor(self.conn.raw_conn, self.raw_cursor) +have triggered a call to ``tp_clear`` to clear the object +(see :ref:`dealloc_intro`). + +In that case, any object references have vanished when ``__dealloc__`` +is called. Now your cleanup code lost access to the objects it has to clean up. +To fix this, you can disable clearing instances of a specific class by using +the ``no_gc_clear`` directive: + +.. tabs:: + + .. group-tab:: Pure Python + + .. code-block:: python + + @cython.no_gc_clear + @cython.cclass + class DBCursor: + conn: DBConnection + raw_cursor: cython.pointer(DBAPI_Cursor) + # ... + def __dealloc__(self): + DBAPI_close_cursor(self.conn.raw_conn, self.raw_cursor) + + .. group-tab:: Cython + + .. code-block:: cython + + @cython.no_gc_clear + cdef class DBCursor: + cdef DBConnection conn + cdef DBAPI_Cursor *raw_cursor + # ... + def __dealloc__(self): + DBAPI_close_cursor(self.conn.raw_conn, self.raw_cursor) This example tries to close a cursor via a database connection when the Python object is destroyed. The ``DBConnection`` object is kept alive by the reference from ``DBCursor``. But if a cursor happens to be in a reference cycle, the -garbage collector may effectively "steal" the database connection reference, +garbage collector may delete the database connection reference, which makes it impossible to clean up the cursor. -Using the ``no_gc_clear`` decorator this can not happen anymore because the -references of a cursor object will not be cleared anymore. +If you use ``no_gc_clear``, it is important that any given reference cycle +contains at least one object *without* ``no_gc_clear``. Otherwise, the cycle +cannot be broken, which is a memory leak. + +Disabling cyclic garbage collection +----------------------------------- In rare cases, extension types can be guaranteed not to participate in cycles, but the compiler won't be able to prove this. This would be the case if the class can never reference itself, even indirectly. In that case, you can manually disable cycle collection by using the -``no_gc`` decorator, but beware that doing so when in fact the extension type -can participate in cycles could cause memory leaks :: +``no_gc`` directive, but beware that doing so when in fact the extension type +can participate in cycles could cause memory leaks: + +.. tabs:: + + .. group-tab:: Pure Python + + .. code-block:: python + + @cython.no_gc + @cython.cclass + class UserInfo: + name: str + addresses: tuple - @cython.no_gc - cdef class UserInfo: - cdef str name - cdef tuple addresses + .. group-tab:: Cython + + .. code-block:: cython + + @cython.no_gc + cdef class UserInfo: + + cdef str name + cdef tuple addresses If you can be sure addresses will contain only references to strings, the above would be safe, and it may yield a significant speedup, depending on your usage pattern. +.. _auto_pickle: + Controlling pickling ==================== @@ -688,6 +1009,13 @@ declaration makes an extension type defined in external C code available to a Cython module. A public extension type declaration makes an extension type defined in a Cython module available to external C code. +.. note:: + + Cython currently does not support Extension types declared as extern or public + in Pure Python mode. This is not considered an issue since public/extern extension + types are most commonly declared in `.pxd` files and not in `.py` files. + + .. _external_extension_types: External extension types @@ -704,7 +1032,7 @@ objects defined in the Python core or in a non-Cython extension module. :ref:`sharing-declarations`. Here is an example which will let you get at the C-level members of the -built-in complex object.:: +built-in complex object:: from __future__ import print_function @@ -730,7 +1058,7 @@ built-in complex object.:: because, in the Python header files, the ``PyComplexObject`` struct is declared with: - .. sourcecode:: c + .. code-block:: c typedef struct { ... @@ -805,8 +1133,7 @@ write ``dtype.itemsize`` in Cython code which will be compiled into direct access of the C struct field, without going through a C-API equivalent of ``dtype.__getattr__('itemsize')``. -For example we may have an extension -module ``foo_extension``:: +For example, we may have an extension module ``foo_extension``:: cdef class Foo: cdef public int field0, field1, field2; @@ -816,7 +1143,9 @@ module ``foo_extension``:: self.field1 = f1 self.field2 = f2 -but a C struct in a file ``foo_nominal.h``:: +but a C struct in a file ``foo_nominal.h``: + +.. code-block:: c typedef struct { PyObject_HEAD @@ -850,6 +1179,7 @@ use the C-API equivalent of:: instead of the desired C equivalent of ``return f->f0 + f->f1 + f->f2``. We can alias the fields by using:: + cdef extern from "foo_nominal.h": ctypedef class foo_extension.Foo [object FooStructNominal]: @@ -867,6 +1197,19 @@ code. No changes to Python need be made to achieve significant speedups, even though the field names in Python and C are different. Of course, one should make sure the fields are equivalent. +C inline properties +------------------- + +Similar to Python property attributes, Cython provides a way to declare C-level +properties on external extension types. This is often used to shadow Python +attributes through faster C level data access, but can also be used to add certain +functionality to existing types when using them from Cython. The declarations +must use `cdef inline`. + +For example, the above ``complex`` type could also be declared like this: + +.. literalinclude:: ../../examples/userguide/extension_types/c_property.pyx + Implicit importing ------------------ @@ -942,5 +1285,37 @@ generated containing declarations for its object struct and type object. By including the ``.h`` file in external C code that you write, that code can access the attributes of the extension type. +Dataclass extension types +========================= + +Cython supports extension types that behave like the dataclasses defined in +the Python 3.7+ standard library. The main benefit of using a dataclass is +that it can auto-generate simple ``__init__``, ``__repr__`` and comparison +functions. The Cython implementation behaves as much like the Python +standard library implementation as possible and therefore the documentation +here only briefly outlines the differences - if you plan on using them +then please read `the documentation for the standard library module +<https://docs.python.org/3/library/dataclasses.html>`_. + +Dataclasses can be declared using the ``@cython.dataclasses.dataclass`` +decorator on a Cython extension type. ``@cython.dataclasses.dataclass`` +can only be applied to extension types (types marked ``cdef`` or created with the +``cython.cclass`` decorator) and not to regular classes. If +you need to define special properties on a field then use ``cython.dataclasses.field`` + +.. tabs:: + + .. group-tab:: Pure Python + + .. literalinclude:: ../../examples/userguide/extension_types/dataclass.py + + .. group-tab:: Cython + .. literalinclude:: ../../examples/userguide/extension_types/dataclass.pyx +You may use C-level types such as structs, pointers, or C++ classes. +However, you may find these types are not compatible with the auto-generated +special methods - for example if they cannot be converted from a Python +type they cannot be passed to a constructor, and so you must use a +``default_factory`` to initialize them. Like with the Python implementation, you can also control +which special functions an attribute is used in using ``field()``. diff --git a/docs/src/userguide/external_C_code.rst b/docs/src/userguide/external_C_code.rst index e6605223d..6108d2cbf 100644 --- a/docs/src/userguide/external_C_code.rst +++ b/docs/src/userguide/external_C_code.rst @@ -189,19 +189,19 @@ same applies equally to union and enum declarations. +-------------------------+---------------------------------------------+-----------------------------------------------------------------------+ | C code | Possibilities for corresponding Cython Code | Comments | +=========================+=============================================+=======================================================================+ -| .. sourcecode:: c | :: | Cython will refer to the as ``struct Foo`` in the generated C code. | -| | | | +| .. code-block:: c | :: | Cython will refer to the type as ``struct Foo`` in | +| | | the generated C code. | | struct Foo { | cdef struct Foo: | | | ... | ... | | | }; | | | +-------------------------+---------------------------------------------+-----------------------------------------------------------------------+ -| .. sourcecode:: c | :: | Cython will refer to the type simply as ``Foo`` in | +| .. code-block:: c | :: | Cython will refer to the type simply as ``Foo`` in | | | | the generated C code. | | typedef struct { | ctypedef struct Foo: | | | ... | ... | | | } Foo; | | | +-------------------------+---------------------------------------------+-----------------------------------------------------------------------+ -| .. sourcecode:: c | :: | If the C header uses both a tag and a typedef with *different* | +| .. code-block:: c | :: | If the C header uses both a tag and a typedef with *different* | | | | names, you can use either form of declaration in Cython | | typedef struct foo { | cdef struct foo: | (although if you need to forward reference the type, | | ... | ... | you'll have to use the first form). | @@ -212,7 +212,7 @@ same applies equally to union and enum declarations. | | ctypedef struct Foo: | | | | ... | | +-------------------------+---------------------------------------------+-----------------------------------------------------------------------+ -| .. sourcecode:: c | :: | If the header uses the *same* name for the tag and typedef, you | +| .. code-block:: c | :: | If the header uses the *same* name for the tag and typedef, you | | | | won't be able to include a :keyword:`ctypedef` for it -- but then, | | typedef struct Foo { | cdef struct Foo: | it's not necessary. | | ... | ... | | @@ -223,6 +223,38 @@ See also use of :ref:`external_extension_types`. Note that in all the cases below, you refer to the type in Cython code simply as :c:type:`Foo`, not ``struct Foo``. +Pointers +-------- +When interacting with a C-api there may be functions that require pointers as arguments. +Pointers are variables that contain a memory address to another variable. + +For example:: + + cdef extern from "<my_lib.h>": + cdef void increase_by_one(int *my_var) + +This function takes a pointer to an integer as argument. Knowing the address of the +integer allows the function to modify the value in place, so that the caller can see +the changes afterwards. In order to get the address from an existing variable, +use the ``&`` operator:: + + cdef int some_int = 42 + cdef int *some_int_pointer = &some_int + increase_by_one(some_int_pointer) + # Or without creating the extra variable + increase_by_one(&some_int) + print(some_int) # prints 44 (== 42+1+1) + +If you want to manipulate the variable the pointer points to, you can access it by +referencing its first element like you would in python ``my_pointer[0]``. For example:: + + cdef void increase_by_one(int *my_var): + my_var[0] += 1 + +For a deeper introduction to pointers, you can read `this tutorial at tutorialspoint +<https://www.tutorialspoint.com/cprogramming/c_pointers.htm>`_. For differences between +Cython and C syntax for manipulating pointers, see :ref:`statements_and_expressions`. + Accessing Python/C API routines --------------------------------- @@ -235,6 +267,16 @@ routines in the Python/C API. For example,:: will allow you to create Python strings containing null bytes. +Note that Cython comes with ready-to-use declarations of (almost) all C-API functions +in the cimportable ``cpython.*`` modules. See the list in +https://github.com/cython/cython/tree/master/Cython/Includes/cpython + +You should always use submodules (e.g. ``cpython.object``, ``cpython.list``) to +access these functions. Historically Cython has made some of the C-API functions +available under directly under the ``cpython`` module. However, this is +deprecated, will be removed eventually, and any new additions will not be added +there. + Special Types -------------- @@ -329,13 +371,16 @@ are entirely on your own with this feature. If you want to declare a name the C file for it, you can do this using a C name declaration. Consider this an advanced feature, only for the rare cases where everything else fails. + +.. _verbatim_c: + Including verbatim C code ------------------------- For advanced use cases, Cython allows you to directly write C code as "docstring" of a ``cdef extern from`` block: -.. literalinclude:: ../../examples/userguide/external_C_code/c_code_docstring.pyx +.. literalinclude:: ../../examples/userguide/external_C_code/verbatim_c_code.pyx The above is essentially equivalent to having the C code in a file ``header.h`` and writing :: @@ -344,6 +389,11 @@ The above is essentially equivalent to having the C code in a file long square(long x) void assign(long& x, long y) +This feature is commonly used for platform specific adaptations at +compile time, for example: + +.. literalinclude:: ../../examples/userguide/external_C_code/platform_adaptation.pyx + It is also possible to combine a header file and verbatim C code:: cdef extern from "badheader.h": @@ -356,6 +406,11 @@ It is also possible to combine a header file and verbatim C code:: In this case, the C code ``#undef int`` is put right after ``#include "badheader.h"`` in the C code generated by Cython. +Verbatim C code can also be used for version specific adaptations, e.g. when +a struct field was added to a library but is not available in older versions: + +.. literalinclude:: ../../examples/userguide/external_C_code/struct_field_adaptation.pyx + Note that the string is parsed like any other docstring in Python. If you require character escapes to be passed into the C code file, use a raw docstring, i.e. ``r""" ... """``. @@ -381,12 +436,12 @@ You can make C types, variables and functions defined in a Cython module accessible to C code that is linked together with the Cython-generated C file, by declaring them with the public keyword:: - cdef public struct Bunny: # public type declaration + cdef public struct Bunny: # a public type declaration int vorpalness - cdef public int spam # public variable declaration + cdef public int spam # a public variable declaration - cdef public void grail(Bunny *) # public function declaration + cdef public void grail(Bunny *) # a public function declaration If there are any public declarations in a Cython module, a header file called :file:`modulename.h` file is generated containing equivalent C declarations for @@ -416,7 +471,9 @@ For example, in the following snippet that includes :file:`grail.h`: } This C code can then be built together with the Cython-generated C code -in a single program (or library). +in a single program (or library). Be aware that this program will not include +any external dependencies that your module uses. Therefore typically this will +not generate a truly portable application for most cases. In Python 3.x, calling the module init function directly should be avoided. Instead, use the `inittab mechanism <https://docs.python.org/3/c-api/import.html#c._inittab>`_ @@ -439,6 +496,30 @@ file consists of the full dotted name of the module, e.g. a module called the resulting ``.so`` file like a dynamic library. Beware that this is not portable, so it should be avoided. +C++ public declarations +^^^^^^^^^^^^^^^^^^^^^^^ + +When a file is compiled as C++, its public functions are declared as C++ API (using ``extern "C++"``) by default. +This disallows to call the functions from C code. If the functions are really meant as a plain C API, +the ``extern`` declaration needs to be manually specified by the user. +This can be done by setting the ``CYTHON_EXTERN_C`` C macro to ``extern "C"`` during the compilation of the generated C++ file:: + + from setuptools import Extension, setup + from Cython.Build import cythonize + + extensions = [ + Extension( + "module", ["module.pyx"], + define_macros=[("CYTHON_EXTERN_C", 'extern "C"')], + language="c++", + ) + ] + + setup( + name="My hello app", + ext_modules=cythonize(extensions), + ) + .. _api: C API Declarations @@ -488,7 +569,7 @@ the call to :func:`import_modulename`, it is likely that this wasn't done. You can use both :keyword:`public` and :keyword:`api` on the same function to make it available by both methods, e.g.:: - cdef public api void belt_and_braces(): + cdef public api void belt_and_braces() except *: ... However, note that you should include either :file:`modulename.h` or @@ -513,8 +594,8 @@ You can declare a whole group of items as :keyword:`public` and/or example,:: cdef public api: - void order_spam(int tons) - char *get_lunch(float tomato_size) + void order_spam(int tons) except * + char *get_lunch(float tomato_size) except NULL This can be a useful thing to do in a ``.pxd`` file (see :ref:`sharing-declarations`) to make the module's public interface @@ -524,7 +605,7 @@ Acquiring and Releasing the GIL --------------------------------- Cython provides facilities for acquiring and releasing the -`Global Interpreter Lock (GIL) <http://docs.python.org/dev/glossary.html#term-global-interpreter-lock>`_. +Global Interpreter Lock (GIL) (see :term:`our glossary<Global Interpreter Lock or GIL>` or `external documentation <https://docs.python.org/dev/glossary.html#term-global-interpreter-lock>`_). This may be useful when calling from multi-threaded code into (external C) code that may block, or when wanting to use Python from a (native) C thread callback. Releasing the GIL should @@ -549,14 +630,18 @@ You can release the GIL around a section of code using the with nogil: <code to be executed with the GIL released> -Code in the body of the with-statement must not raise exceptions or -manipulate Python objects in any way, and must not call anything that -manipulates Python objects without first re-acquiring the GIL. Cython -validates these operations at compile time, but cannot look into -external C functions, for example. They must be correctly declared -as requiring or not requiring the GIL (see below) in order to make +Code in the body of the with-statement must not manipulate Python objects +in any way, and must not call anything that manipulates Python objects without +first re-acquiring the GIL. Cython validates these operations at compile time, +but cannot look into external C functions, for example. They must be correctly +declared as requiring or not requiring the GIL (see below) in order to make Cython's checks effective. +Since Cython 3.0, some simple Python statements can be used inside of ``nogil`` +sections: ``raise``, ``assert`` and ``print`` (the Py2 statement, not the function). +Since they tend to be lone Python statements, Cython will automatically acquire +and release the GIL around them for convenience. + .. _gil: Acquiring the GIL @@ -580,6 +665,26 @@ The GIL may also be acquired through the ``with gil`` statement:: with gil: <execute this block with the GIL acquired> +.. _gil_conditional: + +Conditional Acquiring / Releasing the GIL +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Sometimes it is helpful to use a condition to decide whether to run a +certain piece of code with or without the GIL. This code would run anyway, +the difference is whether the GIL will be held or released. +The condition must be constant (at compile time). + +This could be useful for profiling, debugging, performance testing, and +for fused types (see :ref:`fused_gil_conditional`).:: + + DEF FREE_GIL = True + + with nogil(FREE_GIL): + <code to be executed with the GIL released> + + with gil(False): + <GIL is still released> + Declaring a function as callable without the GIL -------------------------------------------------- diff --git a/docs/src/userguide/fusedtypes.rst b/docs/src/userguide/fusedtypes.rst index b50bb0efd..3167c77d3 100644 --- a/docs/src/userguide/fusedtypes.rst +++ b/docs/src/userguide/fusedtypes.rst @@ -6,6 +6,9 @@ Fused Types (Templates) *********************** +.. include:: + ../two-syntax-variants-used + Fused types allow you to have one type definition that can refer to multiple types. This allows you to write a single static-typed cython algorithm that can operate on values of multiple types. Thus fused types allow `generic @@ -22,9 +25,19 @@ Java / C#. Quickstart ========== -.. literalinclude:: ../../examples/userguide/fusedtypes/char_or_float.pyx +.. tabs:: + + .. group-tab:: Pure Python + + .. literalinclude:: ../../examples/userguide/fusedtypes/char_or_float.py + + .. group-tab:: Cython -This gives:: + .. literalinclude:: ../../examples/userguide/fusedtypes/char_or_float.pyx + +This gives: + +.. code-block:: pycon >>> show_me() char -128 @@ -36,104 +49,247 @@ whereas ``plus_one(b)`` specializes ``char_or_float`` as a ``float``. Declaring Fused Types ===================== -Fused types may be declared as follows:: +Fused types may be declared as follows: - cimport cython +.. tabs:: - ctypedef fused my_fused_type: - cython.int - cython.double + .. group-tab:: Pure Python -This declares a new type called ``my_fused_type`` which can be *either* an -``int`` *or* a ``double``. Alternatively, the declaration may be written as:: + .. code-block:: python + + my_fused_type = cython.fused_type(cython.int, cython.float) - my_fused_type = cython.fused_type(cython.int, cython.float) + .. group-tab:: Cython + + .. code-block:: cython + + ctypedef fused my_fused_type: + int + double + +This declares a new type called ``my_fused_type`` which can be *either* an +``int`` *or* a ``double``. Only names may be used for the constituent types, but they may be any -(non-fused) type, including a typedef. i.e. one may write:: +(non-fused) type, including a typedef. I.e. one may write: + +.. tabs:: + + .. group-tab:: Pure Python + + .. code-block:: python + + my_double = cython.typedef(cython.double) + my_fused_type = cython.fused_type(cython.int, my_double) - ctypedef double my_double - my_fused_type = cython.fused_type(cython.int, my_double) + .. group-tab:: Cython + + .. code-block:: cython + + ctypedef double my_double + ctypedef fused fused_type: + int + my_double Using Fused Types ================= -Fused types can be used to declare parameters of functions or methods:: +Fused types can be used to declare parameters of functions or methods: - cdef cfunc(my_fused_type arg): - return arg + 1 +.. tabs:: -If the you use the same fused type more than once in an argument list, then each -specialization of the fused type must be the same:: + .. group-tab:: Pure Python - cdef cfunc(my_fused_type arg1, my_fused_type arg2): - return cython.typeof(arg1) == cython.typeof(arg2) + .. code-block:: python -In this case, the type of both parameters is either an int, or a double -(according to the previous examples). However, because these arguments use the -same fused type ``my_fused_type``, both ``arg1`` and ``arg2`` are -specialized to the same type. Therefore this function returns True for every -possible valid invocation. You are allowed to mix fused types however:: + @cython.cfunc + def cfunc(arg: my_fused_type): + return arg + 1 - def func(A x, B y): - ... + .. group-tab:: Cython + + .. code-block:: cython + + cdef cfunc(my_fused_type arg): + return arg + 1 + +If the same fused type appears more than once in the function arguments, +then they will all have the same specialised type: + +.. tabs:: + + .. group-tab:: Pure Python + + .. code-block:: python + + @cython.cfunc + def cfunc(arg1: my_fused_type, arg2: my_fused_type): + # arg1 and arg2 always have the same type here + return arg1 + arg2 + + .. group-tab:: Cython + + .. code-block:: cython + + cdef cfunc(my_fused_type arg1, my_fused_type arg2): + # arg1 and arg2 always have the same type here + return arg1 + arg2 + +Here, the type of both parameters is either an int, or a double +(according to the previous examples), because they use the same fused type +name ``my_fused_type``. Mixing different fused types (or differently named +fused types) in the arguments will specialise them independently: + +.. tabs:: + + .. group-tab:: Pure Python + + .. code-block:: python + + def func(x: A, y: B): + ... + + .. group-tab:: Cython + + .. code-block:: cython + + + def func(A x, B y): + ... + +This will result in specialized code paths for all combinations of types +contained in ``A`` and ``B``, e.g.: + +.. tabs:: + + .. group-tab:: Pure Python + + .. code-block:: python + + my_fused_type = cython.fused_type(cython.int, cython.double) + + + + my_fused_type2 = cython.fused_type(cython.int, cython.double) + + + @cython.cfunc + def func(a: my_fused_type, b: my_fused_type2): + # a and b may have the same or different types here + print("SAME!" if my_fused_type is my_fused_type2 else "NOT SAME!") + return a + b + + .. group-tab:: Cython + + .. code-block:: cython + + ctypedef fused my_fused_type: + int + double + + ctypedef fused my_fused_type2: + int + double + + cdef func(my_fused_type a, my_fused_type2 b): + # a and b may have the same or different types here + print("SAME!" if my_fused_type is my_fused_type2 else "NOT SAME!") + return a + b + + + + +.. Note:: A simple typedef to rename the fused type does not currently work here. + See Github issue :issue:`4302`. -where ``A`` and ``B`` are different fused types. This will result in -specialized code paths for all combinations of types contained in ``A`` -and ``B``. Fused types and arrays ---------------------- Note that specializations of only numeric types may not be very useful, as one can usually rely on promotion of types. This is not true for arrays, pointers -and typed views of memory however. Indeed, one may write:: +and typed views of memory however. Indeed, one may write: - def myfunc(A[:, :] x): - ... +.. tabs:: - # and + .. group-tab:: Pure Python - cdef otherfunc(A *x): - ... + .. code-block:: python -Note that in Cython 0.20.x and earlier, the compiler generated the full cross -product of all type combinations when a fused type was used by more than one -memory view in a type signature, e.g. + @cython.cfunc + def myfunc(x: A[:, :]): + ... -:: + # and - def myfunc(A[:] a, A[:] b): - # a and b had independent item types in Cython 0.20.x and earlier. - ... + @cython.cfunc + cdef otherfunc(x: cython.pointer(A)): + ... -This was unexpected for most users, unlikely to be desired, and also inconsistent -with other structured type declarations like C arrays of fused types, which were -considered the same type. It was thus changed in Cython 0.21 to use the same -type for all memory views of a fused type. In order to get the original -behaviour, it suffices to declare the same fused type under different names, and -then use these in the declarations:: - ctypedef fused A: - int - long + .. group-tab:: Cython - ctypedef fused B: - int - long + .. code-block:: cython - def myfunc(A[:] a, B[:] b): - # a and b are independent types here and may have different item types - ... + cdef myfunc(A[:, :] x): + ... -To get only identical types also in older Cython versions (pre-0.21), a ``ctypedef`` -can be used:: + # and - ctypedef A[:] A_1d + cdef otherfunc(A *x): + ... - def myfunc(A_1d a, A_1d b): - # a and b have identical item types here, also in older Cython versions - ... +Following code snippet shows an example with pointer to the fused type: + +.. tabs:: + + .. group-tab:: Pure Python + + .. literalinclude:: ../../examples/userguide/fusedtypes/pointer.py + + .. group-tab:: Cython + + .. literalinclude:: ../../examples/userguide/fusedtypes/pointer.pyx + +.. Note:: + + In Cython 0.20.x and earlier, the compiler generated the full cross + product of all type combinations when a fused type was used by more than one + memory view in a type signature, e.g. + + :: + + def myfunc(A[:] a, A[:] b): + # a and b had independent item types in Cython 0.20.x and earlier. + ... + + This was unexpected for most users, unlikely to be desired, and also inconsistent + with other structured type declarations like C arrays of fused types, which were + considered the same type. It was thus changed in Cython 0.21 to use the same + type for all memory views of a fused type. In order to get the original + behaviour, it suffices to declare the same fused type under different names, and + then use these in the declarations:: + + ctypedef fused A: + int + long + + ctypedef fused B: + int + long + + def myfunc(A[:] a, B[:] b): + # a and b are independent types here and may have different item types + ... + + To get only identical types also in older Cython versions (pre-0.21), a ``ctypedef`` + can be used:: + + ctypedef A[:] A_1d + + def myfunc(A_1d a, A_1d b): + # a and b have identical item types here, also in older Cython versions + ... Selecting Specializations @@ -143,36 +299,125 @@ You can select a specialization (an instance of the function with specific or specialized (i.e., non-fused) argument types) in two ways: either by indexing or by calling. + +.. _fusedtypes_indexing: + Indexing -------- -You can index functions with types to get certain specializations, i.e.:: +You can index functions with types to get certain specializations, i.e.: - cfunc[cython.p_double](p1, p2) +.. tabs:: - # From Cython space - func[float, double](myfloat, mydouble) + .. group-tab:: Pure Python - # From Python space - func[cython.float, cython.double](myfloat, mydouble) + .. literalinclude:: ../../examples/userguide/fusedtypes/indexing.py + :caption: indexing.py -If a fused type is used as a base type, this will mean that the base type is the -fused type, so the base type is what needs to be specialized:: + .. group-tab:: Cython - cdef myfunc(A *x): - ... + .. literalinclude:: ../../examples/userguide/fusedtypes/indexing.pyx + :caption: indexing.pyx + +Indexed functions can be called directly from Python: + +.. code-block:: pycon + + >>> import cython + >>> import indexing + cfunc called: double 5.0 double 1.0 + cpfunc called: float 1.0 double 2.0 + func called: float 1.0 double 2.0 + >>> indexing.cpfunc[cython.float, cython.float](1, 2) + cpfunc called: float 1.0 float 2.0 + >>> indexing.func[cython.float, cython.float](1, 2) + func called: float 1.0 float 2.0 + +If a fused type is used as a component of a more complex type +(for example a pointer to a fused type, or a memoryview of a fused type), +then you should index the function with the individual component and +not the full argument type: + +.. tabs:: + + .. group-tab:: Pure Python + + .. code-block:: python + + @cython.cfunc + def myfunc(x: cython.pointer(A)): + ... + + # Specialize using int, not int * + myfunc[cython.int](myint) + + .. group-tab:: Cython + + .. code-block:: cython + + cdef myfunc(A *x): + ... + + # Specialize using int, not int * + myfunc[int](myint) + +For memoryview indexing from python space we can do the following: + +.. tabs:: - # Specialize using int, not int * - myfunc[int](myint) + .. group-tab:: Pure Python + + .. code-block:: python + + my_fused_type = cython.fused_type(cython.int[:, ::1], cython.float[:, ::1]) + + def func(array: my_fused_type): + print("func called:", cython.typeof(array)) + + my_fused_type[cython.int[:, ::1]](myarray) + + .. group-tab:: Cython + + .. code-block:: cython + + ctypedef fused my_fused_type: + int[:, ::1] + float[:, ::1] + + def func(my_fused_type array): + print("func called:", cython.typeof(array)) + + my_fused_type[cython.int[:, ::1]](myarray) + +The same goes for when using e.g. ``cython.numeric[:, :]``. Calling ------- A fused function can also be called with arguments, where the dispatch is -figured out automatically:: +figured out automatically: + +.. tabs:: + + .. group-tab:: Pure Python + + .. code-block:: python + + def main(): + p1: cython.double = 1.0 + p2: cython.float = 2.0 + cfunc(p1, p1) # prints "cfunc called: double 1.0 double 1.0" + cpfunc(p1, p2) # prints "cpfunc called: double 1.0 float 2.0" + + .. group-tab:: Cython - cfunc(p1, p2) - func(myfloat, mydouble) + .. code-block:: cython + + def main(): + cdef double p1 = 1.0 + cdef float p2 = 2.0 + cfunc(p1, p1) # prints "cfunc called: double 1.0 double 1.0" + cpfunc(p1, p2) # prints "cpfunc called: double 1.0 float 2.0" For a ``cdef`` or ``cpdef`` function called from Cython this means that the specialization is figured out at compile time. For ``def`` functions the @@ -201,6 +446,8 @@ There are some built-in fused types available for convenience, these are:: Casting Fused Functions ======================= +.. note:: Pointers to functions are currently not supported by pure Python mode. (GitHub issue :issue:`4279`) + Fused ``cdef`` and ``cpdef`` functions may be cast or assigned to C function pointers as follows:: cdef myfunc(cython.floating, cython.integral): @@ -226,57 +473,72 @@ False conditions are pruned to avoid invalid code. One may check with ``is``, ``is not`` and ``==`` and ``!=`` to see if a fused type is equal to a certain other non-fused type (to check the specialization), or use ``in`` and ``not in`` to figure out whether a specialization is part of another set of types -(specified as a fused type). In example:: +(specified as a fused type). In example: - ctypedef fused bunch_of_types: - ... +.. tabs:: + + .. group-tab:: Pure Python + + .. literalinclude:: ../../examples/userguide/fusedtypes/type_checking.py + + .. group-tab:: Cython + + .. literalinclude:: ../../examples/userguide/fusedtypes/type_checking.pyx + +.. _fused_gil_conditional: - ctypedef fused string_t: - cython.p_char - bytes - unicode +Conditional GIL Acquiring / Releasing +===================================== - cdef cython.integral myfunc(cython.integral i, bunch_of_types s): - cdef int *int_pointer - cdef long *long_pointer +Acquiring and releasing the GIL can be controlled by a condition +which is known at compile time (see :ref:`gil_conditional`). - # Only one of these branches will be compiled for each specialization! - if cython.integral is int: - int_pointer = &i - else: - long_pointer = &i +.. Note:: Pure python mode currently does not support Conditional GIL Acquiring / Releasing. See Github issue :issue:`5113`. - if bunch_of_types in string_t: - print("s is a string!") +This is most useful when combined with fused types. +A fused type function may have to handle both cython native types +(e.g. cython.int or cython.double) and python types (e.g. object or bytes). +Conditional Acquiring / Releasing the GIL provides a method for running +the same piece of code either with the GIL released (for cython native types) +and with the GIL held (for python types): + +.. literalinclude:: ../../examples/userguide/fusedtypes/conditional_gil.pyx __signatures__ ============== Finally, function objects from ``def`` or ``cpdef`` functions have an attribute -__signatures__, which maps the signature strings to the actual specialized -functions. This may be useful for inspection. Listed signature strings may also -be used as indices to the fused function, but the index format may change between -Cython versions:: +``__signatures__``, which maps the signature strings to the actual specialized +functions. This may be useful for inspection: - specialized_function = fused_function["MyExtensionClass|int|float"] +.. tabs:: -It would usually be preferred to index like this, however:: + .. group-tab:: Pure Python - specialized_function = fused_function[MyExtensionClass, int, float] + .. literalinclude:: ../../examples/userguide/fusedtypes/indexing.py + :lines: 1-9,14-16 + :caption: indexing.py -Although the latter will select the biggest types for ``int`` and ``float`` from -Python space, as they are not type identifiers but builtin types there. Passing -``cython.int`` and ``cython.float`` would resolve that, however. + .. group-tab:: Cython -For memoryview indexing from python space we can do the following:: + .. literalinclude:: ../../examples/userguide/fusedtypes/indexing.pyx + :lines: 1-9,14-16 + :caption: indexing.pyx - ctypedef fused my_fused_type: - int[:, ::1] - float[:, ::1] +.. code-block:: pycon - def func(my_fused_type array): - ... + >>> from indexing import cpfunc + >>> cpfunc.__signatures__, + ({'double|double': <cyfunction __pyx_fuse_0_0cpfunc at 0x107292f20>, 'double|float': <cyfunction __pyx_fuse_0_1cpfunc at 0x1072a6040>, 'float|double': <cyfunction __pyx_fuse_1_0cpfunc at 0x1072a6120>, 'float|float': <cyfunction __pyx_fuse_1_1cpfunc at 0x1072a6200>},) - my_fused_type[cython.int[:, ::1]](myarray) +Listed signature strings may also +be used as indices to the fused function, but the index format may change between +Cython versions -The same goes for when using e.g. ``cython.numeric[:, :]``. +.. code-block:: pycon + + >>> specialized_function = cpfunc["double|float"] + >>> specialized_function(5.0, 1.0) + cpfunc called: double 5.0 float 1.0 + +However, the better way how to index is by providing list of types as mentioned in :ref:`fusedtypes_indexing` section. diff --git a/docs/src/userguide/glossary.rst b/docs/src/userguide/glossary.rst new file mode 100644 index 000000000..ddd38df5f --- /dev/null +++ b/docs/src/userguide/glossary.rst @@ -0,0 +1,60 @@ +Glossary +======== + +.. glossary:: + + Extension type + "Extension type" can refer to either a Cython class defined with ``cdef class`` or ``@cclass``, + or more generally to any Python type that is ultimately implemented as a + native C struct (including the built-in types like `int` or `dict`). + + Dynamic allocation or Heap allocation + A C variable allocated with ``malloc`` (in C) or ``new`` (in C++) is + `allocated dynamically/heap allocated <https://en.wikipedia.org/wiki/C_dynamic_memory_allocation>`_. + Its lifetime is until the user deletes it explicitly (with ``free`` in C or ``del`` in C++). + This can happen in a different function than the allocation. + + Global Interpreter Lock or GIL + A lock inside the Python interpreter to ensure that only one Python thread is run at once. + This lock is purely to ensure that race conditions do not corrupt internal Python state. + Python objects cannot be manipulated unless the GIL is held. + It is most relevant to Cython when writing code that should be run in parallel. If you are + not aiming to write parallel code then there is usually no benefit to releasing the GIL in + Cython. You should not use the GIL as a general locking mechanism in your code since many + operations on Python objects can lead to it being released and to control being passed to + another thread. Also see the `CPython project's glossary entry <https://docs.python.org/dev/glossary.html#term-global-interpreter-lock>`_. + + pointer + A **pointer** is a variable that stores the address of another variable + (i.e. direct address of the memory location). They allow for + dynamic memory allocation and deallocation. They can be used to build + dynamic data structures. + `Read more <https://en.wikipedia.org/wiki/Pointer_(computer_programming)#C_pointers>`__. + + Python object + When using Python, the contents of every variable is a Python object + (including Cython extension types). Key features of Python objects are that + they are passed *by reference* and that their lifetime is *managed* automatically + so that they are destroyed when no more references exist to them. + In Cython, they are distinct from C types, which are passed *by value* and whose + lifetime is managed depending on whether they are allocated on the stack or heap. + To explicitly declare a Python object variable in Cython use ``cdef object abc``. + Internally in C, they are referred to as ``PyObject*``. + + Stack allocation + A C variable declared within a function as ``cdef SomeType a`` + is said to be allocated on the stack. + It exists for the duration of the function only. + + Typed memoryview + A useful Cython type for getting quick access to blocks of memory. + A memoryview alone does not actually own any memory. + However, it can be initialized with a Python object that supports the + `buffer protocol`_ (typically "array" types, for example a Numpy array). + The memoryview keeps a reference to that Python object alive + and provides quick access to the memory without needing to go + through the Python API of the object and its + :meth:`__getitem__` / :meth:`__setitem__` methods. + For more information, see :ref:`memoryviews`. + +.. _buffer protocol: https://docs.python.org/3/c-api/buffer.html diff --git a/docs/src/userguide/index.rst b/docs/src/userguide/index.rst index cfbee2fbd..a89d6d65a 100644 --- a/docs/src/userguide/index.rst +++ b/docs/src/userguide/index.rst @@ -16,6 +16,7 @@ Contents: wrapping_CPlusPlus fusedtypes pypy + migrating_to_cy30 limitations pyrex_differences memoryviews @@ -23,7 +24,9 @@ Contents: parallelism debugging numpy_tutorial + numpy_ufuncs numpy_pythran + nogil Indices and tables ------------------ @@ -31,6 +34,3 @@ Indices and tables * :ref:`genindex` * :ref:`modindex` * :ref:`search` - -.. toctree:: - diff --git a/docs/src/userguide/language_basics.rst b/docs/src/userguide/language_basics.rst index c3b9f36e4..aa057dd98 100644 --- a/docs/src/userguide/language_basics.rst +++ b/docs/src/userguide/language_basics.rst @@ -11,6 +11,10 @@ Language Basics ***************** +.. include:: + ../two-syntax-variants-used + + .. _declaring_data_types: Declaring Data Types @@ -44,74 +48,237 @@ the use of ‘early binding’ programming techniques. C variable and type definitions =============================== -The :keyword:`cdef` statement is used to declare C variables, either local or -module-level:: +C variables can be declared by + +* using the Cython specific :keyword:`cdef` statement, +* using PEP-484/526 type annotations with C data types or +* using the function ``cython.declare()``. + +The :keyword:`cdef` statement and ``declare()`` can define function-local and +module-level variables as well as attributes in classes, but type annotations only +affect local variables and attributes and are ignored at the module level. +This is because type annotations are not Cython specific, so Cython keeps +the variables in the module dict (as Python values) instead of making them +module internal C variables. Use ``declare()`` in Python code to explicitly +define global C variables. + +.. tabs:: + + .. group-tab:: Pure Python + + .. code-block:: python + + a_global_variable = declare(cython.int) + + def func(): + i: cython.int + j: cython.int + k: cython.int + f: cython.float + g: cython.float[42] + h: cython.p_float + + i = j = 5 + + .. group-tab:: Cython + + .. code-block:: cython + + cdef int a_global_variable + + def func(): + cdef int i, j, k + cdef float f + cdef float[42] g + cdef float *h + # cdef float f, g[42], *h # mix of pointers, arrays and values in a single line is deprecated + + i = j = 5 + +As known from C, declared global variables are automatically initialised to +``0``, ``NULL`` or ``None``, depending on their type. However, also as known +from both Python and C, for a local variable, simply declaring it is not enough +to initialise it. If you use a local variable but did not assign a value, both +Cython and the C compiler will issue a warning "local variable ... referenced +before assignment". You need to assign a value at some point before first +using the variable, but you can also assign a value directly as part of +the declaration in most cases: + +.. tabs:: + + .. group-tab:: Pure Python + + .. code-block:: python + + a_global_variable = declare(cython.int, 42) - cdef int i, j, k - cdef float f, g[42], *h + def func(): + i: cython.int = 10 + f: cython.float = 2.5 + g: cython.int[4] = [1, 2, 3, 4] + h: cython.p_float = cython.address(f) -and C :keyword:`struct`, :keyword:`union` or :keyword:`enum` types: + .. group-tab:: Cython -.. literalinclude:: ../../examples/userguide/language_basics/struct_union_enum.pyx + .. code-block:: cython -See also :ref:`struct-union-enum-styles` + cdef int a_global_variable + + def func(): + cdef int i = 10, j, k + cdef float f = 2.5 + cdef int[4] g = [1, 2, 3, 4] + cdef float *h = &f .. note:: - Structs can be declared as ``cdef packed struct``, which has - the same effect as the C directive ``#pragma pack(1)``. + There is also support for giving names to types using the + ``ctypedef`` statement or the ``cython.typedef()`` function, e.g. -Declaring an enum as ``cpdef`` will create a :pep:`435`-style Python wrapper:: + .. tabs:: - cpdef enum CheeseState: - hard = 1 - soft = 2 - runny = 3 + .. group-tab:: Pure Python + .. code-block:: python + ULong = cython.typedef(cython.ulong) -There is currently no special syntax for defining a constant, but you can use -an anonymous :keyword:`enum` declaration for this purpose, for example,:: + IntPtr = cython.typedef(cython.p_int) - cdef enum: - tons_of_spam = 3 + .. group-tab:: Cython + + .. code-block:: cython + + ctypedef unsigned long ULong + + ctypedef int* IntPtr + +C Arrays +-------- + +C array can be declared by adding ``[ARRAY_SIZE]`` to the type of variable: + +.. tabs:: + + .. group-tab:: Pure Python + + .. code-block:: python + + def func(): + g: cython.float[42] + f: cython.int[5][5][5] + + .. group-tab:: Cython + + .. code-block:: cython + + def func(): + cdef float[42] g + cdef int[5][5][5] f .. note:: - the words ``struct``, ``union`` and ``enum`` are used only when - defining a type, not when referring to it. For example, to declare a variable - pointing to a ``Grail`` you would write:: - cdef Grail *gp + Cython syntax currently supports two ways to declare an array: - and not:: + .. code-block:: cython - cdef struct Grail *gp # WRONG + cdef int arr1[4], arr2[4] # C style array declaration + cdef int[4] arr1, arr2 # Java style array declaration - There is also a ``ctypedef`` statement for giving names to types, e.g.:: + Both of them generate the same C code, but the Java style is more + consistent with :ref:`memoryviews` and :ref:`fusedtypes`. The C style + declaration is soft-deprecated and it's recommended to use Java style + declaration instead. - ctypedef unsigned long ULong + The soft-deprecated C style array declaration doesn't support + initialization. - ctypedef int* IntPtr + .. code-block:: cython + cdef int g[4] = [1, 2, 3, 4] # error -It is also possible to declare functions with :keyword:`cdef`, making them c functions. + cdef int[4] g = [1, 2, 3, 4] # OK -:: + cdef int g[4] # OK but not recommended + g = [1, 2, 3, 4] - cdef int eggs(unsigned long l, float f): - ... +.. _structs: -You can read more about them in :ref:`python_functions_vs_c_functions`. +Structs, Unions, Enums +---------------------- -You can declare classes with :keyword:`cdef`, making them :ref:`extension-types`. Those will -have a behavior very close to python classes, but are faster because they use a ``struct`` -internally to store attributes. +In addition to the basic types, C :keyword:`struct`, :keyword:`union` and :keyword:`enum` +are supported: -Here is a simple example: +.. tabs:: -.. literalinclude:: ../../examples/userguide/extension_types/shrubbery.pyx + .. group-tab:: Pure Python + + .. literalinclude:: ../../examples/userguide/language_basics/struct.py + + .. group-tab:: Cython + + .. literalinclude:: ../../examples/userguide/language_basics/struct.pyx + +Structs can be declared as ``cdef packed struct``, which has +the same effect as the C directive ``#pragma pack(1)``:: + + cdef packed struct StructArray: + int[4] spam + signed char[5] eggs + +.. note:: + This declaration removes the empty + space between members that C automatically to ensure that they're aligned in memory + (see `Wikipedia article <https://en.wikipedia.org/wiki/Data_structure_alignment>`_ for more details). + The main use is that numpy structured arrays store their data in packed form, so a ``cdef packed struct`` + can be :ref:`used in a memoryview<using_memoryviews>` to match that. + + Pure python mode does not support packed structs. + +The following example shows a declaration of unions: + +.. tabs:: + + .. group-tab:: Pure Python + + .. literalinclude:: ../../examples/userguide/language_basics/union.py + + .. group-tab:: Cython + + .. literalinclude:: ../../examples/userguide/language_basics/union.pyx + +Enums are created by ``cdef enum`` statement: + +.. literalinclude:: ../../examples/userguide/language_basics/enum.pyx + + +.. note:: Currently, Pure Python mode does not support enums. (GitHub issue :issue:`4252`) + +Declaring an enum as ``cpdef`` will create a :pep:`435`-style Python wrapper:: + + cpdef enum CheeseState: + hard = 1 + soft = 2 + runny = 3 + +There is currently no special syntax for defining a constant, but you can use +an anonymous :keyword:`enum` declaration for this purpose, for example,:: + + cdef enum: + tons_of_spam = 3 + +.. note:: + In the Cython syntax, the words ``struct``, ``union`` and ``enum`` are used only when + defining a type, not when referring to it. For example, to declare a variable + pointing to a ``Grail`` struct, you would write:: + + cdef Grail *gp + + and not:: + + cdef struct Grail *gp # WRONG -You can read more about them in :ref:`extension-types`. .. _typing_types: .. _types: @@ -119,15 +286,21 @@ You can read more about them in :ref:`extension-types`. Types ----- -Cython uses the normal C syntax for C types, including pointers. It provides +The Cython language uses the normal C syntax for C types, including pointers. It provides all the standard C types, namely ``char``, ``short``, ``int``, ``long``, -``long long`` as well as their ``unsigned`` versions, e.g. ``unsigned int``. +``long long`` as well as their ``unsigned`` versions, +e.g. ``unsigned int`` (``cython.uint`` in Python code). The special ``bint`` type is used for C boolean values (``int`` with 0/non-0 values for False/True) and ``Py_ssize_t`` for (signed) sizes of Python containers. -Pointer types are constructed as in C, by appending a ``*`` to the base type -they point to, e.g. ``int**`` for a pointer to a pointer to a C int. +Pointer types are constructed as in C when using Cython syntax, by appending a ``*`` to the base type +they point to, e.g. ``int**`` for a pointer to a pointer to a C int. In Pure python mode, simple pointer types +use a naming scheme with "p"s instead, separated from the type name with an underscore, e.g. ``cython.pp_int`` for a pointer to +a pointer to a C int. Further pointer types can be constructed with the ``cython.pointer()`` function, +e.g. ``cython.pointer(cython.int)``. + + Arrays use the normal C array syntax, e.g. ``int[10]``, and the size must be known at compile time for stack allocated arrays. Cython doesn't support variable length arrays from C99. Note that Cython uses array access for pointer dereferencing, as ``*x`` is not valid Python syntax, @@ -135,23 +308,53 @@ whereas ``x[0]`` is. Also, the Python types ``list``, ``dict``, ``tuple``, etc. may be used for static typing, as well as any user defined :ref:`extension-types`. -For example:: +For example + +.. tabs:: + + .. group-tab:: Pure Python + + .. code-block:: python + + def main(): + foo: list = [] + + .. group-tab:: Cython + + .. code-block:: cython + + cdef list foo = [] + +This requires an *exact* match of the class, it does not allow subclasses. +This allows Cython to optimize code by accessing internals of the builtin class, +which is the main reason for declaring builtin types in the first place. - cdef list foo = [] +For declared builtin types, Cython uses internally a C variable of type ``PyObject*``. -This requires an *exact* match of the class, it does not allow -subclasses. This allows Cython to optimize code by accessing -internals of the builtin class. -For this kind of typing, Cython uses internally a C variable of type ``PyObject*``. -The Python types int, long, and float are not available for static -typing and instead interpreted as C ``int``, ``long``, and ``float`` -respectively, as statically typing variables with these Python -types has zero advantages. +.. note:: The Python types ``int``, ``long``, and ``float`` are not available for static + typing in ``.pyx`` files and instead interpreted as C ``int``, ``long``, and ``float`` + respectively, as statically typing variables with these Python + types has zero advantages. On the other hand, annotating in Pure Python with + ``int``, ``long``, and ``float`` Python types will be interpreted as + Python object types. Cython provides an accelerated and typed equivalent of a Python tuple, the ``ctuple``. -A ``ctuple`` is assembled from any valid C types. For example:: +A ``ctuple`` is assembled from any valid C types. For example - cdef (double, int) bar +.. tabs:: + + .. group-tab:: Pure Python + + .. code-block:: python + + def main(): + bar: tuple[cython.double, cython.int] + + .. group-tab:: Cython + + .. code-block:: cython + + cdef (double, int) bar They compile down to C-structures and can be used as efficient alternatives to Python tuples. @@ -164,12 +367,58 @@ and is typically what one wants). If you want to use these numeric Python types simply omit the type declaration and let them be objects. -It is also possible to declare :ref:`extension-types` (declared with ``cdef class``). -This does allow subclasses. This typing is mostly used to access -``cdef`` methods and attributes of the extension type. + +Type qualifiers +--------------- + +Cython supports ``const`` and ``volatile`` `C type qualifiers <https://en.wikipedia.org/wiki/Type_qualifier>`_:: + + cdef volatile int i = 5 + + cdef const int sum(const int a, const int b): + return a + b + + cdef void print_const_pointer(const int *value): + print(value[0]) + + cdef void print_pointer_to_const_value(int * const value): + print(value[0]) + + cdef void print_const_pointer_to_const_value(const int * const value): + print(value[0]) + +.. Note:: + + Both type qualifiers are not supported by pure python mode. Moreover, the ``const`` modifier is unusable + in a lot of contexts since Cython needs to generate definitions and their assignments separately. Therefore + we suggest using it mainly for function argument and pointer types where ``const`` is necessary to + work with an existing C/C++ interface. + + +Extension Types +--------------- + +It is also possible to declare :ref:`extension-types` (declared with ``cdef class`` or the ``@cclass`` decorator). +Those will have a behaviour very close to python classes (e.g. creating subclasses), +but access to their members is faster from Cython code. Typing a variable +as extension type is mostly used to access ``cdef``/``@cfunc`` methods and attributes of the extension type. The C code uses a variable which is a pointer to a structure of the specific type, something like ``struct MyExtensionTypeObject*``. +Here is a simple example: + +.. tabs:: + + .. group-tab:: Pure Python + + .. literalinclude:: ../../examples/userguide/extension_types/shrubbery.py + + .. group-tab:: Cython + + .. literalinclude:: ../../examples/userguide/extension_types/shrubbery.pyx + +You can read more about them in :ref:`extension-types`. + Grouping multiple C declarations -------------------------------- @@ -177,8 +426,11 @@ Grouping multiple C declarations If you have a series of declarations that all begin with :keyword:`cdef`, you can group them into a :keyword:`cdef` block like this: +.. note:: This is supported only in Cython's ``cdef`` syntax. + .. literalinclude:: ../../examples/userguide/language_basics/cdef_block.pyx + .. _cpdef: .. _cdef: .. _python_functions_vs_c_functions: @@ -188,48 +440,97 @@ Python functions vs. C functions There are two kinds of function definition in Cython: -Python functions are defined using the def statement, as in Python. They take -Python objects as parameters and return Python objects. +Python functions are defined using the :keyword:`def` statement, as in Python. They take +:term:`Python objects<Python object>` as parameters and return Python objects. -C functions are defined using the new :keyword:`cdef` statement. They take +C functions are defined using the :keyword:`cdef` statement in Cython syntax or with the ``@cfunc`` decorator. They take either Python objects or C values as parameters, and can return either Python objects or C values. Within a Cython module, Python functions and C functions can call each other freely, but only Python functions can be called from outside the module by interpreted Python code. So, any functions that you want to "export" from your -Cython module must be declared as Python functions using def. -There is also a hybrid function, called :keyword:`cpdef`. A :keyword:`cpdef` -can be called from anywhere, but uses the faster C calling conventions -when being called from other Cython code. A :keyword:`cpdef` can also be overridden +Cython module must be declared as Python functions using ``def``. +There is also a hybrid function, declared with :keyword:`cpdef` in ``.pyx`` +files or with the ``@ccall`` decorator. These functions +can be called from anywhere, but use the faster C calling convention +when being called from other Cython code. They can also be overridden by a Python method on a subclass or an instance attribute, even when called from Cython. If this happens, most performance gains are of course lost and even if it does not, -there is a tiny overhead in calling a :keyword:`cpdef` method from Cython compared to -calling a :keyword:`cdef` method. +there is a tiny overhead in calling such a method from Cython compared to +calling a C method. Parameters of either type of function can be declared to have C data types, -using normal C declaration syntax. For example,:: +using normal C declaration syntax. For example, - def spam(int i, char *s): - ... +.. tabs:: - cdef int eggs(unsigned long l, float f): - ... + .. group-tab:: Pure Python -``ctuples`` may also be used:: + .. code-block:: python - cdef (int, float) chips((long, long, double) t): - ... + def spam(i: cython.int, s: cython.p_char): + ... + + @cython.cfunc + def eggs(l: cython.ulong, f: cython.float) -> cython.int: + ... + + .. group-tab:: Cython + + .. code-block:: cython + + def spam(int i, char *s): + ... + + + cdef int eggs(unsigned long l, float f): + ... + +``ctuples`` may also be used + +.. tabs:: + + .. group-tab:: Pure Python + + .. code-block:: python + + @cython.cfunc + def chips(t: tuple[cython.long, cython.long, cython.double]) -> tuple[cython.int, cython.float]: + ... + + .. group-tab:: Cython + + .. code-block:: cython + + cdef (int, float) chips((long, long, double) t): + ... When a parameter of a Python function is declared to have a C data type, it is passed in as a Python object and automatically converted to a C value, if possible. In other words, the definition of ``spam`` above is equivalent to -writing:: +writing + +.. tabs:: + + .. group-tab:: Pure Python + + .. code-block:: python + + def spam(python_i, python_s): + i: cython.int = python_i + s: cython.p_char = python_s + ... + + .. group-tab:: Cython + + .. code-block:: cython + + def spam(python_i, python_s): + cdef int i = python_i + cdef char* s = python_s + ... - def spam(python_i, python_s): - cdef int i = python_i - cdef char* s = python_s - ... Automatic conversion is currently only possible for numeric types, string types and structs (composed recursively of any of these types); @@ -242,24 +543,40 @@ with string attributes if they are to be used after the function returns. C functions, on the other hand, can have parameters of any type, since they're passed in directly using a normal C function call. -Functions declared using :keyword:`cdef` with Python object return type, like Python functions, will return a :keyword:`None` +C Functions declared using :keyword:`cdef` or the ``@cfunc`` decorator with a +Python object return type, like Python functions, will return a :keyword:`None` value when execution leaves the function body without an explicit return value. This is in -contrast to C/C++, which leaves the return value undefined. +contrast to C/C++, which leaves the return value undefined. In the case of non-Python object return types, the equivalent of zero is returned, for example, 0 for ``int``, :keyword:`False` for ``bint`` and :keyword:`NULL` for pointer types. A more complete comparison of the pros and cons of these different method types can be found at :ref:`early-binding-for-speed`. + Python objects as parameters and return values ---------------------------------------------- If no type is specified for a parameter or return value, it is assumed to be a -Python object. (Note that this is different from the C convention, where it -would default to int.) For example, the following defines a C function that -takes two Python objects as parameters and returns a Python object:: +Python object. (Note that this is different from the C convention, where it +would default to ``int``.) For example, the following defines a C function that +takes two Python objects as parameters and returns a Python object - cdef spamobjs(x, y): - ... +.. tabs:: + + .. group-tab:: Pure Python + + .. code-block:: python + + @cython.cfunc + def spamobjs(x, y): + ... + + .. group-tab:: Cython + + .. code-block:: cython + + cdef spamobjs(x, y): + ... Reference counting for these objects is performed automatically according to the standard Python/C API rules (i.e. borrowed references are taken as @@ -270,46 +587,104 @@ parameters and a new reference is returned). This only applies to Cython code. Other Python packages which are implemented in C like NumPy may not follow these conventions. - -The name object can also be used to explicitly declare something as a Python +The type name ``object`` can also be used to explicitly declare something as a Python object. This can be useful if the name being declared would otherwise be taken -as the name of a type, for example,:: +as the name of a type, for example, - cdef ftang(object int): - ... +.. tabs:: -declares a parameter called int which is a Python object. You can also use -object as the explicit return type of a function, e.g.:: + .. group-tab:: Pure Python - cdef object ftang(object int): - ... + .. code-block:: python + + @cython.cfunc + def ftang(int: object): + ... + + .. group-tab:: Cython + + .. code-block:: cython + + cdef ftang(object int): + ... + +declares a parameter called ``int`` which is a Python object. You can also use +``object`` as the explicit return type of a function, e.g. + +.. tabs:: + + .. group-tab:: Pure Python + + .. code-block:: python + + @cython.cfunc + def ftang(int: object) -> object: + ... + + .. group-tab:: Cython + + .. code-block:: cython + + cdef object ftang(object int): + ... In the interests of clarity, it is probably a good idea to always be explicit about object parameters in C functions. +To create a borrowed reference, specify the parameter type as ``PyObject*``. +Cython won't perform automatic ``Py_INCREF``, or ``Py_DECREF``, e.g.: + +.. tabs:: + + .. group-tab:: Pure Python + + .. literalinclude:: ../../examples/userguide/language_basics/parameter_refcount.py + + .. group-tab:: Cython + + .. literalinclude:: ../../examples/userguide/language_basics/parameter_refcount.pyx + +will display:: + + Initial refcount: 2 + Inside owned_reference: 3 + Inside borrowed_reference: 2 + + .. _optional_arguments: Optional Arguments ------------------ -Unlike C, it is possible to use optional arguments in ``cdef`` and ``cpdef`` functions. -There are differences though whether you declare them in a ``.pyx`` +Unlike C, it is possible to use optional arguments in C and ``cpdef``/``@ccall`` functions. +There are differences though whether you declare them in a ``.pyx``/``.py`` file or the corresponding ``.pxd`` file. To avoid repetition (and potential future inconsistencies), default argument values are not visible in the declaration (in ``.pxd`` files) but only in the implementation (in ``.pyx`` files). -When in a ``.pyx`` file, the signature is the same as it is in Python itself: +When in a ``.pyx``/``.py`` file, the signature is the same as it is in Python itself: + +.. tabs:: + + .. group-tab:: Pure Python -.. literalinclude:: ../../examples/userguide/language_basics/optional_subclassing.pyx + .. literalinclude:: ../../examples/userguide/language_basics/optional_subclassing.py + :caption: optional_subclassing.py + + .. group-tab:: Cython + + .. literalinclude:: ../../examples/userguide/language_basics/optional_subclassing.pyx + :caption: optional_subclassing.pyx When in a ``.pxd`` file, the signature is different like this example: ``cdef foo(x=*)``. This is because the program calling the function just needs to know what signatures are possible in C, but doesn't need to know the value of the default arguments.: .. literalinclude:: ../../examples/userguide/language_basics/optional_subclassing.pxd + :caption: optional_subclassing.pxd .. note:: The number of arguments may increase when subclassing, @@ -342,14 +717,24 @@ terminate the list of positional arguments: Shown above, the signature takes exactly two positional parameters and has two required keyword parameters. + Function Pointers ----------------- -Functions declared in a ``struct`` are automatically converted to function pointers. +.. note:: Pointers to functions are currently not supported by pure Python mode. (GitHub issue :issue:`4279`) + +The following example shows declaring a ``ptr_add`` function pointer and assigning the ``add`` function to it: + +.. literalinclude:: ../../examples/userguide/language_basics/function_pointer.pyx + +Functions declared in a ``struct`` are automatically converted to function pointers: + +.. literalinclude:: ../../examples/userguide/language_basics/function_pointer_struct.pyx For using error return values with function pointers, see the note at the bottom of :ref:`error_return_values`. + .. _error_return_values: Error return values @@ -362,24 +747,42 @@ through defined error return values. For functions that return a Python object ``NULL`` pointer, so any function returning a Python object has a well-defined error return value. -While this is always the case for :keyword:`def` functions, functions -defined as :keyword:`cdef` or :keyword:`cpdef` can return arbitrary C types, -which do not have such a well-defined error return value. Thus, if an -exception is detected in such a function, a warning message is printed, -the exception is ignored, and the function returns immediately without -propagating the exception to its caller. +While this is always the case for Python functions, functions +defined as C functions or ``cpdef``/``@ccall`` functions can return arbitrary C types, +which do not have such a well-defined error return value. +By default Cython uses a dedicated return value to signal that an exception has been raised from non-external ``cpdef``/``@ccall`` +functions. However, how Cython handles exceptions from these functions can be changed if needed. -If you want such a C function to be able to propagate exceptions, you need -to declare an exception return value for it as a contract with the caller. -Here is an example:: +A ``cdef`` function may be declared with an exception return value for it +as a contract with the caller. Here is an example: - cdef int spam() except -1: - ... +.. tabs:: + + .. group-tab:: Pure Python + + .. code-block:: python + + @cython.cfunc + @cython.exceptval(-1) + def spam() -> cython.int: + ... + + .. group-tab:: Cython + + .. code-block:: cython + + cdef int spam() except -1: + ... With this declaration, whenever an exception occurs inside ``spam``, it will immediately return with the value ``-1``. From the caller's side, whenever a call to spam returns ``-1``, the caller will assume that an exception has -occurred and can now process or propagate it. +occurred and can now process or propagate it. Calling ``spam()`` is roughly translated to the following C code: + +.. code-block:: C + + ret_val = spam(); + if (ret_val == -1) goto error_handler; When you declare an exception value for a function, you should never explicitly or implicitly return that value. This includes empty :keyword:`return` @@ -392,51 +795,123 @@ returns small results. If all possible return values are legal and you can't reserve one entirely for signalling errors, you can use an alternative -form of exception value declaration:: +form of exception value declaration - cdef int spam() except? -1: - ... +.. tabs:: + + .. group-tab:: Pure Python -The "?" indicates that the value ``-1`` only signals a possible error. In this -case, Cython generates a call to :c:func:`PyErr_Occurred` if the exception value + .. code-block:: python + + @cython.cfunc + @cython.exceptval(-1, check=True) + def spam() -> cython.int: + ... + + The keyword argument ``check=True`` indicates that the value ``-1`` **may** signal an error. + + .. group-tab:: Cython + + .. code-block:: cython + + cdef int spam() except? -1: + ... + + The ``?`` indicates that the value ``-1`` **may** signal an error. + +In this case, Cython generates a call to :c:func:`PyErr_Occurred` if the exception value is returned, to make sure it really received an exception and not just a normal -result. +result. Calling ``spam()`` is roughly translated to the following C code: -There is also a third form of exception value declaration:: - cdef int spam() except *: - ... +.. code-block:: C + + ret_val = spam(); + if (ret_val == -1 && PyErr_Occurred()) goto error_handler; + +There is also a third form of exception value declaration + +.. tabs:: + + .. group-tab:: Pure Python + + .. code-block:: python + + @cython.cfunc + @cython.exceptval(check=True) + def spam() -> cython.void: + ... + + .. group-tab:: Cython + + .. code-block:: cython + + cdef void spam() except *: + ... This form causes Cython to generate a call to :c:func:`PyErr_Occurred` after -*every* call to spam, regardless of what value it returns. If you have a +*every* call to spam, regardless of what value it returns. Calling ``spam()`` is roughly translated to the following C code: + +.. code-block:: C + + spam() + if (PyErr_Occurred()) goto error_handler; + +If you have a function returning ``void`` that needs to propagate errors, you will have to use this form, since there isn't any error return value to test. Otherwise, an explicit error return value allows the C compiler to generate more efficient code and is thus generally preferable. -To explicitly mark a function as not returning an exception use -``noexcept``. - - cdef int spam() noexcept: - ... - -This is worth doing because (a) "explicit is better than implicit", and -(b) the default behaviour for ``cdef`` functions will change in Cython 3.0 -so that functions will propagate exceptions by default. Therefore, it is -best to mark them now if you want them to swallow exceptions in the future. - An external C++ function that may raise an exception can be declared with:: cdef int spam() except + +.. note:: These declarations are not used in Python code, only in ``.pxd`` and ``.pyx`` files. + See :ref:`wrapping-cplusplus` for more details. +Finally, if you are certain that your function should not raise an exception, (e.g., it +does not use Python objects at all, or you plan to use it as a callback in C code that +is unaware of Python exceptions), you can declare it as such using ``noexcept`` or by ``@cython.exceptval(check=False)``: + +.. tabs:: + + .. group-tab:: Pure Python + + .. code-block:: python + + @cython.cfunc + @cython.exceptval(check=False) + def spam() -> cython.int: + ... + + .. group-tab:: Cython + + .. code-block:: cython + + cdef int spam() noexcept: + ... + +If a ``noexcept`` function *does* finish with an exception then it will print a warning message but not allow the exception to propagate further. +On the other hand, calling a ``noexcept`` function has zero overhead related to managing exceptions, unlike the previous declarations. + Some things to note: +* ``cdef`` functions that are also ``extern`` are implicitly declared ``noexcept`` or ``@cython.exceptval(check=False)``. + In the uncommon case of external C/C++ functions that **can** raise Python exceptions, + e.g., external functions that use the Python C API, you should explicitly declare + them with an exception value. + +* ``cdef`` functions that are *not* ``extern`` are implicitly declared with a suitable + exception specification for the return type (e.g. ``except *`` or ``@cython.exceptval(check=True)`` for a ``void`` return + type, ``except? -1`` or ``@cython.exceptval(-1, check=True)`` for an ``int`` return type). + * Exception values can only be declared for functions returning a C integer, enum, float or pointer type, and the value must be a constant expression. Functions that return ``void``, or a struct/union by value, can only use - the ``except *`` form. + the ``except *`` or ``exceptval(check=True)`` form. + * The exception value specification is part of the signature of the function. If you're passing a pointer to a function as a parameter or assigning it to a variable, the declared type of the parameter or variable must have @@ -445,10 +920,25 @@ Some things to note: int (*grail)(int, char*) except -1 + .. note:: Pointers to functions are currently not supported by pure Python mode. (GitHub issue :issue:`4279`) + +* If the returning type of a ``cdef`` function with ``except *`` or ``@cython.exceptval(check=True)`` is C integer, + enum, float or pointer type, Cython calls :c:func:`PyErr_Occurred` only when + dedicated value is returned instead of checking after every call of the function. + * You don't need to (and shouldn't) declare exception values for functions which return Python objects. Remember that a function with no declared return type implicitly returns a Python object. (Exceptions on such functions are implicitly propagated by returning ``NULL``.) + +* There's a known performance pitfall when combining ``nogil`` and + ``except *`` \ ``@cython.exceptval(check=True)``. + In this case Cython must always briefly re-acquire the GIL after a function + call to check if an exception has been raised. This can commonly happen with a + function returning nothing (C ``void``). Simple workarounds are to mark the + function as ``noexcept`` if you're certain that exceptions cannot be thrown, or + to change the return type to ``int`` and just let Cython use the return value + as an error flag (by default, ``-1`` triggers the exception check). .. _checking_return_values_of_non_cython_functions: @@ -469,7 +959,16 @@ function or a C function that calls Python/C API routines. To get an exception from a non-Python-aware function such as :func:`fopen`, you will have to check the return value and raise it yourself, for example: -.. literalinclude:: ../../examples/userguide/language_basics/open_file.pyx +.. tabs:: + + .. group-tab:: Pure Python + + .. literalinclude:: ../../examples/userguide/language_basics/open_file.py + + .. group-tab:: Cython + + .. literalinclude:: ../../examples/userguide/language_basics/open_file.pyx + .. _overriding_in_extension_types: @@ -477,15 +976,30 @@ Overriding in extension types ----------------------------- -``cpdef`` methods can override ``cdef`` methods: +``cpdef``/``@ccall`` methods can override C methods: -.. literalinclude:: ../../examples/userguide/language_basics/optional_subclassing.pyx +.. tabs:: + + .. group-tab:: Pure Python + + .. literalinclude:: ../../examples/userguide/language_basics/optional_subclassing.py + + .. group-tab:: Cython + + .. literalinclude:: ../../examples/userguide/language_basics/optional_subclassing.pyx When subclassing an extension type with a Python class, -``def`` methods can override ``cpdef`` methods but not ``cdef`` -methods: +Python methods can override ``cpdef``/``@ccall`` methods but not plain C methods: + +.. tabs:: -.. literalinclude:: ../../examples/userguide/language_basics/override.pyx + .. group-tab:: Pure Python + + .. literalinclude:: ../../examples/userguide/language_basics/override.py + + .. group-tab:: Cython + + .. literalinclude:: ../../examples/userguide/language_basics/override.pyx If ``C`` above would be an extension type (``cdef class``), this would not work correctly. @@ -547,13 +1061,27 @@ as the C string is needed. If you can't guarantee that the Python string will live long enough, you will need to copy the C string. Cython detects and prevents some mistakes of this kind. For instance, if you -attempt something like:: +attempt something like + +.. tabs:: + + .. group-tab:: Pure Python - cdef char *s - s = pystring1 + pystring2 + .. code-block:: python -then Cython will produce the error message ``Obtaining char* from temporary -Python value``. The reason is that concatenating the two Python strings + def main(): + s: cython.p_char + s = pystring1 + pystring2 + + .. group-tab:: Cython + + .. code-block:: cython + + cdef char *s + s = pystring1 + pystring2 + +then Cython will produce the error message ``Storing unsafe C derivative of temporary +Python reference``. The reason is that concatenating the two Python strings produces a new Python string object that is referenced only by a temporary internal variable that Cython generates. As soon as the statement has finished, the temporary variable will be decrefed and the Python string deallocated, @@ -561,11 +1089,26 @@ leaving ``s`` dangling. Since this code could not possibly work, Cython refuses compile it. The solution is to assign the result of the concatenation to a Python -variable, and then obtain the ``char*`` from that, i.e.:: +variable, and then obtain the ``char*`` from that, i.e. + +.. tabs:: - cdef char *s - p = pystring1 + pystring2 - s = p + .. group-tab:: Pure Python + + .. code-block:: python + + def main(): + s: cython.p_char + p = pystring1 + pystring2 + s = p + + .. group-tab:: Cython + + .. code-block:: cython + + cdef char *s + p = pystring1 + pystring2 + s = p It is then your responsibility to hold the reference p for as long as necessary. @@ -575,50 +1118,106 @@ Sometimes Cython will complain unnecessarily, and sometimes it will fail to detect a problem that exists. Ultimately, you need to understand the issue and be careful what you do. + .. _type_casting: Type Casting ------------ -Where C uses ``"("`` and ``")"``, Cython uses ``"<"`` and ``">"``. For example:: +The Cython language supports type casting in a similar way as C. Where C uses ``"("`` and ``")"``, +Cython uses ``"<"`` and ``">"``. In pure python mode, the ``cython.cast()`` function is used. For example: + +.. tabs:: + + .. group-tab:: Pure Python + + .. code-block:: python - cdef char *p - cdef float *q - p = <char*>q + def main(): + p: cython.p_char + q: cython.p_float + p = cython.cast(cython.p_char, q) -When casting a C value to a Python object type or vice versa, -Cython will attempt a coercion. Simple examples are casts like ``<int>pyobj``, -which converts a Python number to a plain C ``int`` value, or ``<bytes>charptr``, -which copies a C ``char*`` string into a new Python bytes object. + When casting a C value to a Python object type or vice versa, + Cython will attempt a coercion. Simple examples are casts like ``cast(int, pyobj_value)``, + which convert a Python number to a plain C ``int`` value, or the statement ``cast(bytes, charptr_value)``, + which copies a C ``char*`` string into a new Python bytes object. - .. note:: Cython will not prevent a redundant cast, but emits a warning for it. + .. note:: Cython will not prevent a redundant cast, but emits a warning for it. -To get the address of some Python object, use a cast to a pointer type -like ``<void*>`` or ``<PyObject*>``. -You can also cast a C pointer back to a Python object reference -with ``<object>``, or a more specific builtin or extension type -(e.g. ``<MyExtType>ptr``). This will increase the reference count of -the object by one, i.e. the cast returns an owned reference. -Here is an example: + To get the address of some Python object, use a cast to a pointer type + like ``cast(p_void, ...)`` or ``cast(pointer(PyObject), ...)``. + You can also cast a C pointer back to a Python object reference + with ``cast(object, ...)``, or to a more specific builtin or extension type + (e.g. ``cast(MyExtType, ptr)``). This will increase the reference count of + the object by one, i.e. the cast returns an owned reference. + Here is an example: -.. literalinclude:: ../../examples/userguide/language_basics/casting_python.pyx -The precedence of ``<...>`` is such that ``<type>a.b.c`` is interpreted as ``<type>(a.b.c)``. + .. group-tab:: Cython + + .. code-block:: cython + + cdef char *p + cdef float *q + p = <char*>q + + When casting a C value to a Python object type or vice versa, + Cython will attempt a coercion. Simple examples are casts like ``<int>pyobj_value``, + which convert a Python number to a plain C ``int`` value, or the statement ``<bytes>charptr_value``, + which copies a C ``char*`` string into a new Python bytes object. + + .. note:: Cython will not prevent a redundant cast, but emits a warning for it. + + To get the address of some Python object, use a cast to a pointer type + like ``<void*>`` or ``<PyObject*>``. + You can also cast a C pointer back to a Python object reference + with ``<object>``, or to a more specific builtin or extension type + (e.g. ``<MyExtType>ptr``). This will increase the reference count of + the object by one, i.e. the cast returns an owned reference. + Here is an example: + +.. tabs:: + + .. group-tab:: Pure Python + + .. literalinclude:: ../../examples/userguide/language_basics/casting_python.pxd + :caption: casting_python.pxd + .. literalinclude:: ../../examples/userguide/language_basics/casting_python.py + :caption: casting_python.py + + Casting with ``cast(object, ...)`` creates an owned reference. Cython will automatically + perform a ``Py_INCREF`` and ``Py_DECREF`` operation. Casting to + ``cast(pointer(PyObject), ...)`` creates a borrowed reference, leaving the refcount unchanged. + + .. group-tab:: Cython + + .. literalinclude:: ../../examples/userguide/language_basics/casting_python.pyx + :caption: casting_python.pyx + + The precedence of ``<...>`` is such that ``<type>a.b.c`` is interpreted as ``<type>(a.b.c)``. + + Casting to ``<object>`` creates an owned reference. Cython will automatically + perform a ``Py_INCREF`` and ``Py_DECREF`` operation. Casting to + ``<PyObject *>`` creates a borrowed reference, leaving the refcount unchanged. + .. _checked_type_casts: Checked Type Casts ------------------ -A cast like ``<MyExtensionType>x`` will cast x to the class +A cast like ``<MyExtensionType>x`` or ``cast(MyExtensionType, x)`` will cast ``x`` to the class ``MyExtensionType`` without any checking at all. -To have a cast checked, use the syntax like: ``<MyExtensionType?>x``. +To have a cast checked, use ``<MyExtensionType?>x`` in Cython syntax +or ``cast(MyExtensionType, x, typecheck=True)``. In this case, Cython will apply a runtime check that raises a ``TypeError`` if ``x`` is not an instance of ``MyExtensionType``. This tests for the exact class for builtin types, but allows subclasses for :ref:`extension-types`. + .. _statements_and_expressions: Statements and expressions @@ -636,6 +1235,7 @@ Reference counts are maintained automatically for all Python objects, and all Python operations are automatically checked for errors, with appropriate action taken. + Differences between C and Cython expressions -------------------------------------------- @@ -645,17 +1245,38 @@ direct equivalent in Python. * An integer literal is treated as a C constant, and will be truncated to whatever size your C compiler thinks appropriate. - To get a Python integer (of arbitrary precision) cast immediately to - an object (e.g. ``<object>100000000000000000000``). The ``L``, ``LL``, - and ``U`` suffixes have the same meaning as in C. + To get a Python integer (of arbitrary precision), cast immediately to + an object (e.g. ``<object>100000000000000000000`` or ``cast(object, 100000000000000000000)``). The ``L``, ``LL``, + and ``U`` suffixes have the same meaning in Cython syntax as in C. * There is no ``->`` operator in Cython. Instead of ``p->x``, use ``p.x`` * There is no unary ``*`` operator in Cython. Instead of ``*p``, use ``p[0]`` -* There is an ``&`` operator, with the same semantics as in C. -* The null C pointer is called ``NULL``, not ``0`` (and ``NULL`` is a reserved word). -* Type casts are written ``<type>value`` , for example,:: +* There is an ``&`` operator in Cython, with the same semantics as in C. + In pure python mode, use the ``cython.address()`` function instead. +* The null C pointer is called ``NULL``, not ``0``. ``NULL`` is a reserved word in Cython + and ``cython.NULL`` is a special object in pure python mode. +* Type casts are written ``<type>value`` or ``cast(type, value)``, for example, + + .. tabs:: + + .. group-tab:: Pure Python + + .. code-block:: python + + def main(): + p: cython.p_char + q: cython.p_float + + p = cython.cast(cython.p_char, q) + + .. group-tab:: Cython + + .. code-block:: cython + + cdef char* p + cdef float* q + + p = <char*>q - cdef char* p, float* q - p = <char*>q Scope rules ----------- @@ -667,6 +1288,7 @@ variable residing in the scope where it is assigned. The type of the variable depends on type inference, except for the global module scope, where it is always a Python object. + .. _built_in_functions: Built-in Functions @@ -734,9 +1356,12 @@ Operator Precedence Keep in mind that there are some differences in operator precedence between Python and C, and that Cython uses the Python precedences, not the C ones. + Integer for-loops ------------------ +.. note:: This syntax is supported only in Cython files. Use a normal `for-in-range()` loop instead. + Cython recognises the usual Python for-in-range integer loop pattern:: for i in range(n): @@ -778,6 +1403,7 @@ Some things to note about the for-from loop: Like other Python looping statements, break and continue may be used in the body, and the loop may have an else clause. + .. _cython_file_types: Cython file types @@ -789,6 +1415,7 @@ There are three file types in Cython: * The definition files, carrying a ``.pxd`` suffix. * The include files, carrying a ``.pxi`` suffix. + The implementation file ----------------------- @@ -877,6 +1504,7 @@ of functions or class bodies. separate parts that may be more appropriate in many cases. See :ref:`sharing-declarations`. + .. _conditional_compilation: Conditional Compilation @@ -885,6 +1513,19 @@ Conditional Compilation Some features are available for conditional compilation and compile-time constants within a Cython source file. +.. note:: + + This feature has very little use cases. Specifically, it is not a good + way to adapt code to platform and environment. Use code generation or + (preferably) C compile time adaptation for this. See, for example, + :ref:`verbatim_c`. + +.. note:: + + Cython currently does not support conditional compilation and compile-time + definitions in Pure Python mode. As it stands, this is unlikely to change. + + Compile-Time Definitions ------------------------ @@ -923,6 +1564,7 @@ expression must evaluate to a Python value of type ``int``, ``long``, .. literalinclude:: ../../examples/userguide/language_basics/compile_time.pyx + Conditional Statements ---------------------- diff --git a/docs/src/userguide/limitations.rst b/docs/src/userguide/limitations.rst index 6128c308a..670f01d03 100644 --- a/docs/src/userguide/limitations.rst +++ b/docs/src/userguide/limitations.rst @@ -8,9 +8,12 @@ Limitations This page used to list bugs in Cython that made the semantics of compiled code differ from that in Python. Most of the missing -features have been fixed in Cython 0.15. Note that a -future version 1.0 of Cython is planned to provide full Python -language compatibility. +features have been fixed in Cython 0.15. A future version of +Cython is planned to provide full Python language compatibility. +For now, the issue tracker can provide an overview of deviations +that we are aware of and would like to see fixed. + +https://github.com/cython/cython/labels/Python%20Semantics Below is a list of differences that we will probably not be addressing. Most of these things that fall more into the implementation details rather diff --git a/docs/src/userguide/memoryviews.rst b/docs/src/userguide/memoryviews.rst index 328831e86..285cc67ea 100644 --- a/docs/src/userguide/memoryviews.rst +++ b/docs/src/userguide/memoryviews.rst @@ -20,6 +20,9 @@ A memoryview can be used in any context (function parameters, module-level, cdef class attribute, etc) and can be obtained from nearly any object that exposes writable buffer through the `PEP 3118`_ buffer interface. +.. _`PEP 3118`: https://www.python.org/dev/peps/pep-3118/ + + .. _view_quickstart: Quickstart @@ -39,6 +42,7 @@ This code should give the following output:: Memoryview sum of Cython array is 1351 Memoryview sum of C memoryview is 451 +.. _using_memoryviews: Using memoryviews ================= @@ -56,16 +60,11 @@ A complete 3D view:: cdef int[:,:,:] view3D = exporting_object -A 2D view that restricts the first dimension of a buffer to 100 rows -starting at the second (index 1) and then skips every second (odd) row:: - - cdef int[1:102:2,:] partial_view = exporting_object - -This also works conveniently as function arguments: +They also work conveniently as function arguments: .. code-block:: cython - def process_3d_buffer(int[1:102:2,:] view not None): + def process_3d_buffer(int[:,:,:] view not None): ... The ``not None`` declaration for the argument automatically rejects @@ -80,6 +79,12 @@ three dimensional buffer into a function that requires a two dimensional buffer will raise a ``ValueError``. +To use a memory view on a numpy array with a custom dtype, you'll need to +declare an equivalent packed struct that mimics the dtype: + +.. literalinclude:: ../../examples/userguide/memoryviews/custom_dtype.pyx + + Indexing -------- @@ -234,6 +239,8 @@ NumPy arrays support this interface, as do :ref:`view_cython_arrays`. The data array to themselves be pointers; Cython memoryviews do not yet support this. +.. _`new style buffers`: https://docs.python.org/3/c-api/buffer.html + .. _view_memory_layout: Memory layout @@ -660,6 +667,16 @@ object handling. For the details of how to compile and call functions in C files, see :ref:`using_c_libraries`. -.. _GIL: http://docs.python.org/dev/glossary.html#term-global-interpreter-lock +Performance: Disabling initialization checks +============================================ + +Every time the memoryview is accessed, Cython adds a check to make sure that it has been initialized. + +If you are looking for performance, you can disable them by setting the +``initializedcheck`` directive to ``False``. +See: :ref:`compiler-directives` for more information about this directive. + + +.. _GIL: https://docs.python.org/dev/glossary.html#term-global-interpreter-lock .. _NumPy: https://docs.scipy.org/doc/numpy/reference/arrays.ndarray.html#memory-layout .. _example: https://docs.scipy.org/doc/numpy/reference/arrays.indexing.html diff --git a/docs/src/userguide/migrating_to_cy30.rst b/docs/src/userguide/migrating_to_cy30.rst new file mode 100644 index 000000000..e870f00e8 --- /dev/null +++ b/docs/src/userguide/migrating_to_cy30.rst @@ -0,0 +1,285 @@ +.. highlight:: cython + +.. _cython30: + +********************************* +Migrating from Cython 0.29 to 3.0 +********************************* + +Cython 3.0 is a major revision of the compiler and the language +that comes with some backwards incompatible changes. +This document lists the important ones and explains how to deal with +them in existing code. + + +Python 3 syntax/semantics +========================= + +Cython 3.0 now uses Python 3 syntax and semantics by default, which previously +required setting the ``language_level`` `directive <compiler-directives>` to +either ``3`` or ``3str``. +The new default setting is now ``language_level=3str``, which means Python 3 +semantics, but unprefixed strings are ``str`` objects, i.e. unicode text strings +under Python 3 and byte strings under Python 2.7. + +You can revert your code to the previous (Python 2.x) semantics by setting +``language_level=2``. + +Further semantic changes due to the language level include: + +* ``/``-division uses the true (float) division operator, unless ``cdivision`` is enabled. +* ``print`` is a function, not a statement. +* Python classes that are defined without bases (``class C: ...``) are "new-style" + classes also in Py2.x (if you never heard about "old-style classes", you're probably + happy without them). +* Annotations (type hints) are now stored as strings. + (`PEP 563 <https://github.com/cython/cython/issues/2863>`_) +* ``StopIteration`` handling in generators has been changed according to + `PEP 479 <https://www.python.org/dev/peps/pep-0479/>`_. + + +Python semantics +================ + +Some Python compatibility bugs were fixed, e.g. + +* Subscripting (``x[1]``) now tries the mapping protocol before the sequence protocol. + (https://github.com/cython/cython/issues/1807) +* Exponentiation of integer literals now follows Python semantics and not C semantics. + (https://github.com/cython/cython/issues/2133) + + +Binding functions +================= + +The :ref:`binding directive <compiler-directives>` is now enabled by default. +This makes Cython compiled Python (``def``) functions mostly compatible +with normal (non-compiled) Python functions, regarding signature introspection, +annotations, etc. + +It also makes them bind as methods in Python classes on attribute assignments, +thus the name. +If this is not intended, i.e. if a function is really meant to be a function +and never a method, you can disable the binding (and all other Python function +features) by setting ``binding=False`` or selectively adding a decorator +``@cython.binding(False)``. +In pure Python mode, the decorator was not available in Cython 0.29.16 yet, +but compiled code does not suffer from this. + +We recommend, however, to keep the new function features and instead deal +with the binding issue using the standard Python ``staticmethod()`` builtin. + +:: + + def func(self, b): ... + + class MyClass(object): + binding_method = func + + no_method = staticmethod(func) + + +Namespace packages +================== + +Cython now has support for loading pxd files also from namespace packages +according to `PEP-420 <https://www.python.org/dev/peps/pep-0420/>`_. +This might have an impact on the import path. + + +NumPy C-API +=========== + +Cython used to generate code that depended on the deprecated pre-NumPy-1.7 C-API. +This is no longer the case with Cython 3.0. + +You can now define the macro ``NPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION`` +to get rid of the long-standing build warnings that the compiled C module +uses a deprecated API. Either per file:: + + # distutils: define_macros=NPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION + +or by setting it in your Extensions in ``setup.py``:: + + Extension(... + define_macros=[("NPY_NO_DEPRECATED_API", "NPY_1_7_API_VERSION")] + ) + +One side-effect of the different C-API usage is that your code may now +require a call to the `NumPy C-API initialisation function +<https://docs.scipy.org/doc/numpy-1.17.0/reference/c-api.array.html#importing-the-api>`_ +where it previously got away without doing so. + +In order to reduce the user impact here, Cython 3.0 will now call it +automatically when it sees ``numpy`` being cimported, but the function +not being used. +In the (hopefully rare) cases where this gets in the way, the internal +C-API initialisation can be disabled by faking the use of the function +without actually calling it, e.g. + +:: + + # Explicitly disable the automatic initialisation of NumPy's C-API. + <void>import_array + +Class-private name mangling +=========================== + +Cython has been updated to follow the `Python rules for class-private names +<https://docs.python.org/3/tutorial/classes.html#private-variables>`_ +more closely. Essentially any name that starts with and doesn't end with +``__`` within a class is mangled with the class name. Most user code +should be unaffected -- unlike in Python unmangled global names will +still be matched to ensure it is possible to access C names +beginning with ``__``:: + + cdef extern void __foo() + + class C: # or "cdef class" + def call_foo(self): + return __foo() # still calls the global name + +What will no-longer work is overriding methods starting with ``__`` in +a ``cdef class``:: + + cdef class Base: + cdef __bar(self): + return 1 + + def call_bar(self): + return self.__bar() + + cdef class Derived(Base): + cdef __bar(self): + return 2 + +Here ``Base.__bar`` is mangled to ``_Base__bar`` and ``Derived.__bar`` +to ``_Derived__bar``. Therefore ``call_bar`` will always call +``_Base__bar``. This matches established Python behaviour and applies +for ``def``, ``cdef`` and ``cpdef`` methods and attributes. + +Arithmetic special methods +========================== + +The behaviour of arithmetic special methods (for example ``__add__`` +and ``__pow__``) of cdef classes has changed in Cython 3.0. They now +support separate "reversed" versions of these methods (e.g. +``__radd__``, ``__rpow__``) that behave like in pure Python. +The main incompatible change is that the type of the first operand +(usually ``__self__``) is now assumed to be that of the defining class, +rather than relying on the user to test and cast the type of each operand. + +The old behaviour can be restored with the +:ref:`directive <compiler-directives>` ``c_api_binop_methods=True``. +More details are given in :ref:`arithmetic_methods`. + +Exception values and ``noexcept`` +================================= + +``cdef`` functions that are not ``extern`` now safely propagate Python +exceptions by default. Previously, they needed to explicitly be declared +with an :ref:`exception value <error_return_values>` to prevent them from +swallowing exceptions. A new ``noexcept`` modifier can be used to declare +``cdef`` functions that really will not raise exceptions. + +In existing code, you should mainly look out for ``cdef`` functions +that are declared without an exception value:: + + cdef int spam(int x): + pass + + cdef void silent(int x): + pass + +If you left out the exception value by mistake, i.e., the function +should propagate Python exceptions, then the new behaviour will take +care of this for you, and correctly propagate any exceptions. +This was a common mistake in Cython code and the main reason to change the behaviour. + +On the other hand, if you didn't declare an exception value because +you want to avoid exceptions propagating out of this function, the new behaviour +will result in slightly less efficient code being generated, now involving an exception check. +To prevent that, you must declare the function explicitly as being +``noexcept``:: + + cdef int spam(int x) noexcept: + pass + + cdef void silent(int x) noexcept: + pass + +The behaviour for ``cdef`` functions that are also ``extern`` is +unchanged as ``extern`` functions are less likely to raise Python +exceptions and rather tend to be plain C functions. This mitigates +the effect of this change for code that talks to C libraries. + +The behaviour for any ``cdef`` function that is declared with an +explicit exception value (e.g., ``cdef int spam(int x) except -1``) is +also unchanged. + +There is an easy-to-encounter performance pitfall here with ``nogil`` functions +with an implicit exception specification of ``except *``. This can happen +most commonly when the return type is ``void`` (but in principle applies +to most non-numeric return types). In this case, Cython is forced to +re-acquire the GIL briefly *after each call* to check the exception state. +To avoid this overhead, either change the signature to ``noexcept`` (if +you have determined that it's suitable to do so), or to returning an ``int`` +instead to let Cython use the ``int`` as an error flag +(by default, ``-1`` triggers the exception check). + +.. note:: + The unsafe legacy behaviour of not propagating exceptions by default can be enabled by + setting ``legacy_implicit_noexcept`` :ref:`compiler directive<compiler-directives>` + to ``True``. + + +Annotation typing +================= + +Cython 3 has made substantial improvements in recognising types in +annotations and it is well worth reading +:ref:`the pure Python tutorial<pep484_type_annotations>` to understand +some of the improvements. + +A notable backwards-incompatible change is that ``x: int`` is now typed +such that ``x`` is an exact Python ``int`` (Cython 0.29 would accept +any Python object for ``x``), unless the language level is explicitly +set to 2. To mitigate the effect, Cython 3.0 still accepts both Python +``int`` and ``long`` values under Python 2.x. + +To make it easier to handle cases where your interpretation of type +annotations differs from Cython's, Cython 3 now supports setting the +``annotation_typing`` :ref:`directive <compiler-directives>` on a +per-class or per-function level. + +C++ postincrement/postdecrement operator +======================================== + +Cython 3 differentiates between pre/post-increment and pre/post-decrement +operators (Cython 0.29 implemented both as pre(in/de)crement operator). +This only has an effect when using ``cython.operator.postdecrement`` / ``cython.operator.postincrement``. +When running into an error it is required to add the corresponding operator:: + + cdef cppclass Example: + Example operator++(int) + Example operator--(int) + +Public Declarations in C++ +========================== + +Public declarations in C++ mode are exported as C++ API in Cython 3, using ``extern "C++"``. +This behaviour can be changed by setting the export keyword using the ``CYTHON_EXTERN_C`` macro +to allow Cython modules to be implemented in C++ but callable from C. + +``**`` power operator +===================== + +Cython 3 has changed the behaviour of the power operator to be +more like Python. The consequences are that + +#. ``a**b`` of two ints may return a floating point type, +#. ``a**b`` of one or more non-complex floating point numbers may + return a complex number. + +The old behaviour can be restored by setting the ``cpow`` +:ref:`compiler directive <compiler-directives>` to ``True``. diff --git a/docs/src/userguide/nogil.rst b/docs/src/userguide/nogil.rst new file mode 100644 index 000000000..bff072a51 --- /dev/null +++ b/docs/src/userguide/nogil.rst @@ -0,0 +1,168 @@ +.. _cython_and_gil: + +Cython and the GIL +================== + +Python has a global lock (:term:`the GIL <Global Interpreter Lock or GIL>`) +to ensure that data related to the Python interpreter is not corrupted. +It is *sometimes* useful to release this lock in Cython when you are not +accessing Python data. + +There are two occasions when you may want to release the GIL: + +#. Using :ref:`Cython's parallelism mechanism <parallel>`. The contents of + a ``prange`` loop for example are required to be ``nogil``. + +#. If you want other (external) Python threads to be able to run at the same time. + + #. if you have a large computationally/IO-intensive block that doesn't need the + GIL then it may be "polite" to release it, just to benefit users of your code + who want to do multi-threading. However, this is mostly useful rather than necessary. + + #. (very, very occasionally) in long-running Cython code that never calls into the + Python interpreter, it can sometimes be useful to briefly release the GIL with a + short ``with nogil: pass`` block. This is because Cython doesn't release it + spontaneously (unlike the Python interpreter), so if you're waiting on another + Python thread to complete a task, this can avoid deadlocks. This sub-point + probably doesn't apply to you unless you're compiling GUI code with Cython. + +If neither of these two points apply then you probably do not need to release the GIL. +The sort of Cython code that can run without the GIL (no calls to Python, purely C-level +numeric operations) is often the sort of code that runs efficiently. This sometimes +gives people the impression that the inverse is true and the trick is releasing the GIL, +rather than the actual code they’re running. Don’t be misled by this -- +your (single-threaded) code will run the same speed with or without the GIL. + +Marking functions as able to run without the GIL +------------------------------------------------ + +You can mark a whole function (either a Cython function or an :ref:`external function <nogil>`) as +``nogil`` by appending this to the function signature or by using the ``@cython.nogil`` decorator: + +.. tabs:: + + .. group-tab:: Pure Python + + .. code-block:: python + + @cython.nogil + @cython.cfunc + @cython.noexcept + def some_func() -> None: + ... + + .. group-tab:: Cython + + .. code-block:: cython + + cdef void some_func() noexcept nogil: + .... + +Be aware that this does not release the GIL when calling the function. It merely indicates that +a function is suitable for use when the GIL is released. It is also fine to call these functions +while holding the GIL. + +In this case we've marked the function as ``noexcept`` to indicate that it cannot raise a Python +exception. Be aware that a function with an ``except *`` exception specification (typically functions +returning ``void``) will be expensive to call because Cython will need to temporarily reacquire +the GIL after every call to check the exception state. Most other exception specifications are +cheap to handle in a ``nogil`` block since the GIL is only acquired if an exception is +actually thrown. + +Releasing (and reacquiring) the GIL +----------------------------------- + +To actually release the GIL you can use context managers + +.. tabs:: + + .. group-tab:: Pure Python + + .. code-block:: python + + with cython.nogil: + ... # some code that runs without the GIL + with cython.gil: + ... # some code that runs with the GIL + ... # some more code without the GIL + + .. group-tab:: Cython + + .. code-block:: cython + + with nogil: + ... # some code that runs without the GIL + with gil: + ... # some code that runs with the GIL + ... # some more code without the GIL + +The ``with gil`` block is a useful trick to allow a small +chunk of Python code or Python object processing inside a non-GIL block. Try not to use it +too much since there is a cost to waiting for and acquiring the GIL, and because these +blocks cannot run in parallel since all executions require the same lock. + +A function may be marked as ``with gil`` to ensure that the +GIL is acquired immediately then calling it. This is currently a Cython-only +feature (no equivalent syntax exists in pure-Python mode):: + + cdef int some_func() with gil: + ... + +Conditionally acquiring the GIL +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +It's possible to release the GIL based on a compile-time condition. +This is most often used when working with :ref:`fusedtypes` + +.. tabs:: + + .. group-tab:: Pure Python + + .. code-block:: python + + with cython.nogil(some_type is not object): + ... # some code that runs without the GIL, unless we're processing objects + + .. group-tab:: Cython + + .. code-block:: cython + + with nogil(some_type is not object): + ... # some code that runs without the GIL, unless we're processing objects + +Exceptions and the GIL +---------------------- + +A small number of "Python operations" may be performed in a ``nogil`` +block without needing to explicitly use ``with gil``. The main example +is throwing exceptions. Here Cython knows that an exception will always +require the GIL and so re-acquires it implicitly. Similarly, if +a ``nogil`` function throws an exception, Cython is able to propagate +it correctly without you needing to write explicit code to handle it. +In most cases this is efficient since Cython is able to use the +function's exception specification to check for an error, and then +acquire the GIL only if needed, but ``except *`` functions are +less efficient since Cython must always re-acquire the GIL. + +Don't use the GIL as a lock +--------------------------- + +It may be tempting to try to use the GIL for your own locking +purposes and to say "the entire contents of a ``with gil`` block will +run atomically since we hold the GIL". Don't do this! + +The GIL is only for the benefit of the interpreter, not for you. +There are two issues here: + +#. that future improvements in the Python interpreter may destroy +your "locking". + +#. Second, that the GIL can be released if any Python code is +executed. The easiest way to run arbitrary Python code is to +destroy a Python object that has a ``__del__`` function, but +there are numerous other creative ways to do so, and it is +almost impossible to know that you aren't going to trigger one +of these. + +If you want a reliable lock then use the tools in the standard library's +``threading`` module. diff --git a/docs/src/userguide/numpy_pythran.rst b/docs/src/userguide/numpy_pythran.rst index 185f7c654..cb075d729 100644 --- a/docs/src/userguide/numpy_pythran.rst +++ b/docs/src/userguide/numpy_pythran.rst @@ -16,22 +16,22 @@ This can lead to really interesting speedup in some cases, going from 2 up to Please note that this feature is experimental. -Usage example with distutils ----------------------------- +Usage example with setuptools +----------------------------- You first need to install Pythran. See its `documentation -<http://pythran.readthedocs.io/en/latest/>`_ for more information. +<https://pythran.readthedocs.io/>`_ for more information. Then, simply add a ``cython: np_pythran=True`` directive at the top of the Python files that needs to be compiled using Pythran numpy support. -Here is an example of a simple ``setup.py`` file using distutils: +Here is an example of a simple ``setup.py`` file using setuptools: .. code:: - from distutils.core import setup + from setuptools import setup from Cython.Build import cythonize - + setup( name = "My hello app", ext_modules = cythonize('hello_pythran.pyx') diff --git a/docs/src/userguide/numpy_tutorial.rst b/docs/src/userguide/numpy_tutorial.rst index 3d1cd5a74..b74c41509 100644 --- a/docs/src/userguide/numpy_tutorial.rst +++ b/docs/src/userguide/numpy_tutorial.rst @@ -31,7 +31,7 @@ Cython at a glance Cython is a compiler which compiles Python-like code files to C code. Still, ''Cython is not a Python to C translator''. That is, it doesn't take your full -program and "turns it into C" -- rather, the result makes full use of the +program and "turn it into C" -- rather, the result makes full use of the Python runtime environment. A way of looking at it may be that your code is still Python in that it runs within the Python runtime environment, but rather than compiling to interpreted Python bytecode one compiles to native machine @@ -61,11 +61,11 @@ Using Cython consists of these steps: However there are several options to automate these steps: -1. The `SAGE <http://sagemath.org>`_ mathematics software system provides +1. The `SAGE <https://sagemath.org>`_ mathematics software system provides excellent support for using Cython and NumPy from an interactive command line or through a notebook interface (like Maple/Mathematica). See `this documentation - <http://doc.sagemath.org/html/en/developer/coding_in_cython.html>`_. + <https://doc.sagemath.org/html/en/developer/coding_in_cython.html>`_. 2. Cython can be used as an extension within a Jupyter notebook, making it easy to compile and use Cython code with just a ``%%cython`` at the top of a cell. For more information see @@ -73,7 +73,7 @@ However there are several options to automate these steps: 3. A version of pyximport is shipped with Cython, so that you can import pyx-files dynamically into Python and have them compiled automatically (See :ref:`pyximport`). -4. Cython supports distutils so that you can very easily create build scripts +4. Cython supports setuptools so that you can very easily create build scripts which automate the process, this is the preferred method for Cython implemented libraries and packages. See :ref:`Basic setup.py <basic_setup.py>`. @@ -88,7 +88,9 @@ However there are several options to automate these steps: Installation ============= -If you already have a C compiler, just do:: +If you already have a C compiler, just do: + +.. code-block:: bash pip install Cython @@ -97,7 +99,9 @@ otherwise, see :ref:`the installation page <install>`. As of this writing SAGE comes with an older release of Cython than required for this tutorial. So if using SAGE you should download the newest Cython and -then execute :: +then execute : + +.. code-block:: bash $ cd path/to/cython-distro $ path-to-sage/sage -python setup.py install @@ -108,7 +112,9 @@ Manual compilation ==================== As it is always important to know what is going on, I'll describe the manual -method here. First Cython is run:: +method here. First Cython is run: + +.. code-block:: bash $ cython yourmod.pyx @@ -120,7 +126,9 @@ line by line. Then we compile the C file. This may vary according to your system, but the C file should be built like Python was built. Python documentation for writing extensions should have some details. On Linux this often means something -like:: +like: + +.. code-block:: bash $ gcc -shared -pthread -fPIC -fwrapv -O2 -Wall -fno-strict-aliasing -I/usr/include/python2.7 -o yourmod.so yourmod.c @@ -166,7 +174,7 @@ This should be compiled to produce :file:`compute_cy.so` for Linux systems run a Python session to test both the Python version (imported from ``.py``-file) and the compiled Cython module. -.. sourcecode:: ipython +.. code-block:: ipythonconsole In [1]: import numpy as np In [2]: array_1 = np.random.uniform(0, 1000, size=(3000, 2000)).astype(np.intc) @@ -218,7 +226,7 @@ of C code to set up while in :file:`compute_typed.c` a normal C for loop is used After building this and continuing my (very informal) benchmarks, I get: -.. sourcecode:: ipython +.. code-block:: ipythonconsole In [13]: %timeit compute_typed.compute(array_1, array_2, a, b, c) 26.5 s ± 422 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) @@ -287,7 +295,7 @@ Here is how to use them in our code: Let's see how much faster accessing is now. -.. sourcecode:: ipython +.. code-block:: ipythonconsole In [22]: %timeit compute_memview.compute(array_1, array_2, a, b, c) 22.9 ms ± 197 µs per loop (mean ± std. dev. of 7 runs, 10 loops each) @@ -326,7 +334,7 @@ mode in many ways, see :ref:`compiler-directives` for more information. -.. sourcecode:: ipython +.. code-block:: ipythonconsole In [23]: %timeit compute_index.compute(array_1, array_2, a, b, c) 16.8 ms ± 25.4 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) @@ -376,7 +384,7 @@ all about, you can see `this answer on StackOverflow For the sake of giving numbers, here are the speed gains that you should get by declaring the memoryviews as contiguous: -.. sourcecode:: ipython +.. code-block:: ipythonconsole In [23]: %timeit compute_contiguous.compute(array_1, array_2, a, b, c) 11.1 ms ± 30.2 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) @@ -405,7 +413,7 @@ be useful when using fused types. We now do a speed test: -.. sourcecode:: ipython +.. code-block:: ipythonconsole In [24]: %timeit compute_infer_types.compute(array_1, array_2, a, b, c) 11.5 ms ± 261 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) @@ -439,14 +447,14 @@ In this case, our function now works for ints, doubles and floats. We can check that the output type is the right one:: - >>>compute(array_1, array_2, a, b, c).dtype + >>> compute(array_1, array_2, a, b, c).dtype dtype('int32') - >>>compute(array_1.astype(np.double), array_2.astype(np.double), a, b, c).dtype + >>> compute(array_1.astype(np.double), array_2.astype(np.double), a, b, c).dtype dtype('float64') We now do a speed test: -.. sourcecode:: ipython +.. code-block:: ipythonconsole In [25]: %timeit compute_fused_types.compute(array_1, array_2, a, b, c) 11.5 ms ± 258 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) @@ -463,7 +471,9 @@ like the function :func:`prange`. You can see more information about Cython and parallelism in :ref:`parallel`. Since we do elementwise operations, we can easily distribute the work among multiple threads. It's important not to forget to pass the correct arguments to the compiler to enable OpenMP. When using the Jupyter notebook, -you should use the cell magic like this:: +you should use the cell magic like this: + +.. code-block:: ipython %%cython --force # distutils: extra_compile_args=-fopenmp @@ -476,7 +486,7 @@ declare our :func:`clip` function ``nogil``. We can have substantial speed gains for minimal effort: -.. sourcecode:: ipython +.. code-block:: ipythonconsole In [25]: %timeit compute_prange.compute(array_1, array_2, a, b, c) 9.33 ms ± 412 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) @@ -487,8 +497,8 @@ than NumPy! Where to go from here? ====================== -* If you want to learn how to make use of `BLAS <http://www.netlib.org/blas/>`_ - or `LAPACK <http://www.netlib.org/lapack/>`_ with Cython, you can watch +* If you want to learn how to make use of `BLAS <https://www.netlib.org/blas/>`_ + or `LAPACK <https://www.netlib.org/lapack/>`_ with Cython, you can watch `the presentation of Ian Henriksen at SciPy 2015 <https://www.youtube.com/watch?v=R4yB-8tB0J0&t=693s&ab_channel=Enthought>`_. * If you want to learn how to use Pythran as backend in Cython, you diff --git a/docs/src/userguide/numpy_ufuncs.rst b/docs/src/userguide/numpy_ufuncs.rst new file mode 100644 index 000000000..b5df36861 --- /dev/null +++ b/docs/src/userguide/numpy_ufuncs.rst @@ -0,0 +1,70 @@ +.. highlight:: cython + +.. _numpy-ufuncs: + +********************* +Creating Numpy ufuncs +********************* + +.. include:: + ../two-syntax-variants-used + +Numpy supports a `special type of function called a ufunc +<https://numpy.org/doc/stable/reference/ufuncs.html>`_ . +These support array broadcasting (i.e. the ability to handle arguments with any +number of dimensions), alongside other useful features. + +Cython can generate a ufunc from a Cython C function by tagging it with the ``@cython.ufunc`` +decorator. The input and output argument types should be scalar variables ("generic ufuncs" are +not yet supported) and should either by Python objects or simple numeric types. The body +of such a function is inserted into an efficient, compiled loop. + +.. tabs:: + + .. group-tab:: Pure Python + + .. literalinclude:: ../../examples/userguide/numpy_ufuncs/ufunc.py + :lines: 2- + + .. group-tab:: Cython + + .. literalinclude:: ../../examples/userguide/numpy_ufuncs/ufunc.pyx + :lines: 2- + +You can have as many arguments to your function as you like. If you want to have multiple +output arguments then you can use the :ref:`ctuple syntax<typing_types>`: + +.. tabs:: + + .. group-tab:: Pure Python + + .. literalinclude:: ../../examples/userguide/numpy_ufuncs/ufunc_ctuple.py + :lines: 2- + + .. group-tab:: Cython + + .. literalinclude:: ../../examples/userguide/numpy_ufuncs/ufunc_ctuple.pyx + :lines: 2- + +If you want to accept multiple different argument types then you can use :ref:`fusedtypes`: + +.. tabs:: + + .. group-tab:: Pure Python + + .. literalinclude:: ../../examples/userguide/numpy_ufuncs/ufunc_fused.py + :lines: 2- + + .. group-tab:: Cython + + .. literalinclude:: ../../examples/userguide/numpy_ufuncs/ufunc_fused.pyx + :lines: 2- + +Finally, if you declare the ``cdef``/``@cfunc`` function as ``nogil`` then Cython will release the +:term:`GIL<Global Interpreter Lock or GIL>` once in the generated ufunc. This is a slight difference +from the general behaviour of ``nogil`` functions (they generally do not automatically +release the GIL, but instead can be run without the GIL). + +This feature relies on Numpy. Therefore if you create a ufunc in +Cython, you must have the Numpy headers available when you build the generated C code, and +users of your module must have Numpy installed when they run it. diff --git a/docs/src/userguide/parallelism.rst b/docs/src/userguide/parallelism.rst index ad97885e1..4d23f0d73 100644 --- a/docs/src/userguide/parallelism.rst +++ b/docs/src/userguide/parallelism.rst @@ -8,8 +8,11 @@ Using Parallelism ********************************** +.. include:: + ../two-syntax-variants-used + Cython supports native parallelism through the :py:mod:`cython.parallel` -module. To use this kind of parallelism, the GIL must be released +module. To use this kind of parallelism, the :term:`GIL<Global Interpreter Lock or GIL>` must be released (see :ref:`Releasing the GIL <nogil>`). It currently supports OpenMP, but later on more backends might be supported. @@ -87,7 +90,7 @@ It currently supports OpenMP, but later on more backends might be supported. runtime: The schedule and chunk size are taken from the runtime scheduling variable, which can be set through the ``openmp.omp_set_schedule()`` - function call, or the OMP_SCHEDULE environment variable. Note that + function call, or the ``OMP_SCHEDULE`` environment variable. Note that this essentially disables any static compile time optimisations of the scheduling code itself and may therefore show a slightly worse performance than when the same scheduling policy is statically @@ -116,17 +119,27 @@ It currently supports OpenMP, but later on more backends might be supported. Example with a reduction: -.. literalinclude:: ../../examples/userguide/parallelism/simple_sum.pyx +.. tabs:: + + .. group-tab:: Pure Python + + .. literalinclude:: ../../examples/userguide/parallelism/simple_sum.py + + .. group-tab:: Cython + + .. literalinclude:: ../../examples/userguide/parallelism/simple_sum.pyx -Example with a typed memoryview (e.g. a NumPy array):: +Example with a :term:`typed memoryview<Typed memoryview>` (e.g. a NumPy array) - from cython.parallel import prange +.. tabs:: - def func(double[:] x, double alpha): - cdef Py_ssize_t i + .. group-tab:: Pure Python - for i in prange(x.shape[0]): - x[i] = alpha * x[i] + .. literalinclude:: ../../examples/userguide/parallelism/memoryview_sum.py + + .. group-tab:: Cython + + .. literalinclude:: ../../examples/userguide/parallelism/memoryview_sum.pyx .. function:: parallel(num_threads=None) @@ -137,29 +150,17 @@ Example with a typed memoryview (e.g. a NumPy array):: is also private to the prange. Variables that are private in the parallel block are unavailable after the parallel block. - Example with thread-local buffers:: - - from cython.parallel import parallel, prange - from libc.stdlib cimport abort, malloc, free + Example with thread-local buffers - cdef Py_ssize_t idx, i, n = 100 - cdef int * local_buf - cdef size_t size = 10 + .. tabs:: - with nogil, parallel(): - local_buf = <int *> malloc(sizeof(int) * size) - if local_buf is NULL: - abort() + .. group-tab:: Pure Python - # populate our local buffer in a sequential loop - for i in xrange(size): - local_buf[i] = i * 2 + .. literalinclude:: ../../examples/userguide/parallelism/parallel.py - # share the work using the thread-local buffer(s) - for i in prange(n, schedule='guided'): - func(local_buf) + .. group-tab:: Cython - free(local_buf) + .. literalinclude:: ../../examples/userguide/parallelism/parallel.pyx Later on sections might be supported in parallel blocks, to distribute code sections of work among threads. @@ -174,9 +175,17 @@ Compiling ========= To actually use the OpenMP support, you need to tell the C or C++ compiler to -enable OpenMP. For gcc this can be done as follows in a setup.py: +enable OpenMP. For gcc this can be done as follows in a ``setup.py``: + +.. tabs:: + + .. group-tab:: Pure Python -.. literalinclude:: ../../examples/userguide/parallelism/setup.py + .. literalinclude:: ../../examples/userguide/parallelism/setup_py.py + + .. group-tab:: Cython + + .. literalinclude:: ../../examples/userguide/parallelism/setup_pyx.py For Microsoft Visual C++ compiler, use ``'/openmp'`` instead of ``'-fopenmp'``. @@ -188,13 +197,21 @@ The parallel with and prange blocks support the statements break, continue and return in nogil mode. Additionally, it is valid to use a ``with gil`` block inside these blocks, and have exceptions propagate from them. However, because the blocks use OpenMP, they can not just be left, so the -exiting procedure is best-effort. For prange() this means that the loop +exiting procedure is best-effort. For ``prange()`` this means that the loop body is skipped after the first break, return or exception for any subsequent iteration in any thread. It is undefined which value shall be returned if multiple different values may be returned, as the iterations are in no particular order: -.. literalinclude:: ../../examples/userguide/parallelism/breaking_loop.pyx +.. tabs:: + + .. group-tab:: Pure Python + + .. literalinclude:: ../../examples/userguide/parallelism/breaking_loop.py + + .. group-tab:: Cython + + .. literalinclude:: ../../examples/userguide/parallelism/breaking_loop.pyx In the example above it is undefined whether an exception shall be raised, whether it will simply break or whether it will return 2. @@ -203,7 +220,17 @@ Using OpenMP Functions ====================== OpenMP functions can be used by cimporting ``openmp``: -.. literalinclude:: ../../examples/userguide/parallelism/cimport_openmp.pyx +.. tabs:: + + .. group-tab:: Pure Python + + .. literalinclude:: ../../examples/userguide/parallelism/cimport_openmp.py + :lines: 3- + + .. group-tab:: Cython + + .. literalinclude:: ../../examples/userguide/parallelism/cimport_openmp.pyx + :lines: 3- .. rubric:: References diff --git a/docs/src/userguide/pypy.rst b/docs/src/userguide/pypy.rst index cf1fbb24d..893277bda 100644 --- a/docs/src/userguide/pypy.rst +++ b/docs/src/userguide/pypy.rst @@ -2,7 +2,7 @@ Porting Cython code to PyPy =========================== Cython has basic support for cpyext, the layer in -`PyPy <http://pypy.org/>`_ that emulates CPython's C-API. This is +`PyPy <https://pypy.org/>`_ that emulates CPython's C-API. This is achieved by making the generated C code adapt at C compile time, so the generated code will compile in both CPython and PyPy unchanged. diff --git a/docs/src/userguide/pyrex_differences.rst b/docs/src/userguide/pyrex_differences.rst index 8a958ec9a..aa2eb75da 100644 --- a/docs/src/userguide/pyrex_differences.rst +++ b/docs/src/userguide/pyrex_differences.rst @@ -310,7 +310,7 @@ Automatic ``typecheck`` Rather than introducing a new keyword ``typecheck`` as explained in the `Pyrex docs -<http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/version/Doc/Manual/special_methods.html>`_, +<https://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/version/Doc/Manual/special_methods.html>`_, Cython emits a (non-spoofable and faster) typecheck whenever :func:`isinstance` is used with an extension type as the second parameter. diff --git a/docs/src/userguide/sharing_declarations.rst b/docs/src/userguide/sharing_declarations.rst index 7c2a49e21..6beceda57 100644 --- a/docs/src/userguide/sharing_declarations.rst +++ b/docs/src/userguide/sharing_declarations.rst @@ -6,6 +6,9 @@ Sharing Declarations Between Cython Modules ******************************************** +.. include:: + ../two-syntax-variants-used + This section describes how to make C declarations, functions and extension types in one Cython module available for use in another Cython module. These facilities are closely modeled on the Python import mechanism, @@ -17,13 +20,13 @@ Definition and Implementation files A Cython module can be split into two parts: a definition file with a ``.pxd`` suffix, containing C declarations that are to be available to other Cython -modules, and an implementation file with a ``.pyx`` suffix, containing +modules, and an implementation file with a ``.pyx``/``.py`` suffix, containing everything else. When a module wants to use something declared in another module's definition file, it imports it using the :keyword:`cimport` -statement. +statement or using special :py:mod:`cython.cimports` package. A ``.pxd`` file that consists solely of extern declarations does not need -to correspond to an actual ``.pyx`` file or Python module. This can make it a +to correspond to an actual ``.pyx``/``.py`` file or Python module. This can make it a convenient place to put common declarations, for example declarations of functions from an :ref:`external library <external-C-code>` that one wants to use in several modules. @@ -41,8 +44,8 @@ A definition file can contain: It cannot contain the implementations of any C or Python functions, or any Python class definitions, or any executable statements. It is needed when one -wants to access :keyword:`cdef` attributes and methods, or to inherit from -:keyword:`cdef` classes defined in this module. +wants to access :keyword:`cdef`/``@cfunc`` attributes and methods, or to inherit from +:keyword:`cdef`/``@cclass`` classes defined in this module. .. note:: @@ -70,23 +73,45 @@ The cimport statement The :keyword:`cimport` statement is used in a definition or implementation file to gain access to names declared in another definition file. Its syntax exactly parallels that of the normal Python import -statement:: +statement. When pure python syntax is used, the same effect can be done by +importing from special :py:mod:`cython.cimports` package. In later text the term +to ``cimport`` refers to using both :keyword:`cimport` statement or +:py:mod:`cython.cimports` package. - cimport module [, module...] +.. tabs:: - from module cimport name [as name] [, name [as name] ...] + .. group-tab:: Pure Python -Here is an example. :file:`dishes.pxd` is a definition file which exports a -C data type. :file:`restaurant.pyx` is an implementation file which imports and -uses it. + .. code-block:: python + + from cython.cimports.module import name [as name][, name [as name] ...] + + .. group-tab:: Cython + + .. code-block:: cython + + cimport module [, module...] + + from module cimport name [as name] [, name [as name] ...] -:file:`dishes.pxd`: +Here is an example. :file:`dishes.pxd` is a definition file which exports a +C data type. :file:`restaurant.pyx`/:file:`restaurant.py` is an implementation file +which imports and uses it. .. literalinclude:: ../../examples/userguide/sharing_declarations/dishes.pxd + :caption: dishes.pxd + +.. tabs:: + + .. group-tab:: Pure Python + + .. literalinclude:: ../../examples/userguide/sharing_declarations/restaurant.py + :caption: dishes.py -:file:`restaurant.pyx`: + .. group-tab:: Cython -.. literalinclude:: ../../examples/userguide/sharing_declarations/restaurant.pyx + .. literalinclude:: ../../examples/userguide/sharing_declarations/restaurant.pyx + :caption: dishes.pyx It is important to understand that the :keyword:`cimport` statement can only be used to import C data types, C functions and variables, and extension @@ -116,8 +141,8 @@ option to ``cythonize()``), as well as ``sys.path``. Using ``package_data`` to install ``.pxd`` files in your ``setup.py`` script allows other packages to cimport items from your module as a dependency. -Also, whenever you compile a file :file:`modulename.pyx`, the corresponding -definition file :file:`modulename.pxd` is first searched for along the +Also, whenever you compile a file :file:`modulename.pyx`/:file:`modulename.py`, +the corresponding definition file :file:`modulename.pxd` is first searched for along the include path (but not ``sys.path``), and if found, it is processed before processing the ``.pyx`` file. @@ -132,16 +157,23 @@ for an imaginary module, and :keyword:`cimport` that module. You can then refer to the C functions by qualifying them with the name of the module. Here's an example: -:file:`c_lunch.pxd`: - .. literalinclude:: ../../examples/userguide/sharing_declarations/c_lunch.pxd + :caption: c_lunch.pxd + +.. tabs:: + + .. group-tab:: Pure Python -:file:`lunch.pyx`: + .. literalinclude:: ../../examples/userguide/sharing_declarations/lunch.py + :caption: lunch.py -.. literalinclude:: ../../examples/userguide/sharing_declarations/lunch.pyx + .. group-tab:: Cython -You don't need any :file:`c_lunch.pyx` file, because the only things defined -in :file:`c_lunch.pxd` are extern C entities. There won't be any actual + .. literalinclude:: ../../examples/userguide/sharing_declarations/lunch.pyx + :caption: lunch.pyx + +You don't need any :file:`c_lunch.pyx`/:file:`c_lunch.py` file, because the only +things defined in :file:`c_lunch.pxd` are extern C entities. There won't be any actual ``c_lunch`` module at run time, but that doesn't matter; the :file:`c_lunch.pxd` file has done its job of providing an additional namespace at compile time. @@ -154,17 +186,32 @@ C functions defined at the top level of a module can be made available via :keyword:`cimport` by putting headers for them in the ``.pxd`` file, for example: -:file:`volume.pxd`: - .. literalinclude:: ../../examples/userguide/sharing_declarations/volume.pxd + :caption: volume.pxd + +.. tabs:: + + .. group-tab:: Pure Python + + .. literalinclude:: ../../examples/userguide/sharing_declarations/volume.py + :caption: volume.py + + .. literalinclude:: ../../examples/userguide/sharing_declarations/spammery.py + :caption: spammery.py + + .. note:: -:file:`volume.pyx`: + Type definitions of function ``cube()`` in :file:`volume.py` are not provided + since they are used from .pxd definition file. See :ref:`augmenting_pxd` and + GitHub issue :issue:`4388`. -.. literalinclude:: ../../examples/userguide/sharing_declarations/volume.pyx + .. group-tab:: Cython -:file:`spammery.pyx`: + .. literalinclude:: ../../examples/userguide/sharing_declarations/volume.pyx + :caption: volume.pyx -.. literalinclude:: ../../examples/userguide/sharing_declarations/spammery.pyx + .. literalinclude:: ../../examples/userguide/sharing_declarations/spammery.pyx + :caption: spammery.pyx .. _sharing_extension_types: @@ -186,36 +233,70 @@ Python methods. Here is an example of a module which defines and exports an extension type, and another module which uses it: -:file:`shrubbing.pxd`: - .. literalinclude:: ../../examples/userguide/sharing_declarations/shrubbing.pxd + :caption: shrubbing.pxd + +.. tabs:: + + .. group-tab:: Pure Python -:file:`shrubbing.pyx`: + .. literalinclude:: ../../examples/userguide/sharing_declarations/shrubbing.py + :caption: shrubbing.py -.. literalinclude:: ../../examples/userguide/sharing_declarations/shrubbing.pyx + .. literalinclude:: ../../examples/userguide/sharing_declarations/landscaping.py + :caption: landscaping.py -:file:`landscaping.pyx`: + One would then need to compile both of these modules, e.g. using -.. literalinclude:: ../../examples/userguide/sharing_declarations/landscaping.pyx + .. literalinclude:: ../../examples/userguide/sharing_declarations/setup_py.py + :caption: setup.py -One would then need to compile both of these modules, e.g. using + .. group-tab:: Cython -:file:`setup.py`: + .. literalinclude:: ../../examples/userguide/sharing_declarations/shrubbing.pyx + :caption: shrubbing.pyx -.. literalinclude:: ../../examples/userguide/sharing_declarations/setup.py + .. literalinclude:: ../../examples/userguide/sharing_declarations/landscaping.pyx + :caption: landscaping.pyx + + One would then need to compile both of these modules, e.g. using + + .. literalinclude:: ../../examples/userguide/sharing_declarations/setup_pyx.py + :caption: setup.py Some things to note about this example: -* There is a :keyword:`cdef` class Shrubbery declaration in both - :file:`Shrubbing.pxd` and :file:`Shrubbing.pyx`. When the Shrubbing module +* There is a :keyword:`cdef`/``@cclass`` class Shrubbery declaration in both + :file:`shrubbing.pxd` and :file:`shrubbing.pyx`. When the shrubbing module is compiled, these two declarations are combined into one. -* In Landscaping.pyx, the :keyword:`cimport` Shrubbing declaration allows us - to refer to the Shrubbery type as :class:`Shrubbing.Shrubbery`. But it - doesn't bind the name Shrubbing in Landscaping's module namespace at run - time, so to access :func:`Shrubbing.standard_shrubbery` we also need to - ``import Shrubbing``. +* In :file:`landscaping.pyx`/:file:`landscaping.py`, the :keyword:`cimport` shrubbing + declaration allows us to refer to the Shrubbery type as :class:`shrubbing.Shrubbery`. + But it doesn't bind the name shrubbing in landscaping's module namespace at run + time, so to access :func:`shrubbing.standard_shrubbery` we also need to + ``import shrubbing``. * One caveat if you use setuptools instead of distutils, the default action when running ``python setup.py install`` is to create a zipped ``egg`` file which will not work with ``cimport`` for ``pxd`` files when you try to use them from a dependent package. To prevent this, include ``zip_safe=False`` in the arguments to ``setup()``. + +.. _versioning: + +Versioning +========== + +``.pxd`` files can be labelled with a minimum Cython version as part of +their file name, similar to the version tagging of ``.so`` files in PEP 3149. +For example a file called :file:`shrubbing.cython-30.pxd` will only be +found by ``cimport shrubbing`` on Cython 3.0 and higher. Cython will use the +file tagged with the highest compatible version number. + +Note that versioned files that are distributed across different directories +will not be found. Only the first directory in the Python module search +path in which a matching ``.pxd`` file is found will be considered. + +The purpose of this feature is to allow third-party packages to release +Cython interfaces to their packages that take advantage of the latest Cython +features while not breaking compatibility for users with older versions of Cython. +Users intending to use ``.pxd`` files solely within their own project +need not produce these tagged files. diff --git a/docs/src/userguide/source_files_and_compilation.rst b/docs/src/userguide/source_files_and_compilation.rst index c4f01806d..9b27433b2 100644 --- a/docs/src/userguide/source_files_and_compilation.rst +++ b/docs/src/userguide/source_files_and_compilation.rst @@ -12,17 +12,22 @@ file named :file:`primes.pyx`. Cython code, unlike Python, must be compiled. This happens in two stages: - * A ``.pyx`` file is compiled by Cython to a ``.c`` file. + * A ``.pyx`` (or ``.py``) file is compiled by Cython to a ``.c`` file. * The ``.c`` file is compiled by a C compiler to a ``.so`` file (or a ``.pyd`` file on Windows) -Once you have written your ``.pyx`` file, there are a couple of ways of turning it -into an extension module. +Once you have written your ``.pyx``/``.py`` file, there are a couple of ways +how to turn it into an extension module. The following sub-sections describe several ways to build your extension modules, and how to pass directives to the Cython compiler. +There are also a number of tools that process ``.pyx`` files apart from Cython, e.g. + +- Linting: https://pypi.org/project/cython-lint/ + + .. _compiling_command_line: Compiling from the command line @@ -44,7 +49,7 @@ Compiling with the ``cython`` command One way is to compile it manually with the Cython compiler, e.g.: -.. sourcecode:: text +.. code-block:: text $ cython primes.pyx @@ -53,7 +58,7 @@ compiled with the C compiler using whatever options are appropriate on your platform for generating an extension module. For these options look at the official Python documentation. -The other, and probably better, way is to use the :mod:`distutils` extension +The other, and probably better, way is to use the :mod:`setuptools` extension provided with Cython. The benefit of this method is that it will give the platform specific compilation options, acting like a stripped down autotools. @@ -62,7 +67,9 @@ Compiling with the ``cythonize`` command ---------------------------------------- Run the ``cythonize`` compiler command with your options and list of -``.pyx`` files to generate an extension module. For example:: +``.pyx`` files to generate an extension module. For example: + +.. code-block:: bash $ cythonize -a -i yourmod.pyx @@ -82,7 +89,9 @@ There simpler command line tool ``cython`` only invokes the source code translat In the case of manual compilation, how to compile your ``.c`` files will vary depending on your operating system and compiler. The Python documentation for writing extension modules should have some details for your system. On a Linux -system, for example, it might look similar to this:: +system, for example, it might look similar to this: + +.. code-block:: bash $ gcc -shared -pthread -fPIC -fwrapv -O2 -Wall -fno-strict-aliasing \ -I/usr/include/python3.5 -o yourmod.so yourmod.c @@ -93,45 +102,58 @@ to libraries you want to link with.) After compilation, a ``yourmod.so`` (:file:`yourmod.pyd` for Windows) file is written into the target directory and your module, ``yourmod``, is available for you to import as with any other -Python module. Note that if you are not relying on ``cythonize`` or distutils, +Python module. Note that if you are not relying on ``cythonize`` or setuptools, you will not automatically benefit from the platform specific file extension that CPython generates for disambiguation, such as ``yourmod.cpython-35m-x86_64-linux-gnu.so`` on a regular 64bit Linux installation of CPython 3.5. + .. _basic_setup.py: Basic setup.py =============== -The distutils extension provided with Cython allows you to pass ``.pyx`` files +The setuptools extension provided with Cython allows you to pass ``.pyx`` files directly to the ``Extension`` constructor in your setup file. If you have a single Cython file that you want to turn into a compiled extension, say with filename :file:`example.pyx` the associated :file:`setup.py` would be:: - from distutils.core import setup + from setuptools import setup from Cython.Build import cythonize setup( ext_modules = cythonize("example.pyx") ) -To understand the :file:`setup.py` more fully look at the official -:mod:`distutils` documentation. To compile the extension for use in the -current directory use: +If your build depends directly on Cython in this way, +then you may also want to inform pip that :mod:`Cython` is required for +:file:`setup.py` to execute, following `PEP 518 +<https://www.python.org/dev/peps/pep-0518/>`, creating a :file:`pyproject.toml` +file containing, at least: -.. sourcecode:: text +.. code-block:: ini + + + [build-system] + requires = ["setuptools", "wheel", "Cython"] + +To understand the :file:`setup.py` more fully look at the official `setuptools +documentation`_. To compile the extension for use in the current directory use: + +.. code-block:: text $ python setup.py build_ext --inplace + Configuring the C-Build ------------------------ If you have include files in non-standard places you can pass an ``include_path`` parameter to ``cythonize``:: - from distutils.core import setup + from setuptools import setup from Cython.Build import cythonize setup( @@ -150,20 +172,38 @@ the necessary include files, e.g. for NumPy:: you have to add the path to NumPy include files. You need to add this path only if you use ``cimport numpy``. -Despite this, you will still get warnings like the -following from the compiler, because Cython is using a deprecated Numpy API:: +Despite this, you may still get warnings like the following from the compiler, +because Cython is not disabling the usage of the old deprecated Numpy API:: .../include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp] -For the time being, it is just a warning that you can ignore. +In Cython 3.0, you can get rid of this warning by defining the C macro +``NPY_NO_DEPRECATED_API`` as ``NPY_1_7_API_VERSION`` +in your build, e.g.:: + + # distutils: define_macros=NPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION + +or (see below):: + + Extension( + ..., + define_macros=[("NPY_NO_DEPRECATED_API", "NPY_1_7_API_VERSION")], + ) + +With older Cython releases, setting this macro will fail the C compilation, +because Cython generates code that uses this deprecated C-API. However, the +warning has no negative effects even in recent NumPy versions including 1.18.x. +You can ignore it until you (or your library's users) switch to a newer NumPy +version that removes this long deprecated API, in which case you also need to +use Cython 3.0 or later. Thus, the earlier you switch to Cython 3.0, the +better for your users. If you need to specify compiler options, libraries to link with or other linker options you will need to create ``Extension`` instances manually (note that glob syntax can still be used to specify multiple extensions in one line):: - from distutils.core import setup - from distutils.extension import Extension + from setuptools import Extension, setup from Cython.Build import cythonize extensions = [ @@ -182,11 +222,10 @@ in one line):: ext_modules=cythonize(extensions), ) -Note that when using setuptools, you should import it before Cython as -setuptools may replace the ``Extension`` class in distutils. Otherwise, +Note that when using setuptools, you should import it before Cython, otherwise, both might disagree about the class to use here. -Note also that if you use setuptools instead of distutils, the default +Note also that if you use setuptools instead of :mod:`distutils`, the default action when running ``python setup.py install`` is to create a zipped ``egg`` file which will not work with ``cimport`` for ``pxd`` files when you try to use them from a dependent package. @@ -204,7 +243,7 @@ merges the list of libraries, so this works as expected (similarly with other options, like ``include_dirs`` above). If you have some C files that have been wrapped with Cython and you want to -compile them into your extension, you can define the distutils ``sources`` +compile them into your extension, you can define the setuptools ``sources`` parameter:: # distutils: sources = helper.c, another_helper.c @@ -213,9 +252,8 @@ Note that these sources are added to the list of sources of the current extension module. Spelling this out in the :file:`setup.py` file looks as follows:: - from distutils.core import setup + from setuptools import Extension, setup from Cython.Build import cythonize - from distutils.extension import Extension sourcefiles = ['example.pyx', 'helper.c', 'another_helper.c'] @@ -226,14 +264,14 @@ as follows:: ) The :class:`Extension` class takes many options, and a fuller explanation can -be found in the `distutils documentation`_. Some useful options to know about +be found in the `setuptools documentation`_. Some useful options to know about are ``include_dirs``, ``libraries``, and ``library_dirs`` which specify where to find the ``.h`` and library files when linking to external libraries. -.. _distutils documentation: https://docs.python.org/extending/building.html +.. _setuptools documentation: https://setuptools.readthedocs.io/ Sometimes this is not enough and you need finer customization of the -distutils :class:`Extension`. +setuptools :class:`Extension`. To do this, you can provide a custom function ``create_extension`` to create the final :class:`Extension` object after Cython has processed the sources, dependencies and ``# distutils`` directives but before the @@ -283,6 +321,7 @@ Just as an example, this adds ``mylib`` as library to every extension:: then the argument to ``create_extension`` must be pickleable. In particular, it cannot be a lambda function. + .. _cythonize_arguments: Cythonize arguments @@ -329,11 +368,10 @@ doesn't want to use it just to install your module. Also, the installed version may not be the same one you used, and may not compile your sources correctly. This simply means that the :file:`setup.py` file that you ship with will just -be a normal distutils file on the generated `.c` files, for the basic example +be a normal setuptools file on the generated `.c` files, for the basic example we would have instead:: - from distutils.core import setup - from distutils.extension import Extension + from setuptools import Extension, setup setup( ext_modules = [Extension("example", ["example.c"])] @@ -342,8 +380,7 @@ we would have instead:: This is easy to combine with :func:`cythonize` by changing the file extension of the extension module sources:: - from distutils.core import setup - from distutils.extension import Extension + from setuptools import Extension, setup USE_CYTHON = ... # command line option, try-import, ... @@ -385,14 +422,17 @@ Another option is to make Cython a setup dependency of your system and use Cython's build_ext module which runs ``cythonize`` as part of the build process:: setup( - setup_requires=[ - 'cython>=0.x', - ], extensions = [Extension("*", ["*.pyx"])], cmdclass={'build_ext': Cython.Build.build_ext}, ... ) +This depends on pip knowing that :mod:`Cython` is a setup dependency, by having +a :file:`pyproject.toml` file:: + + [build-system] + requires = ["setuptools", "wheel", "Cython"] + If you want to expose the C-level interface of your library for other libraries to cimport from, use package_data to install the ``.pxd`` files, e.g.:: @@ -522,7 +562,7 @@ glob must be on a separate line. Pyximport will check the file date for each of those files before deciding whether to rebuild the module. In order to keep track of the fact that the dependency has been handled, Pyximport updates the modification time of your ".pyx" source file. Future versions may do -something more sophisticated like informing distutils of the dependencies +something more sophisticated like informing setuptools of the dependencies directly. @@ -538,7 +578,7 @@ compiled. Usually the defaults are fine. You might run into problems if you wanted to write your program in half-C, half-Cython and build them into a single library. -Pyximport does not hide the Distutils/GCC warnings and errors generated +Pyximport does not hide the setuptools/GCC warnings and errors generated by the import process. Arguably this will give you better feedback if something went wrong and why. And if nothing went wrong it will give you the warm fuzzy feeling that pyximport really did rebuild your module as it @@ -573,6 +613,37 @@ Unbound variables are automatically pulled from the surrounding local and global scopes, and the result of the compilation is cached for efficient re-use. + +Compiling with ``cython.compile`` +================================= + +Cython supports transparent compiling of the cython code in a function using the +``@cython.compile`` decorator:: + + @cython.compile + def plus(a, b): + return a + b + +Parameters of the decorated function cannot have type declarations. Their types are +automatically determined from values passed to the function, thus leading to one or more +specialised compiled functions for the respective argument types. +Executing example:: + + import cython + + @cython.compile + def plus(a, b): + return a + b + + print(plus('3', '5')) + print(plus(3, 5)) + +will produce following output:: + + 35 + 8 + + .. _compiling_with_sage: Compiling with Sage @@ -582,11 +653,12 @@ The Sage notebook allows transparently editing and compiling Cython code simply by typing ``%cython`` at the top of a cell and evaluate it. Variables and functions defined in a Cython cell are imported into the running session. Please check `Sage documentation -<http://www.sagemath.org/doc/>`_ for details. +<https://www.sagemath.org/doc/>`_ for details. You can tailor the behavior of the Cython compiler by specifying the directives below. + .. _compiling_notebook: Compiling with a Jupyter Notebook @@ -623,6 +695,8 @@ You can see them also by typing ```%%cython?`` in IPython or a Jupyter notebook. -a, --annotate Produce a colorized HTML version of the source. +--annotate-fullc Produce a colorized HTML version of the source which includes entire generated C/C++-code. + -+, --cplus Output a C++ rather than C file. -f, --force Force the compilation of a new module, even if the source has been previously compiled. @@ -648,6 +722,7 @@ You can see them also by typing ```%%cython?`` in IPython or a Jupyter notebook. --pgo Enable profile guided optimisation in the C compiler. Compiles the cell twice and executes it in between to generate a runtime profile. --verbose Print debug information like generated .c/.cpp file location and exact gcc/g++ command invoked. + ============================================ ======================================================================================================================================= @@ -659,7 +734,7 @@ Compiler options Compiler options can be set in the :file:`setup.py`, before calling :func:`cythonize`, like this:: - from distutils.core import setup + from setuptools import setup from Cython.Build import cythonize from Cython.Compiler import Options @@ -675,7 +750,6 @@ Here are the options that are available: .. autodata:: Cython.Compiler.Options.docstrings .. autodata:: Cython.Compiler.Options.embed_pos_in_docstring -.. autodata:: Cython.Compiler.Options.emit_code_comments .. pre_import .. autodata:: Cython.Compiler.Options.generate_cleanup_code .. autodata:: Cython.Compiler.Options.clear_to_none @@ -711,7 +785,11 @@ Cython code. Here is the list of currently supported directives: class attribute (hence the name) and will emulate the attributes of Python functions, including introspections like argument names and annotations. - Default is False. + + Default is True. + + .. versionchanged:: 3.0.0 + Default changed from False to True ``boundscheck`` (True / False) If set to False, Cython is free to assume that indexing operations @@ -740,9 +818,13 @@ Cython code. Here is the list of currently supported directives: Default is True. ``initializedcheck`` (True / False) - If set to True, Cython checks that a memoryview is initialized - whenever its elements are accessed or assigned to. Setting this - to False disables these checks. + If set to True, Cython checks that + - a memoryview is initialized whenever its elements are accessed + or assigned to. + - a C++ class is initialized when it is accessed + (only when ``cpp_locals`` is on) + + Setting this to False disables these checks. Default is True. ``nonecheck`` (True / False) @@ -786,14 +868,39 @@ Cython code. Here is the list of currently supported directives: division is performed with negative operands. See `CEP 516 <https://github.com/cython/cython/wiki/enhancements-division>`_. Default is False. + +``cpow`` (True / False) + ``cpow`` modifies the return type of ``a**b``, as shown in the + table below: + + .. csv-table:: cpow behaviour + :file: cpow_table.csv + :header-rows: 1 + :class: longtable + :widths: 1 1 3 3 + + The ``cpow==True`` behaviour largely keeps the result type the + same as the operand types, while the ``cpow==False`` behaviour + follows Python and returns a flexible type depending on the + inputs. + + Introduced in Cython 3.0 with a default of False; + before that, the behaviour matched the ``cpow=True`` version. ``always_allow_keywords`` (True / False) - Avoid the ``METH_NOARGS`` and ``METH_O`` when constructing - functions/methods which take zero or one arguments. Has no effect - on special methods and functions with more than one argument. The - ``METH_NOARGS`` and ``METH_O`` signatures provide faster + When disabled, uses the ``METH_NOARGS`` and ``METH_O`` signatures when + constructing functions/methods which take zero or one arguments. Has no + effect on special methods and functions with more than one argument. The + ``METH_NOARGS`` and ``METH_O`` signatures provide slightly faster calling conventions but disallow the use of keywords. +``c_api_binop_methods`` (True / False) + When enabled, makes the special binary operator methods (``__add__``, etc.) + behave according to the low-level C-API slot semantics, i.e. only a single + method implements both the normal and reversed operator. This used to be + the default in Cython 0.x and was now replaced by Python semantics, i.e. the + default in Cython 3.x and later is ``False``. + ``profile`` (True / False) Write hooks for Python profilers into the compiled C code. Default is False. @@ -803,7 +910,7 @@ Cython code. Here is the list of currently supported directives: into the compiled C code. This also enables profiling. Default is False. Note that the generated module will not actually use line tracing, unless you additionally pass the C macro definition - ``CYTHON_TRACE=1`` to the C compiler (e.g. using the distutils option + ``CYTHON_TRACE=1`` to the C compiler (e.g. using the setuptools option ``define_macros``). Define ``CYTHON_TRACE_NOGIL=1`` to also include ``nogil`` functions and sections. @@ -816,13 +923,15 @@ Cython code. Here is the list of currently supported directives: explicitly requested. ``language_level`` (2/3/3str) - Globally set the Python language level to be used for module - compilation. Default is compatibility with Python 2. To enable - Python 3 source code semantics, set this to 3 (or 3str) at the start + Globally set the Python language level to be used for module compilation. + Default is compatibility with Python 3 in Cython 3.x and with Python 2 in Cython 0.x. + To enable Python 3 source code semantics, set this to 3 (or 3str) at the start of a module or pass the "-3" or "--3str" command line options to the - compiler. The ``3str`` option enables Python 3 semantics but does + compiler. For Python 2 semantics, use 2 and "-2" accordingly. The ``3str`` + option enables Python 3 semantics but does not change the ``str`` type and unprefixed string literals to ``unicode`` when the compiled code runs in Python 2.x. + Language level 2 ignores ``x: int`` type annotations due to the int/long ambiguity. Note that cimported files inherit this setting from the module being compiled, unless they explicitly set their own language level. Included source files always inherit this setting. @@ -857,6 +966,30 @@ Cython code. Here is the list of currently supported directives: selectively as decorator on an async-def coroutine to make the affected coroutine(s) iterable and thus directly interoperable with yield-from. +``annotation_typing`` (True / False) + Uses function argument annotations to determine the type of variables. Default + is True, but can be disabled. Since Python does not enforce types given in + annotations, setting to False gives greater compatibility with Python code. + From Cython 3.0, ``annotation_typing`` can be set on a per-function or + per-class basis. + +``emit_code_comments`` (True / False) + Copy the original source code line by line into C code comments in the generated + code file to help with understanding the output. + This is also required for coverage analysis. + +``cpp_locals`` (True / False) + Make C++ variables behave more like Python variables by allowing them to be + "unbound" instead of always default-constructing them at the start of a + function. See :ref:`cpp_locals directive` for more detail. + +``legacy_implicit_noexcept`` (True / False) + When enabled, ``cdef`` functions will not propagate raised exceptions by default. Hence, + the function will behave in the same way as if declared with `noexcept` keyword. See + :ref:`error_return_values` for details. Setting this directive to ``True`` will + cause Cython 3.0 to have the same semantics as Cython 0.x. This directive was solely added + to help migrate legacy code written before Cython 3. It will be removed in a future release. + .. _configurable_optimisations: @@ -878,6 +1011,7 @@ Configurable optimisations completely wrong. Disabling this option can also reduce the code size. Default is True. + .. _warnings: Warnings @@ -927,7 +1061,9 @@ One can set compiler directives through a special header comment near the top of The comment must appear before any code (but can appear after other comments or whitespace). -One can also pass a directive on the command line by using the -X switch:: +One can also pass a directive on the command line by using the -X switch: + +.. code-block:: bash $ cython -X boundscheck=True ... @@ -964,7 +1100,7 @@ In :file:`setup.py` Compiler directives can also be set in the :file:`setup.py` file by passing a keyword argument to ``cythonize``:: - from distutils.core import setup + from setuptools import setup from Cython.Build import cythonize setup( diff --git a/docs/src/userguide/special_methods.rst b/docs/src/userguide/special_methods.rst index 8bb4d9be4..d085aa284 100644 --- a/docs/src/userguide/special_methods.rst +++ b/docs/src/userguide/special_methods.rst @@ -3,6 +3,9 @@ Special Methods of Extension Types =================================== +.. include:: + ../two-syntax-variants-used + This page describes the special methods currently supported by Cython extension types. A complete list of all the special methods appears in the table at the bottom. Some of these methods behave differently from their Python @@ -12,7 +15,8 @@ mention. .. Note:: Everything said on this page applies only to extension types, defined - with the :keyword:`cdef class` statement. It doesn't apply to classes defined with the + with the :keyword:`cdef` class statement or decorated using ``@cclass`` decorator. + It doesn't apply to classes defined with the Python :keyword:`class` statement, where the normal Python rules apply. .. _declaration: @@ -20,7 +24,7 @@ mention. Declaration ------------ Special methods of extension types must be declared with :keyword:`def`, not -:keyword:`cdef`. This does not impact their performance--Python uses different +:keyword:`cdef`/``@cfunc``. This does not impact their performance--Python uses different calling conventions to invoke these special methods. .. _docstrings: @@ -34,68 +38,89 @@ won't show up in the corresponding :attr:`__doc__` attribute at run time. (This seems to be is a Python limitation -- there's nowhere in the `PyTypeObject` data structure to put such docstrings.) + .. _initialisation_methods: Initialisation methods: :meth:`__cinit__` and :meth:`__init__` --------------------------------------------------------------- -There are two methods concerned with initialising the object. - -The :meth:`__cinit__` method is where you should perform basic C-level -initialisation of the object, including allocation of any C data structures -that your object will own. You need to be careful what you do in the -:meth:`__cinit__` method, because the object may not yet be fully valid Python -object when it is called. Therefore, you should be careful invoking any Python -operations which might touch the object; in particular, its methods. - -By the time your :meth:`__cinit__` method is called, memory has been allocated for the -object and any C attributes it has have been initialised to 0 or null. (Any -Python attributes have also been initialised to None, but you probably -shouldn't rely on that.) Your :meth:`__cinit__` method is guaranteed to be called -exactly once. - -If your extension type has a base type, the :meth:`__cinit__` method of the base type -is automatically called before your :meth:`__cinit__` method is called; you cannot -explicitly call the inherited :meth:`__cinit__` method. If you need to pass a modified -argument list to the base type, you will have to do the relevant part of the -initialisation in the :meth:`__init__` method instead (where the normal rules for -calling inherited methods apply). - -Any initialisation which cannot safely be done in the :meth:`__cinit__` method should -be done in the :meth:`__init__` method. By the time :meth:`__init__` is called, the object is -a fully valid Python object and all operations are safe. Under some -circumstances it is possible for :meth:`__init__` to be called more than once or not -to be called at all, so your other methods should be designed to be robust in -such situations. +There are two methods concerned with initialising the object, the normal Python +:meth:`__init__` method and a special :meth:`__cinit__` method where basic +C level initialisation can be performed. + +The main difference between the two is when they are called. +The :meth:`__cinit__` method is guaranteed to be called as part of the object +allocation, but before the object is fully initialised. Specifically, methods +and object attributes that belong to subclasses or that were overridden by +subclasses may not have been initialised at all yet and must not be used by +:meth:`__cinit__` in a base class. Note that the object allocation in Python +clears all fields and sets them to zero (or ``NULL``). Cython additionally +takes responsibility of setting all object attributes to ``None``, but again, +this may not yet have been done for the attributes defined or overridden by +subclasses. If your object needs anything more than this basic attribute +clearing in order to get into a correct and safe state, :meth:`__cinit__` +may be a good place to do it. + +The :meth:`__init__` method, on the other hand, works exactly like in Python. +It is called after allocation and basic initialisation of the object, including +the complete inheritance chain. +By the time :meth:`__init__` is called, the object is a fully valid Python object +and all operations are safe. Any initialisation which cannot safely be done in +the :meth:`__cinit__` method should be done in the :meth:`__init__` method. +However, as in Python, it is the responsibility of the subclasses to call up the +hierarchy and make sure that the :meth:`__init__` methods in the base class are +called correctly. If a subclass forgets (or refuses) to call the :meth:`__init__` +method of one of its base classes, that method will not be called. +Also, if the object gets created by calling directly its :meth:`__new__` method [#]_ +(as opposed to calling the class itself), then none of the :meth:`__init__` +methods will be called. + +The :meth:`__cinit__` method is where you should perform basic safety C-level +initialisation of the object, possibly including allocation of any C data +structures that your object will own. In contrast to :meth:`__init__`, +your :meth:`__cinit__` method is guaranteed to be called exactly once. + +If your extension type has a base type, any existing :meth:`__cinit__` methods in +the base type hierarchy are automatically called before your :meth:`__cinit__` +method. You cannot explicitly call the inherited :meth:`__cinit__` methods, and the +base types are free to choose whether they implement :meth:`__cinit__` at all. +If you need to pass a modified argument list to the base type, you will have to do +the relevant part of the initialisation in the :meth:`__init__` method instead, +where the normal rules for calling inherited methods apply. Any arguments passed to the constructor will be passed to both the -:meth:`__cinit__` method and the :meth:`__init__` method. If you anticipate -subclassing your extension type in Python, you may find it useful to give the -:meth:`__cinit__` method `*` and `**` arguments so that it can accept and -ignore extra arguments. Otherwise, any Python subclass which has an -:meth:`__init__` with a different signature will have to override -:meth:`__new__` [#]_ as well as :meth:`__init__`, which the writer of a Python -class wouldn't expect to have to do. Alternatively, as a convenience, if you declare -your :meth:`__cinit__`` method to take no arguments (other than self) it -will simply ignore any extra arguments passed to the constructor without -complaining about the signature mismatch. +:meth:`__cinit__` method and the :meth:`__init__` method. If you anticipate +subclassing your extension type, you may find it useful to give the +:meth:`__cinit__` method ``*`` and ``**`` arguments so that it can accept and +ignore arbitrary extra arguments, since the arguments that are passed through +the hierarchy during allocation cannot be changed by subclasses. +Alternatively, as a convenience, if you declare your :meth:`__cinit__` method +to take no arguments (other than self) it will simply ignore any extra arguments +passed to the constructor without complaining about the signature mismatch. .. Note:: All constructor arguments will be passed as Python objects. This implies that non-convertible C types such as pointers or C++ objects - cannot be passed into the constructor from Cython code. If this is needed, - use a factory function instead that handles the object initialisation. - It often helps to directly call ``__new__()`` in this function to bypass the - call to the ``__init__()`` constructor. + cannot be passed into the constructor, neither from Python nor from Cython code. + If this is needed, use a factory function or method instead that handles the + object initialisation. + It often helps to directly call the :meth:`__new__` method in this function to + explicitly bypass the call to the :meth:`__init__` constructor. See :ref:`existing-pointers-instantiation` for an example. +.. Note:: + + Implementing a :meth:`__cinit__` method currently excludes the type from + :ref:`auto-pickling <auto_pickle>`. + .. [#] https://docs.python.org/reference/datamodel.html#object.__new__ + .. _finalization_method: -Finalization method: :meth:`__dealloc__` ----------------------------------------- +Finalization methods: :meth:`__dealloc__` and :meth:`__del__` +------------------------------------------------------------- The counterpart to the :meth:`__cinit__` method is the :meth:`__dealloc__` method, which should perform the inverse of the :meth:`__cinit__` method. Any @@ -119,41 +144,64 @@ of the superclass will always be called, even if it is overridden. This is in contrast to typical Python behavior where superclass methods will not be executed unless they are explicitly called by the subclass. -.. Note:: There is no :meth:`__del__` method for extension types. +Python 3.4 made it possible for extension types to safely define +finalizers for objects. When running a Cython module on Python 3.4 and +higher you can add a :meth:`__del__` method to extension types in +order to perform Python cleanup operations. When the :meth:`__del__` +is called the object is still in a valid state (unlike in the case of +:meth:`__dealloc__`), permitting the use of Python operations +on its class members. On Python <3.4 :meth:`__del__` will not be called. .. _arithmetic_methods: Arithmetic methods ------------------- -Arithmetic operator methods, such as :meth:`__add__`, behave differently from their -Python counterparts. There are no separate "reversed" versions of these -methods (:meth:`__radd__`, etc.) Instead, if the first operand cannot perform the -operation, the same method of the second operand is called, with the operands -in the same order. +Arithmetic operator methods, such as :meth:`__add__`, used to behave differently +from their Python counterparts in Cython 0.x, following the low-level semantics +of the C-API slot functions. Since Cython 3.0, they are called in the same way +as in Python, including the separate "reversed" versions of these methods +(:meth:`__radd__`, etc.). + +Previously, if the first operand could not perform the operation, the same method +of the second operand was called, with the operands in the same order. +This means that you could not rely on the first parameter of these methods being +"self" or being the right type, and you needed to test the types of both operands +before deciding what to do. + +If backwards compatibility is needed, the normal operator method (``__add__``, etc.) +can still be implemented to support both variants, applying a type check to the +arguments. The reversed method (``__radd__``, etc.) can always be implemented +with ``self`` as first argument and will be ignored by older Cython versions, whereas +Cython 3.x and later will only call the normal method with the expected argument order, +and otherwise call the reversed method instead. -This means that you can't rely on the first parameter of these methods being -"self" or being the right type, and you should test the types of both operands -before deciding what to do. If you can't handle the combination of types you've -been given, you should return `NotImplemented`. +Alternatively, the old Cython 0.x (or native C-API) behaviour is still available with +the directive ``c_api_binop_methods=True``. -This also applies to the in-place arithmetic method :meth:`__ipow__`. It doesn't apply -to any of the other in-place methods (:meth:`__iadd__`, etc.) which always -take `self` as the first argument. +If you can't handle the combination of types you've been given, you should return +``NotImplemented``. This will let Python's operator implementation first try to apply +the reversed operator to the second operand, and failing that as well, report an +appropriate error to the user. -.. _righ_comparisons: +This change in behaviour also applies to the in-place arithmetic method :meth:`__ipow__`. +It does not apply to any of the other in-place methods (:meth:`__iadd__`, etc.) +which always take ``self`` as the first argument. + +.. _rich_comparisons: Rich comparisons ----------------- -There are two ways to implement comparison methods. +There are a few ways to implement comparison methods. Depending on the application, one way or the other may be better: -* The first way uses the 6 Python +* Use the 6 Python `special methods <https://docs.python.org/3/reference/datamodel.html#basic-customization>`_ :meth:`__eq__`, :meth:`__lt__`, etc. - This is new since Cython 0.27 and works exactly as in plain Python classes. -* The second way uses a single special method :meth:`__richcmp__`. + This is supported since Cython 0.27 and works exactly as in plain Python classes. + +* Use a single special method :meth:`__richcmp__`. This implements all rich comparison operations in one method. The signature is ``def __richcmp__(self, other, int op)``. The integer argument ``op`` indicates which operation is to be performed @@ -175,6 +223,23 @@ Depending on the application, one way or the other may be better: These constants can be cimported from the ``cpython.object`` module. +* Use the ``@cython.total_ordering`` decorator, which is a low-level + re-implementation of the `functools.total_ordering + <https://docs.python.org/3/library/functools.html#functools.total_ordering>`_ + decorator specifically for ``cdef`` classes. (Normal Python classes can use + the original ``functools`` decorator.) + +.. tabs:: + + .. group-tab:: Pure Python + + .. literalinclude:: ../../examples/userguide/special_methods/total_ordering.py + + .. group-tab:: Cython + + .. literalinclude:: ../../examples/userguide/special_methods/total_ordering.pyx + + .. _the__next__method: The :meth:`__next__` method @@ -212,13 +277,13 @@ https://docs.python.org/3/reference/datamodel.html#special-method-names +-----------------------+---------------------------------------+-------------+-----------------------------------------------------+ | __dealloc__ |self | | Basic deallocation (no direct Python equivalent) | +-----------------------+---------------------------------------+-------------+-----------------------------------------------------+ -| __cmp__ |x, y | int | 3-way comparison | +| __cmp__ |x, y | int | 3-way comparison (Python 2 only) | +-----------------------+---------------------------------------+-------------+-----------------------------------------------------+ | __str__ |self | object | str(self) | +-----------------------+---------------------------------------+-------------+-----------------------------------------------------+ | __repr__ |self | object | repr(self) | +-----------------------+---------------------------------------+-------------+-----------------------------------------------------+ -| __hash__ |self | int | Hash function | +| __hash__ |self | Py_hash_t | Hash function (returns 32/64 bit integer) | +-----------------------+---------------------------------------+-------------+-----------------------------------------------------+ | __call__ |self, ... | object | self(...) | +-----------------------+---------------------------------------+-------------+-----------------------------------------------------+ @@ -266,47 +331,55 @@ Arithmetic operators https://docs.python.org/3/reference/datamodel.html#emulating-numeric-types -+-----------------------+---------------------------------------+-------------+-----------------------------------------------------+ -| Name | Parameters | Return type | Description | -+=======================+=======================================+=============+=====================================================+ -| __add__ | x, y | object | binary `+` operator | -+-----------------------+---------------------------------------+-------------+-----------------------------------------------------+ -| __sub__ | x, y | object | binary `-` operator | -+-----------------------+---------------------------------------+-------------+-----------------------------------------------------+ -| __mul__ | x, y | object | `*` operator | -+-----------------------+---------------------------------------+-------------+-----------------------------------------------------+ -| __div__ | x, y | object | `/` operator for old-style division | -+-----------------------+---------------------------------------+-------------+-----------------------------------------------------+ -| __floordiv__ | x, y | object | `//` operator | -+-----------------------+---------------------------------------+-------------+-----------------------------------------------------+ -| __truediv__ | x, y | object | `/` operator for new-style division | -+-----------------------+---------------------------------------+-------------+-----------------------------------------------------+ -| __mod__ | x, y | object | `%` operator | -+-----------------------+---------------------------------------+-------------+-----------------------------------------------------+ -| __divmod__ | x, y | object | combined div and mod | -+-----------------------+---------------------------------------+-------------+-----------------------------------------------------+ -| __pow__ | x, y, z | object | `**` operator or pow(x, y, z) | -+-----------------------+---------------------------------------+-------------+-----------------------------------------------------+ -| __neg__ | self | object | unary `-` operator | -+-----------------------+---------------------------------------+-------------+-----------------------------------------------------+ -| __pos__ | self | object | unary `+` operator | -+-----------------------+---------------------------------------+-------------+-----------------------------------------------------+ -| __abs__ | self | object | absolute value | -+-----------------------+---------------------------------------+-------------+-----------------------------------------------------+ -| __nonzero__ | self | int | convert to boolean | -+-----------------------+---------------------------------------+-------------+-----------------------------------------------------+ -| __invert__ | self | object | `~` operator | -+-----------------------+---------------------------------------+-------------+-----------------------------------------------------+ -| __lshift__ | x, y | object | `<<` operator | -+-----------------------+---------------------------------------+-------------+-----------------------------------------------------+ -| __rshift__ | x, y | object | `>>` operator | -+-----------------------+---------------------------------------+-------------+-----------------------------------------------------+ -| __and__ | x, y | object | `&` operator | -+-----------------------+---------------------------------------+-------------+-----------------------------------------------------+ -| __or__ | x, y | object | `|` operator | -+-----------------------+---------------------------------------+-------------+-----------------------------------------------------+ -| __xor__ | x, y | object | `^` operator | -+-----------------------+---------------------------------------+-------------+-----------------------------------------------------+ ++-----------------------------+--------------------+-------------+-----------------------------------------------------+ +| Name | Parameters | Return type | Description | ++=============================+====================+=============+=====================================================+ +| __add__, __radd__ | self, other | object | binary `+` operator | ++-----------------------------+--------------------+-------------+-----------------------------------------------------+ +| __sub__, __rsub__ | self, other | object | binary `-` operator | ++-----------------------------+--------------------+-------------+-----------------------------------------------------+ +| __mul__, __rmul__ | self, other | object | `*` operator | ++-----------------------------+--------------------+-------------+-----------------------------------------------------+ +| __div__, __rdiv__ | self, other | object | `/` operator for old-style division | ++-----------------------------+--------------------+-------------+-----------------------------------------------------+ +| __floordiv__, __rfloordiv__ | self, other | object | `//` operator | ++-----------------------------+--------------------+-------------+-----------------------------------------------------+ +| __truediv__, __rtruediv__ | self, other | object | `/` operator for new-style division | ++-----------------------------+--------------------+-------------+-----------------------------------------------------+ +| __mod__, __rmod__ | self, other | object | `%` operator | ++-----------------------------+--------------------+-------------+-----------------------------------------------------+ +| __divmod__, __rdivmod__ | self, other | object | combined div and mod | ++-----------------------------+--------------------+-------------+-----------------------------------------------------+ +| __pow__, __rpow__ | self, other, [mod] | object | `**` operator or pow(x, y, [mod]) | ++-----------------------------+--------------------+-------------+-----------------------------------------------------+ +| __neg__ | self | object | unary `-` operator | ++-----------------------------+--------------------+-------------+-----------------------------------------------------+ +| __pos__ | self | object | unary `+` operator | ++-----------------------------+--------------------+-------------+-----------------------------------------------------+ +| __abs__ | self | object | absolute value | ++-----------------------------+--------------------+-------------+-----------------------------------------------------+ +| __nonzero__ | self | int | convert to boolean | ++-----------------------------+--------------------+-------------+-----------------------------------------------------+ +| __invert__ | self | object | `~` operator | ++-----------------------------+--------------------+-------------+-----------------------------------------------------+ +| __lshift__, __rlshift__ | self, other | object | `<<` operator | ++-----------------------------+--------------------+-------------+-----------------------------------------------------+ +| __rshift__, __rrshift__ | self, other | object | `>>` operator | ++-----------------------------+--------------------+-------------+-----------------------------------------------------+ +| __and__, __rand__ | self, other | object | `&` operator | ++-----------------------------+--------------------+-------------+-----------------------------------------------------+ +| __or__, __ror__ | self, other | object | `|` operator | ++-----------------------------+--------------------+-------------+-----------------------------------------------------+ +| __xor__, __rxor__ | self, other | object | `^` operator | ++-----------------------------+--------------------+-------------+-----------------------------------------------------+ + +Note that Cython 0.x did not make use of the ``__r...__`` variants and instead +used the bidirectional C slot signature for the regular methods, thus making the +first argument ambiguous (not 'self' typed). +Since Cython 3.0, the operator calls are passed to the respective special methods. +See the section on :ref:`Arithmetic methods <arithmetic_methods>` above. +Cython 0.x also did not support the 2 argument version of ``__pow__`` and +``__rpow__``, or the 3 argument version of ``__ipow__``. Numeric conversions ^^^^^^^^^^^^^^^^^^^ @@ -326,7 +399,7 @@ https://docs.python.org/3/reference/datamodel.html#emulating-numeric-types +-----------------------+---------------------------------------+-------------+-----------------------------------------------------+ | __hex__ | self | object | Convert to hexadecimal | +-----------------------+---------------------------------------+-------------+-----------------------------------------------------+ -| __index__ (2.5+ only) | self | object | Convert to sequence index | +| __index__ | self | object | Convert to sequence index | +-----------------------+---------------------------------------+-------------+-----------------------------------------------------+ In-place arithmetic operators @@ -351,7 +424,7 @@ https://docs.python.org/3/reference/datamodel.html#emulating-numeric-types +-----------------------+---------------------------------------+-------------+-----------------------------------------------------+ | __imod__ | self, x | object | `%=` operator | +-----------------------+---------------------------------------+-------------+-----------------------------------------------------+ -| __ipow__ | x, y, z | object | `**=` operator | +| __ipow__ | self, y, [z] | object | `**=` operator (3-arg form only on Python >= 3.8) | +-----------------------+---------------------------------------+-------------+-----------------------------------------------------+ | __ilshift__ | self, x | object | `<<=` operator | +-----------------------+---------------------------------------+-------------+-----------------------------------------------------+ @@ -372,7 +445,7 @@ https://docs.python.org/3/reference/datamodel.html#emulating-container-types +-----------------------+---------------------------------------+-------------+-----------------------------------------------------+ | Name | Parameters | Return type | Description | +=======================+=======================================+=============+=====================================================+ -| __len__ | self int | | len(self) | +| __len__ | self | Py_ssize_t | len(self) | +-----------------------+---------------------------------------+-------------+-----------------------------------------------------+ | __getitem__ | self, x | object | self[x] | +-----------------------+---------------------------------------+-------------+-----------------------------------------------------+ diff --git a/docs/src/userguide/wrapping_CPlusPlus.rst b/docs/src/userguide/wrapping_CPlusPlus.rst index e12bf38be..47b55245c 100644 --- a/docs/src/userguide/wrapping_CPlusPlus.rst +++ b/docs/src/userguide/wrapping_CPlusPlus.rst @@ -11,8 +11,8 @@ Overview Cython has native support for most of the C++ language. Specifically: -* C++ objects can be dynamically allocated with ``new`` and ``del`` keywords. -* C++ objects can be stack-allocated. +* C++ objects can be :term:`dynamically allocated<Dynamic allocation or Heap allocation>` with ``new`` and ``del`` keywords. +* C++ objects can be :term:`stack-allocated<Stack allocation>`. * C++ classes can be declared with the new keyword ``cppclass``. * Templated classes and functions are supported. * Overloaded functions are supported. @@ -97,7 +97,7 @@ We use the lines:: pass to include the C++ code from :file:`Rectangle.cpp`. It is also possible to specify to -distutils that :file:`Rectangle.cpp` is a source. To do that, you can add this directive at the +setuptools that :file:`Rectangle.cpp` is a source. To do that, you can add this directive at the top of the ``.pyx`` (not ``.pxd``) file:: # distutils: sources = Rectangle.cpp @@ -136,6 +136,9 @@ a "default" constructor:: def func(): cdef Foo foo ... + +See the section on the :ref:`cpp_locals directive` for a way +to avoid requiring a nullary/default constructor. Note that, like C++, if the class has only one constructor and it is a nullary one, it's not necessary to declare it. @@ -162,7 +165,9 @@ attribute access, you could just implement some properties: Cython initializes C++ class attributes of a cdef class using the nullary constructor. If the class you're wrapping does not have a nullary constructor, you must store a pointer -to the wrapped class and manually allocate and deallocate it. +to the wrapped class and manually allocate and deallocate it. Alternatively, the +:ref:`cpp_locals directive` avoids the need for the pointer and only initializes the +C++ class attribute when it is assigned to. A convenient and safe place to do so is in the `__cinit__` and `__dealloc__` methods which are guaranteed to be called exactly once upon creation and deletion of the Python instance. @@ -331,19 +336,27 @@ arguments) or by an explicit cast, e.g.: The following coercions are available: -+------------------+----------------+-----------------+ -| Python type => | *C++ type* | => Python type | -+==================+================+=================+ -| bytes | std::string | bytes | -+------------------+----------------+-----------------+ -| iterable | std::vector | list | -+------------------+----------------+-----------------+ -| iterable | std::list | list | -+------------------+----------------+-----------------+ -| iterable | std::set | set | -+------------------+----------------+-----------------+ -| iterable (len 2) | std::pair | tuple (len 2) | -+------------------+----------------+-----------------+ ++------------------+------------------------+-----------------+ +| Python type => | *C++ type* | => Python type | ++==================+========================+=================+ +| bytes | std::string | bytes | ++------------------+------------------------+-----------------+ +| iterable | std::vector | list | ++------------------+------------------------+-----------------+ +| iterable | std::list | list | ++------------------+------------------------+-----------------+ +| iterable | std::set | set | ++------------------+------------------------+-----------------+ +| iterable | std::unordered_set | set | ++------------------+------------------------+-----------------+ +| mapping | std::map | dict | ++------------------+------------------------+-----------------+ +| mapping | std::unordered_map | dict | ++------------------+------------------------+-----------------+ +| iterable (len 2) | std::pair | tuple (len 2) | ++------------------+------------------------+-----------------+ +| complex | std::complex | complex | ++------------------+------------------------+-----------------+ All conversions create a new container and copy the data into it. The items in the containers are converted to a corresponding type @@ -432,7 +445,10 @@ for Cython to discern that, so watch out with exception masks on IO streams. :: cdef int bar() except +MemoryError This will catch any C++ error and raise a Python MemoryError in its place. -(Any Python exception is valid here.) :: +(Any Python exception is valid here.) + +Cython also supports using a custom exception handler. This is an advanced feature +that most users won't need, but for those that do a full example follows:: cdef int raise_py_error() cdef int something_dangerous() except +raise_py_error @@ -440,7 +456,90 @@ This will catch any C++ error and raise a Python MemoryError in its place. If something_dangerous raises a C++ exception then raise_py_error will be called, which allows one to do custom C++ to Python error "translations." If raise_py_error does not actually raise an exception a RuntimeError will be -raised. +raised. This approach may also be used to manage custom Python exceptions +created using the Python C API. :: + + # raising.pxd + cdef extern from "Python.h" nogil: + ctypedef struct PyObject + + cdef extern from *: + """ + #include <Python.h> + #include <stdexcept> + #include <ios> + + PyObject *CustomLogicError; + + void create_custom_exceptions() { + CustomLogicError = PyErr_NewException("raiser.CustomLogicError", NULL, NULL); + } + + void custom_exception_handler() { + try { + if (PyErr_Occurred()) { + ; // let the latest Python exn pass through and ignore the current one + } else { + throw; + } + } catch (const std::logic_error& exn) { + // Add mapping of std::logic_error -> CustomLogicError + PyErr_SetString(CustomLogicError, exn.what()); + } catch (...) { + PyErr_SetString(PyExc_RuntimeError, "Unknown exception"); + } + } + + class Raiser { + public: + Raiser () {} + void raise_exception() { + throw std::logic_error("Failure"); + } + }; + """ + cdef PyObject* CustomLogicError + cdef void create_custom_exceptions() + cdef void custom_exception_handler() + + cdef cppclass Raiser: + Raiser() noexcept + void raise_exception() except +custom_exception_handler + + + # raising.pyx + create_custom_exceptions() + PyCustomLogicError = <object> CustomLogicError + + + cdef class PyRaiser: + cdef Raiser c_obj + + def raise_exception(self): + self.c_obj.raise_exception() + +The above example leverages Cython's ability to include :ref:`verbatim C code +<verbatim_c>` in pxd files to create a new Python exception type +``CustomLogicError`` and map it to the standard C++ ``std::logic_error`` using +the ``custom_exception_handler`` function. There is nothing special about using +a standard exception class here, ``std::logic_error`` could easily be replaced +with some new C++ exception type defined in this file. The +``Raiser::raise_exception`` is marked with ``+custom_exception_handler`` to +indicate that this function should be called whenever an exception is raised. +The corresponding Python function ``PyRaiser.raise_exception`` will raise a +``CustomLogicError`` whenever it is called. Defining ``PyCustomLogicError`` +allows other code to catch this exception, as shown below: :: + + try: + PyRaiser().raise_exception() + except PyCustomLogicError: + print("Caught the exception") + +When defining custom exception handlers it is typically good to also include +logic to handle all the standard exceptions that Cython typically handles as +listed in the table above. The code for this standard exception handler can be +found `here +<https://github.com/cython/cython/blob/master/Cython/Utility/CppSupport.cpp>`__. There is also the special form:: @@ -454,7 +553,7 @@ Static member method If the Rectangle class has a static member: -.. sourcecode:: c++ +.. code-block:: c++ namespace shapes { class Rectangle { @@ -482,6 +581,33 @@ Note, however, that it is unnecessary to declare the arguments of extern functions as references (const or otherwise) as it has no impact on the caller's syntax. +Scoped Enumerations +------------------- + +Cython supports scoped enumerations (:keyword:`enum class`) in C++ mode:: + + cdef enum class Cheese: + cheddar = 1 + camembert = 2 + +As with "plain" enums, you may access the enumerators as attributes of the type. +Unlike plain enums however, the enumerators are not visible to the +enclosing scope:: + + cdef Cheese c1 = Cheese.cheddar # OK + cdef Cheese c2 = cheddar # ERROR! + +Optionally, you may specify the underlying type of a scoped enumeration. +This is especially important when declaring an external scoped enumeration +with an underlying type:: + + cdef extern from "Foo.h": + cdef enum class Spam(unsigned int): + x = 10 + y = 20 + ... + +Declaring an enum class as ``cpdef`` will create a :pep:`435`-style Python wrapper. ``auto`` Keyword ---------------- @@ -527,7 +653,7 @@ Specify C++ language in setup.py Instead of specifying the language and the sources in the source files, it is possible to declare them in the :file:`setup.py` file:: - from distutils.core import setup + from setuptools import setup from Cython.Build import cythonize setup(ext_modules = cythonize( @@ -553,7 +679,7 @@ recognize the ``language`` option and it needs to be specified as an option to an :class:`Extension` that describes your extension and that is then handled by ``cythonize()`` as follows:: - from distutils.core import setup, Extension + from setuptools import Extension, setup from Cython.Build import cythonize setup(ext_modules = cythonize(Extension( @@ -568,7 +694,7 @@ often preferable (and overrides any global option). Starting with version 0.17, Cython also allows passing external source files into the ``cythonize()`` command this way. Here is a simplified setup.py file:: - from distutils.core import setup + from setuptools import setup from Cython.Build import cythonize setup( @@ -586,14 +712,52 @@ any source code, to compile it in C++ mode and link it statically against the .. note:: When using distutils directives, the paths are relative to the working - directory of the distutils run (which is usually the - project root where the :file:`setup.py` resides). + directory of the setuptools run (which is usually the project root where + the :file:`setup.py` resides). To compile manually (e.g. using ``make``), the ``cython`` command-line utility can be used to generate a C++ ``.cpp`` file, and then compile it into a python extension. C++ mode for the ``cython`` command is turned on with the ``--cplus`` option. +.. _cpp_locals directive: + +``cpp_locals`` directive +======================== + +The ``cpp_locals`` compiler directive is an experimental feature that makes +C++ variables behave like normal Python object variables. With this +directive they are only initialized at their first assignment, and thus +they no longer require a nullary constructor to be stack-allocated. Trying to +access an uninitialized C++ variable will generate an ``UnboundLocalError`` +(or similar) in the same way as a Python variable would. For example:: + + def function(dont_write): + cdef SomeCppClass c # not initialized + if dont_write: + return c.some_cpp_function() # UnboundLocalError + else: + c = SomeCppClass(...) # initialized + return c.some_cpp_function() # OK + +Additionally, the directive avoids initializing temporary C++ objects before +they are assigned, for cases where Cython needs to use such objects in its +own code-generation (often for return values of functions that can throw +exceptions). + +For extra speed, the ``initializedcheck`` directive disables the check for an +unbound-local. With this directive on, accessing a variable that has not +been initialized will trigger undefined behaviour, and it is entirely the user's +responsibility to avoid such access. + +The ``cpp_locals`` directive is currently implemented using ``std::optional`` +and thus requires a C++17 compatible compiler. Defining +``CYTHON_USE_BOOST_OPTIONAL`` (as define for the C++ compiler) uses ``boost::optional`` +instead (but is even more experimental and untested). The directive may +come with a memory and performance cost due to the need to store and check +a boolean that tracks if a variable is initialized, but the C++ compiler should +be able to eliminate the check in most cases. + Caveats and Limitations ======================== |
