diff options
Diffstat (limited to 'Doc/c-api/memory.rst')
| -rw-r--r-- | Doc/c-api/memory.rst | 171 |
1 files changed, 170 insertions, 1 deletions
diff --git a/Doc/c-api/memory.rst b/Doc/c-api/memory.rst index 8afa56ac6c..e4a759b563 100644 --- a/Doc/c-api/memory.rst +++ b/Doc/c-api/memory.rst @@ -84,6 +84,48 @@ the C library allocator as shown in the previous example, the allocated memory for the I/O buffer escapes completely the Python memory manager. +Raw Memory Interface +==================== + +The following function sets are wrappers to the system allocator. These +functions are thread-safe, the :term:`GIL <global interpreter lock>` does not +need to be held. + +The default raw memory block allocator uses the following functions: +:c:func:`malloc`, :c:func:`realloc` and :c:func:`free`; call ``malloc(1)`` when +requesting zero bytes. + +.. versionadded:: 3.4 + +.. c:function:: void* PyMem_RawMalloc(size_t n) + + Allocates *n* bytes and returns a pointer of type :c:type:`void\*` to the + allocated memory, or *NULL* if the request fails. Requesting zero bytes + returns a distinct non-*NULL* pointer if possible, as if + ``PyMem_RawMalloc(1)`` had been called instead. The memory will not have + been initialized in any way. + + +.. c:function:: void* PyMem_RawRealloc(void *p, size_t n) + + Resizes the memory block pointed to by *p* to *n* bytes. The contents will + be unchanged to the minimum of the old and the new sizes. If *p* is *NULL*, + the call is equivalent to ``PyMem_RawMalloc(n)``; else if *n* is equal to + zero, the memory block is resized but is not freed, and the returned pointer + is non-*NULL*. Unless *p* is *NULL*, it must have been returned by a + previous call to :c:func:`PyMem_RawMalloc` or :c:func:`PyMem_RawRealloc`. If + the request fails, :c:func:`PyMem_RawRealloc` returns *NULL* and *p* remains + a valid pointer to the previous memory area. + + +.. c:function:: void PyMem_RawFree(void *p) + + Frees the memory block pointed to by *p*, which must have been returned by a + previous call to :c:func:`PyMem_RawMalloc` or :c:func:`PyMem_RawRealloc`. + Otherwise, or if ``PyMem_Free(p)`` has been called before, undefined + behavior occurs. If *p* is *NULL*, no operation is performed. + + .. _memoryinterface: Memory Interface @@ -91,8 +133,16 @@ Memory Interface The following function sets, modeled after the ANSI C standard, but specifying behavior when requesting zero bytes, are available for allocating and releasing -memory from the Python heap: +memory from the Python heap. + +The default memory block allocator uses the following functions: +:c:func:`malloc`, :c:func:`realloc` and :c:func:`free`; call ``malloc(1)`` when +requesting zero bytes. + +.. warning:: + The :term:`GIL <global interpreter lock>` must be held when using these + functions. .. c:function:: void* PyMem_Malloc(size_t n) @@ -155,6 +205,125 @@ versions and is therefore deprecated in extension modules. :c:func:`PyMem_NEW`, :c:func:`PyMem_RESIZE`, :c:func:`PyMem_DEL`. +Customize Memory Allocators +=========================== + +.. versionadded:: 3.4 + +.. c:type:: PyMemAllocator + + Structure used to describe a memory block allocator. The structure has + four fields: + + +----------------------------------------------------------+---------------------------------------+ + | Field | Meaning | + +==========================================================+=======================================+ + | ``void *ctx`` | user context passed as first argument | + +----------------------------------------------------------+---------------------------------------+ + | ``void* malloc(void *ctx, size_t size)`` | allocate a memory block | + +----------------------------------------------------------+---------------------------------------+ + | ``void* realloc(void *ctx, void *ptr, size_t new_size)`` | allocate or resize a memory block | + +----------------------------------------------------------+---------------------------------------+ + | ``void free(void *ctx, void *ptr)`` | free a memory block | + +----------------------------------------------------------+---------------------------------------+ + +.. c:type:: PyMemAllocatorDomain + + Enum used to identify an allocator domain. Domains: + + * :c:data:`PYMEM_DOMAIN_RAW`: functions :c:func:`PyMem_RawMalloc`, + :c:func:`PyMem_RawRealloc` and :c:func:`PyMem_RawFree` + * :c:data:`PYMEM_DOMAIN_MEM`: functions :c:func:`PyMem_Malloc`, + :c:func:`PyMem_Realloc` and :c:func:`PyMem_Free` + * :c:data:`PYMEM_DOMAIN_OBJ`: functions :c:func:`PyObject_Malloc`, + :c:func:`PyObject_Realloc` and :c:func:`PyObject_Free` + + +.. c:function:: void PyMem_GetAllocator(PyMemAllocatorDomain domain, PyMemAllocator *allocator) + + Get the memory block allocator of the specified domain. + + +.. c:function:: void PyMem_SetAllocator(PyMemAllocatorDomain domain, PyMemAllocator *allocator) + + Set the memory block allocator of the specified domain. + + The new allocator must return a distinct non-NULL pointer when requesting + zero bytes. + + For the :c:data:`PYMEM_DOMAIN_RAW` domain, the allocator must be + thread-safe: the :term:`GIL <global interpreter lock>` is not held when the + allocator is called. + + If the new allocator is not a hook (does not call the previous allocator), + the :c:func:`PyMem_SetupDebugHooks` function must be called to reinstall the + debug hooks on top on the new allocator. + + +.. c:function:: void PyMem_SetupDebugHooks(void) + + Setup hooks to detect bugs in the following Python memory allocator + functions: + + - :c:func:`PyMem_RawMalloc`, :c:func:`PyMem_RawRealloc`, + :c:func:`PyMem_RawFree` + - :c:func:`PyMem_Malloc`, :c:func:`PyMem_Realloc`, :c:func:`PyMem_Free` + - :c:func:`PyObject_Malloc`, :c:func:`PyObject_Realloc`, + :c:func:`PyObject_Free` + + Newly allocated memory is filled with the byte ``0xCB``, freed memory is + filled with the byte ``0xDB``. Additionnal checks: + + - detect API violations, ex: :c:func:`PyObject_Free` called on a buffer + allocated by :c:func:`PyMem_Malloc` + - detect write before the start of the buffer (buffer underflow) + - detect write after the end of the buffer (buffer overflow) + + The function does nothing if Python is not compiled is debug mode. + + +Customize PyObject Arena Allocator +================================== + +Python has a *pymalloc* allocator for allocations smaller than 512 bytes. This +allocator is optimized for small objects with a short lifetime. It uses memory +mappings called "arenas" with a fixed size of 256 KB. It falls back to +:c:func:`PyMem_Malloc` and :c:func:`PyMem_Realloc` for allocations larger than +512 bytes. *pymalloc* is the default allocator used by +:c:func:`PyObject_Malloc`. + +The default arena allocator uses the following functions: + +* :c:func:`VirtualAlloc` and :c:func:`VirtualFree` on Windows, +* :c:func:`mmap` and :c:func:`munmap` if available, +* :c:func:`malloc` and :c:func:`free` otherwise. + +.. versionadded:: 3.4 + +.. c:type:: PyObjectArenaAllocator + + Structure used to describe an arena allocator. The structure has + three fields: + + +--------------------------------------------------+---------------------------------------+ + | Field | Meaning | + +==================================================+=======================================+ + | ``void *ctx`` | user context passed as first argument | + +--------------------------------------------------+---------------------------------------+ + | ``void* alloc(void *ctx, size_t size)`` | allocate an arena of size bytes | + +--------------------------------------------------+---------------------------------------+ + | ``void free(void *ctx, size_t size, void *ptr)`` | free an arena | + +--------------------------------------------------+---------------------------------------+ + +.. c:function:: PyObject_GetArenaAllocator(PyObjectArenaAllocator *allocator) + + Get the arena allocator. + +.. c:function:: PyObject_SetArenaAllocator(PyObjectArenaAllocator *allocator) + + Set the arena allocator. + + .. _memoryexamples: Examples |