mirror of
https://github.com/python/cpython
synced 2026-09-29 12:10:30 +03:00
Issue #3329: Implement the PEP 445
Add new enum: * PyMemAllocatorDomain Add new structures: * PyMemAllocator * PyObjectArenaAllocator Add new functions: * PyMem_RawMalloc(), PyMem_RawRealloc(), PyMem_RawFree() * PyMem_GetAllocator(), PyMem_SetAllocator() * PyObject_GetArenaAllocator(), PyObject_SetArenaAllocator() * PyMem_SetupDebugHooks() Changes: * PyMem_Malloc()/PyObject_Realloc() now always call malloc()/realloc(), instead of calling PyObject_Malloc()/PyObject_Realloc() in debug mode. * PyObject_Malloc()/PyObject_Realloc() now falls back to PyMem_Malloc()/PyMem_Realloc() for allocations larger than 512 bytes. * Redesign debug checks on memory block allocators as hooks, instead of using C macros
This commit is contained in:
Victor Stinner
parent
df715ba54d
commit
0507bf56f0
8 changed files with 797 additions and 238 deletions
|
|
@ -84,6 +84,48 @@ the C library allocator as shown in the previous example, the allocated memory
|
|||
for the I/O buffer escapes completely the Python memory manager.
|
||||
|
||||
|
||||
Raw Memory Interface
|
||||
====================
|
||||
|
||||
The following function sets are wrappers to the system allocator. These
|
||||
functions are thread-safe, the :term:`GIL <global interpreter lock>` does not
|
||||
need to be held.
|
||||
|
||||
The default raw memory block allocator uses the following functions:
|
||||
:c:func:`malloc`, :c:func:`realloc` and :c:func:`free`; call ``malloc(1)`` when
|
||||
requesting zero bytes.
|
||||
|
||||
.. versionadded:: 3.4
|
||||
|
||||
.. c:function:: void* PyMem_RawMalloc(size_t n)
|
||||
|
||||
Allocates *n* bytes and returns a pointer of type :c:type:`void\*` to the
|
||||
allocated memory, or *NULL* if the request fails. Requesting zero bytes
|
||||
returns a distinct non-*NULL* pointer if possible, as if
|
||||
``PyMem_RawMalloc(1)`` had been called instead. The memory will not have
|
||||
been initialized in any way.
|
||||
|
||||
|
||||
.. c:function:: void* PyMem_RawRealloc(void *p, size_t n)
|
||||
|
||||
Resizes the memory block pointed to by *p* to *n* bytes. The contents will
|
||||
be unchanged to the minimum of the old and the new sizes. If *p* is *NULL*,
|
||||
the call is equivalent to ``PyMem_RawMalloc(n)``; else if *n* is equal to
|
||||
zero, the memory block is resized but is not freed, and the returned pointer
|
||||
is non-*NULL*. Unless *p* is *NULL*, it must have been returned by a
|
||||
previous call to :c:func:`PyMem_RawMalloc` or :c:func:`PyMem_RawRealloc`. If
|
||||
the request fails, :c:func:`PyMem_RawRealloc` returns *NULL* and *p* remains
|
||||
a valid pointer to the previous memory area.
|
||||
|
||||
|
||||
.. c:function:: void PyMem_RawFree(void *p)
|
||||
|
||||
Frees the memory block pointed to by *p*, which must have been returned by a
|
||||
previous call to :c:func:`PyMem_RawMalloc` or :c:func:`PyMem_RawRealloc`.
|
||||
Otherwise, or if ``PyMem_Free(p)`` has been called before, undefined
|
||||
behavior occurs. If *p* is *NULL*, no operation is performed.
|
||||
|
||||
|
||||
.. _memoryinterface:
|
||||
|
||||
Memory Interface
|
||||
|
|
@ -91,8 +133,16 @@ Memory Interface
|
|||
|
||||
The following function sets, modeled after the ANSI C standard, but specifying
|
||||
behavior when requesting zero bytes, are available for allocating and releasing
|
||||
memory from the Python heap:
|
||||
memory from the Python heap.
|
||||
|
||||
The default memory block allocator uses the following functions:
|
||||
:c:func:`malloc`, :c:func:`realloc` and :c:func:`free`; call ``malloc(1)`` when
|
||||
requesting zero bytes.
|
||||
|
||||
.. warning::
|
||||
|
||||
The :term:`GIL <global interpreter lock>` must be held when using these
|
||||
functions.
|
||||
|
||||
.. c:function:: void* PyMem_Malloc(size_t n)
|
||||
|
||||
|
|
@ -155,6 +205,125 @@ versions and is therefore deprecated in extension modules.
|
|||
:c:func:`PyMem_NEW`, :c:func:`PyMem_RESIZE`, :c:func:`PyMem_DEL`.
|
||||
|
||||
|
||||
Customize Memory Allocators
|
||||
===========================
|
||||
|
||||
.. versionadded:: 3.4
|
||||
|
||||
.. c:type:: PyMemAllocator
|
||||
|
||||
Structure used to describe a memory block allocator. The structure has
|
||||
four fields:
|
||||
|
||||
+----------------------------------------------------------+---------------------------------------+
|
||||
| Field | Meaning |
|
||||
+==========================================================+=======================================+
|
||||
| ``void *ctx`` | user context passed as first argument |
|
||||
+----------------------------------------------------------+---------------------------------------+
|
||||
| ``void* malloc(void *ctx, size_t size)`` | allocate a memory block |
|
||||
+----------------------------------------------------------+---------------------------------------+
|
||||
| ``void* realloc(void *ctx, void *ptr, size_t new_size)`` | allocate or resize a memory block |
|
||||
+----------------------------------------------------------+---------------------------------------+
|
||||
| ``void free(void *ctx, void *ptr)`` | free a memory block |
|
||||
+----------------------------------------------------------+---------------------------------------+
|
||||
|
||||
.. c:type:: PyMemAllocatorDomain
|
||||
|
||||
Enum used to identify an allocator domain. Domains:
|
||||
|
||||
* :c:data:`PYMEM_DOMAIN_RAW`: functions :c:func:`PyMem_RawMalloc`,
|
||||
:c:func:`PyMem_RawRealloc` and :c:func:`PyMem_RawFree`
|
||||
* :c:data:`PYMEM_DOMAIN_MEM`: functions :c:func:`PyMem_Malloc`,
|
||||
:c:func:`PyMem_Realloc` and :c:func:`PyMem_Free`
|
||||
* :c:data:`PYMEM_DOMAIN_OBJ`: functions :c:func:`PyObject_Malloc`,
|
||||
:c:func:`PyObject_Realloc` and :c:func:`PyObject_Free`
|
||||
|
||||
|
||||
.. c:function:: void PyMem_GetAllocator(PyMemAllocatorDomain domain, PyMemAllocator *allocator)
|
||||
|
||||
Get the memory block allocator of the specified domain.
|
||||
|
||||
|
||||
.. c:function:: void PyMem_SetAllocator(PyMemAllocatorDomain domain, PyMemAllocator *allocator)
|
||||
|
||||
Set the memory block allocator of the specified domain.
|
||||
|
||||
The new allocator must return a distinct non-NULL pointer when requesting
|
||||
zero bytes.
|
||||
|
||||
For the :c:data:`PYMEM_DOMAIN_RAW` domain, the allocator must be
|
||||
thread-safe: the :term:`GIL <global interpreter lock>` is not held when the
|
||||
allocator is called.
|
||||
|
||||
If the new allocator is not a hook (does not call the previous allocator),
|
||||
the :c:func:`PyMem_SetupDebugHooks` function must be called to reinstall the
|
||||
debug hooks on top on the new allocator.
|
||||
|
||||
|
||||
.. c:function:: void PyMem_SetupDebugHooks(void)
|
||||
|
||||
Setup hooks to detect bugs in the following Python memory allocator
|
||||
functions:
|
||||
|
||||
- :c:func:`PyMem_RawMalloc`, :c:func:`PyMem_RawRealloc`,
|
||||
:c:func:`PyMem_RawFree`
|
||||
- :c:func:`PyMem_Malloc`, :c:func:`PyMem_Realloc`, :c:func:`PyMem_Free`
|
||||
- :c:func:`PyObject_Malloc`, :c:func:`PyObject_Realloc`,
|
||||
:c:func:`PyObject_Free`
|
||||
|
||||
Newly allocated memory is filled with the byte ``0xCB``, freed memory is
|
||||
filled with the byte ``0xDB``. Additionnal checks:
|
||||
|
||||
- detect API violations, ex: :c:func:`PyObject_Free` called on a buffer
|
||||
allocated by :c:func:`PyMem_Malloc`
|
||||
- detect write before the start of the buffer (buffer underflow)
|
||||
- detect write after the end of the buffer (buffer overflow)
|
||||
|
||||
The function does nothing if Python is not compiled is debug mode.
|
||||
|
||||
|
||||
Customize PyObject Arena Allocator
|
||||
==================================
|
||||
|
||||
Python has a *pymalloc* allocator for allocations smaller than 512 bytes. This
|
||||
allocator is optimized for small objects with a short lifetime. It uses memory
|
||||
mappings called "arenas" with a fixed size of 256 KB. It falls back to
|
||||
:c:func:`PyMem_Malloc` and :c:func:`PyMem_Realloc` for allocations larger than
|
||||
512 bytes. *pymalloc* is the default allocator used by
|
||||
:c:func:`PyObject_Malloc`.
|
||||
|
||||
The default arena allocator uses the following functions:
|
||||
|
||||
* :c:func:`VirtualAlloc` and :c:func:`VirtualFree` on Windows,
|
||||
* :c:func:`mmap` and :c:func:`munmap` if available,
|
||||
* :c:func:`malloc` and :c:func:`free` otherwise.
|
||||
|
||||
.. versionadded:: 3.4
|
||||
|
||||
.. c:type:: PyObjectArenaAllocator
|
||||
|
||||
Structure used to describe an arena allocator. The structure has
|
||||
three fields:
|
||||
|
||||
+--------------------------------------------------+---------------------------------------+
|
||||
| Field | Meaning |
|
||||
+==================================================+=======================================+
|
||||
| ``void *ctx`` | user context passed as first argument |
|
||||
+--------------------------------------------------+---------------------------------------+
|
||||
| ``void* alloc(void *ctx, size_t size)`` | allocate an arena of size bytes |
|
||||
+--------------------------------------------------+---------------------------------------+
|
||||
| ``void free(void *ctx, size_t size, void *ptr)`` | free an arena |
|
||||
+--------------------------------------------------+---------------------------------------+
|
||||
|
||||
.. c:function:: PyObject_GetArenaAllocator(PyObjectArenaAllocator *allocator)
|
||||
|
||||
Get the arena allocator.
|
||||
|
||||
.. c:function:: PyObject_SetArenaAllocator(PyObjectArenaAllocator *allocator)
|
||||
|
||||
Set the arena allocator.
|
||||
|
||||
|
||||
.. _memoryexamples:
|
||||
|
||||
Examples
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue