np.ctypeslib.as_array leaks memory when used on a pointer #6511
Description
I have a C++ function returning an array, which I convert to a numpy array using np.ctypeslib.as_array(pointer_from_C++_function, (size_of_array,)).
This works as expected, but when I repeatedly call this function (about a million times) I found a substantial increase of memory usage for my python process.
Running a sample script (attached, output attached) through valgrind's memcheck (output attached), it appears that the problem is in ctors.c, which calls the python C API function PyErr_WarnEx. The strange thing is that I never see a warning appear in my python output, so this could also be a bug in python.
For now I will try to work around this problem, but it would be great if this could be fixed.
Details of installation:
Ubuntu 15.04 x86-64
Python: 3.4.3 (installed from ubuntu repo), but problem arises with python 2.7.9 too (also from ubuntu repo)
Numpy 1.10.1 (from pip), but problem also present in numpy 1.8.2 (from ubuntu repo)
Working example:
#!/usr/bin/env python
from __future__ import print_function
import ctypes
import sys
import numpy as np
print("python version:", sys.version)
print("numpy version:", np.version.full_version)
def printvminfo():
with open('/proc/self/status', 'r') as f:
print(''.join(line for line in f.readlines() if line.startswith('Vm')))
# create array to work with
N = 100
a = np.arange(N)
# get pointer to array
pnt = np.ctypeslib.as_ctypes(a)
printvminfo()
Nrun = 1000
if len(sys.argv)>=2:
Nrun = int(sys.argv[1])
print("Running", Nrun, "times")
for i in range(Nrun):
# create a raw pointer (this is how my real c function works)
newpnt = ctypes.cast(pnt, ctypes.POINTER(ctypes.c_long))
# and construct an array using this data
b = np.ctypeslib.as_array(newpnt, (N,))
# now delete both, which should cleanup both objects
del newpnt, b
# except it doesn't, RSS memory increases as a function of Nrun!
printvminfo()Output of script
python version: 3.4.3 (default, Mar 26 2015, 22:03:40)
[GCC 4.9.2]
numpy version: 1.10.1
VmPeak: 75088 kB
VmSize: 75088 kB
VmLck: 0 kB
VmPin: 0 kB
VmHWM: 20820 kB
VmRSS: 20820 kB
VmData: 13448 kB
VmStk: 136 kB
VmExe: 3408 kB
VmLib: 10920 kB
VmPTE: 168 kB
VmSwap: 0 kB
Running 10000 times
VmPeak: 86960 kB
VmSize: 86960 kB
VmLck: 0 kB
VmPin: 0 kB
VmHWM: 32636 kB
VmRSS: 32636 kB
VmData: 25320 kB
VmStk: 136 kB
VmExe: 3408 kB
VmLib: 10920 kB
VmPTE: 188 kB
VmSwap: 0 kB
Relevant valgrind output
<lots of output before>
==2956== 11,086,528 bytes in 9,952 blocks are definitely lost in loss record 720 of 720
==2956== at 0x4C2BBA0: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==2956== by 0x4C2DF4F: realloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==2956== by 0x4D5123: _PyMem_RawRealloc (obmalloc.c:68)
==2956== by 0x4D5123: PyMem_Realloc (obmalloc.c:308)
==2956== by 0x4D5123: data_stack_grow (_sre.c:237)
==2956== by 0x4D5123: sre_ucs1_match (sre_lib.h:506)
==2956== by 0x4D8442: sre_match (_sre.c:511)
==2956== by 0x4D8442: pattern_match (_sre.c:579)
==2956== by 0x594194: PyObject_Call (abstract.c:2040)
==2956== by 0x594194: call_function_tail.lto_priv.2483 (abstract.c:2078)
==2956== by 0x5946ED: callmethod (abstract.c:2147)
==2956== by 0x5946ED: _PyObject_CallMethodId (abstract.c:2192)
==2956== by 0x5074D6: check_matched.part.1.lto_priv.2234 (_warnings.c:29)
==2956== by 0x53369B: check_matched (_warnings.c:154)
==2956== by 0x53369B: get_filter (_warnings.c:149)
==2956== by 0x53369B: warn_explicit (_warnings.c:414)
==2956== by 0x533884: do_warn (_warnings.c:665)
==2956== by 0x53398C: warn_unicode (_warnings.c:790)
==2956== by 0x5E503E: PyErr_WarnEx (_warnings.c:829)
==2956== by 0x72D4599: _array_from_buffer_3118 (ctors.c:1277)
==2956==
==2956== LEAK SUMMARY:
==2956== definitely lost: 11,246,547 bytes in 10,035 blocks
==2956== indirectly lost: 0 bytes in 0 blocks
==2956== possibly lost: 208,484 bytes in 130 blocks
==2956== still reachable: 1,390,813 bytes in 2,937 blocks
==2956== suppressed: 0 bytes in 0 blocks
<some output after>