Doc/c-api/memory.rst: extend --without-pymalloc doc with ASan information #136790

disconnect3d · 2025-07-19T11:22:44Z

This commit extends the documentation for disabling pymalloc with the --without-pymalloc flag regarding why it is worth to use it when enabling AddressSanitizer for Python build (which is done, e.g., in CPython's CI builds).

I have tested the CPython latest main build with both ASan and pymalloc enabled and it seems to work just fine. I did run the python -m test suite which didn't uncover any ASan crashes (though, it detected some memory leaks, which I believe are irrelevant here).

I have discussed ASan and this flag with @encukou on the CPython Core sprint on EuroPython 2025. We initially thought that the --without-pymalloc flag is needed for ASan builds due to the fact pymalloc must hit the begining of page when determining if the memory to be freed comes from pymalloc or was allocated by the system malloc. In other words, we thought, that ASan would crash CPython during free of big objects (allocated by system malloc). It may be that this was the case in the past, but it is not the case anymore as the address_in_range function used by pymalloc is annotated to be skipped from the ASan instrumentation.

This code can be seen here:

cpython/Objects/obmalloc.c

Lines 2096 to 2110 in acefb97

    
           static bool _Py_NO_SANITIZE_ADDRESS 
        
                       _Py_NO_SANITIZE_THREAD 
        
                       _Py_NO_SANITIZE_MEMORY 
        
           address_in_range(OMState *state, void *p, poolp pool) 
        
           { 
        
               // Since address_in_range may be reading from memory which was not allocated 
        
               // by Python, it is important that pool->arenaindex is read only once, as 
        
               // another thread may be concurrently modifying the value without holding 
        
               // the GIL. The following dance forces the compiler to read pool->arenaindex 
        
               // only once. 
        
               uint arenaindex = *((volatile uint *)&pool->arenaindex); 
        
               return arenaindex < maxarenas && 
        
                   (uintptr_t)p - allarenas[arenaindex].address < ARENA_SIZE && 
        
                   allarenas[arenaindex].address != 0; 
        
           }

While the annotation macro is defined here:

cpython/Include/pyport.h

Lines 582 to 598 in acefb97

    
           #  if __has_feature(address_sanitizer) 
        
           #    if !defined(_Py_ADDRESS_SANITIZER) 
        
           #      define _Py_ADDRESS_SANITIZER 
        
           #      define _Py_NO_SANITIZE_ADDRESS __attribute__((no_sanitize_address)) 
        
           #    endif 
        
           #  endif 
        
           #  if __has_feature(thread_sanitizer) 
        
           #    if !defined(_Py_THREAD_SANITIZER) 
        
           #      define _Py_THREAD_SANITIZER 
        
           #      define _Py_NO_SANITIZE_THREAD __attribute__((no_sanitize_thread)) 
        
           #    endif 
        
           #  endif 
        
           #elif defined(__GNUC__) 
        
           #  if defined(__SANITIZE_ADDRESS__) 
        
           #    define _Py_ADDRESS_SANITIZER 
        
           #    define _Py_NO_SANITIZE_ADDRESS __attribute__((no_sanitize_address)) 
        
           #  endif

And the corresponding attribute is documented in:

📚 Documentation preview 📚: https://cpython-previews--136790.org.readthedocs.build/

@encukou

…tion This commit extends the documentation for disabling pymalloc with the `--without-pymalloc` flag regarding why it is worth to use it when enabling AddressSanitizer for Python build (which is done, e.g., in CPython's CI builds). I have tested the CPython latest main build with both ASan and pymalloc enabled and it seems to work just fine. I did run the `python -m test` suite which didn't uncover any ASan crashes (though, it detected some memory leaks, which I believe are irrelevant here). I have discussed ASan and this flag with @encukou on the CPython Core sprint on EuroPython 2025. We initially thought that the `--without-pymalloc` flag is needed for ASan builds due to the fact pymalloc must hit the begining of page when determining if the memory to be freed comes from pymalloc or was allocated by the system malloc. In other words, we thought, that ASan would crash CPython during free of big objects (allocated by system malloc). It may be that this was the case in the past, but it is not the case anymore as the `address_in_range` function used by pymalloc is annotated to be skipped from the ASan instrumentation. This code can be seen here: https://github.com/python/cpython/blob/acefb978dcb5dd554e3c49a3015ee5c2ad6bfda1/Objects/obmalloc.c#L2096-L2110 While the annotation macro is defined here: https://github.com/python/cpython/blob/acefb978dcb5dd554e3c49a3015ee5c2ad6bfda1/Include/pyport.h#L582-L598 And the corresponding attribute is documented in: * for gcc: https://gcc.gnu.org/onlinedocs/gcc/Common-Function-Attributes.html#index-no_005fsanitize_005faddress-function-attribute * for clang: https://clang.llvm.org/docs/AttributeReference.html#no-sanitize-address-no-address-safety-analysis

Doc/c-api/memory.rst

encukou · 2025-07-19T11:52:42Z

Thank you the PR, and for taking the time to make it this short!

miss-islington-app · 2025-07-19T11:53:05Z

Thanks @disconnect3d for the PR, and @encukou for merging it 🌮🎉.. I'm working now to backport this PR to: 3.14.
🐍🍒⛏🤖

miss-islington-app · 2025-07-19T11:53:05Z

Thanks @disconnect3d for the PR, and @encukou for merging it 🌮🎉.. I'm working now to backport this PR to: 3.13.
🐍🍒⛏🤖

@encukou

…tion (pythonGH-136790) * Doc/c-api/memory.rst: extend --without-pymalloc doc with ASan information This commit extends the documentation for disabling pymalloc with the `--without-pymalloc` flag regarding why it is worth to use it when enabling AddressSanitizer for Python build (which is done, e.g., in CPython's CI builds). I have tested the CPython latest main build with both ASan and pymalloc enabled and it seems to work just fine. I did run the `python -m test` suite which didn't uncover any ASan crashes (though, it detected some memory leaks, which I believe are irrelevant here). I have discussed ASan and this flag with @encukou on the CPython Core sprint on EuroPython 2025. We initially thought that the `--without-pymalloc` flag is needed for ASan builds due to the fact pymalloc must hit the begining of page when determining if the memory to be freed comes from pymalloc or was allocated by the system malloc. In other words, we thought, that ASan would crash CPython during free of big objects (allocated by system malloc). It may be that this was the case in the past, but it is not the case anymore as the `address_in_range` function used by pymalloc is annotated to be skipped from the ASan instrumentation. This code can be seen here: https://github.com/python/cpython/blob/acefb978dcb5dd554e3c49a3015ee5c2ad6bfda1/Objects/obmalloc.c#L2096-L2110 While the annotation macro is defined here: https://github.com/python/cpython/blob/acefb978dcb5dd554e3c49a3015ee5c2ad6bfda1/Include/pyport.h#L582-L598 And the corresponding attribute is documented in: * for gcc: https://gcc.gnu.org/onlinedocs/gcc/Common-Function-Attributes.html#index-no_005fsanitize_005faddress-function-attribute * for clang: https://clang.llvm.org/docs/AttributeReference.html#no-sanitize-address-no-address-safety-analysis * Update Doc/c-api/memory.rst * Improve --with-address-sanitizer and pymalloc docs --------- (cherry picked from commit d19bb44) Co-authored-by: Disconnect3d <dominik.b.czarnota@gmail.com> Co-authored-by: Petr Viktorin <encukou@gmail.com>

@encukou

…tion (pythonGH-136790) * Doc/c-api/memory.rst: extend --without-pymalloc doc with ASan information This commit extends the documentation for disabling pymalloc with the `--without-pymalloc` flag regarding why it is worth to use it when enabling AddressSanitizer for Python build (which is done, e.g., in CPython's CI builds). I have tested the CPython latest main build with both ASan and pymalloc enabled and it seems to work just fine. I did run the `python -m test` suite which didn't uncover any ASan crashes (though, it detected some memory leaks, which I believe are irrelevant here). I have discussed ASan and this flag with @encukou on the CPython Core sprint on EuroPython 2025. We initially thought that the `--without-pymalloc` flag is needed for ASan builds due to the fact pymalloc must hit the begining of page when determining if the memory to be freed comes from pymalloc or was allocated by the system malloc. In other words, we thought, that ASan would crash CPython during free of big objects (allocated by system malloc). It may be that this was the case in the past, but it is not the case anymore as the `address_in_range` function used by pymalloc is annotated to be skipped from the ASan instrumentation. This code can be seen here: https://github.com/python/cpython/blob/acefb978dcb5dd554e3c49a3015ee5c2ad6bfda1/Objects/obmalloc.c#L2096-L2110 While the annotation macro is defined here: https://github.com/python/cpython/blob/acefb978dcb5dd554e3c49a3015ee5c2ad6bfda1/Include/pyport.h#L582-L598 And the corresponding attribute is documented in: * for gcc: https://gcc.gnu.org/onlinedocs/gcc/Common-Function-Attributes.html#index-no_005fsanitize_005faddress-function-attribute * for clang: https://clang.llvm.org/docs/AttributeReference.html#no-sanitize-address-no-address-safety-analysis * Update Doc/c-api/memory.rst * Improve --with-address-sanitizer and pymalloc docs --------- (cherry picked from commit d19bb44) Co-authored-by: Disconnect3d <dominik.b.czarnota@gmail.com> Co-authored-by: Petr Viktorin <encukou@gmail.com>

bedevere-app · 2025-07-19T11:53:15Z

GH-136798 is a backport of this pull request to the 3.14 branch.

bedevere-app · 2025-07-19T11:53:20Z

GH-136799 is a backport of this pull request to the 3.13 branch.

@encukou

…information (GH-136790) (GH-136798) Extend the documentation for disabling pymalloc with the `--without-pymalloc` flag regarding why it is worth to use it when enabling AddressSanitizer for Python build (which is done, e.g., in CPython's CI builds). I have tested the CPython latest main build with both ASan and pymalloc enabled and it seems to work just fine. I did run the `python -m test` suite which didn't uncover any ASan crashes (though, it detected some memory leaks, which I believe are irrelevant here). I have discussed ASan and this flag with @encukou on the CPython Core sprint on EuroPython 2025. We initially thought that the `--without-pymalloc` flag is needed for ASan builds due to the fact pymalloc must hit the begining of page when determining if the memory to be freed comes from pymalloc or was allocated by the system malloc. In other words, we thought, that ASan would crash CPython during free of big objects (allocated by system malloc). It may be that this was the case in the past, but it is not the case anymore as the `address_in_range` function used by pymalloc is annotated to be skipped from the ASan instrumentation. (cherry picked from commit d19bb44) Co-authored-by: Disconnect3d <dominik.b.czarnota@gmail.com> Co-authored-by: Petr Viktorin <encukou@gmail.com>

…information (GH-136790) (#136799) Extends the documentation for disabling pymalloc with the `--without-pymalloc` flag regarding why it is worth to use it when enabling AddressSanitizer for Python build (which is done, e.g., in CPython's CI builds). (cherry picked from commit d19bb44) Co-authored-by: Disconnect3d <dominik.b.czarnota@gmail.com> Co-authored-by: Petr Viktorin <encukou@gmail.com>

bedevere-app bot added awaiting review docs Documentation in the Doc dir skip news labels Jul 19, 2025

github-project-automation bot added this to Docs PRs Jul 19, 2025

github-project-automation bot moved this to Todo in Docs PRs Jul 19, 2025

encukou reviewed Jul 19, 2025

View reviewed changes

Doc/c-api/memory.rst Outdated Show resolved Hide resolved

encukou added the skip issue label Jul 19, 2025

encukou and others added 2 commits July 19, 2025 13:27

Update Doc/c-api/memory.rst

7659c24

Improve --with-address-sanitizer and pymalloc docs

d3483fc

disconnect3d requested review from erlend-aasland and corona10 as code owners July 19, 2025 11:45

encukou approved these changes Jul 19, 2025

View reviewed changes

bedevere-app bot added awaiting merge and removed awaiting review labels Jul 19, 2025

encukou merged commit d19bb44 into python:main Jul 19, 2025
30 checks passed

github-project-automation bot moved this from Todo to Done in Docs PRs Jul 19, 2025

bedevere-app bot removed the awaiting merge label Jul 19, 2025

encukou added needs backport to 3.13 bugs and security fixes needs backport to 3.14 bugs and security fixes labels Jul 19, 2025

bedevere-app bot removed the needs backport to 3.14 bugs and security fixes label Jul 19, 2025

bedevere-app bot removed the needs backport to 3.13 bugs and security fixes label Jul 19, 2025

AA-Turner added the sprint label Jul 19, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Doc/c-api/memory.rst: extend --without-pymalloc doc with ASan information #136790

Doc/c-api/memory.rst: extend --without-pymalloc doc with ASan information #136790

disconnect3d commented Jul 19, 2025 •

edited by github-actions bot

Loading

Uh oh!

Uh oh!

encukou commented Jul 19, 2025

Uh oh!

Uh oh!

miss-islington-app bot commented Jul 19, 2025

Uh oh!

miss-islington-app bot commented Jul 19, 2025

Uh oh!

bedevere-app bot commented Jul 19, 2025

Uh oh!

bedevere-app bot commented Jul 19, 2025

Uh oh!

Uh oh!

	static bool _Py_NO_SANITIZE_ADDRESS
	_Py_NO_SANITIZE_THREAD
	_Py_NO_SANITIZE_MEMORY
	address_in_range(OMState state, void p, poolp pool)
	{
	// Since address_in_range may be reading from memory which was not allocated
	// by Python, it is important that pool->arenaindex is read only once, as
	// another thread may be concurrently modifying the value without holding
	// the GIL. The following dance forces the compiler to read pool->arenaindex
	// only once.
	uint arenaindex = ((volatile uint )&pool->arenaindex);
	return arenaindex < maxarenas &&
	(uintptr_t)p - allarenas[arenaindex].address < ARENA_SIZE &&
	allarenas[arenaindex].address != 0;
	}

	# if __has_feature(address_sanitizer)
	# if !defined(_Py_ADDRESS_SANITIZER)
	# define _Py_ADDRESS_SANITIZER
	# define _Py_NO_SANITIZE_ADDRESS __attribute__((no_sanitize_address))
	# endif
	# endif
	# if __has_feature(thread_sanitizer)
	# if !defined(_Py_THREAD_SANITIZER)
	# define _Py_THREAD_SANITIZER
	# define _Py_NO_SANITIZE_THREAD __attribute__((no_sanitize_thread))
	# endif
	# endif
	#elif defined(__GNUC__)
	# if defined(__SANITIZE_ADDRESS__)
	# define _Py_ADDRESS_SANITIZER
	# define _Py_NO_SANITIZE_ADDRESS __attribute__((no_sanitize_address))
	# endif

Uh oh!

Doc/c-api/memory.rst: extend --without-pymalloc doc with ASan information #136790

Doc/c-api/memory.rst: extend --without-pymalloc doc with ASan information #136790

Conversation

disconnect3d commented Jul 19, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

encukou commented Jul 19, 2025

Uh oh!

Uh oh!

miss-islington-app bot commented Jul 19, 2025

Uh oh!

miss-islington-app bot commented Jul 19, 2025

Uh oh!

bedevere-app bot commented Jul 19, 2025

Uh oh!

bedevere-app bot commented Jul 19, 2025

Uh oh!

Uh oh!

disconnect3d commented Jul 19, 2025 •

edited by github-actions bot

Loading