Race condition in Numpy distutils w/ Python distutils -j #15957
Description
There's a race condition in Numpy distutils when using the Python distutils parallelization (e.g. python setup.py build -j 3). It can make Fortran compilation to use wrong extra_f*_compiler_flags.
The race is here: (numpy/distutils/commands/build_ext.py:build_ext.build_extension)
numpy/numpy/distutils/command/build_ext.py
Line 356 in b489287
The fcompiler comes from self._f77/f90_compiler where self is the global build_ext object. So the compiler is a global object. Python distutils launches several calls to build_extension in parallel
so they race each other here, and so an extension build can end up with flags from wrong extension.
Reproducing code example:
#!/bin/bash
set -e
mkdir -p tmp
# Fortran file that requires -DMACRO flag given
cat <<EOF > tmp/foo0.f90
#ifndef MACRO
#error fail
#endif
EOF
# Fortran file that requires -DMACRO flag NOT given
cat <<EOF > tmp/foo1.f90
#ifdef MACRO
#error fail
#endif
EOF
# Add filler to increase compile time, to better trigger races
for k in `seq 1 100`; do
cat <<EOF >> tmp/foo0.f90
subroutine foo_$k()
end subroutine foo_$k
EOF
cat <<EOF >> tmp/foo1.f90
subroutine foo_$k()
end subroutine foo_$k
EOF
done
# 10 .pyf wrappers with no routines
for j in `seq 1 10`; do
cat <<EOF > tmp/foo$j.pyf
python module foo$j
interface
end interface
end python module
EOF
done
# Simple setup.py
cat <<EOF > tmp/setup.py
import setuptools
import time
from numpy.distutils.core import setup, Extension
extensions = [
Extension(name="foo{}".format(j),
sources=["foo{}.pyf".format(j), "foo{}.f90".format(j % 2)],
extra_f90_compile_args=["-cpp", "-DMACRO=''"] if j % 2 == 0 else ["-cpp"])
for j in range(1, 11)
]
setup(ext_modules=extensions)
EOF
# Check build status
echo "Serial build:"
rm -rf tmp/build
(cd tmp && python setup.py build) > tmp/build1.log 2>&1 || { cat tmp/build1.log; echo "FAIL"; exit 1; }
echo "OK"
echo "Parallel build:"
rm -rf tmp/build
(cd tmp && python setup.py build -j 10) > tmp/build2.log 2>&1 || { cat tmp/build2.log; echo "FAIL"; exit 1; }
echo "OK"
Numpy/Python version information:
1.18.2 3.8.2 (default, Feb 28 2020, 00:00:00)