-
-
Notifications
You must be signed in to change notification settings - Fork 32.4k
Open
Labels
stdlibPython modules in the Lib dirPython modules in the Lib dirtype-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error
Description
Bug report
Bug description:
Summary
ProcessPoolExecutor.map
may hang forever when the submitted iterable is far greater than max_tasks_per_child
Internally ProcessPoolExecutor.map
submits to the executor all items from the iterable (via submit
). The worker process exits when the first max_tasks_per_child
work items are processed but a new worker process isn't created because ProcessPoolExecutor._adjust_process_count
can be invoked only in submit
Reproducible example
import time
from concurrent.futures import ProcessPoolExecutor
def work(i):
time.sleep(1)
return i * i
def main():
with ProcessPoolExecutor(max_workers=1) as executor:
_ = list(executor.map(work, range(100)))
# completes
with ProcessPoolExecutor(max_workers=1, max_tasks_per_child=5) as executor:
_ = list(executor.map(work, range(100)))
# hangs forever
if __name__ == '__main__':
main()
Expected behavior
Program exits
Actual behavor
Program hangs forever
CPython versions tested on:
3.11
Operating systems tested on:
Windows
cwindolf, harshit-singhania2000 and pratyushmittal
Metadata
Metadata
Assignees
Labels
stdlibPython modules in the Lib dirPython modules in the Lib dirtype-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error