Bug report

Bug description:

From @corona10:

import concurrent.futures
from concurrent.futures import InterpreterPoolExecutor, ThreadPoolExecutor


def fib(n):
    if n < 2:
        return n
    return fib(n-1) + fib(n-2)

with InterpreterPoolExecutor(max_workers=5) as executor:
    futures = {executor.submit(fib, n): n for n in range(10)}
    for future in concurrent.futures.as_completed(futures):
        n = futures[future]
        data = future.result()
        print(f"fib({n}): {data}")

This fails with:

Exception: AttributeError: module '__main__' has no attribute 'fib'

The above exception was the direct cause of the following exception:

concurrent.interpreters.NotShareableError: object could not be unpickled

The above exception was the direct cause of the following exception:
...

The failure goes away if the "executable" part of the script is wrapped in an if __name__ == '__main__': block.

This is because of how Interpreter.call() sends functions between interpreters. It mostly pickles them. On the other side, functions are looked up by name in the defining module. For the __main__ module, this is a problem.

The original script only runs in the main interpreter, so the function will not be found in the other interpreter's __main__ module. We must run the script in the other interpreter, but without polluting __main__ and avoiding executing anything but the desired function definition. We partially solve this by calling runpy.run_path() with a different module name than __main__.

multiprocessing faces a similar situation. See the "Safe importing of main module" entry in the https://docs.python.org/3/library/multiprocessing.html#the-spawn-and-forkserver-start-methods section.

The solution here is probably the same: document that users should apply if __name__ == '__main__':. We may also be able to accommodate this in the code somewhat, since we know when we are running the script in the other interpreter. However, there's a limit to how much we can do so without introducing nontrivial complexity.

CPython versions tested on:

CPython main branch

Operating systems tested on:

No response