Skip to content

Hermetic toolchains incur a significant overhead to running py_binary and py_test #1653

@martis42

Description

@martis42

🐞 bug report

Affected Rule

py_binary and py_test

Is this a regression?

No

Description

Hermetic toolchains incur a significant overhead to executing Python targets due to the large amount of files which are added to the runfiles tree.
You find an analysis of the issue here: https://github.com/martis42/show_hermetic_python_overhead

Bazel caching of course negates much of this issue in daily working.
Still, whenever a test has to rerun we pay this additional overhead.

I don't know if there is a "perfect" solution possible given how Bazel sandboxing works.
However, it would be great if rues_python would add a note in its documentation for hermetic toolchains that adding --nolegacy_external_runfiles to the .bazelrc file is recommended.

Maybe the rules_python maintainer can even convince the Bazel maintainer to prioritize working on flipping --legacy_external_runfiles in Bazel 8 ?

🔬 Minimal Reproduction

You see reproduce this with this small example workspace https://github.com/martis42/show_hermetic_python_overhead

🔥 Exception or Error

NA

🌍 Your Environment

Operating System:

  
Linux Mint 21.1
  

Output of bazel version:

  
Bazelisk version: v1.16.0
Build label: 7.0.0
Build target: @@//src/main/java/com/google/devtools/build/lib/bazel:BazelServer
Build time: Mon Dec 11 16:51:49 2023 (1702313509)
Build timestamp: 1702313509
Build timestamp as int: 1702313509
  

Rules_python version:

  
0.27.0
  

Anything else relevant?

#1624 might be related.
It is however maybe Windows specific.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions