rules_python not able to build offline from vendor only directory

[amfern]

🐞 bug report

Description

rules_python isn't able to build things completely offline, and still downloads python dependencies even through everything supposed to be available in the --vendor_dir to support complete offline build.

🔬 Minimal Reproduction

git clone git@github.com:bazel-contrib/rules_python.git

# prepare vendor dir
docker run -e USER="$(id -u)" -u="$(id -u)" -v $HOME/.netrc:/.netrc -v $(pwd):/src/workspace -w /src/workspace/examples/bzlmod gcr.io/bazel-public/bazel:8.4.2 --output_user_root=/tmp/build_output vendor --vendor_dir=my_vendor_dir //...

# run build offline using only the data from my_vendor_dir
docker run --network none -e USER="$(id -u)" -u="$(id -u)" -v $HOME/.netrc:/.netrc -v $(pwd):/src/workspace -w /src/workspace/examples/bzlmod gcr.io/bazel-public/bazel:8.4.2 --output_user_root=/tmp/build_output build --lockfile_mode=error --vendor_dir=my_vendor_dir //...

🔥 Exception or Error

Extracting Bazel installation...
Starting local Bazel server and connecting to it...
Computing main repo mapping:
WARNING: For repository 'bazel_skylib', the root module requires module version bazel_skylib@1.7.1, but got bazel_skylib@1.8.2 in the resolved dependency graph.
WARNING: For repository 'platforms', the root module requires module version platforms@0.0.4, but got platforms@1.0.0 in the resolved dependency graph.
WARNING: For repository 'rules_shell', the root module requires module version rules_shell@0.3.0, but got rules_shell@0.6.1 in the resolved dependency graph.
Loading:
Loading: 0 packages loaded
WARNING: Download from https://pypi.org/simple/ failed: class com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException Unknown host: pypi.org
ERROR: /tmp/build_output_offline/6a4437257408fba6059e0f41368f396b/external/rules_python~/python/private/pypi/simpleapi_download.bzl:258:39: Traceback (most recent call last):
        File "/tmp/build_output_offline/6a4437257408fba6059e0f41368f396b/external/rules_python~/python/private/pypi/extension.bzl", line 378, column 25, in _pip_impl
                mods = parse_modules(
        File "/tmp/build_output_offline/6a4437257408fba6059e0f41368f396b/external/rules_python~/python/private/pypi/extension.bzl", line 267, column 30, in parse_modules
                builder.pip_parse(
        File "/tmp/build_output_offline/6a4437257408fba6059e0f41368f396b/external/rules_python~/python/private/pypi/hub_builder.bzl", line 60, column 47, in lambda
                pip_parse = lambda *a, **k: _pip_parse(self, *a, **k),
        File "/tmp/build_output_offline/6a4437257408fba6059e0f41368f396b/external/rules_python~/python/private/pypi/hub_builder.bzl", line 193, column 22, in _pip_parse
                _create_whl_repos(
        File "/tmp/build_output_offline/6a4437257408fba6059e0f41368f396b/external/rules_python~/python/private/pypi/hub_builder.bzl", line 542, column 50, in _create_whl_repos
                requirements_by_platform = parse_requirements(
        File "/tmp/build_output_offline/6a4437257408fba6059e0f41368f396b/external/rules_python~/python/private/pypi/parse_requirements.bzl", line 196, column 36, in parse_requirements
                index_urls = get_index_urls(
        File "/tmp/build_output_offline/6a4437257408fba6059e0f41368f396b/external/rules_python~/python/private/pypi/hub_builder.bzl", line 381, column 144, in lambda
                self._get_index_urls[python_version] = lambda ctx, distributions, *, index_url = None, extra_index_urls = None: self._simpleapi_download_fn(
        File "/tmp/build_output_offline/6a4437257408fba6059e0f41368f396b/external/rules_python~/python/private/pypi/simpleapi_download.bzl", line 92, column 31, in simpleapi_download
                dist_urls = _get_dist_urls(
        File "/tmp/build_output_offline/6a4437257408fba6059e0f41368f396b/external/rules_python~/python/private/pypi/simpleapi_download.bzl", line 164, column 43, in _get_dist_urls
                results[index_url] = download.wait()
        File "/tmp/build_output_offline/6a4437257408fba6059e0f41368f396b/external/rules_python~/python/private/pypi/simpleapi_download.bzl", line 258, column 39, in lambda
                result = download.wait(),
Error in wait: java.io.IOException: Error downloading [https://pypi.org/simple/] to /tmp/build_output_offline/6a4437257408fba6059e0f41368f396b/modextwd/rules_python~~pip/https___pypi_org_simple.html: Unknown host: pypi.org
ERROR: error evaluating module extension pip in @@rules_python~//python/extensions:pip.bzl
INFO: Elapsed time: 7.499s
INFO: 0 processes.
ERROR: Build did NOT complete successfully
FAILED:

🌍 Your Environment

**Operating System: **

  
ubuntu 24.04
  

Output of bazel version:

  
bazel 8.4.2
  

Rules_python version:

  
https://github.com/bazel-contrib/rules_python/commit/c33ba2383b7ce23f92fd97fbaa58132341dec2ca
  

Anything else relevant?

There is a test for building from vendor, however it allows network access which makes it ineffective.

rules_python/.bazelci/presubmit.yml

Lines 360 to 362 in c33ba23

	- "bazel vendor --vendor_dir=./vendor //..."
	- "bazel build --vendor_dir=./vendor //..."
	- "rm -rf ./vendor"

[aignas]

Could you please check if #3719 fixes your issue? You will have to use the latest 8.5 release though which contains the facts support.

[aignas]

Since this should start working with at least 8.5 and 8.6, I'll mark it as a duplicate of another issue. PRs to support previous bazels are welcome.

Unable to find image 'gcr.io/bazel-public/bazel:8.5.1' locally
8.5.1: Pulling from bazel-public/bazel
7e49dc6156b0: Already exists
cfdee132a9c3: Pull complete
1a08e8a37b4e: Pull complete
4f4fb700ef54: Pull complete
cefd52a73966: Pull complete
Digest: sha256:c8fd8a0a0c21be95d516dc6a82d844ed1e38b66cc9cd27085ce6006acd55a011
Status: Downloaded newer image for gcr.io/bazel-public/bazel:8.5.1
Extracting Bazel installation...
Starting local Bazel server (8.5.1) and connecting to it...
Computing main repo mapping:
Computing main repo mapping:
Computing main repo mapping:
Computing main repo mapping:
Computing main repo mapping:
Computing main repo mapping:
WARNING: For repository 'bazel_skylib', the root module requires module version bazel_skylib@1.7.1, but got bazel_skylib@1.8.2 in the resolved dependency graph. Please update the version in your MODULE.bazel or set --check_direct_dependencies=off
WARNING: For repository 'platforms', the root module requires module version platforms@0.0.4, but got platforms@1.0.0 in the resolved dependency graph. Please update the version in your MODULE.bazel or set --check_direct_dependencies=off
WARNING: For repository 'rules_shell', the root module requires module version rules_shell@0.3.0, but got rules_shell@0.6.1 in the resolved dependency graph. Please update the version in your MODULE.bazel or set --check_direct_dependencies=off
Computing main repo mapping:
Loading:
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 7 packages loaded
    currently loading: libs/my_lib ... (2 packages)
Loading: 7 packages loaded
    currently loading: libs/my_lib ... (2 packages)
Loading: 7 packages loaded
    currently loading: libs/my_lib ... (2 packages)
Loading: 7 packages loaded
    currently loading: libs/my_lib ... (2 packages)
Loading: 7 packages loaded
    currently loading: libs/my_lib ... (2 packages)
Loading: 7 packages loaded
    currently loading: libs/my_lib ... (2 packages)
Analyzing: 46 targets (9 packages loaded, 0 targets configured)
Analyzing: 46 targets (9 packages loaded, 0 targets configured)
Analyzing: 46 targets (93 packages loaded, 70 targets configured)
Analyzing: 46 targets (114 packages loaded, 82 targets configured)
Analyzing: 46 targets (117 packages loaded, 89 targets configured)
Analyzing: 46 targets (133 packages loaded, 853 targets configured)
Analyzing: 46 targets (149 packages loaded, 3136 targets configured)
Analyzing: 46 targets (172 packages loaded, 3705 targets configured)
Analyzing: 46 targets (200 packages loaded, 8142 targets configured)
Analyzing: 46 targets (205 packages loaded, 11075 targets configured)
Analyzing: 46 targets (213 packages loaded, 12264 targets configured)
Analyzing: 46 targets (213 packages loaded, 12358 targets configured)
Analyzing: 46 targets (213 packages loaded, 12358 targets configured)
Analyzing: 46 targets (213 packages loaded, 12358 targets configured)
Analyzing: 46 targets (213 packages loaded, 12358 targets configured)
Analyzing: 46 targets (221 packages loaded, 16709 targets configured)
Analyzing: 46 targets (225 packages loaded, 19211 targets configured)
INFO: Analyzed 46 targets (226 packages loaded, 21484 targets configured).
INFO: Found 46 targets...
INFO: Elapsed time: 68.715s, Critical Path: 0.00s
INFO: 0 processes.
INFO: Build completed successfully, 0 total actions
INFO: Vendoring dependencies for targets...
INFO: All external dependencies for the requested targets vendored successfully.
aignas@panda ~/src/github/aignas/rules_python aignas.fix.index-contents-partial-gets
$ docker run --network none -e USER="$(id -u)" -u="$(id -u)" -v $HOME/.netrc:/.netrc -v $(pwd):/src/workspace -w /src/workspace/examples/bzlmod gcr.io/bazel-public/bazel:8.5.1 --output_user_root=/tmp/build_output build --lockfile_mode=error --vendor_dir=my_vendor_dir //...
Extracting Bazel installation...
Starting local Bazel server (8.5.1) and connecting to it...
Computing main repo mapping:
WARNING: For repository 'bazel_skylib', the root module requires module version bazel_skylib@1.7.1, but got bazel_skylib@1.8.2 in the resolved dependency graph. Please update the version in your MODULE.bazel or set --check_direct_dependencies=off
WARNING: For repository 'platforms', the root module requires module version platforms@0.0.4, but got platforms@1.0.0 in the resolved dependency graph. Please update the version in your MODULE.bazel or set --check_direct_dependencies=off
WARNING: For repository 'rules_shell', the root module requires module version rules_shell@0.3.0, but got rules_shell@0.6.1 in the resolved dependency graph. Please update the version in your MODULE.bazel or set --check_direct_dependencies=off
Computing main repo mapping:
Loading:
Loading: 0 packages loaded
Loading: 7 packages loaded
    currently loading: libs/my_lib ... (2 packages)
Loading: 7 packages loaded
    currently loading: libs/my_lib ... (2 packages)
Analyzing: 46 targets (9 packages loaded, 0 targets configured)
Analyzing: 46 targets (9 packages loaded, 0 targets configured)

Analyzing: 46 targets (90 packages loaded, 108 targets configured)
[1 / 1] no actions running
Analyzing: 46 targets (156 packages loaded, 342 targets configured)
[1 / 1] no actions running
Analyzing: 46 targets (176 packages loaded, 3015 targets configured)
[1 / 1] no actions running
Analyzing: 46 targets (206 packages loaded, 7851 targets configured)
[1 / 1] no actions running
Analyzing: 46 targets (210 packages loaded, 16746 targets configured)
[53 / 53] no actions running
Analyzing: 46 targets (225 packages loaded, 19185 targets configured)
[53 / 53] no actions running
Analyzing: 46 targets (226 packages loaded, 20684 targets configured)
[120 / 162] [Prepa] Creating source manifest for //tests:version_3_10_test ... (4 actions, 3 running)
INFO: Analyzed 46 targets (226 packages loaded, 21484 targets configured).
[148 / 196] Creating runfiles tree bazel-out/k8-fastbuild-ST-0d3860f234bd/bin/tests/version_versioned_test.runfiles; 0s local ... (7 actions running)
[165 / 206] Creating runfiles tree bazel-out/k8-fastbuild-ST-0d3860f234bd/bin/tests/version_3_10_test.runfiles; 1s local ... (7 actions running)
[210 / 315] Creating runfiles tree bazel-out/k8-fastbuild/bin/tests/version_default_takes_3_10_subprocess_test.runfiles; 2s local ... (7 actions running)
[241 / 337] Creating runfiles tree bazel-out/k8-fastbuild-ST-0d3860f234bd/bin/requirements_3_10.update.runfiles; 2s local ... (8 actions, 6 running)
[270 / 369] Creating runfiles tree bazel-out/k8-fastbuild-ST-9303e9d96368/bin/test_with_transition.runfiles; 2s local ... (7 actions, 6 running)
[300 / 411] Creating runfiles tree bazel-out/k8-fastbuild/bin/test.runfiles; 2s local ... (7 actions, 6 running)
[375 / 466] Creating runfiles tree bazel-out/k8-fastbuild/bin/test.runfiles; 3s local ... (7 actions, 6 running)
[422 / 518] Creating runfiles tree bazel-out/k8-fastbuild-ST-0d3860f234bd/bin/tests/version_3_10_takes_3_9_subprocess_test_2.runfiles; 1s local ... (8 actions, 7 running)
[488 / 542] Creating runfiles tree bazel-out/k8-fastbuild/bin/bzlmod.runfiles; 2s local ... (7 actions, 6 running)
[536 / 542] Generating zipapp __main__.py: @our_other_module//other_module/pkg:bin_zipapp; 1s processwrapper-sandbox ... (3 actions running)
[539 / 542] RunBinary entry_points/tests/pylint_report.txt; 0s processwrapper-sandbox ... (2 actions, 1 running)
[539 / 542] RunBinary entry_points/tests/pylint_report.txt; 1s processwrapper-sandbox ... (2 actions running)
[541 / 542] Reticulating zipapp executable: @our_other_module//other_module/pkg:bin_zipapp into external/other_module+/other_module/pkg/bin_zipapp.pyz; 1s processwrapper-sandbox
INFO: Found 46 targets...
INFO: Elapsed time: 34.921s, Critical Path: 5.37s
INFO: 542 processes: 353 internal, 2 local, 187 processwrapper-sandbox.
INFO: Build completed successfully, 542 total actions

[amfern]

@aignas Thank you very much for your work.

I tried your branch with bazel 8.5.1 and it works, i am able to build //:bzlmod target completely offline, thank you!