`rules_python_publish_deps` causing pip extension resolution to always require network access

[meteorcloudy]

🐞 bug report

Affected Rule

rules_python/MODULE.bazel

Lines 63 to 75 in b40d96a

	pip.parse(
	# NOTE @aignas 2024-10-26: We have an integration test that depends on us
	# being able to build sdists for this hub, so explicitly set this to False.
	download_only = False,
	experimental_index_url = "https://pypi.org/simple",
	hub_name = "rules_python_publish_deps",
	python_version = "3.11",
	requirements_by_platform = {
	"//tools/publish:requirements_darwin.txt": "osx_*",
	"//tools/publish:requirements_linux.txt": "linux_*",
	"//tools/publish:requirements_windows.txt": "windows_*",
	},
	)

Is this a regression?

Yes, after b9b0948, moving from pip_internal to pip.

Description

The rules_python_publish_deps python hub uses the same module extension (pip) end users use, which causes the module extension resolution to always require network access. In offline build without an up-to-date lockfile, the module extension resolution will fail even though rules_python_publish_deps is never required by the end user.

This was discovered in bazelbuild/bazel#26163, where upgrading rules_python broke Bazel's bootstrap build.

🔬 Minimal Reproduction

🔥 Exception or Error

�[1A�[K�[31m�[1mERROR: �[0m/tmp/bazel_8RGzY3p4/out/external/rules_python+/python/private/pypi/simpleapi_download.bzl:130:14: Traceback (most recent call last):
	File "/tmp/bazel_8RGzY3p4/out/external/rules_python+/python/private/pypi/extension.bzl", line 615, column 25, in _pip_impl
		mods = parse_modules(module_ctx)
	File "/tmp/bazel_8RGzY3p4/out/external/rules_python+/python/private/pypi/extension.bzl", line 490, column 36, in parse_modules
		out = _create_whl_repos(
	File "/tmp/bazel_8RGzY3p4/out/external/rules_python+/python/private/pypi/extension.bzl", line 155, column 50, in _create_whl_repos
		requirements_by_platform = parse_requirements(
	File "/tmp/bazel_8RGzY3p4/out/external/rules_python+/python/private/pypi/parse_requirements.bzl", line 180, column 36, in parse_requirements
		index_urls = get_index_urls(
	File "/tmp/bazel_8RGzY3p4/out/external/rules_python+/python/private/pypi/extension.bzl", line 470, column 79, in lambda
		get_index_urls = lambda ctx, distributions: simpleapi_download(
	File "/tmp/bazel_8RGzY3p4/out/external/rules_python+/python/private/pypi/simpleapi_download.bzl", line 130, column 14, in simpleapi_download
		_fail("Failed to download metadata for {} for from urls: {}".format(
Error in fail: Failed to download metadata for ["nh3", "idna", "rich", "zipp", "mdurl", "twine", "certifi", "keyring", "pkginfo", "rfc3986", "urllib3", "docutils", "pygments", "requests", "jaraco-classes", "jaraco-context", "markdown-it-py", "more-itertools", "readme-renderer", "jaraco-functools", "backports-tarfile", "requests-toolbelt", "charset-normalizer", "importlib-metadata", "cffi", "jeepney", "pycparser", "cryptography", "secretstorage", "pywin32-ctypes"] for from urls: ["https://pypi.org/simple"]

🌍 Your Environment

Operating System:

  
macOS, Linux, Windows
  

Output of bazel version:

  
8.2.1
  

Rules_python version:

  
1.3.1
  

Anything else relevant?

This might be working as intended, but should rules_python_publish_deps be dev_dependency?

[aignas]

This is unfortunately working as intended since the publishing dep is something that the user can consume and it is not a dev_dependency. We could use the isolation of the extension, but I think that would require all of the consumers of rules_python to modify their .bazelrc.

Other solutions for this would be:

• Use a PEP751 compliant lock file Support PEP751 lockfile format in bzlmod #2787. This is not yet supported by pip.parse but this would fix the issue.
• Remove the usage of experimental_index_url in that dependency - it's fine for now - this should remove the network dependency.

However, in general I don't see how users could get around this unless extension isolation or have an alias extension for internal Python usage. Any thoughts on this topic?

[meteorcloudy]

Thanks for the quick response! Yes, that's also my understanding. This isn't a big issue for Bazel, we've workaround this for our bootstrap build (we don't really need the pip dependencies). And I agree using extension isolation is probably the solution, but we have to make that feature stable first. @Wyverald

[meteorcloudy]

Also using the lock file can partially mitigate the issue since it skips the resolution.

[aignas]

In 1.4 the extension has been marked as reproducible, so the lock file will not help. See #2434. So for now just dropping the experimental_index_url usage is good enough for now.

PRs dropping that would be welcome.

[rickeylev]

It is unfortunate that this extra network activity is necessary, even if twine isn't used.

Couple other ideas

Is it possible to have a file-based simple api URL?

Ignore the network failure, just generate some stubs?

It does seem problematic that, if a bzlmod extension does any network activity, it can't work in offline mode

[meteorcloudy]

It does seem problematic that, if a bzlmod extension does any network activity, it can't work in offline mode

One major use case of module extension is to call package manger to resolve dependencies, requiring network for that does make sense to me. The only fix is using the lock file to bypass the resolution, which means for reproducible module extension that requires network access, checking in the lockfile still might be useful. We are generating lockfile for reproducible extensions in bazelbuild/bazel#24757, but not under a place that can be checked in. @fmeum

[fmeum]

Would bazelbuild/bazel#24777 here? It would definitely help rules_go in a similar case (getting SDK URLs and hashes from just a version identifier).

[meteorcloudy]

I think it would help with incremental build but if you just clone a repo containing all vendored repos, you can still not build offline due to the module extension resolution?

[fmeum]

With an up-to-date lockfile checked in you would be since the reproducible data fetched over the network would be persisted in the facts, which are stored in the regular (not the hidden) lockfile.

[aignas]

Does it mean that rules_python lock file would affect the root module?

[fmeum]

Does it mean that rules_python lock file would affect the root module?

It would result in information being added to the MODULE.bazel.lock file of any module transitively using rules_python. Hopefully that information could be very concise though and, at some point, we might want to suggest users update their lockfiles with a command that doesn't blindly run all extensions, even the ones they never use in their build.

[aignas]

This is unrelated to the ticket, but in the past it has been problematic to have MODULE.bazel.lock and test against multiple bazel versions using --lockmode=error (because bazel version affects the lock file), I hope whatever clever solution we come up with here does not have the same issue.