Files in underscore directories do not show up as files of a py_library target

[RobertClarke64]

🐞 bug report

Affected Rule

I think this is likely a bug with the py_library rule, could potentially be something to do with a pip rule though.

Description

I'm trying to copy PyPI packages to a specific folder for putting on an OCI image. We're creating an Azure Function App and it's particular about how it runs the image, and so it's not finding the PyPI packages in the bazel runfiles tree (also it overrides the python path, so we couldn't just set that).

I've solved most of this by just copying the @third_party_python_deps//package:pkg and @third_party_python_deps//package:dist_info into the /usr/local/lib/python3.10/ folder on our image. However, we're getting errors because it cannot find files which are within a directory starting with an underscore. For us, it's cryptography/hazmat/bindings/_rust/*. I also noticed that those files do not show up in a bazel cquery --output=files @third_party_python_deps//cryptography:pkg.

🔬 Minimal Reproduction

Add cryptography as a third party PyPI dependency, then run bazel cquery --output=files @third_party_python_deps//cryptography:pkg. This should show most of the package's files, except the ones in external/third_party_python_deps_cryptography/site-packages/cryptography/hazmat/bindings/_rust/*.

🌍 Your Environment

Operating System:

  
macOS Sonoma 14.4.1
  

Output of bazel version:

  
Bazelisk version: 1.19.0
INFO: Invocation ID: 916bcf6b-c490-4217-9aa4-a9183b95af3b
Build label: 7.0.2
Build target: @@//src/main/java/com/google/devtools/build/lib/bazel:BazelServer
Build time: Thu Jan 25 16:14:15 2024 (1706199255)
Build timestamp: 1706199255
Build timestamp as int: 1706199255
  

Rules_python version:

  
0.29.0
  

[vonschultz]

It looks like the missing files are in the data attribute of the py_library that is the :pkg of cryptography. The included files are in the srcs attribute.

[RobertClarke64]

Thanks, that's helpful! I hadn't noticed that. When I run bazel cquery --output=files @third_party_python_deps//cryptography:data, it returns nothing. Is that expected?

[vonschultz]

I used bazel query --output=build to inspect the relevant targets.

The third_party_python_deps//cryptography:data target, introduced in #519, is for the data directory that might be included in the wheel file. It does not currently include any data that's in site-packages/**/* (which does get included in the data attribute of the :pkg).

Files are added to srcs if they match site-packages/**/*.py. Anything else in in site-packages/**/* (with some exceptions) goes in data.

Assuming I'm reading this right. I wasn't involved in writing that code. I'm looking at python/pip_install/private/generate_whl_library_build_bazel.bzl, which looks very relevant.

[RobertClarke64]

Amazing, thanks! So I've managed to get a list of all required files (I think) using

sources = src[DefaultInfo].default_runfiles.files.to_list()

in a rule. Going to work on copying these files to the right place now, then try switch them out for symlinks to the runfiles versions instead.

Looks like this is not a bug, going to close this issue.