[Python] Python does not find the DLLs correctly on Windows

[asfimport]

It seems that after the Python refactoring PR #13311 (ARROW-16340) Python is unable to find arrow_python even though the library it is imported into the correct directory.

Currently this issue is fixed with setting an additional environment variable:

CONDA_DLL_SEARCH_MODIFICATION_ENABLE=1

We need to investigate further why this error is happening after the refactoring and make sure Python is able to find the libraries on Windows without the additional env vars being specified.

Reporter: Alenka Frim / @AlenkaF
Assignee: Alenka Frim / @AlenkaF

Related issues:

• [C++][Python] test_cython_api fails on windows (is related to)

PRs and other links:

• GitHub Pull Request #14302

_{Note: This issue was originally created as ARROW-17771. Please see the migration documentation for further details.}

[asfimport]

Antoine Pitrou / @pitrou:
cc [~h-vetinari] for potential guidance.

[asfimport]

H. Vetinari:

Currently this issue is fixed with setting an additional environment variable CONDA_DLL_SEARCH_MODIFICATION_ENABLE=1

I've only ever seen this be necessary for prioritizing conda-DLLs over stuff in C:/System32 (e.g. openssl), but AFAIU the efforts of Isuru and other people in core is to make this variable obsolete (by having things work correctly by default). I think newer CPython versions by conda-forge might not even have it anymore. Which version of Python is this failing with, and what libraries are missing?

[asfimport]

Alenka Frim / @AlenkaF:
Thank you for helping with this issue [~h-vetinari]!

It is failing with Python 3.8 and 3.9. I haven't tried with version 3.10 locally but can do so later today.

Could the issue be that Arrow DLLs and Arrow Python (PyArrow C++) DLL are in a different location?

• Arrow libraries are installed into ARROW_HOME location
• arrow_python is installed into arrow/python/pyarrow location which is different from ARROW_HOME and this lib is the one that is missing/can't be found

[asfimport]

H. Vetinari:

Could the issue be that Arrow DLLs and Arrow Python (PyArrow C++) DLL are in a different location?

Yes, it sounds like the new location of the Arrow Python DLL is not taken into account (curious that it's still found with the ENABLE-stuff).

In any case, from the POV of conda-forge (i.e. when we get around to packaging the next released version), this would probably get installed somewhere in site-packages/arrow/.... I think it might be best to open a PR to the arrow-cpp-feedstock that points to the current master here (use git_url: etc., I can help if necessary), and we can figure out what needs to be done (either on the build script side or through patches) so that this runs through.

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
Based on some of the other issues where I found some discussion about this (ContinuumIO/anaconda-issues#12475 (comment), scipy/scipy#11062 (comment), conda/conda#10897 (comment)), it might be that this is only an issue for inplace development builds (pip install --editable or python setup.py develop), which is also what we are doing on Appveyor.

And for inplace builds, the libarrow_python.dll now no longer lives in the conda env prefix (where Arrow C++ build artifacts end up), but inside the arrow/python/pyarrow directory.
But what I don't understand is why this is a problem, since in either case (having the env variable or not), the same directory as the other DLL (the cython one that links to libarrow_python) should always be searched.

See https://github.com/conda-forge/python-feedstock/pull/481/files#diff-f43d77242cb37d5edf48db1155b6a2ef37972c418918229ea1d00a56c2b961ee for more details (the {{Add-CondaEcosystemModifyDllSearchPath.patch} file) and the comment at conda-forge/python-feedstock#481 (comment) for some additional context and discussion. Isuru indeed changed this for Python 3.10 (as far as I understand, this env variable should no longer be set by default starting with Python 3.10, but if it's the default for 3.8/3.9, I don't really understand why we have to set it explicitly here ...).

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
The updated conda patch for Python 3.11 (which is removing the effect of this env variable, conda-forge/python-feedstock#577) has the following explainer:

Note that in conda python <3.11, there was an option
CONDA_DLL_SEARCH_MODIFICATION=1 to add directories in PATH to the search
order, but this was deprecated in 3.10 and removed in 3.11 in favour of using
AddDllDirectory

[asfimport]

Alenka Frim / @AlenkaF:
I did some testing on my local Windows machine and this is what I found:

• With Python 3.8, without setting CONDA_DLL_SEARCH_MODIFICATION=1 and running

  python install -e .

  instead of making an inplace build there is no difference, the {}arrow_python lib is not found{}. I had to add CONDA_DLL_SEARCH_MODIFICATION env var to make it work.
  https://gist.github.com/AlenkaF/bb12bbe2051db5854e10747e13ca5b91

• With Python 3.10 the inplace build works and there is no need to set CONDA_DLL_SEARCH_MODIFICATION as the libs are found correctly.
  https://gist.github.com/AlenkaF/0dc8bc837472f83731dd6fa22fce558b

  I still do not understand why is this happening and why is CONDA_DLL_SEARCH_MODIFICATION needed as we do not use os.add_dll_directory in the project. But am happy to see that the issue is resolved with Python 3.10.

  I propose keeping the CONDA_DLL_SEARCH_MODIFICATION env var set to 1 in case older versions of Python are used. What I can do is to add a check for Python version in setup.py and set the CONDA_DLL_SEARCH_MODIFICATION env var automatically for older versions of Python.

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
Issue resolved by pull request 14302
#14302