[Python] pyarrow missing py.typed marker file

[asfimport]

I understand that, in general, pyarrow does not support type hints. However, I think it is still sensible to add a py.typed marker file to the library. Let me demonstrate why,

$ pip install mypy pyarrow 

# test.py
import pyarrow as pa
 
table = pa.Table()
 
reveal_type(table) 

$ mypy test.py
test.py:1: error: Skipping analyzing "pyarrow": module is installed, but missing library stubs or py.typed marker
test.py:1: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
test.py:5: note: Revealed type is "Any"
Found 1 error in 1 file (checked 1 source file) 

Note that mypy identifies table as being an Any type, when obviously it is a {}Table{}. If we include a py.typed file, mypy will be able to make these trivial inferences. The motivating example is this,

@overload
def from_arrow(a: pa.Table) -> DataFrame:
    ...

@overload
def from_arrow(a: pa.Array | pa.ChunkedArray) -> Series:
    ...

def from_arrow(a: pa.Table | pa.Array | pa.ChunkedArray) -> DataFrame | Series:
    pass 

The problem is that since all of {}pa.Table{}, {}pa.Array{}, and pa.ChunkedArray are determined to be {}Any{}, the overloads effectively become 

@overload
def from_arrow(a: Any) -> DataFrame:
    ...

@overload
def from_arrow(a: Any) -> Series:
    ... 

and mypy complains that overload 2 is covered entirely by overload 1.

 

I tried to test what adding a py.typed file would do, but I ran into compilation issues. I was hoping someone with a little more experience here could quickly test this out for me :)

Reporter: Matteo Santamaria

_{Note: This issue was originally created as ARROW-17901. Please see the migration documentation for further details.}

[asfimport]

Alenka Frim / @AlenkaF:
Hi [~matteosantama] , thank you for opening up this issue!

Do you have a branch open that I could run locally?

[asfimport]

Matteo Santamaria:
@AlenkaF the proposed changes are so minimal that it might be easiest just to describe them, although I am happy to open a branch if that would help.

Basically, we just need to add an (empty) file called py.typed in arrow/python/pyarrow and then make sure it gets included in the distribution. PEP-561 suggests this might be as easy as adding it to the package data here.

[asfimport]

Alenka Frim / @AlenkaF:
I think Will gave a great answer on GitHub issues: #14291, -DARROW_INSTALL_NAME_RPATH=OFF is what came to my mind also.