Improve protocol return types

[srittau]

• Dunder comparisons must return bool.
• write() return type should be ignored.

[AlexWaygood]

FWIW, the reason I annotated the dunder comparison functions as returning Any rather than bool was @JelleZijlstra's comment here. But I don't have strong opinions about this :)

[srittau]

Returning a non-bool from these methods isn't all that uncommon; numpy arrays do it.

I believe that's the case when the comparison operator are used for non-comparison purposes? For example, like pathlib uses the / operator for path concatenation? In that case I would suggest to use a custom protocol, instead of our typeshed protocol.

[AlexWaygood]

Returning a non-bool from these methods isn't all that uncommon; numpy arrays do it.

I believe that's the case when the comparison operator are used for non-comparison purposes?

I don't think so; I think the issue is that the runtime doesn't care what these methods return, as long as it's something that bool() can be called on (which is nearly everything). The most common approach is obviously to return bool; but that's not always the case, as the pandas example shows.

[srittau]

I feel that this exactly one of the cases that type checkers should catch. While of course everything is "boolean-like", returning arbitrary non-bools from those dunders seems error-prone. I am not making sense of the typing of numpy, but the documentation for ndarray.__bool__ says: self != 0, so it seems it should return a bool.

[srittau]

Also, it's just one primer hit, despite the length of the output indicating something different.

[AlexWaygood]

I feel that this exactly one of the cases that type checkers should catch. While of course everything is "boolean-like", returning arbitrary non-bools from those dunders seems error-prone. I am not making sense of the typing of numpy, but the documentation for ndarray.__bool__ says: self != 0, so it seems it should return a bool.

Makes sense!

[Akuli]

self != 0 doesn't return a bool in numpy:

>>> import numpy as np
>>> a = np.array([0,1,2,3])
>>> a != 0
array([False,  True,  True,  True])
>>> if a != 0: print("lol")
... 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

[JelleZijlstra]

Yes, in numpy all array operations do elementwise work where possible, so x OP y is broadly equivalent to [a OP b for a, b in zip(x, y)]. And numpy is very widely used, so I'm hesitant to make a change that drops some usage of numpy arrays.

However, we use these protocols both for functions like min() that really do need a bool (or something bool-like), and for the operator.* family of functions where the return value doesn't matter. We could perhaps split those up.

[srittau]

Splitting them sounds like a good idea. It would make most sense to me to keep the protocols in the PR as they are in the PR, and add custom protocols to operator.pyi directly.

[AlexWaygood]

It would make most sense to me to keep the protocols in the PR as they are in the PR, and add custom protocols to operator.pyi directly.

Let's add a short comment in operator, too, pointing people to this discussion — I'll admit I didn't understand that the use of these methods in operator required a different return type to elsewhere when I wrote #6583 :)

[srittau]

Primer looks good now.

[github-actions]

Diff from mypy_primer, showing the effect of this PR on open source code:

pandas (https://github.com/pandas-dev/pandas)
- pandas/core/indexes/base.py:4053: error: Argument 1 has incompatible type "Union[ExtensionArray, ndarray[Any, Any]]"; expected "Union[SupportsDunderLE, SupportsDunderGE, SupportsDunderGT, SupportsDunderLT]"  [arg-type]
+ pandas/core/indexes/base.py:4053: error: Argument 1 has incompatible type "Union[ExtensionArray, ndarray[Any, Any]]"; expected "Union[_SupportsDunderLE, _SupportsDunderGE, _SupportsDunderGT, _SupportsDunderLT]"  [arg-type]
- pandas/core/indexes/base.py:4053: error: Argument 2 has incompatible type "Union[ExtensionArray, ndarray[Any, Any]]"; expected "Union[SupportsDunderLE, SupportsDunderGE, SupportsDunderGT, SupportsDunderLT]"  [arg-type]
+ pandas/core/indexes/base.py:4053: error: Argument 2 has incompatible type "Union[ExtensionArray, ndarray[Any, Any]]"; expected "Union[_SupportsDunderLE, _SupportsDunderGE, _SupportsDunderGT, _SupportsDunderLT]"  [arg-type]
- pandas/io/formats/style.py:3709: error: Argument 2 to "ge" has incompatible type "Union[str, Any, float, Period, Timestamp, Timedelta, Sequence[Any], ndarray[Any, Any], NDFrame]"; expected "Union[SupportsDunderLE, SupportsDunderGE, SupportsDunderGT, SupportsDunderLT]"  [arg-type]
+ pandas/io/formats/style.py:3709: error: Argument 2 to "ge" has incompatible type "Union[str, Any, float, Period, Timestamp, Timedelta, Sequence[Any], ndarray[Any, Any], NDFrame]"; expected "Union[_SupportsDunderLE, _SupportsDunderGE, _SupportsDunderGT, _SupportsDunderLT]"  [arg-type]
- pandas/io/formats/style.py:3714: error: Argument 2 to "le" has incompatible type "Union[str, Any, float, Period, Timestamp, Timedelta, Sequence[Any], ndarray[Any, Any], NDFrame]"; expected "Union[SupportsDunderLE, SupportsDunderGE, SupportsDunderGT, SupportsDunderLT]"  [arg-type]
+ pandas/io/formats/style.py:3714: error: Argument 2 to "le" has incompatible type "Union[str, Any, float, Period, Timestamp, Timedelta, Sequence[Any], ndarray[Any, Any], NDFrame]"; expected "Union[_SupportsDunderLE, _SupportsDunderGE, _SupportsDunderGT, _SupportsDunderLT]"  [arg-type]