Intersections with `Callable`, in particular `Any` & `Callable`

[randolf-scholz]

In this issue, we shall discuss intersections with Callable. There is an unresolved issue with respect to Any & Callable, that stems from the following 2 axioms:

1. Intersections with Callable are resolved by intersecting with an equivalent Callback-Protocol (see current draft)
2. (x & y).foo = x.foo & y.foo (see Eric's outline: Handling of Any within an Intersection #1 (comment))

Both of these together can lead to infinite regress.

Therefore, we cannot use ① when intersecting function-types with function-types. One can distinguish 3 cases:

1. Intersections of function-types with non-function-types: then we can apply rule ①
2. Intersections of two function-types: regular subtyping and the more complicated function signature subtyping rules apply.
3. Intersections of function-types with Any

---

EDIT: The suggestions below are outdated. The intersection with Any is irreducible. What needs to be discussed is what happens when the type is attempted to be evaluated with arguments args.

---

As I see it, there are 2 options of how to resolve ③:

Ⓐ For a strict interpretation of Any as an unknown type, Any could feasibly be Callable[..., R], which is a "consistent" subtype of Callable[[T1, T2, ..., Tn], R]. It follows that:

Any & Callable[[T1, T2, ..., Tn], R] = Callable[..., R]

The consequence is that (Any & T).foo(*args, **kwargs), is valid for any method foo of T, which as expressed in the Any intersection thread causes some pains, as it makes Any & T behave very similar to Any itself (only attribute types and function return types are preserved)

Ⓑ An alternative approach would hinge on a slight re-interpretation of Any in the context of intersections. One could demand that Any shall be interpreted as a disjoint type from whatever it is intersected with (= maximum compatibility). That is, in Any & T, we interpret Any as an unknown type that presumably shares no overlapping methods with T. This leads to

Any & Callable[[T1, T2, ..., Tn], R] = Callable[[T1, T2, ..., Tn], R]

I.e. we come back full circle to question if Any can be simplified away from intersections. Under this premise, we would have 3 simple rules, as I mentioned in #1 (comment)

1. Any & T is an irreducible form
2. (Any & T).foo ≅ T.foo if T implements foo
3. (Any & T).bar ≅ Any if T does not implement bar

---

So, does it make sense to have a general exemption to the rules #1 (comment) when intersecting Any with function-types?

[randolf-scholz]

EDIT: There is possibly also a sort of middle ground option:

Ⓒ Only subtyping rules of Callable apply, but not the "consistent" subtyping rules. (see draft document for the distinction). That is, Any & Callable can mess with the argument types, but not the signature.

Any & Callable[[T1, T2, ..., Tn], R] = Callable[[Any, Any, ..., Any], R]

Under this approach, (Any & T).foo would have the same function signature as T.foo, but any argument type would be allowed. However, I kind of doubt either camp would be happy with Ⓒ.

[DiscordLiz]

If all of these are accurate:

1. That type checkers are willing to provide an optional warning for Any as an intersection operand
2. That there are no shortcuts that allow skipping certain checks that are not able to be shown as formally equivalent.
3. That these are the distinct places in this example that the intersection is consulted and how

x: A & B = ... # Here, we have to still check that x is both A and B

def f(x: A & B): 
    x.foo()  # here, we have to check that this is consistent use with definitions for all existing defintions of foo in A & B

f(something)  # here we check that something is both A and B

No special case needed.

[randolf-scholz]

Wait a second, I think this actually means that this whole example from earlier should already work fine, we don't even need

1. That type checkers are willing to provide an optional warning for Any as an intersection operand

from where import UntypedLibraryType
from typing import Protocol

class Mine(Protocol):
    def foo(self, x: int, /) -> int: ...

def fn(x: UntypedLibraryType & Mine):
    x.foo(1)  # this is how I expect it to be used
    x.foo("blah")  # this works because UntypedLibraryType is Any, and I would intend this as an error

The type-checker will test whether the signature foo(__x: str) is compatible with Mine.foo and Any.foo, and will consequently fail. If this wasn't how it is working already, then it would apply in the same vein to

from typing import Protocol

class Mine(Protocol):
    def foo(self, x: int, /) -> int: ...

def fn(x: Mine):
    x.foo(1)
    x.foo("blah")

I think this really does mean that we can simplify Any & C = C for any function-type C.

[diabolo-dan]

The type-checker will test whether the signature foo(__x: str) is compatible with Mine.foo and Any.foo, and will consequently fail.

It need only be consistent with one of Mine.foo and Any.foo, and it is consistent with Any.foo.

[randolf-scholz]

@diabolo-dan why only with one of them?

[erictraut]

Here's how type checkers handle calls on unions today in Python:

When A | B is called with args:

• If either A or B is not callable with args, the operation is an error
• If A and B are both callable with args, the return result is the union of the return results from A and B
• Any is treated as Callable[..., Any]

By symmetry, I would think that the rule for intersections would be:

When A & B is called with args:

• If neither A or B are callable with args, the operation is an error
• If A is callable with args but B is not, the return result comes from A
• If B is callable with args but A is not, the return result comes from B
• If A and B are both callable with args, the return result is the intersection of the return results from A and B
• Any is treated as Callable[..., Any]

And yes, this means that if an Any appears in either a union or an intersection, the resulting return type will include an Any. That's the nature of Any, and we should be consistent here.

Interestingly, TypeScript deviates from the above slightly in the case of intersections. If A and B are both callable with args, TypeScript uses the return type from A only. It apparently sorts the types (using an undocumented sorting algorithm) to maintain determinism and order independence of intersections. You get the same answer for A & B and B & A, but if you swap the names of the two interfaces, the result changes. So the sorting algorithm probably takes into account the names of the types.

interface A {
    y: (val: string) => number;
    z: () => number;
}

interface B {
    y: (val: number) => string;
    z: () => string;
}

function f(c: A & B) {
    const x = c.x(); // Error
    const y1 = c.y(0); // string
    const y2 = c.y(''); // number
    const z = c.z(); // number (but becomes string if A and B interface names are swapped)
}

The behavior in TypeScript seems inconsistent to me, and the "undocumented sorting algorithm" means that it cannot be replicated from a spec unless that spec fully documents the way types should be sorted. I'd prefer to avoid that for Python.

[erictraut]

Edit: The following was a response to an earlier post that was deleted. I'll leave it here, but it's probably no longer valid.

---

@randolf-scholz, good point. That means my proposed rule above is flawed. "If A and B are both callable with args, the return result is the intersection of the return results from A and B" doesn't hold.

This would explain TypeScript's behavior. It is effectively creating an overload from the two callables. The challenge is that overloads depend on ordering, but intersections are order-independent. That means some deterministic order must be imposed, which explains the sorting behavior in TypeScript.

In TypeScript, overloads are simple. The first matching overload is always selected. In Python, thanks to mypy's implementation, overloads are extremely complex and involve a bunch of special casing for union expansion and handling of Any arguments. To avoid this complexity, we could avoid the mention of "overload" and instead define the rule as follows:

If A and B are both callable with args, the return result comes from the return type of Sort(A, B) where Sort is a sorting algorithm that deterministically chooses either A or B.

We would then need to define the sorting algorithm. (Yuck.)

I have a type sorting algorithm in pyright that I added in an attempt to make union-related operations more deterministic and order-independent, but I consider this algorithm to be an implementation detail. I presume that mypy has a similar sorting algorithm.

Maybe it's OK to leave the sorting algorithm out of the spec and say that it's up to each type checker to implement this? But this will lead to different behaviors across type checkers.

[randolf-scholz]

Hm, I think there is a problematic case though, which is incompatible signatures:

class A:
    def foo(self, *, key: str) -> T: ... # takes 1 mandatory keyword argument "key"
class B:
    def foo(self) -> S: ...

The issue here is that by the LSP, one is not allowed to have a subclass of B where foo has mandatory key arguments.
One is generally only allowed to add optional arguments, i.e. arguments with defaults, *args, and **kwargs.

I think one has to amend the rules as follows: When A & B is called with args:

• If neither A or B are callable with args, the operation is an error
• If A is callable with args, and signature(args) ≤ signature(B), the result comes from A.
• If A is callable with args, but signature(args) incompatible with signature(B), the operation is an error
• If B is callable with args, and signature(args) ≤ signature(A), the result comes from B.
• If B is callable with args, but signature(args) incompatible with signature(A), the operation is an error
• If A and B are both callable with args, the return result is the intersection of the return results from A and B
• Any is treated as Callable[..., Any]

Also, I wonder if "the operation is an error" can generally just be replaced by returning Never and type-checker raising whenever an instance of Never is created.

[randolf-scholz]

@erictraut I deleted the earlier comment, the argument was flawed. In the example

class A:
    def foo(self, x: int) -> int: ...
class B:
    def foo(self, x: str) -> str: ...
class C(A, B):
    @overload
    def foo(self, x: int) -> int: ...
    @overload
    def foo(self, x: str) -> str: ...

C is not a proper subclass of A & B since C.foo(int & str) would yield int, violating LSP for class B.

Actually, I am not the only one who fell for this, in the current specification draft for Protocol-merging the overloads are used as well.

[mikeshardmind]

I think most of this is ruled out by the original assignability check.

I don't think it should be picking one of A or B as typescript does. If the two are incompatible, the two are incompatible. If they aren't, compatible use should be acceptable, but the assignability concern says it needs to be compatible with both. We aren't stating that there is a direct subclass with a specific MRO with intersections, and even if we were, replacement says such a subclass would have to be compatible this way too.

@DiscordLiz linked an accepted (rust) RFC in #14 that has a section detailing detecting overlapping implementations that feels relevant here

[mikeshardmind]

There's also my comment here

quick relevant part:

As an exhaustive pattern for example
(A & B).foo
if A provides foo and B doesn't, A.foo
If B provides foo and A doesn't, B.foo
if both A and B provide foo, A.foo & B.foo
if neither A nor B provide foo, Never

Other than this not checking for compatibility between operands at the time of use (consistent subtyping, we should assume it from assignability), and only needing to check that use is compatible with each operand individually, this should align with how rust does this, and align with a general expectation of "I specified being compatible with each of these"

[DiscordLiz]

I don't think it should be picking one of A or B as typescript does. If the two are incompatible, the two are incompatible. If they aren't, compatible use should be acceptable

Yeah, but there's still a problem with that here:

class A:

    def foo(self) -> None:
        ...

class B:
    def foo(self, wait: bool = False) -> None:
        ...

def f(x: A & B):
    x.foo()  # fine
    x.foo(wait=True)  # not fine, not consistent with A.foo? But something consistent with A.foo and B.foo should allow this.

I think this shows that we have to check that overlapping attributes and methods have compatible overlap and determine what that compatible overlap is. We can't split assignment from use without first calculating a compatible overlap.

This means that #5 actually does need to be a type checking concern and mandated. This behavior needs to be part of the spec.

We also shouldn't arbitrarily pick a class, or try to determine what the mro would need to be, this is an intersection, not a direct subclass. someone could inherit from both and reconcile in a way that would be impossible without further additions than just these as base classes

class A:
    def foo(self) -> None:
        ...

    def bar(self, wait: bool = False) -> None:
        ...

class B:
    def foo(self, wait: bool = False) -> None:
        ...

    def bar(self) -> None:
        ...

This example shows that anything there are cases that would need additional reconciliation, even as just a subclass, let alone an intersection.

[DiscordLiz]

Going from that minimal overlap calculation, here would be some of my expectations

() -> None & (optional: bool=False) -> None = (optional: bool=False) -> None
(int) -> int & (int) -> str = Never

But in the above, we could modify it only slightly and also have a solvable case:

(int) -> int & (str) -> str = (T) -> T, for T: int | str

we also have to consider names as part of the api here when discussing something being compatible with both

(not_pos_only_name: int) -> int  & (different_not_pos_only_name: int) -> int = Never