Various stubgen regressions in mypy 1.7.0

[bluenote10]

Bug Report

Trying to update mypy from 1.6.1 to 1.7.0 has revealed a few regressions in the stub generator output related to pybind11 modules.

To Reproduce

Consider this pybind11 module:

#include <filesystem>
#include <optional>
#include <utility>
#include <vector>

#include <pybind11/pybind11.h>
#include <pybind11/stl.h>

namespace py = pybind11;

std::vector<float> funcReturningVector()
{
  return std::vector<float>{1.0, 2.0, 3.0, 4.0};
}

std::pair<int, float> funcReturningPair()
{
  return std::pair{42, 1.0};
}

std::optional<int> funcReturningOptional()
{
  return std::nullopt;
}

std::filesystem::path funcIncompleteSignature()
{
  // This example does not include <pybind11/stl/filesystem.h> on purpose
  // to demonstrate the signature of an incomplete binding.
  return std::filesystem::path{"foobar"};
}

struct MyStruct
{
  int a;
  int b;
  int c;
};

PYBIND11_MODULE(my_native_module, m)
{
  m.def("func_returning_vector", &funcReturningVector);
  m.def("func_returning_pair", &funcReturningPair);
  m.def("func_returning_optional", &funcReturningOptional);

  m.def("func_incomplete_signature", &funcIncompleteSignature);

  py::class_<MyStruct>(m, "MyStruct")
      .def_readwrite("a", &MyStruct::a)
      .def_readwrite("b", &MyStruct::b, "some docstring")
      .def_property_readonly(
          "c",
          [](const MyStruct& x) {
            return x.c;
          },
          "some docstring");
}

To reproduce, store the above snippet as my_native_module.cpp and then:

# Install latest pybind11:
pip install pybind11==2.11.1

# Compile pybind11 module:
c++ -O3 -Wall -shared -std=c++17 -fPIC $(python3 -m pybind11 --includes) my_native_module.cpp -o my_native_module.so

# Run the stub generator on it:
stubgen -p my_native_module

Expected Behavior

mypy 1.6.1 outputs the following, which is pretty reasonable:

from typing import Any, List, Optional, Tuple

class MyStruct:
    a: int
    b: int
    def __init__(self, *args, **kwargs) -> None: ...
    @property
    def c(self) -> int: ...

def func_incomplete_signature(*args, **kwargs) -> Any: ...
def func_returning_optional() -> Optional[int]: ...
def func_returning_pair() -> Tuple[int,float]: ...
def func_returning_vector() -> List[float]: ...

Actual Behavior

mypy 1.7.0 outputs the following:

from _typeshed import Incomplete

class MyStruct:
    a: int
    b: Incomplete
    def __init__(self, *args, **kwargs) -> None: ...
    @property
    def c(self): ...

def func_incomplete_signature(*args, **kwargs): ...
def func_returning_optional() -> Optional[int]: ...
def func_returning_pair() -> Tuple[int, float]: ...
def func_returning_vector() -> List[float]: ...

The regressions are:

• The most obvious thing is that the imports for List, Tuple, and Optional are missing. This makes the resulting .pyi invalid, because it uses symbols that have not been imported. The stub output does not type check. This is not limited to these three types, but applies to other types like Iterable, Iterator, Dict, Set, Union, etc.
• The type annotation of field b has regressed from int to Incomplete, apparently due to the explicit docstring.
• The type annotation of field c has regressed from int to not annotated at all, probably also because of the explicit docstring.
• The signature of the func_incomplete_signature has also slightly worsened: Before, it was at least annotated as -> Any but now it misses any return type annotation at all. This is problematic when users want to validate their generated stubs for completeness, i.e., run mypy --disallow-untyped-defs on the output. Of course semantically it is the same, but still it would be nicer to always produce some return type (arguably a broken signature may be rather annotated as -> object instead of -> Incomplete to avoid running into bugs from incomplete stub output).

Your Environment

• Mypy version used: 1.7.0
• Mypy command-line flags: irrelevant for stubgen
• Mypy configuration options from mypy.ini (and other config files): irrelevant for stubgen
• Python version used: 3.10

[bluenote10]

It also looks to me like --include-docstrings has no effect in general:

If I copy the contents of test-data/pybind11_mypy_demo/src/main.cpp locally, compile it, and run the stub generator with --include-docstrings, the output is exactly the same as if I omit it, i.e., no docstrings in the result. I've verified that the __doc__ are properly set at runtime.

I've also checked out the mypy repo and ran ./misc/test-stubgenc.sh locally. This also results in all docstrings getting removed from the test-data/pybind11_mypy_demo/stubgen-include-docs outputs. Surprisingly the bash scripts exits with return code 0 in this case, which it shouldn't, because the output does not match to the expected output. It looks like the CI got broken in #13284. I guess the issues is in line 27:

mypy/misc/test-stubgenc.sh

Line 27 in 0699dde

EXIT=$?

This $? is always 0 so it swallows the test failure. It should probably be just EXIT=1.

[chadrik]

The most obvious thing is that the imports for List, Tuple, and Optional are missing. This makes the resulting .pyi invalid, because it uses symbols that have not been imported. The stub output does not type check. This is not limited to these three types, but applies to other types like Iterable, Iterator, Dict, Set, Union, etc.

The MR that was merged prior to 1.7 unified the code and many of the behaviors of the C-extension and pure-python stubgen processes. In the pure-python code path, imports are only added for modules that have been imported: there is limited specialized behavior to detect and import types whose names match those in the typing or typing_extensions modules and these are shared by the C and pure-python code paths:

        known_imports = {
            "_typeshed": ["Incomplete"],
            "typing": ["Any", "TypeVar", "NamedTuple"],
            "collections.abc": ["Generator"],
            "typing_extensions": ["TypedDict", "ParamSpec", "TypeVarTuple"],
        }

There are real-world projects that I use stubgen on (PySide2) which have classes that match the names of those in the typing module, and the automatic generation of imports broke the stubs. The simple solution to the problem is for docstrings to use native types (list, dict, set, etc), or specify the module (typing.List, typing.Dict), but I do think that it makes sense to add a new option to stubgen to automatically add typing imports for known types.

The signature of the func_incomplete_signature has also slightly worsened: Before, it was at least annotated as -> Any but now it misses any return type annotation at all. This is problematic when users want to validate their generated stubs for completeness, i.e., run mypy --disallow-untyped-defs on the output. Of course semantically it is the same, but still it would be nicer to always produce some return type (arguably a broken signature may be rather annotated as -> object instead of -> Incomplete to avoid running into bugs from incomplete stub output).

This is another example of the C-extension behavior being adjusted to match the pure-python behavior (by sharing code paths). The C-extension behavior now correctly distinguishes types that are known to be Any (via annotations or docstrings), from those which are not known. This is a feature that ensures that stub authors can find and correct unknown types. In the case of unknown types, stubgen now consistently omits the type from the signature, or it uses the placeholder type _typeshed.Incomplete if the type can only be partially inferred, for example Iterator[_typeshed.Incomplete] in the case of a function that has a yield statement. Iterator[Any] would be over-specific in scenarios where the contained type cannot be inferred.

It would be reasonable to add a flag to use Any instead of Incomplete and generate them in cases that would otherwise be omitted, and with the new unified behavior, this flag would work for both C and pure-python generators. Such a hypothetical flag would likely produce output like this:

def func_returning_array(*args: Any, **kwargs: Any) -> Any: ...

The type annotation of field b has regressed from int to Incomplete, apparently due to the explicit docstring.
The type annotation of field c has regressed from int to not annotated at all, probably also because of the explicit docstring.

I'll add these to the pybind11 demo and fix them in the generator.

[bluenote10]

I just had time to re-check the behavior with mypy 1.8 that includes the fix by @chadrik. The example now produces the follow, which looks pretty:

from typing import List, Optional, Tuple

class FieldAnnotationTest:
    a: int
    b: int
    def __init__(self, *args, **kwargs) -> None: ...
    @property
    def c(self) -> int: ...

def answer() -> int: ...
def func_incomplete_signature(*args, **kwargs): ...
def func_returning_optional() -> Optional[int]: ...
def func_returning_pair() -> Tuple[int, float]: ...
def func_returning_vector() -> List[float]: ...

There is only the very minor regression left that the signature of the func_incomplete_signature lacks a return type, which only affects people who try to make their stubs conform to --disallow-untyped-defs in a post-processing (we apply a post-processing to *args and **kwargs to be typed as Incomplete -- now we also have to include the return type in that post-processing logic). I guess this is good enough to close this issue, and perhaps I will also find the time to contribute a small fix for that remaining aspect.

@chadrik Thanks again for the fix!