Python 3.8 incompatibility: test_enum_to_int failure in pybind versions 2.2.4, 2.3.0 and 2.4.0

[susilehtola]

Fedora build of pybind-2.2.4 for Python 3.8 with patch for e7ef34f fails the test phase with

    def test_enum_to_int():
>       m.test_enum_to_int(m.Flags.Read)
E       TypeError: test_enum_to_int(): incompatible function arguments. The following argument types are supported:
E           1. (arg0: int) -> None
E       
E       Invoked with: Flags.Read

Full log at https://copr-be.cloud.fedoraproject.org/results/@python/python3.8/fedora-rawhide-x86_64/00903558-pybind11/builder-live.log

[eacousineau]

Confirmed this on Ubuntu 18.04, building CPython 3.8.0a3 (9a448855b5) from source (but using --with-pydebug):

cd cpython
git checkout v3.8.0a3
install_prefix=~/.local/python/3.8.0a3
./configure \
   --with-pydebug --enable-shared CFLAGS=-O3 \
   --prefix=${install_prefix} LDFLAGS=-Wl,-rpath=${install_prefix}/lib \
   --with-ensurepip=upgrade
make install
${install_prefix}/bin/python3 -m pip install pytest

cd pybind11
mkdir build && cd build
cmake .. -DPYTHON_EXECUTABLE=${install_prefix}/bin/python3
make pytest

Not sure if this is an upstream issue, since 3.8 is alpha, though?

Trying without --with-pydebug CFLAGS=..., adding in --enable-optimizations...

UPDATE: Confirmed that this happens with release flags, i.e.:

cd cpython
./configure \
   --enable-optimizations \
   --prefix=${install_prefix} LDFLAGS=-Wl,-rpath=${install_prefix}/lib \
   --with-ensurepip=upgrade

[susilehtola]

Same problem with pybind-2.3.0.

[susilehtola]

And also with pybind-2.4.0.

[susilehtola]

This is getting to be a huge problem, since the next release of Fedora is going to ship with Python 3.8, and if pybind11 doesn't build it will be withdrawn from the distribution.

[wjakob]

It's not a problem of pybind11, just a behavioral change of Python 3.8.

Fixed in 21d0eb4 by just disabling that test.

[sizmailov]

@wjakob I think you refer to next paragraphs from release notes

Functions that convert Python number to C integer like PyLong_AsLong() and argument parsing functions like PyArg_ParseTuple() with integer converting format units like 'i' will now use the index() special method instead of int(), if available. The deprecation warning will be emitted for objects with the int() method but without the index() method (like Decimal and Fraction). PyNumber_Check() will now return 1 for objects implementing index(). PyNumber_Long(), PyNumber_Float() and PyFloat_AsDouble() also now use the index() method if available. (Contributed by Serhiy Storchaka in bpo-36048 and bpo-20092.)

Constructors of int, float and complex will now use the index() special method, if available and the corresponding method int(), float() or complex() is not available. (Contributed by Serhiy Storchaka in bpo-20092.)

And https://docs.python.org/3.8/reference/datamodel.html#object.__index__ :

Presence of index method indicates that the numeric object is an integer type. Must return an integer.

I think enums should be considered as integer types after all, therefore __index__ method should be provided.

This patch fixes the test. Note cast to int instead of Scalar is only to avoid problems with char as Scalar.

--- a/include/pybind11/pybind11.h
+++ b/include/pybind11/pybind11.h
@@ -1566,6 +1566,9 @@ public:
         #if PY_MAJOR_VERSION < 3
             def("__long__", [](Type value) { return (Scalar) value; });
         #endif
+        #if (PY_MAJOR_VERSION >= 3) && (PY_MINOR_VERSION >= 8)
+                def("__index__", [](Type value) { return (int   ) value; });
+        #endif

[wjakob]

Indeed, that is a much nicer fix -- thank you. I made one change and cast to Scalar instead of int (the enum type might e.g. be a 64 bit integer, it's best to let C++ decide)

[wjakob]

Fixed in 31680e6. I will push out v2.4.1 with this fix.