[C++][Python] Move all Python related code into PyArrow

[asfimport]

Move src/arrow/python directory into pyarrow and arrange PyArrow to build it.

More details can be found on this thread:

https://lists.apache.org/thread/jbxyldhqff4p9z53whhs95y4jcomdgd2

Reporter: Alenka Frim / @AlenkaF
Assignee: Alenka Frim / @AlenkaF

Subtasks:

• [C++] xxhash not installed into dist/lib/include when building C++
• [C++][Flight] Remove the need for serialization_internal.h inside python/flight.cc
• [C++][Flight] Remove the need for serialization_internal.h inside python/flight.cc
• [C++] Split utf8.h to avoid exposing xsimd dependency to the outside

PRs and other links:

• GitHub Pull Request #13311

_{Note: This issue was originally created as ARROW-16340. Please see the migration documentation for further details.}

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
Moving to the 10.0.0 milestone. Given that this is a large change, the goal is to merge this as soon as possible after the 9.0.0 release is cut, so we have the full 10.0.0 cycle to test and stabilize the change.

[asfimport]

Kouhei Sutou / @kou:
Issue resolved by pull request 13311
#13311

[asfimport]

Yue Ni / @niyue:
I have a project relying on arrow/python/pyarrow.h, and I installed arrow from vcpkg https://github.com/microsoft/vcpkg/tree/master/ports/arrow. Previously arrow/python/pyarrow.h is available in include directory after installing arrow, but after upgrading to the latest version of arrow, more specifically commit ec579df, I am not able to find pyarrow.h in include directory of vcpkg. 

I think it is probably caused by this change, do we have any recommendation what projects relying on arrow/python/pyarrow.h should do to overcome this issue?

[asfimport]

Alenka Frim / @AlenkaF:
Can you check if arrow/python/pyarrow.h is available in arrow/python/pyarrow/include/? And if it is, will you be able to include it from there?

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
The pyarrow.h file will indeed no longer be installed by installing Arrow C++ with vcpkg, but it will still be installed by the python package, only not in the standard "include" directory. For building against pyarrow, you can use pyarrow.get_include() to get the location of those files, as is documented at https://arrow.apache.org/docs/dev/python/integration/extending.html#c-api

[asfimport]

Yue Ni / @niyue:
@AlenkaF   @jorisvandenbossche  Thanks for the help.

it will still be installed by the python package, only not in the standard "include" directory

Where is the header is expected to be installed? Is it expected to be installed by pyarrow's python wheel to some where and I have to add this path to my compiler's include path?

In my C++ project, I use vcpkg to manage dependency. And one of the module is a python binding for the C++ library, where I use pyarrow's C++ API like arrow::py::wrap_table together with pybind11 to create the python binding. Since I use vcpkg to manage dependency, I expect all C++ dependencies available via vcpkg.

1. Previously pyarrow C++ is part of vcpkg arrow port, and I can use CMake's find_library(arrow_python) to find the library

2. and use find_path(arrow/python/pyarrow.h) to find the path to the include directory

What I find for the latest arrow version:

1. the libarrow_python.a is not built, even if I set ARROW_PYTHON CMake option to ON. 

2. the arrow/python/pyarrow.h cannot be found in include directory after building the C++ library (at least using vcpkg arrow port)

I went through most of the comments in PR for this issue [1] and read this ARROW_PYTHON option issue [2] as well, and the current behavior seems to be the expected behavior. The python directory will NOT be built even if ARROW_PYTHON=ON.

I am not sure what the recommended approach for using pyarrow in C++. According to the document here [3], to make the build automated, it seems these are the steps:

1. install pyarrow package

2. launch python, run pyarrow.get_include() to get the include directory

3. add the include directory to compiler's include search path (probably via CMake)

4. where is it expected to find the libarrow_python so that CMake can find and use it for link?

5. build

Is this the recommended approach for doing this? I am not quite sure step #4, any comments on this?

I can think of another approach, which is creating another vcpkg port like arrow_python and build the arrow_python library explicitly, so that projects can use this port for such purpose. Is this a recommended approach after this issue? Thanks.

 

[1] #13311

[2]https://issues.apache.org/jira/browse/ARROW-17868

[3]Using pyarrow from C++ and Cython Code, https://arrow.apache.org/docs/dev/python/integration/extending.html#c-api

 

[asfimport]

Kouhei Sutou / @kou:
Is your project an open source project? If so, could you share your code? I want to see an use case for pyarrow.h.

[asfimport]

Yue Ni / @niyue:
@kou  sorry it is not open source at this moment, but the binding itself is a just very thin wrapper for the C++ library. The code looks like this:

// pyarrow.h is included here
#include <arrow/python/pyarrow.h>
#include <pybind11/pybind11.h>

class SearchContext {
  public:
  // ...
  SearchContext() {
    // here pyarrow module is initialized
    auto result = arrow::py::import_pyarrow();
    // ...
  }

  pybind11::object search(const string &sql) {
    // call the C++ library to run the search and get the result as an arrow table
    auto table = get_exec_context()->search(sql); 
    // use pyarrow's C++ API to wrap the Arrow C++ table as a pyarrow table instance
    return _wrap_table(table);
  }

  private:
  pybind11::object _wrap_table(std::shared_ptr<arrow::Table> table) {
    // this is where pyarrow.h is used, the wrap_table API   
    return pybind11::reinterpret_steal<pybind11::object(arrow::py::wrap_table(table)); 
  }
}

// the above SearchContext class is exposed via pybind11 so that users can use them in Python 
PYBIND11_MODULE(my_py_binding, m) { 
  py::class_<SearchContext>(m, "SearchContext")       
    // ...   
    .def("search", &SearchContext::search, py::arg("sql") = ""); 
  // .... 
} 

Here are the materials we referred to when we authored this binding initially (probably two years ago), we don't revisit if there is better approach since it is working for us until now.

 [1] https://arrow.apache.org/docs/python/integration/extending.html#_CPPv4N5arrow5arrow2py10wrap_tableERKNSt10shared_ptrI5TableEE

 [2] https://stackoverflow.com/questions/57863751/how-to-convert-pyarrow-table-to-arrow-table-when-interfacing-between-pyarrow-in

[asfimport]

Alenka Frim / @AlenkaF:
I have created a new Jira issue that will add extra info on the Apache Arrow documentation on how to use functions mentioned above to find the header files and the libraries for PyArrow C++ API: https://issues.apache.org/jira/browse/ARROW-17950

[asfimport]

Kouhei Sutou / @kou:
@niyue Thanks. How about using the C data interface / the C stream interface instead of arrow/python/pyarrow.h?

auto table = get_exec_context()->search(sql);
auto reader = arrow::tableBatchReader(table);
struct ArrowArrayStream c_stream;
auto status = arrow::ExportRecordBatchReader(reader, &c_stream); // TODO: Check status
// Call the following Python code from pybind11 (sorry. I'm not familiar with Python's C API):
//   py_reader = pyarrow.ipc.RecordBatchReader._import_from_c(&c_stream);
//   py_table = pyarrow.Table.from_batches(py_reader);
return pybind11::reinterpret_steal<pybind11::object>(py_table);

The C data interface and the C stream interface are included in Apache Arrow C++.

[asfimport]

Yue Ni / @niyue:
@kou  thanks so much. I will give it a try and get back to you later (probably not very soon because I will ask someone else for verifying the python part).

[asfimport]

Yue Ni / @niyue:
@kou 

// Call the following Python code from pybind11

py_reader = pyarrow.ipc.RecordBatchReader._import_from_c(&c_stream);

Could you please briefly explain more how this works? I would like to confirm, do you mean the pyarrow.ipc.RecordBatchReader._import_from_c is python code and not C++ code, and we could use pybind11 to execute the pyarrow python code to do this (probably using pybind11's API like https://pybind11.readthedocs.io/en/stable/advanced/embedding.html#executing-python-code)?

[asfimport]

Kouhei Sutou / @kou:
Yes, pyarrow.ipc.RecordBatchReader._import_from_c is a Python code.

Thanks for sharing the pybind11's API document. It seems that we can write the code like the following:

{
  py::scoped_interpreter guard{};
  py::module_ pyarrow = py::module_::import("pyarrow"); // We can use auto
  py::object py_reader_import_from_c = pyarrow.attr("ipc").attr("RecordBatchReader").attr("_import_from_c")
  // uintptr_t isn't listed here: https://pybind11.readthedocs.io/en/stable/advanced/cast/overview.html#list-of-all-builtin-conversions
  py::object py_reader = py_reader_import_from_c(static_cast<uint64_t>(static_cast<uintptr_t>(&c_stream)));
  py::object py_table = pyarrow.attr("Table").attr("from_batches")(py_reader);
  return pybind11::reinterpret_steal<pybind11::object>(py_table);
}

[asfimport]

Yue Ni / @niyue:
@kou I follow your instruction (with some slight modification) to make it compiled and running but ran into some segfault during importingpyarrow module.

1) Since I create a python binding, I use this python binding in my Python code. py::scoped_interpreter guard{} seems not necessary because that will embed another interpreter, otherwise, pybind will report the interpreter is already running error. This is the only thing I revised.

2) In my python code:

import my_python_binding
import pyarrow # this lines triggers segfault

Some error like below is reported:

 

......
free(): invalid pointer
Fatal Python error: Aborted
Current thread 0x00007fae68f939c0 (most recent call first):
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1176 in create_module
  File "<frozen importlib._bootstrap>", line 571 in module_from_spec
  File "<frozen importlib._bootstrap>", line 674 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1006 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1027 in _find_and_load
  File "/root/.cache/pypoetry/virtualenvs/<redated>-sOknH4PM-py3.10/lib/python3.10/site-packages/pyarrow/__init__.py", line 65 in <module>
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 883 in exec_module
  File "<frozen importlib._bootstrap>", line 688 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1006 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1027 in _find_and_load
  File "/usr/src/stonewave/bindings/python/<redated>/<redated>/search.py", line 16 in search

The line 65 of pyarrow/\_\_init\_\_.py is:

import pyarrow.lib as _lib

 

 

{}3) a core dump was generated{}, and I use gdb to briefly view the core dump, it looks like this:

#0  raise (sig=<optimized out>) at ../sysdeps/unix/sysv/linux/raise.c:50
        set = {__val = {0 <repeats 16 times>}}
        pid = <optimized out>
        tid = <optimized out>
        ret = <optimized out>
#1  <signal handler called>
No locals.
#2  0x00007fefa03eb735 in {}gnu_cxx::{}exchange_and_add ({_}_mem=0xa, __val=-1) at /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/ext/atomicity.h:50
No locals.
#3  {}gnu_cxx::{}exchange_and_add_dispatch ({_}_mem=0xa, __val=-1) at /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/ext/atomicity.h:84
No locals.
#4  std::Sp_counted_base<(_gnu_cxx::_Lock_policy)2>::_M_release (this=0x2) at /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/bits/shared_ptr_base.h:155
No locals.
#5  0x00007fefa03eb6da in std::{}shared_count<({}gnu_cxx::Lock_policy)2>::~_shared_count (this=0x1bca350) at /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/bits/shared_ptr_base.h:733
No locals.
#6  0x00007fefa0463c5e in std::{}shared_ptr<arrow::StatusDetail, ({}gnu_cxx::Lock_policy)2>::~_shared_ptr (this=0x1bca348)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/bits/shared_ptr_base.h:1183
No locals.
#7  0x00007fefa0463c38 in std::shared_ptr<arrow::StatusDetail>::~shared_ptr (this=0x1bca348) at /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/bits/shared_ptr.h:121
No locals.
#8  0x00007fefa0463c06 in arrow::Status::State::~State (this=0x1bca320) at /vcpkg/installed/x64-linux-haswell/include/arrow/status.h:347
No locals.
#9  0x00007fefa0463b9e in arrow::Status::DeleteState (this=0x7ffdcda1dd20) at /vcpkg/installed/x64-linux-haswell/include/arrow/status.h:357
No locals.
#10 0x00007fef93345f0c in arrow::internal::GetEnvVar(char const*) () from /root/.cache/pypoetry/virtualenvs/<redated>-sOknH4PM-py3.10/lib/python3.10/site-packages/pyarrow/libarrow.so.1000
No symbol table info available.
#11 0x00007fef93398924 in arrow::io::internal::GetIOThreadPool() () from /root/.cache/pypoetry/virtualenvs/<redated>-sOknH4PM-py3.10/lib/python3.10/site-packages/pyarrow/libarrow.so.1000
No symbol table info available.
#12 0x00007fef928ca51e in GLOBAL_sub_I_unity_6_cxx.cxx () from /root/.cache/pypoetry/virtualenvs/<redated>-sOknH4PM-py3.10/lib/python3.10/site-packages/pyarrow/libarrow.so.1000
No symbol table info available.
#13 0x00007fefae7fcfe2 in call_init (l=<optimized out>, argc=argc@entry=3, argv=argv@entry=0x7ffdcda20dc8, env=env@entry=0x7ffdcda20de8) at dl-init.c:72
        j = 0
        jm = <optimized out>
        addrs = <optimized out>
        init_array = <optimized out>
#14 0x00007fefae7fd0e9 in call_init (env=0x7ffdcda20de8, argv=0x7ffdcda20dc8, argc=3, l=<optimized out>) at dl-init.c:30
        init_array = <optimized out>
        j = <optimized out>
        jm = <optimized out>
        addrs = <optimized out>
#15 _dl_init (main_map=0x1baa930, argc=3, argv=0x7ffdcda20dc8, env=0x7ffdcda20de8) at dl-init.c:119
        preinit_array = <optimized out>
        preinit_array_size = <optimized out>
        i = <optimized out>
#16 0x00007fef9a56faed in {}GI{_}_dl_catch_exception (exception=exception@entry=0x0, operate=operate@entry=0x7fefae8005d0 <call_dl_init>, args=args@entry=0x7ffdcda1dfd0) at dl-error-skeleton.c:182
        old = <optimized out>
        errcode = 0
{}Type <RET> for more, q to quit, c to continue without paging{}
        c = {exception = 0x7ffdcda1df30, errcode = 0x0, env = {{__jmpbuf =
{140728053391152, 0, 0, 140666978835288, 29011096, 140666978907692, 140666978805312, 140666981352408}
, __mask_was_saved = -1790078976,
              {}saved_mask = {{_}_val =
{0, 140728053391248, 140667076916640, 0, 0, 0, 0, 0, 0, 0, 0, 12468370135039750697, 140728053391728, 140728053391216, 140728053391728, 140728053391392}
}}}}
        old = <optimized out>
#17 0x00007fefae801058 in dl_open_worker (a=a@entry=0x7ffdcda1e170) at dl-open.c:758
        init_args = {new = 0x1baa930, argc = 3, argv = 0x7ffdcda20dc8, env = 0x7ffdcda20de8}
        args = 0x7ffdcda1e170
        file = <optimized out>
        mode = <optimized out>
        call_map = <optimized out>
        dst = <optimized out>
        new = 0x1baa930
        {}PRETTY_FUNCTION{} = "dl_open_worker"
        r = <optimized out>
        reloc_mode = <optimized out>
        nmaps = <optimized out>
        l = <optimized out>
        maps = <error reading variable maps (value requires 34359738360 bytes, which is more than max-value-size)>
        relocation_in_progress = <optimized out>
        any_tls = <optimized out>
#18 0x00007fef9a56fa90 in {}GI{_}_dl_catch_exception (exception=exception@entry=0x7ffdcda1e150, operate=operate@entry=0x7fefae800ca0 <dl_open_worker>, args=args@entry=0x7ffdcda1e170) at dl-error-skeleton.c:208
        errcode = 0
        c = {exception = 0x7ffdcda1e150, errcode = 0x7ffdcda1e05c, env = {{__jmpbuf =
{2147483650, -855320820084358115, -74376, 3, 5369100, -2, -863008342291758051, -855308368215391203}
, __mask_was_saved = 0,
              {}saved_mask = {{_}_val =
{140667001712800, 140667001712800, 4506233, 140667001207448, 15, 140667001207392, 140667001712848, 140728053391680, 140728053391712, 140728053392737, 660865357547929600,                   140728053392480, 0, 257, 140728053391712, 6671076}
}}}}
        old = 0x7ffdcda1e240
#19 0x00007fefae8008fa in _dl_open (file=0x7fef95808100 "/root/.cache/pypoetry/virtualenvs/<redated>-sOknH4PM-py3.10/lib/python3.10/site-packages/pyarrow/lib.cpython-310-x86_64-linux-gnu.so", mode=-2147483646,
    caller_dlopen=0x51ed0c <_PyImport_FindSharedFuncptr+284>, nsid=-2, argc=3, argv=0x7ffdcda1e150, env=0x7ffdcda20de8) at dl-open.c:837
        args =
{file = 0x7fef95808100 "/root/.cache/pypoetry/virtualenvs/<redated>-sOknH4PM-py3.10/lib/python3.10/site-packages/pyarrow/lib.cpython-310-x86_64-linux-gnu.so", mode = -2147483646,           caller_dlopen = 0x51ed0c <_PyImport_FindSharedFuncptr+284>, map = 0x1baa930, nsid = 0, original_global_scope_pending_adds = 0, argc = 3, argv = 0x7ffdcda20dc8, env = 0x7ffdcda20de8}
        exception =
{objname = 0x439830 <list_richcompare> "UAWAVAUATSPH\213G\b\366\200\253", errstring = 0x7ffdcda1e250 "8Y\217\001",           message_buffer = 0xfbad8001 <error: Cannot access memory at address 0xfbad8001>}
        errcode = <optimized out>
        {}PRETTY_FUNCTION{} = "_dl_open"
#20 0x00007fef9a75a258 in dlopen_doit (a=a@entry=0x7ffdcda1e390) at dlopen.c:66
        args = 0x7ffdcda1e390
#21 0x00007fef9a56fa90 in {}GI{_}_dl_catch_exception (exception=exception@entry=0x7ffdcda1e330, operate=operate@entry=0x7fef9a75a200 <dlopen_doit>, args=args@entry=0x7ffdcda1e390) at dl-error-skeleton.c:208
        errcode = 32751
        c = {exception = 0x7ffdcda1e330, errcode = 0x7ffdcda1e23c, env = {{__jmpbuf =
{26171704, 864236073942795293, -74376, 140666982652160, 140666983577264, 140666982138112, -863008342354672611,                 -855308368215391203}
, {}mask_was_saved = 0, _saved_mask = {_val =
{660865357547929600, 9223372036854775807, 258, 0, 0, 140666982652160, 140728053392480, 140666982138112, 5216203, 206158430248,                   140728053392336, 140728053392128, 140667001246696, 140667001246696, 2, 140667001308992}
}}}}
        old = 0x0
#22 0x00007fef9a56fb4f in {}GI{_}_dl_catch_error (objname=objname@entry=0x18f5940, errstring=errstring@entry=0x18f5948, mallocedp=mallocedp@entry=0x18f5938, operate=operate@entry=0x7fef9a75a200 <dlopen_doit>,
    args=args@entry=0x7ffdcda1e390) at dl-error-skeleton.c:227
        exception = {objname = 0x0, errstring = 0x2 <error: Cannot access memory at address 0x2>, message_buffer = 0x1 <error: Cannot access memory at address 0x1>}
        errorcode = <optimized out>
#23 0x00007fef9a75aa65 in _dlerror_run (operate=operate@entry=0x7fef9a75a200 <dlopen_doit>, args=args@entry=0x7ffdcda1e390) at dlerror.c:170
        result = 0x18f5930
{}Type <RET> for more, q to quit, c to continue without paging{}
#24 0x00007fef9a75a2e4 in __dlopen (file=<optimized out>, mode=<optimized out>) at dlopen.c:87
        args =
{file = 0x7fef95808100 "/root/.cache/pypoetry/virtualenvs/<redated>-sOknH4PM-py3.10/lib/python3.10/site-packages/pyarrow/lib.cpython-310-x86_64-linux-gnu.so", mode = 2, new = 0x1a4acd0,           caller = 0x51ed0c <_PyImport_FindSharedFuncptr+284>}
#25 0x000000000051ed0c in _PyImport_FindSharedFuncptr (prefix=0x64fb3d "PyInit", shortname=0x7fef959676b0 "lib",
    pathname=0x7fef95808100 "/root/.cache/pypoetry/virtualenvs/<redated>-sOknH4PM-py3.10/lib/python3.10/site-packages/pyarrow/lib.cpython-310-x86_64-linux-gnu.so", fp=0x0) at ./Python/dynload_shlib.c:100
        funcname = "PyInit_lib", '\000' <repeats 22 times>, "\340\371\234\225\357\177\000\000\000\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000\001\000\000\000\001", '\000' <repeats 43 times>, "O\373d\000\000\000\000\000\320\254\244\001\000\000\000\000\060=\204\225\357\177\000\000P\345\241\315\375\177\000\000\002\000\000\000\000\000\000\000P\345\241\315\375\177\000\000\000\000\000\000\000\000\000\000V\357Y\000\000\000\000\000\240\356Y\000\000\000\000\000\220v\226\225\357\177\000\000\070\004\212\225\357\177\000\000"...
        pathbuf = "\002\000\000\000\000\000\000\000\350\357`\331\210,eY\000\321\254\226\357\177\000\000\236\272D\000\000\000\000\000\230z\243\226\357\177\000\000\000\000\000\000\000\000\000\000p\307\177\225\357\177\000\000\000Y\210\225\357\177\000\000\260A\212\225\357\177\000\000\220v\226\225\357\177\000\000\000\000\000\000\000\000\000\000\023\305P\000\000\000\000\000\020\000\000\000\060\000\000\000\260\346\241\315\375\177\000\000\360\345\241\315\375\177\000\000\000Y\210\225\357\177\000\000\000\000\000\000\000\000\000\000Q\274D\000\000\000\000\000p\307\177\225\357\177\000\000\260A\212\225\357\177\000\000\340\251p\000\000\000\000\000\340\251p\000\000\000\000\000\360\071j\000\000\000\000\000\003", '\000' <repeats 15 times>...
        dlopenflags = 29139752
        handle = <optimized out>
        p = <optimized out>
#26 0x00000000004eaa4e in _PyImport_LoadDynamicModuleWithSpec (spec=0x7fef95885900, fp=0x0) at ./Python/importdl.c:137
        m = 0x0
        path = <optimized out>
        name = 0x7fef95967690
        name_unicode = 0x7fef957fc770
        pathbytes = 0x7fef958080e0
        hook_prefix = 0x64fb3d "PyInit"
        name_buf = 0x7fef959676b0 "lib"
        exportfunc = <optimized out>
        p0 = <optimized out>
        oldcontext = <optimized out>
        def = <optimized out>
        modules = <optimized out>
#27 0x00000000004ea40e in _imp_create_dynamic_impl (module=<optimized out>, spec=0x7fef95885900, file=<optimized out>) at Python/import.c:2049
        name = 0x7fef957fc770
        path = 0x7fef958a41b0
        tstate = <optimized out>
        mod = 0x0
        fp = 0x0
#28 _imp_create_dynamic (module=<optimized out>, args=<optimized out>, nargs=<optimized out>) at Python/clinic/import.c.h:330
        file = <optimized out>
        return_value = 0x0
        spec = 0x7fef95885900
#29 0x000000000059ef56 in cfunction_vectorcall_FASTCALL (func=0x7fef96ac58a0, args=0x7fef95885168, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/methodobject.c:430
        tstate = 0x1a4acd0
        nargs = 1
        meth = 0xa
        result = <optimized out>
#30 0x00000000004c1fe7 in do_call_core (tstate=0x1a4acd0, trace_info=0x7ffdcda1e7c0, func=0x7fef96ac58a0, callargs=0x7fef95885150, kwdict=0x0) at Python/ceval.c:5915
        result = <optimized out>
        nargs = <optimized out>
        self = <optimized out>
{}Type <RET> for more, q to quit, c to continue without paging{}
#31 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=0x7fef96a5f490, throwflag=<optimized out>) at Python/ceval.c:4277
        func = 0x7fef96ac58a0
        kwargs = <optimized out>
        callargs = 0x7fef95885150
        result = <optimized out>
        opcode_targets =
{0x4c29f3 <_PyEval_EvalFrameDefault+29555>, 0x4bbe90 <_PyEval_EvalFrameDefault+2064>, 0x4bbeda <_PyEval_EvalFrameDefault+2138>, 0x4bbf20 <_PyEval_EvalFrameDefault+2208>,           0x4bbfba <_PyEval_EvalFrameDefault+2362>, 0x4bc00f <_PyEval_EvalFrameDefault+2447>, 0x4bbf69 <_PyEval_EvalFrameDefault+2281>, 0x4c29f3 <_PyEval_EvalFrameDefault+29555>,           0x4c29f3 <_PyEval_EvalFrameDefault+29555>, 0x4bbd3f <_PyEval_EvalFrameDefault+1727>, 0x4bc070 <_PyEval_EvalFrameDefault+2544>, 0x4bc0e7 <_PyEval_EvalFrameDefault+2663>,           0x4bc15e <_PyEval_EvalFrameDefault+2782>, 0x4c29f3 <_PyEval_EvalFrameDefault+29555>, 0x4c29f3 <_PyEval_EvalFrameDefault+29555>, 0x4bc23f <_PyEval_EvalFrameDefault+3007>,           0x4bc3e8 <_PyEval_EvalFrameDefault+3432>, 0x4bcda4 <_PyEval_EvalFrameDefault+5924>, 0x4c29f3 <_PyEval_EvalFrameDefault+29555>, 0x4bc2b6 <_PyEval_EvalFrameDefault+3126>,           0x4bc353 <_PyEval_EvalFrameDefault+3283>, 0x4c29f3 <_PyEval_EvalFrameDefault+29555>, 0x4bc5a7 <_PyEval_EvalFrameDefault+3879>, 0x4bc6af <_PyEval_EvalFrameDefault+4143>,           0x4bc76f <_PyEval_EvalFrameDefault+4335>, 0x4bc804 <_PyEval_EvalFrameDefault+4484>, 0x4bc512 <_PyEval_EvalFrameDefault+3730>, 0x4bc47d <_PyEval_EvalFrameDefault+3581>,           0x4bcece <_PyEval_EvalFrameDefault+6222>, 0x4bce39 <_PyEval_EvalFrameDefault+6073>, 0x4c0bbe <_PyEval_EvalFrameDefault+21822>, 0x4c0ff9 <_PyEval_EvalFrameDefault+22905>,           0x4c1067 <_PyEval_EvalFrameDefault+23015>, 0x4c10d5 <_PyEval_EvalFrameDefault+23125>, 0x4c142b <_PyEval_EvalFrameDefault+23979>, 0x4c29f3 <_PyEval_EvalFrameDefault+29555> <repeats 14 times>,           0x4c19fa <_PyEval_EvalFrameDefault+25466>, 0x4bd6de <_PyEval_EvalFrameDefault+8286>, 0x4bd78d <_PyEval_EvalFrameDefault+8461>, 0x4c175b <_PyEval_EvalFrameDefault+24795>,           0x4c29f3 <_PyEval_EvalFrameDefault+29555>, 0x4bdc9f <_PyEval_EvalFrameDefault+9759>, 0x4bd01c <_PyEval_EvalFrameDefault+6556>, 0x4bd0dc <_PyEval_EvalFrameDefault+6748>,           0x4bcd0f <_PyEval_EvalFrameDefault+5775>, 0x4c29f3 <_PyEval_EvalFrameDefault+29555>, 0x4bcf63 <_PyEval_EvalFrameDefault+6371>, 0x4bd45a <_PyEval_EvalFrameDefault+7642>,           0x4bd51b <_PyEval_EvalFrameDefault+7835>, 0x4bc899 <_PyEval_EvalFrameDefault+4633>, 0x4bc92e <_PyEval_EvalFrameDefault+4782>, 0x4bc9c3 <_PyEval_EvalFrameDefault+4931>,           0x4bca58 <_PyEval_EvalFrameDefault+5080>, 0x4bcaed <_PyEval_EvalFrameDefault+5229>, 0x4bcc72 <_PyEval_EvalFrameDefault+5618>, 0x4c14e1 <_PyEval_EvalFrameDefault+24161>,           0x4c1558 <_PyEval_EvalFrameDefault+24280>, 0x4bd5cb <_PyEval_EvalFrameDefault+8011>, 0x4bde54 <_PyEval_EvalFrameDefault+10196>, 0x4bd962 <_PyEval_EvalFrameDefault+8930>,           0x4bd855 <_PyEval_EvalFrameDefault+8661>, 0x4bddfc <_PyEval_EvalFrameDefault+10108>, 0x4bd171 <_PyEval_EvalFrameDefault+6897>, 0x4bd206 <_PyEval_EvalFrameDefault+7046>,           0x4bd29b <_PyEval_EvalFrameDefault+7195>, 0x4bd330 <_PyEval_EvalFrameDefault+7344>, 0x4bd3c5 <_PyEval_EvalFrameDefault+7493>, 0x4c29f3 <_PyEval_EvalFrameDefault+29555>,           0x4c29f3 <_PyEval_EvalFrameDefault+29555>, 0x4bed46 <_PyEval_EvalFrameDefault+14022>, 0x4c2ef5 <_PyEval_EvalFrameDefault+30837>, 0x4c008f <_PyEval_EvalFrameDefault+18959>,           0x4bf0e3 <_PyEval_EvalFrameDefault+14947>, 0x4c2f5b <_PyEval_EvalFrameDefault+30939>, 0x4bdc60 <_PyEval_EvalFrameDefault+9696>, 0x4c29f3 <_PyEval_EvalFrameDefault+29555>,           0x4bdb97 <_PyEval_EvalFrameDefault+9495>, 0x4bdee7 <_PyEval_EvalFrameDefault+10343>, 0x4bdf9a <_PyEval_EvalFrameDefault+10522>, 0x4bdffe <_PyEval_EvalFrameDefault+10622>,           0x4c1604 <_PyEval_EvalFrameDefault+24452>, 0x4be190 <_PyEval_EvalFrameDefault+11024>, 0x4be223 <_PyEval_EvalFrameDefault+11171>, 0x4be2c3 <_PyEval_EvalFrameDefault+11331>,           0x4be342 <_PyEval_EvalFrameDefault+11458>, 0x4be3c3 <_PyEval_EvalFrameDefault+11587>, 0x4c26d1 <_PyEval_EvalFrameDefault+28753>, 0x4bbdcd <_PyEval_EvalFrameDefault+1869>,           0x4be465 <_PyEval_EvalFrameDefault+11749>, 0x4bea95 <_PyEval_EvalFrameDefault+13333>, 0x4bec25 <_PyEval_EvalFrameDefault+13733>, 0x4beec7 <_PyEval_EvalFrameDefault+14407>,           0x4befda <_PyEval_EvalFrameDefault+14682>, 0x4bf953 <_PyEval_EvalFrameDefault+17107>, 0x4bfb3b <_PyEval_EvalFrameDefault+17595>, 0x4bfec6 <_PyEval_EvalFrameDefault+18502>,           0x4c047d <_PyEval_EvalFrameDefault+19965>, 0x4c05a9 <_PyEval_EvalFrameDefault+20265>, 0x4c08f0 <_PyEval_EvalFrameDefault+21104>, 0x4c0a2a <_PyEval_EvalFrameDefault+21418>,           0x4c0b66 <_PyEval_EvalFrameDefault+21734>, 0x4c05e7 <_PyEval_EvalFrameDefault+20327>, 0x4c0764 <_PyEval_EvalFrameDefault+20708>, 0x4be53b <_PyEval_EvalFrameDefault+11963>,           0x4bfbd3 <_PyEval_EvalFrameDefault+17747>, 0x4bfc69 <_PyEval_EvalFrameDefault+17897>, 0x4c2f1b <_PyEval_EvalFrameDefault+30875>, 0x4c29f3 <_PyEval_EvalFrameDefault+29555>,           0x4bfd26 <_PyEval_EvalFrameDefault+18086>, 0x4c1706 <_PyEval_EvalFrameDefault+24710>, 0x4c29f3 <_PyEval_EvalFrameDefault+29555>, 0x4bbd76 <_PyEval_EvalFrameDefault+1782>,           0x4bbe2e <_PyEval_EvalFrameDefault+1966>, 0x4be715 <_PyEval_EvalFrameDefault+12437>, 0x4c29f3 <_PyEval_EvalFrameDefault+29555>, 0x4c29f3 <_PyEval_EvalFrameDefault+29555>,           0x4bdb4d <_PyEval_EvalFrameDefault+9421>, 0x4c2ec8 <_PyEval_EvalFrameDefault+30792>, 0x4c1c7d <_PyEval_EvalFrameDefault+26109>, 0x4bbc00 <_PyEval_EvalFrameDefault+1408>,           0x4c2502 <_PyEval_EvalFrameDefault+28290>, 0x4c29f3 <_PyEval_EvalFrameDefault+29555>, 0x4be7dd <_PyEval_EvalFrameDefault+12637>, 0x4be912 <_PyEval_EvalFrameDefault+12946>,           0x4be971 <_PyEval_EvalFrameDefault+13041>, 0x4be777 <_PyEval_EvalFrameDefault+12535>, 0x4c29f3 <_PyEval_EvalFrameDefault+29555>, 0x4c29f3 <_PyEval_EvalFrameDefault+29555>,           0x4c1d13 <_PyEval_EvalFrameDefault+26259>, 0x4c1dcf <_PyEval_EvalFrameDefault+26447>, 0x4c18cc <_PyEval_EvalFrameDefault+25164>, 0x4c2750 <_PyEval_EvalFrameDefault+28880>,           0x4bcb82 <_PyEval_EvalFrameDefault+5378>, 0x4bcbfa <_PyEval_EvalFrameDefault+5498>, 0x4bf45b <_PyEval_EvalFrameDefault+15835>, 0x4be83d <_PyEval_EvalFrameDefault+12733>,           0x4c29f3 <_PyEval_EvalFrameDefault+29555>, 0x4c29f3 <_PyEval_EvalFrameDefault+29555>, 0x4c29f3 <_PyEval_EvalFrameDefault+29555>, 0x4c0c25 <_PyEval_EvalFrameDefault+21925>,           0x4c29f3 <_PyEval_EvalFrameDefault+29555>, 0x4c1860 <_PyEval_EvalFrameDefault+25056>, 0x4c25d0 <_PyEval_EvalFrameDefault+28496>, 0x4bf239 <_PyEval_EvalFrameDefault+15289>,           0x4be9d7 <_PyEval_EvalFrameDefault+13143>, 0x4c29f3 <_PyEval_EvalFrameDefault+29555>, 0x4c29f3 <_PyEval_EvalFrameDefault+29555>, 0x4c1af3 <_PyEval_EvalFrameDefault+25715>,           0x4c1b9f <_PyEval_EvalFrameDefault+25887>, 0x4bedb8 <_PyEval_EvalFrameDefault+14136>, 0x4bee4f <_PyEval_EvalFrameDefault+14287>, 0x4bf3e1 <_PyEval_EvalFrameDefault+15713>,           0x4bf36e <_PyEval_EvalFrameDefault+15598>, 0x4c29f3 <_PyEval_EvalFrameDefault+29555> <repeats 90 times>}
        PyId_displayhook = {string = 0x64aae2 "displayhook", index = -1}
        PyId_send = {string = 0x6a362a "send", index = -1}
        PyId_{}build_class{} = {string = 0x64ac69 "{}build_class{}", index = 18}
{}Type <RET> for more, q to quit, c to continue without paging{}
        PyId_{}annotations{} = {string = 0x655175 "{}annotations{}", index = -1}
        PyId_{}aenter{} = {string = 0x64ae53 "{}aenter{}", index = -1}
        PyId_{}aexit{} = {string = 0x64ae5e "{}aexit{}", index = -1}
        PyId_{}enter{} = {string = 0x606990 "{}enter{}", index = 40}
        PyId_{}exit{} = {string = 0x60699a "{}exit{}", index = 41}
        trace_info = {code = 0x0, bounds = {ar_start = 15, ar_end = 0, ar_line = -1767671200, opaque = {computed_line = -1767152416, lo_next = 0x7ffdcda1e830 "\220\364\245\226\357\177",
              limit = 0x5989e3 <frame_dealloc+163> "\353\333A\213D$P\205\300~>1\333\353\032f.\017\037\204"}}, cframe = {use_tracing = 0, previous = 0x7ffdcda1ea40}}
        _unknown_opcode = <optimized out>
#32 0x00000000004bb57d in _PyEval_EvalFrame (tstate=0x1a4acd0, f=0x7fef96a5f490, throwflag=0) at ./Include/internal/pycore_ceval.h:46
No locals.
#33 _PyEval_Vector (tstate=0x1a4acd0, con=0x7fef96a79520, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:5065
        f = 0x7fef96a5f490
        retval = <optimized out>
#34 0x00000000004c4823 in call_function (tstate=<optimized out>, tstate@entry=0x1a4acd0, trace_info=trace_info@entry=0x7ffdcda1ea10, pp_stack=pp_stack@entry=0x7ffdcda1eac8, oparg=<optimized out>,
    oparg@entry=27569360, kwnames=0xffffffff, kwnames@entry=0x0) at ./Include/cpython/abstract.h:115
        pfunc = 0x7fef96ad5140
        func = 0x7fef96a79510
        nkwargs = <optimized out>
        nargs = <optimized out>
        stack = 0x7fef96ad5148
        x = <optimized out>
        w = <optimized out>
#35 0x00000000004c1c0b in _PyEval_EvalFrameDefault (tstate=<optimized out>, f=0x7fef96ad4fc0, throwflag=<optimized out>) at Python/ceval.c:4181
        sp = 0x7fef96ad5158
        meth = <optimized out>
        res = <optimized out>
        opcode_targets =
{0x4c29f3 <_PyEval_EvalFrameDefault+29555>, 0x4bbe90 <_PyEval_EvalFrameDefault+2064>, 0x4bbeda <_PyEval_EvalFrameDefault+2138>, 0x4bbf20 <_PyEval_EvalFrameDefault+2208>,           0x4bbfba <_PyEval_EvalFrameDefault+2362>, 0x4bc00f <_PyEval_EvalFrameDefault+2447>, 0x4bbf69 <_PyEval_EvalFrameDefault+2281>, 0x4c29f3 <_PyEval_EvalFrameDefault+29555>,           0x4c29f3 <_PyEval_EvalFrameDefault+29555>, 0x4bbd3f <_PyEval_EvalFrameDefault+1727>, 0x4bc070 <_PyEval_EvalFrameDefault+2544>, 0x4bc0e7 <_PyEval_EvalFrameDefault+2663>,           0x4bc15e <_PyEval_EvalFrameDefault+2782>, 0x4c29f3 <_PyEval_EvalFrameDefault+29555>, 0x4c29f3 <_PyEval_EvalFrameDefault+29555>, 0x4bc23f <_PyEval_EvalFrameDefault+3007>,           0x4bc3e8 <_PyEval_EvalFrameDefault+3432>, 0x4bcda4 <_PyEval_EvalFrameDefault+5924>, 0x4c29f3 <_PyEval_EvalFrameDefault+29555>, 0x4bc2b6 <_PyEval_EvalFrameDefault+3126>,           0x4bc353 <_PyEval_EvalFrameDefault+3283>, 0x4c29f3 <_PyEval_EvalFrameDefault+29555>, 0x4bc5a7 <_PyEval_EvalFrameDefault+3879>, 0x4bc6af <_PyEval_EvalFrameDefault+4143>,           0x4bc76f <_PyEval_EvalFrameDefault+4335>, 0x4bc804 <_PyEval_EvalFrameDefault+4484>, 0x4bc512 <_PyEval_EvalFrameDefault+3730>, 0x4bc47d <_PyEval_EvalFrameDefault+3581>,           0x4bcece <_PyEval_EvalFrameDefault+6222>, 0x4bce39 <_PyEval_EvalFrameDefault+6073>, 0x4c0bbe <_PyEval_EvalFrameDefault+21822>, 0x4c0ff9 <_PyEval_EvalFrameDefault+22905>,           0x4c1067 <_PyEval_EvalFrameDefault+23015>, 0x4c10d5 <_PyEval_EvalFrameDefault+23125>, 0x4c142b <_PyEval_EvalFrameDefault+23979>, 0x4c29f3 <_PyEval_EvalFrameDefault+29555> <repeats 14 times>,           0x4c19fa <_PyEval_EvalFrameDefault+25466>, 0x4bd6de <_PyEval_EvalFrameDefault+8286>, 0x4bd78d <_PyEval_EvalFrameDefault+8461>, 0x4c175b <_PyEval_EvalFrameDefault+24795>,           0x4c29f3 <_PyEval_EvalFrameDefault+29555>, 0x4bdc9f <_PyEval_EvalFrameDefault+9759>, 0x4bd01c <_PyEval_EvalFrameDefault+6556>, 0x4bd0dc <_PyEval_EvalFrameDefault+6748>,           0x4bcd0f <_PyEval_EvalFrameDefault+5775>, 0x4c29f3 <_PyEval_EvalFrameDefault+29555>, 0x4bcf63 <_PyEval_EvalFrameDefault+6371>, 0x4bd45a <_PyEval_EvalFrameDefault+7642>,           0x4bd51b <_PyEval_EvalFrameDefault+7835>, 0x4bc899 <_PyEval_EvalFrameDefault+4633>, 0x4bc92e <_PyEval_EvalFrameDefault+4782>, 0x4bc9c3 <_PyEval_EvalFrameDefault+4931>,           0x4bca58 <_PyEval_EvalFrameDefault+5080>, 0x4bcaed <_PyEval_EvalFrameDefault+5229>, 0x4bcc72 <_PyEval_EvalFrameDefault+5618>, 0x4c14e1 <_PyEval_EvalFrameDefault+24161>,           0x4c1558 <_PyEval_EvalFrameDefault+24280>, 0x4bd5cb <_PyEval_EvalFrameDefault+8011>, 0x4bde54 <_PyEval_EvalFrameDefault+10196>, 0x4bd962 <_PyEval_EvalFrameDefault+8930>,           0x4bd855 <_PyEval_EvalFrameDefault+8661>, 0x4bddfc <_PyEval_EvalFrameDefault+10108>, 0x4bd171 <_PyEval_EvalFrameDefault+6897>, 0x4bd206 <_PyEval_EvalFrameDefault+7046>,           0x4bd29b <_PyEval_EvalFrameDefault+7195>, 0x4bd330 <_PyEval_EvalFrameDefault+7344>, 0x4bd3c5 <_PyEval_EvalFrameDefault+7493>, 0x4c29f3 <_PyEval_EvalFrameDefault+29555>,           0x4c29f3 <_PyEval_EvalFrameDefault+29555>, 0x4bed46 <_PyEval_EvalFrameDefault+14022>, 0x4c2ef5 <_PyEval_EvalFrameDefault+30837>, 0x4c008f <_PyEval_EvalFrameDefault+18959>,           0x4bf0e3 <_PyEval_EvalFrameDefault+14947>, 0x4c2f5b <_PyEval_EvalFrameDefault+30939>, 0x4bdc60 <_PyEval_EvalFrameDefault+9696>, 0x4c29f3 <_PyEval_EvalFrameDefault+29555>,           0x4bdb97 <_PyEval_EvalFrameDefault+9495>, 0x4bdee7 <_PyEval_EvalFrameDefault+10343>, 0x4bdf9a <_PyEval_EvalFrameDefault+10522>, 0x4bdffe <_PyEval_EvalFrameDefault+10622>,           0x4c1604 <_PyEval_EvalFrameDefault+24452>, 0x4be190 <_PyEval_EvalFrameDefault+11024>, 0x4be223 <_PyEval_EvalFrameDefault+11171>, 0x4be2c3 <_PyEval_EvalFrameDefault+11331>, --Type <RET> for more, q to quit, c to continue without paging--           0x4be342 <_PyEval_EvalFrameDefault+11458>, 0x4be3c3 <_PyEval_EvalFrameDefault+11587>, 0x4c26d1 <_PyEval_EvalFrameDefault+28753>, 0x4bbdcd <_PyEval_EvalFrameDefault+1869>,           0x4be465 <_PyEval_EvalFrameDefault+11749>, 0x4bea95 <_PyEval_EvalFrameDefault+13333>, 0x4bec25 <_PyEval_EvalFrameDefault+13733>, 0x4beec7 <_PyEval_EvalFrameDefault+14407>,           0x4befda <_PyEval_EvalFrameDefault+14682>, 0x4bf953 <_PyEval_EvalFrameDefault+17107>, 0x4bfb3b <_PyEval_EvalFrameDefault+17595>, 0x4bfec6 <_PyEval_EvalFrameDefault+18502>,           0x4c047d <_PyEval_EvalFrameDefault+19965>, 0x4c05a9 <_PyEval_EvalFrameDefault+20265>, 0x4c08f0 <_PyEval_EvalFrameDefault+21104>, 0x4c0a2a <_PyEval_EvalFrameDefault+21418>,           0x4c0b66 <_PyEval_EvalFrameDefault+21734>, 0x4c05e7 <_PyEval_EvalFrameDefault+20327>, 0x4c0764 <_PyEval_EvalFrameDefault+20708>, 0x4be53b <_PyEval_EvalFrameDefault+11963>,           0x4bfbd3 <_PyEval_EvalFrameDefault+17747>, 0x4bfc69 <_PyEval_EvalFrameDefault+17897>, 0x4c2f1b <_PyEval_EvalFrameDefault+30875>, 0x4c29f3 <_PyEval_EvalFrameDefault+29555>,           0x4bfd26 <_PyEval_EvalFrameDefault+18086>, 0x4c1706 <_PyEval_EvalFrameDefault+24710>, 0x4c29f3 <_PyEval_EvalFrameDefault+29555>, 0x4bbd76 <_PyEval_EvalFrameDefault+1782>,           0x4bbe2e <_PyEval_EvalFrameDefault+1966>, 0x4be715 <_PyEval_EvalFrameDefault+12437>, 0x4c29f3 <_PyEval_EvalFrameDefault+29555>, 0x4c29f3 <_PyEval_EvalFrameDefault+29555>,           0x4bdb4d <_PyEval_EvalFrameDefault+9421>, 0x4c2ec8 <_PyEval_EvalFrameDefault+30792>, 0x4c1c7d <_PyEval_EvalFrameDefault+26109>, 0x4bbc00 <_PyEval_EvalFrameDefault+1408>,           0x4c2502 <_PyEval_EvalFrameDefault+28290>, 0x4c29f3 <_PyEval_EvalFrameDefault+29555>, 0x4be7dd <_PyEval_EvalFrameDefault+12637>, 0x4be912 <_PyEval_EvalFrameDefault+12946>,           0x4be971 <_PyEval_EvalFrameDefault+13041>, 0x4be777 <_PyEval_EvalFrameDefault+12535>, 0x4c29f3 <_PyEval_EvalFrameDefault+29555>, 0x4c29f3 <_PyEval_EvalFrameDefault+29555>,           0x4c1d13 <_PyEval_EvalFrameDefault+26259>, 0x4c1dcf <_PyEval_EvalFrameDefault+26447>, 0x4c18cc <_PyEval_EvalFrameDefault+25164>, 0x4c2750 <_PyEval_EvalFrameDefault+28880>,           0x4bcb82 <_PyEval_EvalFrameDefault+5378>, 0x4bcbfa <_PyEval_EvalFrameDefault+5498>, 0x4bf45b <_PyEval_EvalFrameDefault+15835>, 0x4be83d <_PyEval_EvalFrameDefault+12733>,           0x4c29f3 <_PyEval_EvalFrameDefault+29555>, 0x4c29f3 <_PyEval_EvalFrameDefault+29555>, 0x4c29f3 <_PyEval_EvalFrameDefault+29555>, 0x4c0c25 <_PyEval_EvalFrameDefault+21925>,           0x4c29f3 <_PyEval_EvalFrameDefault+29555>, 0x4c1860 <_PyEval_EvalFrameDefault+25056>, 0x4c25d0 <_PyEval_EvalFrameDefault+28496>, 0x4bf239 <_PyEval_EvalFrameDefault+15289>,           0x4be9d7 <_PyEval_EvalFrameDefault+13143>, 0x4c29f3 <_PyEval_EvalFrameDefault+29555>, 0x4c29f3 <_PyEval_EvalFrameDefault+29555>, 0x4c1af3 <_PyEval_EvalFrameDefault+25715>,           0x4c1b9f <_PyEval_EvalFrameDefault+25887>, 0x4bedb8 <_PyEval_EvalFrameDefault+14136>, 0x4bee4f <_PyEval_EvalFrameDefault+14287>, 0x4bf3e1 <_PyEval_EvalFrameDefault+15713>,           0x4bf36e <_PyEval_EvalFrameDefault+15598>, 0x4c29f3 <_PyEval_EvalFrameDefault+29555> <repeats 90 times>}
        PyId_displayhook = {string = 0x64aae2 "displayhook", index = -1}
        PyId_send = {string = 0x6a362a "send", index = -1}
        PyId_{}build_class{} = {string = 0x64ac69 "{}build_class{}", index = 18}
        PyId_{}annotations{} = {string = 0x655175 "{}annotations{}", index = -1}
        PyId_{}aenter{} = {string = 0x64ae53 "{}aenter{}", index = -1}
        PyId_{}aexit{} = {string = 0x64ae5e "{}aexit{}", index = -1}
        PyId_{}enter{} = {string = 0x606990 "{}enter{}", index = 40}
        PyId_{}exit{} = {string = 0x60699a "{}exit{}", index = 41}
        trace_info = {code = 0x0, bounds = {ar_start = 64, ar_end = 0, ar_line = -1786174976, opaque = {computed_line = 28322560, lo_next = 0x725a70 <PyMethod_Type> "\026", limit = 0x7fef95884eb0 "\003"}},
          cframe = {use_tracing = 0, previous = 0x7ffdcda1ec90}}
        _unknown_opcode = <optimized out>
#36 0x00000000004bb57d in _PyEval_EvalFrame (tstate=0x1a4acd0, f=0x7fef96ad4fc0, throwflag=0) at ./Include/internal/pycore_ceval.h:46
No locals.
#37 _PyEval_Vector (tstate=0x1a4acd0, con=0x7fef96afe600, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:5065
        f = 0x7fef96ad4fc0
        retval = <optimized out>

It seems both my python binding and pyarrow uses arrow's code to do some initialization, the arrow::io::internal::GetIOThreadPool for some reason fails to work in this case and caused the segfault.

 

{}4) Since it is related with ::arrow::internal::GetEnvVar("ARROW_IO_THREADS");{}, I tried setting the ARROW_IO_THREADS env var to a number and gave it another try. And the program still caused a segfault, but reporting a different error (there is no free(): invalid pointer printed this time):

Fatal Python error: Segmentation fault
Current thread 0x00007fea55bd09c0 (most recent call first):
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1176 in create_module
  File "<frozen importlib._bootstrap>", line 571 in module_from_spec
  File "<frozen importlib._bootstrap>", line 674 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1006 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1027 in _find_and_load
  File "/root/.cache/pypoetry/virtualenvs/<redated>-sOknH4PM-py3.10/lib/python3.10/site-packages/pyarrow/{}init{}.py", line 65 in <module>
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 883 in exec_module
  File "<frozen importlib._bootstrap>", line 688 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1006 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1027 in _find_and_load
  File "/usr/src/stonewave/bindings/python/<redated>/<redated>/search.py", line 16 in search

{}5) I used gdb to view the core dump again{}, and it reported error like below:

#0  raise (sig=<optimized out>) at ../sysdeps/unix/sysv/linux/raise.c:50
        set = {__val = {0 <repeats 16 times>}}
        pid = <optimized out>
        tid = <optimized out>
        ret = <optimized out>
#1  <signal handler called>
No locals.
#2  0x00007fea5d4a60bb in std::{}shared_ptr<arrow::DataType, ({}gnu_cxx::Lock_policy)2>::_shared_ptr (this=0x5)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/bits/shared_ptr_base.h:1181
No locals.
#3  0x00007fea5d4a6093 in std::shared_ptr<arrow::DataType>::shared_ptr (this=0x5) at /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/bits/shared_ptr.h:149
No locals.
#4  0x00007fea5d9da822 in arrow::int32 () at /vcpkg/buildtrees/arrow/src/c4eb0a0543-8e1ffd6d16.clean/cpp/src/arrow/type.cc:2165
        result = warning: RTTI symbol not found for class 'std::Sp_counted_ptr_inplace<arrow::Int32Type, std::allocator<arrow::Int32Type>, (_gnu_cxx::_Lock_policy)2>'
warning: RTTI symbol not found for class 'std::Sp_counted_ptr_inplace<arrow::Int32Type, std::allocator<arrow::Int32Type>, (_gnu_cxx::_Lock_policy)2>'
std::shared_ptr<arrow::DataType> (use count 1053, weak count 0) = {get() = 0x108ba70}
#5  0x00007fea364ff4af in GLOBAL_sub_I_unity_1_cxx.cxx () from /root/.cache/pypoetry/virtualenvs/<redated>-sOknH4PM-py3.10/lib/python3.10/site-packages/pyarrow/libarrow_dataset.so.1000
No symbol table info available.
#6  0x00007fea6b82ffe2 in call_init (l=<optimized out>, argc=argc@entry=5, argv=argv@entry=0x7fff106aae38, env=env@entry=0x168fd50) at dl-init.c:72
        j = 0
        jm = <optimized out>
        addrs = <optimized out>
        init_array = <optimized out>
#7  0x00007fea6b8300e9 in call_init (env=0x168fd50, argv=0x7fff106aae38, argc=5, l=<optimized out>) at dl-init.c:30
        init_array = <optimized out>
        j = <optimized out>
        jm = <optimized out>
        addrs = <optimized out>
#8  _dl_init (main_map=0x1918190, argc=5, argv=0x7fff106aae38, env=0x168fd50) at dl-init.c:119
        preinit_array = <optimized out>
        preinit_array_size = <optimized out>
        i = <optimized out>
#9  0x00007fea575a2aed in {}GI{_}_dl_catch_exception (exception=exception@entry=0x0, operate=operate@entry=0x7fea6b8335d0 <call_dl_init>, args=args@entry=0x7fff106a1cd0)
    at dl-error-skeleton.c:182
        old = <optimized out>
        errcode = 0
        c = {exception = 0x7fff106a1c30, errcode = 0x0, env = {{__jmpbuf =
{140733468777520, 0, 0, 140643321125720, 26313976, 140643321198124, 140643321095744, 140643323642840}
,
              {}mask_was_saved = 322015232, _saved_mask = {_val =
{0, 140733468777616, 140644478215584, 0, 0, 0, 0, 0, 0, 0, 0, 12468370135039750697, 140733468778096, 140733468777584,                   140733468778096, 140733468777760}
}}}}
        old = <optimized out>
#10 0x00007fea6b834058 in dl_open_worker (a=a@entry=0x7fff106a1e70) at dl-open.c:758
        init_args = {new = 0x1918190, argc = 5, argv = 0x7fff106aae38, env = 0x168fd50}
        args = 0x7fff106a1e70
        file = <optimized out>
        mode = <optimized out>
        call_map = <optimized out>
        dst = <optimized out>
        new = 0x1918190
        {}PRETTY_FUNCTION{} = "dl_open_worker"
        r = <optimized out>
{}Type <RET> for more, q to quit, c to continue without paging{}
        reloc_mode = <optimized out>
        nmaps = <optimized out>
        l = <optimized out>
        maps = <error reading variable maps (value requires 138793856 bytes, which is more than max-value-size)>
        relocation_in_progress = <optimized out>
        any_tls = <optimized out>
#11 0x00007fea575a2a90 in {}GI{_}_dl_catch_exception (exception=exception@entry=0x7fff106a1e50, operate=operate@entry=0x7fea6b833ca0 <dl_open_worker>, args=args@entry=0x7fff106a1e70)
    at dl-error-skeleton.c:208
        errcode = 0
        c = {exception = 0x7fff106a1e50, errcode = 0x7fff106a1d5c, env = {{__jmpbuf =
{2147483650, 1061564265845182786, -74376, 5, 5369100, -2, 1049880826869928258, 1061551455913726274}
,
              {}mask_was_saved = 0, _saved_mask = {_val =
{140644403015840, 140644403015840, 4506233, 140644402510488, 15, 140644402510432, 140644403015888, 140733468778048,                   140733468778080, 140733468779105, 10865718445837529600, 140733468778848, 0, 257, 140733468778080, 6671076}
}}}}
        old = 0x7fff106a1f40
#12 0x00007fea6b8338fa in _dl_open (file=0x7fea5165efc0 "/root/.cache/pypoetry/virtualenvs/<redated>-sOknH4PM-py3.10/lib/python3.10/site-packages/pyarrow/lib.cpython-310-x86_64-linux-gnu.so",
    mode=-2147483646, caller_dlopen=0x51ed0c <_PyImport_FindSharedFuncptr+284>, nsid=-2, argc=5, argv=0x7fff106a1e50, env=0x168fd50) at dl-open.c:837
        args =
{file = 0x7fea5165efc0 "/root/.cache/pypoetry/virtualenvs/<redated>-sOknH4PM-py3.10/lib/python3.10/site-packages/pyarrow/lib.cpython-310-x86_64-linux-gnu.so",           mode = -2147483646, caller_dlopen = 0x51ed0c <_PyImport_FindSharedFuncptr+284>, map = 0x1918190, nsid = 0, original_global_scope_pending_adds = 0, argc = 5,           argv = 0x7fff106aae38, env = 0x168fd50}
        exception =
{objname = 0x439830 <list_richcompare> "UAWAVAUATSPH\213G\b\366\200\253", errstring = 0x7fff106a1f50 "8Y\b\001",           message_buffer = 0xfbad8001 <error: Cannot access memory at address 0xfbad8001>}
        errcode = <optimized out>
        {}PRETTY_FUNCTION{} = "_dl_open"
#13 0x00007fea5778d258 in dlopen_doit (a=a@entry=0x7fff106a2090) at dlopen.c:66
        args = 0x7fff106a2090
#14 0x00007fea575a2a90 in {}GI{_}_dl_catch_exception (exception=exception@entry=0x7fff106a2030, operate=operate@entry=0x7fea5778d200 <dlopen_doit>, args=args@entry=0x7fff106a2090)
    at dl-error-skeleton.c:208
        errcode = 32746
        c = {exception = 0x7fff106a2030, errcode = 0x7fff106a1f3c, env = {{__jmpbuf =
{17324344, -1049391729601195710, -74376, 140644364656960, 140644364656832, 140644364709824,                 1049880826807013698, 1061551455913726274}
, {}mask_was_saved = 0, _saved_mask = {_val =
{10865718445837529600, 9223372036854775807, 258, 0, 0, 140644364656960,                   140733468778848, 140644364709824, 5216203, 206158430248, 140733468778704, 140733468778496, 140644402549736, 140644402549736, 2, 140644402612032}
}}}}
        old = 0x0
#15 0x00007fea575a2b4f in {}GI{_}_dl_catch_error (objname=objname@entry=0x1085940, errstring=errstring@entry=0x1085948, mallocedp=mallocedp@entry=0x1085938,
    operate=operate@entry=0x7fea5778d200 <dlopen_doit>, args=args@entry=0x7fff106a2090) at dl-error-skeleton.c:227
        exception = {objname = 0x0, errstring = 0x2 <error: Cannot access memory at address 0x2>, message_buffer = 0x1 <error: Cannot access memory at address 0x1>}
        errorcode = <optimized out>
#16 0x00007fea5778da65 in _dlerror_run (operate=operate@entry=0x7fea5778d200 <dlopen_doit>, args=args@entry=0x7fff106a2090) at dlerror.c:170
        result = 0x1085930
#17 0x00007fea5778d2e4 in __dlopen (file=<optimized out>, mode=<optimized out>) at dlopen.c:87
        args =
{file = 0x7fea5165efc0 "/root/.cache/pypoetry/virtualenvs/<redated>-sOknH4PM-py3.10/lib/python3.10/site-packages/pyarrow/lib.cpython-310-x86_64-linux-gnu.so", mode = 2,           new = 0x11f6700, caller = 0x51ed0c <_PyImport_FindSharedFuncptr+284>}
#18 0x000000000051ed0c in _PyImport_FindSharedFuncptr (prefix=0x64fb3d "PyInit", shortname=0x7fea516520c0 "lib",
    pathname=0x7fea5165efc0 "/root/.cache/pypoetry/virtualenvs/<redated>-sOknH4PM-py3.10/lib/python3.10/site-packages/pyarrow/lib.cpython-310-x86_64-linux-gnu.so", fp=0x0)
    at ./Python/dynload_shlib.c:100
        funcname = "PyInit_lib", '\000' <repeats 22 times>, "\320\315qR\352\177\000\000\000\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000\001\000\000\000\001", '\000' <repeats 43 times>, "O\373d\000\000\000\000\000\000g\037\001\000\000\000\000\200sfQ\352\177\000\000P\"j\020\377\177\000\000\002\000\000\000\000\000\000\000P\"j\020\377\177\000\000\000\000\000\000\000\000\000\000V\357Y\000\000\000\000\000\240\356Y\000\000\000\000\000\240 eQ\352\177\000\000\070\067hQ\352\177\000\000"...
        pathbuf = "\002\000\000\000\000\000\000\000\264\212\035\305yz\306d\000\021\260S\352\177\000\000\236\272D\000\000\000\000\000\230\272\246S\352\177\000\000\000\000\000\000\000\000\000\000p\036hQ\352\177\000\000@!eQ\352\177\000\000p;xQ\352\177\000\000\240 eQ\352\177\000\000\000\000\000\000\000\000\000\000\023\305P\000\000\000\000\000\020\000\000\000\060\000\000\000\260#j\0
20\377\177\000\000\360\"j\020\377\177\000\000@!eQ\352\177\000\000\000\000\000\000\000\000\000\000Q\274D\000\000\000\000\000p\036hQ\352\177\000\000p;xQ\352\177\000\000\340\251p\000\000\000\000\000\340\251p\000\000\000\000\000\360\071j\000\000\000\000\000\003", '\000' <repeats 15 times>...
        dlopenflags = 1799967928
        handle = <optimized out>
        p = <optimized out>
#19 0x00000000004eaa4e in _PyImport_LoadDynamicModuleWithSpec (spec=0x7fea51652140, fp=0x0) at ./Python/importdl.c:137
        m = 0x0
        path = <optimized out>
        name = 0x7fea516520a0
        name_unicode = 0x7fea51681e70
        pathbytes = 0x7fea5165efa0
        hook_prefix = 0x64fb3d "PyInit"
        name_buf = 0x7fea516520c0 "lib"
        exportfunc = <optimized out>
        p0 = <optimized out>
        oldcontext = <optimized out>
        def = <optimized out>
        modules = <optimized out>

It is different with previously core dump, but I have no idea how this happens now. And the behavior is pretty consistent, if I unset the ARROW_IO_THREADS env var and run the program again, the previous core dump will be generated again.

 

Summary:

I don't know how to proceed any more. Basically:

1. I create a C++ library with arrow C++ library

2. I create a python binding for the C++ library using pybind

3. Instead of importing pyarrow.h header, I now import <arrow/c/abi.h> and <arrow/c/bridge.h> for bridging

4. In my python program, I use both my python binding and pyarrow wheel

Could you please shed some light on this issue? Do you think if this kind usage is supported or if it is some bug for arrow project?