[C++][Python] Casting extension types with list-based storage segfaults

[spenczar]

Describe the bug, including details regarding any error messages, version, and platform.

I'm taking a table and casting it to its own schema. This should always work, but it errors in a certain case.

(this is deep in some library code - please forgive how strange this scenario is; just trust me that it really does happen!)

# An extension type which wraps list-of-int32:
class MyExtensionType(pa.PyExtensionType):
    def __init__(self):
        pa.PyExtensionType.__init__(self, pa.list_(pa.int32()))
    def __reduce__(self):
        return MyExtensionType, (),

# an array of this type:
ext_array = pa.array([[1, 2, 3]], MyExtensionType())
# Wrap that in a struct:
struct_array = pa.StructArray.from_arrays([ext_array], "x")
# Wrap the struct in a table:
table = pa.Table.from_arrays([struct_array], ["field1"])

All of the above is fine, and then:

table.cast(table.schema)

Blows up:

File ~/.pyenv/versions/3.10.10/envs/general-3.10/lib/python3.10/site-packages/pyarrow/table.pxi:3616, in pyarrow.lib.Table.cast()

File ~/.pyenv/versions/3.10.10/envs/general-3.10/lib/python3.10/site-packages/pyarrow/table.pxi:3798, in pyarrow.lib.Table.from_arrays()

File ~/.pyenv/versions/3.10.10/envs/general-3.10/lib/python3.10/site-packages/pyarrow/table.pxi:2962, in pyarrow.lib.Table.validate()

File ~/.pyenv/versions/3.10.10/envs/general-3.10/lib/python3.10/site-packages/pyarrow/error.pxi:100, in pyarrow.lib.check_status()

ArrowInvalid: Column 0: In chunk 0: Invalid: Struct child array #0 invalid: Invalid: Expected 1 child arrays in array of type list<item: int32>, got 0

I can print table just fine:

pyarrow.Table
field1: struct<x: extension<arrow.py_extension_type<MyExtensionType>>>
  child 0, x: extension<arrow.py_extension_type<MyExtensionType>>
----
field1: [
  -- is_valid: all not null
  -- child 0 type: extension<arrow.py_extension_type<MyExtensionType>>
[[1,2,3]]]

This is pyarrow version 12.0.1.

Component(s)

Python

[spenczar]

This seems to happen as well for FixedShapeTensor tensor types, which indicates it happens for fixed-size list types.

[AlenkaF]

I can reproduce the first issue but cannot reproduce it for FixedShapeTensor array:

>>> import pyarrow as pa
>>> import pyarrow as pa

>>> tensor_type = pa.fixed_shape_tensor(pa.int32(), (2, 2))
>>> arr = [[1, 2, 3, 4], [10, 20, 30, 40], [100, 200, 300, 400]]
>>> storage = pa.array(arr, pa.list_(pa.int32(), 4))
>>> tensor_array = pa.ExtensionArray.from_storage(tensor_type, storage)

>>> data = [
...     pa.array([1, 2, 3]),
...     pa.array(['foo', 'bar', None]),
...     tensor_array,
... ]
>>> my_schema = pa.schema([('f0', pa.int8()),
...                        ('f1', pa.string()),
...                        ('tensors_int', tensor_type)])
>>> table = pa.Table.from_arrays(data, schema=my_schema)

>>> table.cast(table.schema)
pyarrow.Table
f0: int8
f1: string
tensors_int: extension<arrow.fixed_shape_tensor>
----
f0: [[1,2,3]]
f1: [["foo","bar",null]]
tensors_int: [[[1,2,3,4],[10,20,30,40],[100,200,300,400]]]

I am not sure if the issue is in the struct or the underlying list type. But it seems to me that the casting of the column in the table.cast is behaving erroneous

arrow/python/pyarrow/table.pxi

Line 3694 in 4fc2731

casted = column.cast(field.type, safe=safe, options=options)

and I think this should work but it segfaults:

>>> struct_array = pa.StructArray.from_arrays([ext_array], "x")               
>>> struct_array.type
StructType(struct<x: extension<arrow.py_extension_type<MyExtensionType>>>)

>>> struct_array.cast(struct_array.type)
/Users/alenkafrim/repos/arrow/cpp/src/arrow/array/array_nested.cc:210:  Check failed: (data->child_data.size()) == (1) 
0   libarrow.1300.0.0.dylib             0x0000000142366294 _ZN5arrow4util7CerrLog14PrintBackTraceEv + 52
1   libarrow.1300.0.0.dylib             0x000000014236623c _ZN5arrow4util7CerrLogD2Ev + 120
2   libarrow.1300.0.0.dylib             0x0000000142366184 _ZN5arrow4util7CerrLogD1Ev + 28
3   libarrow.1300.0.0.dylib             0x00000001423661b0 _ZN5arrow4util7CerrLogD0Ev + 28
4   libarrow.1300.0.0.dylib             0x0000000142366078 _ZN5arrow4util8ArrowLogD2Ev + 112
5   libarrow.1300.0.0.dylib             0x00000001423660bc _ZN5arrow4util8ArrowLogD1Ev + 28
6   libarrow.1300.0.0.dylib             0x0000000141d4f5f4 _ZN5arrow8internal11SetListDataINS_8ListTypeEEEvPNS_13BaseListArrayIT_EERKNSt3__110shared_ptrINS_9ArrayDataEEENS_4Type4typeE + 536
7   libarrow.1300.0.0.dylib             0x0000000141d4eee8 _ZN5arrow9ListArray7SetDataERKNSt3__110shared_ptrINS_9ArrayDataEEE + 36
8   libarrow.1300.0.0.dylib             0x0000000141d4ee34 _ZN5arrow9ListArrayC2ENSt3__110shared_ptrINS_9ArrayDataEEE + 68
9   libarrow.1300.0.0.dylib             0x0000000141d4ef58 _ZN5arrow9ListArrayC1ENSt3__110shared_ptrINS_9ArrayDataEEE + 28
10  libarrow.1300.0.0.dylib             0x0000000141ec14bc _ZNSt3__120__shared_ptr_emplaceIN5arrow9ListArrayENS_9allocatorIS2_EEEC2B6v15006IJRKNS_10shared_ptrINS1_9ArrayDataEEEEEES4_DpOT_ + 124
11  libarrow.1300.0.0.dylib             0x0000000141ec1430 _ZNSt3__120__shared_ptr_emplaceIN5arrow9ListArrayENS_9allocatorIS2_EEEC1B6v15006IJRKNS_10shared_ptrINS1_9ArrayDataEEEEEES4_DpOT_ + 36
12  libarrow.1300.0.0.dylib             0x0000000141ec13b0 _ZNSt3__115allocate_sharedB6v15006IN5arrow9ListArrayENS_9allocatorIS2_EEJRKNS_10shared_ptrINS1_9ArrayDataEEEEvEENS5_IT_EERKT0_DpOT1_ + 60
13  libarrow.1300.0.0.dylib             0x0000000141ec1368 _ZNSt3__111make_sharedB6v15006IN5arrow9ListArrayEJRKNS_10shared_ptrINS1_9ArrayDataEEEEvEENS3_IT_EEDpOT0_ + 52
14  libarrow.1300.0.0.dylib             0x0000000141eb0494 _ZN5arrow12_GLOBAL__N_116ArrayDataWrapper5VisitINS_8ListTypeEEENS_6StatusERKT_ + 52
15  libarrow.1300.0.0.dylib             0x0000000141e97ccc _ZN5arrow15VisitTypeInlineINS_12_GLOBAL__N_116ArrayDataWrapperEJEEENS_6StatusERKNS_8DataTypeEPT_DpOT0_ + 1164
16  libarrow.1300.0.0.dylib             0x0000000141e974d8 _ZN5arrow9MakeArrayERKNSt3__110shared_ptrINS_9ArrayDataEEE + 124
17  libarrow.1300.0.0.dylib             0x000000014209c47c _ZN5arrow14ExtensionArray7SetDataERKNSt3__110shared_ptrINS_9ArrayDataEEE + 284
18  libarrow.1300.0.0.dylib             0x000000014209c320 _ZN5arrow14ExtensionArrayC2ERKNSt3__110shared_ptrINS_9ArrayDataEEE + 84
19  libarrow.1300.0.0.dylib             0x000000014209c51c _ZN5arrow14ExtensionArrayC1ERKNSt3__110shared_ptrINS_9ArrayDataEEE + 36
20  libarrow_python.dylib               0x00000001066d2914 _ZNSt3__120__shared_ptr_emplaceIN5arrow14ExtensionArrayENS_9allocatorIS2_EEEC2B6v15006IJRNS_10shared_ptrINS1_9ArrayDataEEEEEES4_DpOT_ + 100
21  libarrow_python.dylib               0x00000001066d25f0 _ZNSt3__120__shared_ptr_emplaceIN5arrow14ExtensionArrayENS_9allocatorIS2_EEEC1B6v15006IJRNS_10shared_ptrINS1_9ArrayDataEEEEEES4_DpOT_ + 36
22  libarrow_python.dylib               0x00000001066d24f8 _ZNSt3__115allocate_sharedB6v15006IN5arrow14ExtensionArrayENS_9allocatorIS2_EEJRNS_10shared_ptrINS1_9ArrayDataEEEEvEENS5_IT_EERKT0_DpOT1_ + 60
23  libarrow_python.dylib               0x00000001066d1240 _ZNSt3__111make_sharedB6v15006IN5arrow14ExtensionArrayEJRNS_10shared_ptrINS1_9ArrayDataEEEEvEENS3_IT_EEDpOT0_ + 52
24  libarrow_python.dylib               0x00000001066d1160 _ZNK5arrow2py15PyExtensionType9MakeArrayENSt3__110shared_ptrINS_9ArrayDataEEE + 224
25  libarrow.1300.0.0.dylib             0x0000000141eb0814 _ZN5arrow12_GLOBAL__N_116ArrayDataWrapper5VisitERKNS_13ExtensionTypeE + 84
26  libarrow.1300.0.0.dylib             0x0000000141e97e10 _ZN5arrow15VisitTypeInlineINS_12_GLOBAL__N_116ArrayDataWrapperEJEEENS_6StatusERKNS_8DataTypeEPT_DpOT0_ + 1488
27  libarrow.1300.0.0.dylib             0x0000000141e974d8 _ZN5arrow9MakeArrayERKNSt3__110shared_ptrINS_9ArrayDataEEE + 124
28  libarrow.1300.0.0.dylib             0x0000000141d578d8 _ZNK5arrow11StructArray5fieldEi + 368
29  libarrow.1300.0.0.dylib             0x00000001420b907c _ZN5arrow12_GLOBAL__N_112ArrayPrinter5VisitERKNS_11StructArrayE + 468
30  libarrow.1300.0.0.dylib             0x00000001420b3aa0 _ZN5arrow16VisitArrayInlineINS_12_GLOBAL__N_112ArrayPrinterEJEEENS_6StatusERKNS_5ArrayEPT_DpOT0_ + 1308
31  libarrow.1300.0.0.dylib             0x00000001420b19d0 _ZN5arrow12_GLOBAL__N_112ArrayPrinter5PrintERKNS_5ArrayE + 60
32  libarrow.1300.0.0.dylib             0x00000001420b1b6c _ZN5arrow11PrettyPrintERKNS_5ArrayERKNS_18PrettyPrintOptionsEPNSt3__113basic_ostreamIcNS6_11char_traitsIcEEEE + 68
33  libarrow.1300.0.0.dylib             0x00000001420b1bbc _ZN5arrow11PrettyPrintERKNS_5ArrayERKNS_18PrettyPrintOptionsEPNSt3__112basic_stringIcNS6_11char_traitsIcEENS6_9allocatorIcEEEE + 68
34  lib.cpython-310-darwin.so           0x0000000105490a6c _ZL41__pyx_pf_7pyarrow_3lib_5Array_33to_stringP29__pyx_obj_7pyarrow_3lib_Arrayiiiib + 488
35  lib.cpython-310-darwin.so           0x0000000105481a68 _ZL41__pyx_pw_7pyarrow_3lib_5Array_34to_stringP7_objectS0_S0_ + 1628
36  Python                              0x0000000104787cfc cfunction_call + 60
37  lib.cpython-310-darwin.so           0x00000001053a1e8c _ZL19__Pyx_PyObject_CallP7_objectS0_S0_ + 140
38  lib.cpython-310-darwin.so           0x00000001053a7280 _ZL24__Pyx_PyObject_CallNoArgP7_object + 120
39  lib.cpython-310-darwin.so           0x000000010547e380 _ZL39__pyx_pf_7pyarrow_3lib_5Array_37__str__P29__pyx_obj_7pyarrow_3lib_Array + 324
40  lib.cpython-310-darwin.so           0x000000010547ce30 _ZL39__pyx_pw_7pyarrow_3lib_5Array_38__str__P7_object + 28
41  Python                              0x000000010478cfac PyObject_Str + 172
42  Python                              0x00000001047b97cc unicode_new + 116
43  Python                              0x00000001047a4ad8 type_call + 84
44  lib.cpython-310-darwin.so           0x00000001053a1e8c _ZL19__Pyx_PyObject_CallP7_objectS0_S0_ + 140
45  lib.cpython-310-darwin.so           0x0000000105653be0 _ZL26__Pyx__PyObject_CallOneArgP7_objectS0_ + 176
46  lib.cpython-310-darwin.so           0x00000001053a2b24 _ZL25__Pyx_PyObject_CallOneArgP7_objectS0_ + 116
47  lib.cpython-310-darwin.so           0x000000010547d6b8 _ZL40__pyx_pf_7pyarrow_3lib_5Array_31__repr__P29__pyx_obj_7pyarrow_3lib_Array + 552
48  lib.cpython-310-darwin.so           0x000000010547ce00 _ZL40__pyx_pw_7pyarrow_3lib_5Array_32__repr__P7_object + 28
49  Python                              0x000000010478d17c PyObject_Repr + 156
50  Python                              0x000000010474fe78 PyFile_WriteObject + 392
51  Python                              0x000000010488d7c4 sys_displayhook + 200
52  Python                              0x00000001047886fc cfunction_vectorcall_O + 100
53  Python                              0x00000001048277d8 _PyEval_EvalFrameDefault + 15056
54  Python                              0x0000000104822ab4 _PyEval_Vector + 376
55  Python                              0x0000000104822928 PyEval_EvalCode + 104
56  Python                              0x000000010487e998 run_eval_code_obj + 84
57  Python                              0x000000010487e8fc run_mod + 112
58  Python                              0x000000010487ec9c PyRun_InteractiveOneObjectEx + 656
59  Python                              0x000000010487dbcc _PyRun_InteractiveLoopObject + 124
60  Python                              0x000000010487d46c _PyRun_AnyFileObject + 76
61  Python                              0x00000001048819a4 PyRun_AnyFileExFlags + 84
62  Python                              0x00000001048a04a0 pymain_run_stdin + 160
63  Python                              0x000000010489fa98 Py_RunMain + 512
64  Python                              0x00000001048a1000 Py_BytesMain + 40
65  dyld                                0x0000000191d6ff28 start + 2236
zsh: abort      python

[spenczar]

Thanks for taking a look @AlenkaF. Here is a reproducer for FixedShapeTensor - it only happens if you go through the struct. The issue is mis-titled. It's actually a bug with Tables which contain a Struct field which have an ExtensionType which has a ListTyped-storage (whew, that's a lot!):

In [38]: import pyarrow as pa

In [39]: tensor_type = pa.fixed_shape_tensor(pa.int32(), (2, 2))

In [40]: arr = [[1, 2, 3, 4], [10, 20, 30, 40], [100, 200, 300, 400]]

In [41]: storage = pa.array(arr, pa.list_(pa.int32(), 4))

In [42]: tensor_array = pa.ExtensionArray.from_storage(tensor_type, storage)
    ...:

In [43]: pa.StructArray.from_arrays([tensor_array], "x")
Out[43]:
<pyarrow.lib.StructArray object at 0x154e5a680>
-- is_valid: all not null
-- child 0 type: extension<arrow.fixed_shape_tensor>
  [
    [
      1,
      2,
      3,
      4
    ],
    [
      10,
      20,
      30,
      40
    ],
    [
      100,
      200,
      300,
      400
    ]
  ]

In [44]: struct_array = pa.StructArray.from_arrays([tensor_array], "x")

In [45]: table = pa.Table.from_arrays([struct_array], ["field1"])

In [46]: table.cast(table.schema)
---------------------------------------------------------------------------
ArrowInvalid                              Traceback (most recent call last)
Cell In[46], line 1
----> 1 table.cast(table.schema)

File ~/.pyenv/versions/3.10.12/envs/adam-core/lib/python3.10/site-packages/pyarrow/table.pxi:3616, in pyarrow.lib.Table.cast()

File ~/.pyenv/versions/3.10.12/envs/adam-core/lib/python3.10/site-packages/pyarrow/table.pxi:3798, in pyarrow.lib.Table.from_arrays()

File ~/.pyenv/versions/3.10.12/envs/adam-core/lib/python3.10/site-packages/pyarrow/table.pxi:2962, in pyarrow.lib.Table.validate()

File ~/.pyenv/versions/3.10.12/envs/adam-core/lib/python3.10/site-packages/pyarrow/error.pxi:100, in pyarrow.lib.check_status()

ArrowInvalid: Column 0: In chunk 0: Invalid: Struct child array #0 invalid: Invalid: Expected 1 child arrays in array of type fixed_size_list<item: int32>[4], got 0

[NellyWhads]

Is this related to #34985?

[AlenkaF]

I asked Copilot to go through the backtrace and I checked the result. It does seem that, in both cases, when casting an extension type with list-based storage, the cast kernel produces ArrayData with the correct type but drops the child data. When the extension type then tries to wrap this into an ExtensionArray and calls MakeArray, it tries to construct a list array from that incomplete ArrayData and crashes.

In case that helps anybody starting to look for a fix. Will also update the title to reflect the underlying bug.

[connorsimms]

Hello! I've looked into this issue and I've diagnosed the cause in array/data.cc. I've got a fix with C++/Python tests, and i've also checked it against the Python snippets in this thread. If no one is working on this, I'd be more than happy to open a PR :).

[connorsimms]

take