Feature is not implemeneted: Unsupported cast with list of structs

[liamphmurphy]

Describe the bug

This bug for me originated when encountering schema evolutions on Delta tables using the delta-rs library. Whenever a schema evolution occurred on my table that contains a field with a list of structs, Datafusion is returning this error:

This feature is not implemented: Unsupported CAST from Struct([Field { name: "properties", data_type: Struct([Field { name: "someNewField", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "fields", data_type: List(Field { name: "item", data_type: Struct([Field { name: "messageId", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]) to Struct([Field { name: "properties", data_type: Struct([Field { name: "fields", data_type: List(Field { name: "element", data_type: Struct([Field { name: "messageId", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "someNewField", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }])

It appears that the issue isn't necessarily the exact casting itself, but the ORDER of the new field that's added someNewField. If it's put at the end, then it works, but inserting the field in another position yields this error.

To Reproduce

Below is the python code using delta-rs (which is currently on Datafusion 46) that shows this error

import pyarrow as pa
from deltalake import write_deltalake

# Define the path for the Delta table
delta_table_path = "./datafusion-repro-test-table"

# Define the data for the first write
data_first_write = [
    {
        "uid": "ws_2",
        "event": {
            "properties": {
                "fields": [
                    {
                        "messageId": "veniam sed et elit adipisicing"
                    }
                ],
            },
        }
    }
]

schema = pa.schema([
    pa.field("uid", pa.string()),
    pa.field("event", pa.struct([
        pa.field("properties", pa.struct([
            pa.field("fields", pa.list_(pa.struct([
                pa.field("messageId", pa.string()),
            ]))),
        ])),
    ])),
])

print(schema)



first_write = pa.Table.from_pylist(data_first_write, schema=schema)

# Write data to Delta table for the first write
write_deltalake(delta_table_path, first_write, mode="append", engine="rust", schema_mode="merge")

#### NOW FOR THE SECOND WRITE THAT BREAKS ####

data_second_write = [
    {
        "uid": "ws_2",
        "event": {
            "properties": {
                "someNewField": "test-value", # New field
                "fields": [
                    {
                        "messageId": "veniam sed et elit adipisicing"
                    }
                ],
            },
        }
    }
]

second_schema = pa.schema([
    pa.field("uid", pa.string()),
    pa.field("event", pa.struct([
        pa.field("properties", pa.struct([
            pa.field("someNewField", pa.string()), # New field
            pa.field("fields", pa.list_(pa.struct([
                pa.field("messageId", pa.string()),
            ]))),
        ])),
    ])),
])

second_write = pa.Table.from_pylist(data_second_write, schema=second_schema)

# Write data to Delta table for the second write
write_deltalake(delta_table_path, second_write, mode="append", engine="rust", schema_mode="merge")

Expected behavior

Datafusion would support casting a schema when said schema contains a list of structs, no matter where in the order a new field is added to the schema.

Additional context

Originating bug report in delta-rs: delta-io/delta-rs#3339

[bodubas]

Got exactly the same issue. Would be great if datafusion could sort the input schema to alight with the target schema.

[alamb]

• Related discussion: Arrow can't seem to cast evolving structs arrow-rs#7176

I think @kosiew may have a PR up that is related

• Enhance Schema adapter to accommodate evolving struct #15295

[ion-elgreco]

• Related discussion: Arrow can't seem to cast evolving structs arrow-rs#7176

I think @kosiew may have a PR up that is related

• Enhance Schema adapter to accommodate evolving struct #15295

I don't think the PR solves it for us, to my understanding that PR helps when you read parquet data.

But we are evolving the schema within a dataframe cast

[dabljues]

Same issue for me, explained here: delta-io/delta-rs#3595

[alamb]

@kosiew filed a ticket that I think describes a solution to this problem:

• Expose and generalize cast_column to enable struct → struct casting in more contexts #16579

[adriangb]

This appears fixed on main (verified against datafusion-cli 54.1.0). Name-based nested struct casting now handles field reordering, so the specific complaint here (that inserting someNewField before fields fails while appending it at the end work) no longer reproduces.

Repro, using a Parquet file whose physical struct order differs from the declared table order, which is the path the delta-rs failure took:

COPY (
   SELECT 'ws_2' AS uid,
          {'properties': {'someNewField': 'test-value',
                          'fields': [{'messageId': 'veniam sed et elit'}]}} AS event
) TO '/tmp/df15338/a.parquet' STORED AS PARQUET;

-- declared order is {fields, someNewField}; the file's is {someNewField, fields}
CREATE EXTERNAL TABLE a (
   uid VARCHAR,
   event STRUCT<"properties" STRUCT<"fields" STRUCT<"messageId" VARCHAR>[], "someNewField" VARCHAR>>
) STORED AS PARQUET LOCATION '/tmp/df15338/a.parquet';

SELECT * FROM a;
-- | ws_2 | {properties: {fields: [{messageId: veniam sed et elit}], someNewField: test-value}} |

(The quoted identifiers matter — unquoted ones are lowercased and then genuinely won't match by name.)

Schema evolution across two files in one directory works too, with the older file's missing field null-filled:

| ws_1 | {properties: {fields: [{messageId: first write}],  someNewField: NULL}}       |
| ws_2 | {properties: {fields: [{messageId: second write}], someNewField: test-value}} |

The item vs element list-item name difference visible in the original error is also handled: cast_column matches names only inside Struct, and rebuilds list types from the target's item field.

Closing as fixed. @ion-elgreco @dabljues @bodubas if you still hit this through delta-rs please let us know and we can re-open.