[C++][Python] Casting nested structs fails when inner struct are non-nullable, but the field for the inner struct is nullable

[davlee1972]

Describe the bug, including details regarding any error messages, version, and platform.

Is this a bug and if it isn't is there a preferred way of handling this? I've got some rather complex data structures I'm trying to save into parquet using a common schema and using cast() on a struct array would be the preferred way to align data.

Shouldn't NULL in an inner array be acceptable when its validity mask is set to False? I know I can call pyarrow.compute.fill_null() and replace the NULL with a ZERO, but this can get complicated when I got thousands of fields with a mix of all kinds of datatypes: int, string, decimal, date, timestamp, boolean, etc.. and there isn't a clear way to stuff dummy scalars: pc.scalar(0), pc.scalar(""), pc.scalar(True), etc.. for every possible datatype in the Array ecosystem to replace NULL.

Here are working and non-working example using cast() on a the same struct array type.

Define struct types and setup data using a pylist vs a list of arrays. inner_struct is nullable in the outer struct definition.

If we examine the two structs, to_pylist() returns the same values, but the inner struct for struct a contains a "0" while struct b has a "null".

Casting both structs to the same outer struct type fails for struct b because it has a NULL value in it even though the field it belongs to in the outer struct is nullable and the NULL entry has a False validity mask.

Component(s)

C++, Python

[davlee1972]

FYI. I'm working on a XML to Arrow converter. It's pretty much working except saving XML to Parquet.

https://github.com/davlee1972/xmlwiz

Instead of parsing and validating elements one at a time, the XML Wizard will load XML into data vectors.
Once in vector format, operations like string to numeric data casting and validation can be performed in bulk.

To even out array lengths to turn them into structs from the lowest level of a XML tree up to the XML root, I'm injecting None(s) for missing "row" data.

[davlee1972]

to_pylist() can handle this, but cast() cannot.. I can use it as a work around, but turning a bunch of vectors into rows and then back to vectors isn't ideal.