[Python][C++][Parquet] Support reading Parquet files having a permutation of column order

[asfimport]

See discussion in dask/fastparquet#320

Reporter: Wes McKinney / @wesm
Assignee: Alenka Frim / @AlenkaF

Related issues:

• [Python] More graceful reading of empty String columns in ParquetDataset (relates to)
• [C++][Dataset] Schema evolution in Dataset scanning (is related to)
• [Python][Dataset] Support using dataset API in pyarrow.parquet with a minimal ParquetDataset shim (depends upon)

PRs and other links:

• GitHub Pull Request #11561

_{Note: This issue was originally created as ARROW-2366. Please see the migration documentation for further details.}

[asfimport]

Uwe Korn / @xhochy:
This is the first iteration of schema evolution. While the issue here is quite simple, I would like to add general schema evolution / resolution rules to Arrow. My favorite would be to stick as close as possible to Avro's rules: http://avro.apache.org/docs/current/spec.html#Schema+Resolution

[asfimport]

Wes McKinney / @wesm:
This will need to be addressed as part of general schema conformance in the C++ Datasets API

cc @pitrou @nealrichardson

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
This is now implemented in the C++ Datasets project:

import pyarrow as pa
import pyarrow.parquet as pq
import pyarrow.dataset as ds

# create dummy dataset with column order permutation
import pathlib
basedir = pathlib.Path(".")
case = basedir / "dataset_column_order_permutation"
case.mkdir(exist_ok=True)

table1 = pa.table([[1, 2, 3], [.1, .2, .3]], names=['a', 'b'])
pq.write_table(table1, case / "data1.parquet")

table2 = pa.table([[.4, .5, .6], [4, 5, 6]], names=['b', 'a'])
pq.write_table(table2, case / "data2.parquet")

# reading with the old python implementation indeed raises on schema mismatch
pq.read_table(str(case))

# this reads fine
ds.dataset(str(case)).to_table().to_pandas()

So once we use the datasets API under the hood in pyarrow.parquet (ARROW-8039), this issue should be solved (we can still add a test for it to close this issue)

[asfimport]

Antoine Pitrou / @pitrou:
@jorisvandenbossche Can we close this now? Perhaps we just need to add a test to ensure this works properly?

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
Yes, it's on my "needs a test and then can close it" list, so would prefer to keep it open for now

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
Issue resolved by pull request 11561
#11561