ARROW-8039: [Python] Use dataset API in existing parquet readers and tests#6303

Closed

jorisvandenbossche wants to merge 26 commits into

apache:masterfrom

jorisvandenbossche:parquet-dataset-experiment

Commits on Apr 8, 2020

POC: use dataset API in existing parquet tests
jorisvandenbossche
committed
support old-style filters
jorisvandenbossche
committed
add ParquetDatasetV2 shim and use in tests
jorisvandenbossche
committed
parametrize read_table tests
jorisvandenbossche
committed
do not disallow null characters in strings in filters when use_dataset=True
jorisvandenbossche
committed
add pytest.mark.dataset mark
jorisvandenbossche
committed
non-deterministic cases due to use_threads
jorisvandenbossche
committed
move dataset creation into helper function
jorisvandenbossche
committed
add support for use_pandas_metadata + some cleanup of the tests
jorisvandenbossche
committed
rename use_dataset -> use_legacy_dataset
jorisvandenbossche
committed
consolidate read_table/ParquetDataset code + add errors for unsupported keywords
jorisvandenbossche
committed
fix expression syntax + add docstring
jorisvandenbossche
committed
fix paths test on Windows
jorisvandenbossche
committed
Update python/pyarrow/parquet.py

jorisvandenbossche
and
bkietz
committed
Update python/pyarrow/parquet.py

jorisvandenbossche
and
bkietz
committed
Update python/pyarrow/tests/test_parquet.py

jorisvandenbossche
and
bkietz
committed
Update python/pyarrow/tests/test_parquet.py

jorisvandenbossche
and
bkietz
committed
consolidate filters docstring
jorisvandenbossche
committed
support memory_map
jorisvandenbossche
committed
feedback
jorisvandenbossche
committed
enable different partitioning schemes
jorisvandenbossche
committed
remove ARROW:schema removal from metadata in read_table for new API
jorisvandenbossche
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

ARROW-8039: [Python] Use dataset API in existing parquet readers and tests#6303

ARROW-8039: [Python] Use dataset API in existing parquet readers and tests#6303
jorisvandenbossche wants to merge 26 commits into
apache:masterfrom
jorisvandenbossche:parquet-dataset-experiment

Commits on Apr 8, 2020

POC: use dataset API in existing parquet tests

support old-style filters

add ParquetDatasetV2 shim and use in tests

parametrize read_table tests

do not disallow null characters in strings in filters when use_dataset=True

add pytest.mark.dataset mark

non-deterministic cases due to use_threads

move dataset creation into helper function

add support for use_pandas_metadata + some cleanup of the tests

rename use_dataset -> use_legacy_dataset

consolidate read_table/ParquetDataset code + add errors for unsupported keywords

fix expression syntax + add docstring

fix paths test on Windows

Update python/pyarrow/parquet.py

Update python/pyarrow/parquet.py

Update python/pyarrow/tests/test_parquet.py

Update python/pyarrow/tests/test_parquet.py

consolidate filters docstring

support memory_map

feedback

enable different partitioning schemes

remove ARROW:schema removal from metadata in read_table for new API

Commits on Apr 9, 2020

Apply suggestions from code review

Apply suggestions from code review

update docstrings

deterministic_row_order helper function