Skip to content

Read: fetch file_schema directly from pyarrow_to_schema #597

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
HonahX merged 8 commits into from
Apr 13, 2024
Merged

Read: fetch file_schema directly from pyarrow_to_schema #597

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
29839da
init commit
HonahX Apr 11, 2024
5ee09ac
project with actual file schema
HonahX Apr 11, 2024
760c594
add version 1 & 2
HonahX Apr 11, 2024
9837faa
fix tests
HonahX Apr 11, 2024
12c5021
fix lint
HonahX Apr 11, 2024
fdc25c8
add test for writing special character column name
kevinjqliu Apr 11, 2024
f6b81f6
parameterize format_version
kevinjqliu Apr 11, 2024
68ba943
cherry-pick
HonahX Apr 13, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix tests
  • Loading branch information
@HonahX
HonahX committed Apr 11, 2024
commit 9837faab944361173ff59c52de20c66f04b2bad1
5 changes: 3 additions & 2 deletions tests/conftest.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@
from moto.server import ThreadedMotoServer # type: ignore
from pyspark.sql import SparkSession

from pyiceberg.io.pyarrow import PyArrowFileIO
from pyiceberg.io.pyarrow import PyArrowFileIO, schema_to_pyarrow


def pytest_collection_modifyitems(items: List[pytest.Item]) -> None:
Expand Down Expand Up @@ -1930,10 +1930,11 @@ def clean_up(test_catalog: Catalog) -> None:
def data_file(table_schema_simple: Schema, tmp_path: str) -> str:
import pyarrow as pa
from pyarrow import parquet as pq
from pyiceberg.io.pyarrow import schema_to_pyarrow

table = pa.table(
{"foo": ["a", "b", "c"], "bar": [1, 2, 3], "baz": [True, False, None]},
metadata={"iceberg.schema": table_schema_simple.model_dump_json()},
schema=schema_to_pyarrow(table_schema_simple),
)

file_path = f"{tmp_path}/0000-data.parquet"
Expand Down
6 changes: 3 additions & 3 deletions tests/io/test_pyarrow.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1373,7 +1373,7 @@ def test_delete(deletes_file: str, example_task: FileScanTask, table_schema_simp
str(with_deletes)
== """pyarrow.Table
foo: string
bar: int64 not null
bar: int32 not null
baz: bool
----
foo: [["a","c"]]
Expand Down Expand Up @@ -1411,7 +1411,7 @@ def test_delete_duplicates(deletes_file: str, example_task: FileScanTask, table_
str(with_deletes)
== """pyarrow.Table
foo: string
bar: int64 not null
bar: int32 not null
baz: bool
----
foo: [["a","c"]]
Expand Down Expand Up @@ -1442,7 +1442,7 @@ def test_pyarrow_wrap_fsspec(example_task: FileScanTask, table_schema_simple: Sc
str(projection)
== """pyarrow.Table
foo: string
bar: int64 not null
bar: int32 not null
baz: bool
----
foo: [["a","b","c"]]
Expand Down