[BEAM-11460] Support reading Parquet files with unknown schema#13554
Closed
anantdamle wants to merge 6 commits intoapache:masterfrom
anantdamle:master
Closed
[BEAM-11460] Support reading Parquet files with unknown schema#13554anantdamle wants to merge 6 commits intoapache:masterfrom anantdamle:master
anantdamle wants to merge 6 commits intoapache:masterfrom
anantdamle:master
Conversation
Contributor
Author
|
@danielxjd @lgajowy @jbonofre Can you review the feature for reading Parquet files with unknown schema |
anantdamle
changed the title
…arseFiles<T>` implementation for supporting files with unknown schema.
iemejia
reviewed
Dec 24, 2020
Member
iemejia
left a comment
iemejia
left a comment
There was a problem hiding this comment.
Minor comments, looks pretty good thanks for this contribution @anantdamle
sdks/java/io/parquet/src/main/java/org/apache/beam/sdk/io/parquet/ParquetIO.java
Outdated
Show resolved
Hide resolved
sdks/java/io/parquet/src/main/java/org/apache/beam/sdk/io/parquet/ParquetIO.java
Outdated
Show resolved
Hide resolved
sdks/java/io/parquet/src/main/java/org/apache/beam/sdk/io/parquet/ParquetIO.java
Outdated
Show resolved
Hide resolved
sdks/java/io/parquet/src/main/java/org/apache/beam/sdk/io/parquet/ParquetIO.java
Outdated
Show resolved
Hide resolved
sdks/java/io/parquet/src/main/java/org/apache/beam/sdk/io/parquet/ParquetIO.java
Outdated
Show resolved
Hide resolved
1. Fix Javadoc example by using consistent words 1. Other indentation and space fixes
Contributor
|
@anantdamle Thank you for contribution! Please, do your changes in the feature branch, not in your master. |
iemejia
reviewed
Dec 24, 2020
Member
iemejia
left a comment
iemejia
left a comment
There was a problem hiding this comment.
Looks nice now. I think I am going to merge this eagerly once the last fix is done @aromanenko-dev. Don't hesitate to bring any extra comments afterwards that we consider we can still improve.
| * ReadFromMongoDB/WriteToMongoDB will mask password in display_data (Python) ([BEAM-11444](https://issues.apache.org/jira/browse/BEAM-11444).) | ||
| * Support for X source added (Java/Python) ([BEAM-X](https://issues.apache.org/jira/browse/BEAM-X)). | ||
| * There is a new transform `ReadAllFromBigQuery` that can receive multiple requests to read data from BigQuery at pipeline runtime. See [PR 13170](https://github.com/apache/beam/pull/13170), and [BEAM-9650](https://issues.apache.org/jira/browse/BEAM-9650). | ||
| * ParquetIO can now read files with an unknown schema. See [PR-13554](https://github.com/apache/beam/pull/13554) and ([BEAM-11460](https://issues.apache.org/jira/browse/BEAM-11460)) |
Member
There was a problem hiding this comment.
Can you move this up to the 2.28.0 section. The 2.27.0 was already merged (sorry I missed this in previous check.
4 tasks
Contributor
Author
|
As the branch needs to change. Created new PR/13616 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Data engineers encounter times when schema of Parquet file is unknown at the time of writing the pipeline or multiple schema may be present in different files. Reading Parquet files using ParquetIO requires providing an Avro (equivalent) schema, Many a times its not possible to know the schema of the Parquet files.
On the other hand AvroIO supports reading unknow schema files by providing a parse function :
#parseGenericRecords(SerializableFunction<GenericRecord,T>)Supporting this functionality in ParquetIO is simple and requires minimal changes to the ParquetIO surface.
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
ParquetIOTestR: @lgajowy and @jbonofre).CHANGES.mdwith noteworthy changes.Post-Commit Tests Status (on master branch)
Pre-Commit Tests Status (on master branch)
See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI.