[SPARK-10522] [SQL] Nanoseconds of Timestamp in Parquet should be positive#8674
[SPARK-10522] [SQL] Nanoseconds of Timestamp in Parquet should be positive#8674davies wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
Without the checks I'm adding in #8606 this can overflow and write garbage to the file...
There was a problem hiding this comment.
We haven't handle overflow in many places, the behavior is undefined, it means will be garbage in the file, I think it's fine for now.
There was a problem hiding this comment.
If we throw an exception, the job will crash if there are some garbage rows in it.
There was a problem hiding this comment.
ok. I'll close my PR and leave the bug open.
There was a problem hiding this comment.
Long (2^63) is enough for 584942 years before overflow, so don't worry about it.
There was a problem hiding this comment.
How about adding some comments at here?
|
LGTM. |
|
Test build #42221 has finished for PR 8674 at commit
|
|
Merged into master and 1.5 branch (added comment while merging). |
…itive Or Hive can't read it back correctly. Thanks vanzin for report this. Author: Davies Liu <davies@databricks.com> Closes apache#8674 from davies/positive_nano. (cherry picked from commit 7e32387)
4 participants
Or Hive can't read it back correctly.
Thanks @vanzin for report this.