Skip to content

HIVE-26809: Upgrade ORC to 1.8.3#4121

Merged
abstractdog merged 1 commit into
apache:masterfrom
zratkai:HIVE-26809
Apr 11, 2023
Merged

HIVE-26809: Upgrade ORC to 1.8.3#4121
abstractdog merged 1 commit into
apache:masterfrom
zratkai:HIVE-26809

Conversation

@zratkai

@zratkai zratkai commented Mar 16, 2023

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Upgrade ORC dependency to 1.8.3.

Why are the changes needed?

ORC version was old, newer version contains many new features like SARG filters.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

By running the tests in the pipeline.

@zhangbutao

Copy link
Copy Markdown
Contributor

Good job! I think you can check previous PR which tried to upgrade ORC to 1.8.1 #3833

@aturoczy

aturoczy commented Apr 3, 2023

Copy link
Copy Markdown

As I see this change mainly contains the new ORC file sizes changes in the tests, and minor code modifications on the hive itself. I do not see to many critical changes here.

@zhangbutao

Copy link
Copy Markdown
Contributor

I am thinking if we can mask or remove the ORC file sizes related qtest items. In this pr, It is so wasting time to fix qtests about orc file size when upgrading orc version. Also, other OSS eg. Spark seems not treat orc file size as a test item.

@aturoczy

aturoczy commented Apr 3, 2023

Copy link
Copy Markdown

I think in a long term we should. Now it is fixed is fine. But you right pretty over checked. I guess we can create a following ticket. Because this sizes check does not protect the code, and It does not create to much value. Also with new ORC's the files would be always different and check manually is just beh.
For the first upgrade I would be OK with these if it is already there. (The last upgrade was in 1972 :) Finally the Hive will have the latest one)

@zhangbutao

Copy link
Copy Markdown
Contributor

Sure, i am also ok with this change. Maybe we can create a following ticket to mask orc file size, then we can easily upgrade orc version next time.

Comment thread ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcFile.java
Comment thread beeline/pom.xml Outdated
@akshat0395

Copy link
Copy Markdown
Contributor

Had a minor query which is covered by @abstractdog regarding hadoop-hdfs dependency in beeline pom.
Thanks for this change @zratkai
LGTM, +1

@sonarqubecloud

sonarqubecloud Bot commented Apr 6, 2023

Copy link
Copy Markdown

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 13 Code Smells

No Coverage information No Coverage information
No Duplication information No Duplication information

Comment thread beeline/pom.xml
@dongjoon-hyun

Copy link
Copy Markdown
Member

Thank you everyone. It's a great milestone!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants