HIVE-29225: Premature deletion of scratch directories during output streaming

[sercanCyberVision]

ROOT-CAUSE:
Please see https://issues.apache.org/jira/browse/HIVE-29225 for the detailed problem description and steps to reproduce. In short, scratch directories may be deleted prematurely while Hive is still streaming output to the client.

SOLUTION:
A new configuration property has been introduced:

hive.scratchdir.cleanup.grace.period.hours

This property defines a grace period to prevent cleanup of scratch directories that have been modified within the specified time window.

• Default value is 0h (disabled by default).
• When Hive finishes output streaming, it updates the scratch directory’s last modified time.
• The cleanup logic now checks this timestamp and skips deletion if the directory was updated within the grace period.

This ensures directories are not removed prematurely while still in use.

TESTING:
Added a dedicated unit test to validate the new functionality:

Verified that all existing tests in TestClearDanglingScratchDir continue to pass:

[sonarqubecloud]

Quality Gate passed

Issues
4 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[sercanCyberVision]

Hello @deniskuzZ ,
You previously reviewed and approved the ClearDanglingScratchDir.java changes in PR #4403.
Could you please take a look at this PR as well?

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
Feel free to reach out on the dev@hive.apache.org list if the patch is in need of reviews.

[sercanCyberVision]

Hi @difin,
Can you please review this PR?

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
Feel free to reach out on the dev@hive.apache.org list if the patch is in need of reviews.