Skip to content

more resilient segment cache legacy path file moves during startup#18484

Merged
clintropolis merged 1 commit into
apache:masterfrom
clintropolis:segment-cache-bootstrap-legacy-move-resilience
Sep 5, 2025
Merged

more resilient segment cache legacy path file moves during startup#18484
clintropolis merged 1 commit into
apache:masterfrom
clintropolis:segment-cache-bootstrap-legacy-move-resilience

Conversation

@clintropolis
Copy link
Copy Markdown
Member

Description

Follow up to #18176, which moved segment cache files from being stored in deeply nested paths of the form {location}/{datasource}/ {interval}/{version}/{partition}/ to being stored in a flatly in the location using the segmentId as the path and includes some logic to move the files from the old paths to the new paths on startup.

This PR improves the file legacy segment cache file location move logic which lives in SegmentLocalCacheManager.getCachedSegments to be more resilient in cases where startup is interrupted and cases where files exist in both locations due to upgrade/downgrade/upgrade cycles.

The added test covers the latter scenario, and would fail with something like

025-09-04T22:11:47,199 ERROR [main] org.apache.druid.segment.loading.SegmentLocalCacheManager - Failed to load segment from segment cache file.: {exceptionType=java.nio.file.DirectoryNotEmptyException, exceptionMessage=/var/folders/8y/mhfmxp391pl9m2h103s_kn200000gn/T/junit6723303392002826755/segment_cache/test_segment_loader_2011-01-12T00:00:00.000Z_2011-04-15T00:00:00.001Z_2015-05-27T03:38:35.683Z, class=org.apache.druid.segment.loading.SegmentLocalCacheManager, file=/var/folders/8y/mhfmxp391pl9m2h103s_kn200000gn/T/junit6723303392002826755/segment_cache/info_dir/test_segment_loader_2011-01-12T00:00:00.000Z_2011-04-15T00:00:00.001Z_2015-05-27T03:38:35.683Z}
java.nio.file.DirectoryNotEmptyException: /var/folders/8y/mhfmxp391pl9m2h103s_kn200000gn/T/junit6723303392002826755/segment_cache/test_segment_loader_2011-01-12T00:00:00.000Z_2011-04-15T00:00:00.001Z_2015-05-27T03:38:35.683Z
	at java.base/sun.nio.fs.UnixFileSystem.move(UnixFileSystem.java:927) ~[?:?]
	at java.base/sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:309) ~[?:?]
	at java.base/java.nio.file.Files.move(Files.java:1431) ~[?:?]
	at org.apache.druid.segment.loading.SegmentLocalCacheManager.getCachedSegments(SegmentLocalCacheManager.java:217) [classes/:?]

without the changes in this PR

@clintropolis clintropolis merged commit 50edbe7 into apache:master Sep 5, 2025
62 checks passed
@clintropolis clintropolis deleted the segment-cache-bootstrap-legacy-move-resilience branch September 5, 2025 05:13
@cecemei cecemei added this to the 35.0.0 milestone Oct 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants