Skip to content

feat: clustered segments DataSegment changes and PartialLoadMatcher#19462

Merged
clintropolis merged 11 commits into
apache:masterfrom
clintropolis:clustered-datasegment-and-partial-load-matcher
May 15, 2026
Merged

feat: clustered segments DataSegment changes and PartialLoadMatcher#19462
clintropolis merged 11 commits into
apache:masterfrom
clintropolis:clustered-datasegment-and-partial-load-matcher

Conversation

@clintropolis
Copy link
Copy Markdown
Member

Description

Companion PR to #19460 that adds adds the DataSegment changes so that clustered segments are visible to coordinators/brokers and a PartialLoadMatcher implementation that can use wildcards to match cluster groups for partial loads.

@github-actions github-actions Bot added Area - Batch Ingestion Area - Segment Format and Ser/De Area - MSQ For multi stage queries - https://github.com/apache/druid/issues/12262 labels May 13, 2026
@clintropolis
Copy link
Copy Markdown
Member Author

if this PR goes in after #19461 it needs adjusted to use the PartialLoadSpec base type introduced there (likewise if this is merged first that PR needs to update)

Copy link
Copy Markdown
Member

@FrankChen021 FrankChen021 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity Findings
P0 0
P1 0
P2 1
P3 0
Total 1

Reviewed 19 of 19 changed files.


This is an automated review by Codex GPT-5.5

@Nullable List<String> dimensions,
@Nullable List<String> metrics,
@Nullable List<String> projections,
@Nullable ClusterGroupTuples clusterGroups,
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Keep the existing public constructor signature

This inserts clusterGroups into the public constructor that previously took (... projections, ShardSpec, CompactionState, ...). DataSegment is annotated @publicapi, and extension or application code can construct projection-aware segments with that old signature; after this change those callers either fail to compile or, if already compiled, hit NoSuchMethodError even though they do not use cluster groups. Please keep the old overload and delegate with clusterGroups = null instead of replacing the existing public signature.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fine, I can add it... DataSegment is a bit of a graveyard of deprecated constructors and builder creators though... we need to delete these at some point and just call it out in release notes. I would expect creating DataSegment directly is kind of uncommon in extensions (outside of tests at least).

Copy link
Copy Markdown
Contributor

@capistrant capistrant left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 with or without the DataSegment constructor business

Copy link
Copy Markdown
Member

@FrankChen021 FrankChen021 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed the follow-up on the DataSegment constructor compatibility thread and the current incremental update. The previous public constructor overload has been restored and delegates with clusterGroups = null, so the original concern is handled.

Reviewed 4 of 19 changed files.


This is an automated review by Codex GPT-5.5

@clintropolis clintropolis merged commit 174b3a7 into apache:master May 15, 2026
38 checks passed
@clintropolis clintropolis deleted the clustered-datasegment-and-partial-load-matcher branch May 15, 2026 08:56
@github-actions github-actions Bot added this to the 38.0.0 milestone May 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Area - Batch Ingestion Area - MSQ For multi stage queries - https://github.com/apache/druid/issues/12262 Area - Segment Format and Ser/De

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants