feat: clustered segments pt.1#19460
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| @Override | ||
| public List<String> getColumnNames() | ||
| { | ||
| List<String> columns = new ArrayList<>(columnNames.size() + aggregators.length); |
| }; | ||
| } | ||
|
|
||
| private CursorHolder makeClusteredCursorHolder(CursorBuildSpec spec, ClusteredValueGroupsBaseTableSchema clusterSummary) |
| } | ||
| final Object raw = parent.currentValue(idx); | ||
| final String stringValue = raw == null ? null : String.valueOf(raw); | ||
| cachedSelector = DimensionSelector.constant(stringValue, spec.getExtractionFn()); |
| final Object raw = parent.currentValue(idx); | ||
| final String stringValue = raw == null ? null : String.valueOf(raw); | ||
| final String afterExtraction = | ||
| spec.getExtractionFn() == null ? stringValue : spec.getExtractionFn().apply(stringValue); |
| final Object raw = parent.currentValue(idx); | ||
| final String stringValue = raw == null ? null : String.valueOf(raw); | ||
| final String afterExtraction = | ||
| spec.getExtractionFn() == null ? stringValue : spec.getExtractionFn().apply(stringValue); |
FrankChen021
left a comment
FrankChen021
left a comment
There was a problem hiding this comment.
| Severity | Findings |
|---|---|
| P0 | 0 |
| P1 | 2 |
| P2 | 0 |
| P3 | 0 |
| Total | 2 |
Reviewed 25 of 25 changed files.
This is an automated review by Codex GPT-5.5
| ); | ||
| } | ||
|
|
||
| return new QueryableIndexCursorHolder( |
There was a problem hiding this comment.
[P1] Cluster-column filters are reapplied against sub-indexes that do not contain them
After pruneClusterGroups keeps a matching group, the original CursorBuildSpec is passed unchanged into a QueryableIndexCursorHolder for the group sub-index. That sub-index intentionally does not contain the clustering columns, so EqualityFilter.getBitmapColumnIndex sees a missing column and builds an exact all-false/all-unknown bitmap before the ClusteringColumnSelectorFactory wrapper can provide the constant value to a matcher. Queries such as tenant = 'acme' therefore prune to the acme group and then scan zero rows. The multi-group supplier path repeats the same pattern, so clustering predicates need to be stripped/residualized after pruning or the group holder needs a cluster-aware ColumnIndexSelector/filter-bundle path.
| if (clusteringValues[idx] == null) { | ||
| return DruidPredicateMatch.FALSE; | ||
| } | ||
| return DruidPredicateMatch.of(Objects.equals(clusteringValues[idx], eq.getMatchValue())); |
There was a problem hiding this comment.
[P1] Pruning ignores EqualityFilter type coercion semantics
The pruner compares the stored clustering value directly with eq.getMatchValue(), but EqualityFilter normal matcher casts the literal to the input column type. For example, a LONG clustering value 5 with an EqualityFilter typed as STRING value "5" would match when evaluated by the real matcher, but Objects.equals(5L, "5") returns false here and the only live cluster group is removed before scanning. TypedInFilter has the same risk below with direct Objects.equals over the raw sorted values. The pruning check should use the same predicate/cast semantics as the filter matcher, or stay UNKNOWN when the literal type does not exactly match the clustering column type.
3 participants
Description
This PR builds on the foundations laid by projections (#17214) and the v10 segment format (#18880) to introduce 'clustered' segments, to give operators the option to push a
CLUSTERED BYclause inside of a segment, as a companion to partitioning data is distributed between segments in this manner. Internally, the 'base' table is decmposed into separate cluster groups, which are combined together to form the 'complete' view of all rows stored in the segment via concatenation. This optimizes for use cases where the typical queries are filtering down to a small subset of the cluster groups (ideally a single grouping), which like the effect from using aggregate projections, can greatly reduce the number of rows to be scanned. The expected use cases are things like multi-tenant-with-shared datasource clusters, metrics use cases which typically filter to a single type of service, etcThis PR contains only the read side, to get feedback on the internal segment metadata shapes and query engine integration. The write side (ingestion support) will come in a follow-up PR, so this PR uses some test fixtures to exercise the read paths until segment building is actually in place.
Since this is an experimental/mostly new feature, the most important part for reviewers is the new internal segment metadata,
ClusteredValueGroupsBaseTableSchemaand its internal new stuff likeTableClusterGroupSpecandClusteringDictionaries, so that we can ensure the metadata we will be storing in the segment is "good" since changing it after segments have been written is very hard.todo: elaborate on design