Skip to content

[FLINK-40002][table] Fix PushFilterInCalcIntoTableSourceScanRule for metadata filter push-down#28556

Open
jnh5y wants to merge 1 commit into
apache:masterfrom
jnh5y:FLINK-40002
Open

[FLINK-40002][table] Fix PushFilterInCalcIntoTableSourceScanRule for metadata filter push-down#28556
jnh5y wants to merge 1 commit into
apache:masterfrom
jnh5y:FLINK-40002

Conversation

@jnh5y

@jnh5y jnh5y commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

What is the purpose of the change

  • Mixed physical+metadata predicates now stay as runtime Calc filters
    instead of being routed through FilterPushDownSpec, which crashed on
    compiled-plan restore when ProjectPushDownSpec narrowed the row type.
  • Fix PushFilterInCalcIntoTableSourceScanRule to separate metadata-only
    predicates from physical predicates, matching PushFilterIntoTableSourceScanRule.
  • Extract the duplicated classify-push logic into a shared
    classifyAndPushFilters() method in PushFilterIntoSourceScanRuleBase.
  • Document the identity contract on applyMetadataFilters Javadoc.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): no
  • The public API, i.e., is any changed class annotated with @Public(Evolving): yes (SupportsReadingMetadata — Javadoc only, no interface change)
  • The serializers: no
  • The runtime per-record code paths (performance sensitive): no
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
  • The S3 file system connector: no

Documentation

  • Does this pull request introduce a new feature? no
  • If yes, how is the feature documented? not applicable

Was generative AI tooling used to co-author this PR?
  • Yes (Claude Sonnet 4.6)

…metadata filter push-down

- Mixed physical+metadata predicates now stay as runtime Calc filters
  instead of being routed through FilterPushDownSpec, which crashed on
  compiled-plan restore when ProjectPushDownSpec narrowed the row type.
- Fix PushFilterInCalcIntoTableSourceScanRule to separate metadata-only
  predicates from physical predicates, matching PushFilterIntoTableSourceScanRule.
- Extract the duplicated classify-push logic into a shared
  classifyAndPushFilters() method in PushFilterIntoSourceScanRuleBase.
- Document the identity contract on applyMetadataFilters Javadoc.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@flinkbot

flinkbot commented Jun 26, 2026

Copy link
Copy Markdown
Collaborator

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

Comment on lines +416 to +420
@Override
public Result applyFilters(List<ResolvedExpression> filters) {
acceptedPhysicalFilters.addAll(filters);
return Result.of(Collections.emptyList(), filters);
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can be simplified and the initisialisation of array dropped?

Suggested change
@Override
public Result applyFilters(List<ResolvedExpression> filters) {
acceptedPhysicalFilters.addAll(filters);
return Result.of(Collections.emptyList(), filters);
}
@Override
public Result applyFilters(List<ResolvedExpression> filters) {
return Result.of(Collections.emptyList(), filters);
}

@github-actions github-actions Bot added the community-reviewed PR has been reviewed by the community. label Jun 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-reviewed PR has been reviewed by the community.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants