Skip to content

[SPARK-41398][SQL][FOLLOWUP] Update runtime filtering javadoc to reflect relaxed partition constraints #54046

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
yyanyy wants to merge 1 commit into from
Closed

[SPARK-41398][SQL][FOLLOWUP] Update runtime filtering javadoc to reflect relaxed partition constraints #54046

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/SupportsRuntimeFiltering.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,8 +51,10 @@ public interface SupportsRuntimeFiltering extends SupportsRuntimeV2Filtering {
* the originally reported partitioning during runtime filtering. While applying runtime filters,
* the scan may detect that some {@link InputPartition}s have no matching data. It can omit
* such partitions entirely only if it does not report a specific partitioning. Otherwise,
* the scan can replace the initially planned {@link InputPartition}s that have no matching
* data with empty {@link InputPartition}s but must preserve the overall number of partitions.
* the scan can either replace the initially planned {@link InputPartition}s that have no
* matching data with empty {@link InputPartition}s, or report only a subset of the original
* partition values (omitting those with no data). The scan must not report new partition values
* that were not present in the original partitioning.
* <p>
* Note that Spark will call {@link Scan#toBatch()} again after filtering the scan at runtime.
*
Expand Down
7 changes: 4 additions & 3 deletions ...atalyst/src/main/java/org/apache/spark/sql/connector/read/SupportsRuntimeV2Filtering.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,9 +55,10 @@ public interface SupportsRuntimeV2Filtering extends Scan {
* the originally reported partitioning during runtime filtering. While applying runtime
* predicates, the scan may detect that some {@link InputPartition}s have no matching data. It
* can omit such partitions entirely only if it does not report a specific partitioning.
* Otherwise, the scan can replace the initially planned {@link InputPartition}s that have no
* matching data with empty {@link InputPartition}s but must preserve the overall number of
* partitions.
* Otherwise, the scan can either replace the initially planned {@link InputPartition}s that
* have no matching data with empty {@link InputPartition}s, or report only a subset of the
* original partition values (omitting those with no data). The scan must not report new
* partition values that were not present in the original partitioning.
* <p>
* Note that Spark will call {@link Scan#toBatch()} again after filtering the scan at runtime.
*
Expand Down