Skip to content

[SPARK-53942][SS] Support changing stateless shuffle partitions upon restart of streaming query #52645

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
HeartSaVioR wants to merge 8 commits into from
Closed

[SPARK-53942][SS] Support changing stateless shuffle partitions upon restart of streaming query #52645

Changes from 1 commit
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
0c879dc
Initial commit
HeartSaVioR Oct 17, 2025
89bdc62
fix
HeartSaVioR Oct 17, 2025
57c0395
SPARK-53942 updates
HeartSaVioR Oct 17, 2025
d354e92
[empty] this is a empty commit
HeartSaVioR Oct 17, 2025
5d9629e
update the doc since it's no longer valid
HeartSaVioR Oct 17, 2025
e5a5554
protect ourselves from the freedom of changing the internal config
HeartSaVioR Oct 18, 2025
4b1638d
golden file test
HeartSaVioR Oct 21, 2025
0aab8d5
reflect review comment
HeartSaVioR Oct 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
reflect review comment
  • Loading branch information
@HeartSaVioR
HeartSaVioR committed Oct 25, 2025
commit 0aab8d5985afe69babc2cf6388ccc08aa86a84bb
4 changes: 2 additions & 2 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2627,8 +2627,8 @@ object SQLConf {

val STATEFUL_SHUFFLE_PARTITIONS_INTERNAL =
buildConf("spark.sql.streaming.internal.stateStore.partitions")
.doc("WARN: This is used internally and not intended to be a user facing one. This config " +
"can be removed without support of compatibility in any time. " +
.doc("WARN: This config is used internally and is not intended to be user-facing. This " +
"config can be removed without support of compatibility in any time. " +
"DO NOT USE THIS CONFIG DIRECTLY AND USE THE CONFIG `spark.sql.shuffle.partitions`. " +
"The default number of partitions to use when shuffling data for stateful operations. " +
"If not specified, this config picks up the value of `spark.sql.shuffle.partitions`. " +
Expand Down