[SPARK-52716][SDP][SQL] Remove comment from Flow trait and references#51406
[SPARK-52716][SDP][SQL] Remove comment from Flow trait and references#51406JiaqiWang18 wants to merge 1 commit into
Conversation
dongjoon-hyun
left a comment
There was a problem hiding this comment.
Thank you for making a PR, @JiaqiWang18 .
This feature is still under development in the time frame of Apache Spark 4.1.0. Are we sure that this is not going to be used in the future? It seems too early for me to decide.
Since there is no way for user to see flow comments as of now, this PR removes the field.
|
cc @sryza , @HyukjinKwon , @peter-toth |
sryza
left a comment
There was a problem hiding this comment.
As someone who helped author the original PR that added this, my understanding is that this was added accidentally (out of unneeded consistency with the Table class), so I think removing it is the right thing to do. We can add it back if comments become meaningful in the context of flows.
Yeah, I think SDP don't have a way to display this to the user. We can add it back in if it gets one |
dongjoon-hyun
left a comment
There was a problem hiding this comment.
+1, LGTM per the above discussion.
|
Merged to master. |
What changes were proposed in this pull request?
commentfield from theFlowtrait, childUnresolvedFlowand `ResolutionCompletedFlowWhy are the changes needed?
In Spark Declarative Pipelines (SDP), the Flow trait has an unused
commentfield that is not being referenced anywhere.Since there is no way for user to see flow comments as of now, this PR removes the field.
Does this PR introduce any user-facing change?
No
How was this patch tested?
Updated unit test that references the field and made sure all other test passes.
Was this patch authored or co-authored using generative AI tooling?
No