[SPARK-37984][SHUFFLE] Avoid calculating all outstanding requests to improve performance.#35276
[SPARK-37984][SHUFFLE] Avoid calculating all outstanding requests to improve performance.#35276weixiuli wants to merge 2 commits into
Conversation
…improve performance.
|
Friendly cc @srowen @dongjoon-hyun |
|
Does this change performance much? it's still getting the size of every collection. isEmpty might make it a little more optimal. OK either way just might not make any difference |
Done. |
|
Can one of the admins verify this patch? |
srowen
left a comment
srowen
left a comment
There was a problem hiding this comment.
Are there are any other places where numOutstandingRequests is compared to 0?
This seems OK though I doubt it will make a difference
No, I checked again and confirmed that the PR included all of them. @srowen |
|
Merged to master |
|
Thanks @srowen |
… performance ### What changes were proposed in this pull request? Refer [SPARK-37894](https://issues.apache.org/jira/browse/SPARK-37984)/ apache/spark#35276 Avoid calculating all outstanding requests to improve performance ### Why are the changes needed? Ditto. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Not needed. Closes #2319 from turboFei/SPARK-37984_backport. Authored-by: Fei Wang <fwang12@ebay.com> Signed-off-by: mingji <fengmingxiao.fmx@alibaba-inc.com>
… performance ### What changes were proposed in this pull request? Refer [SPARK-37894](https://issues.apache.org/jira/browse/SPARK-37984)/ apache/spark#35276 Avoid calculating all outstanding requests to improve performance ### Why are the changes needed? Ditto. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Not needed. Closes #2319 from turboFei/SPARK-37984_backport. Authored-by: Fei Wang <fwang12@ebay.com> Signed-off-by: mingji <fengmingxiao.fmx@alibaba-inc.com> (cherry picked from commit 621a719) Signed-off-by: mingji <fengmingxiao.fmx@alibaba-inc.com>
3 participants
What changes were proposed in this pull request?
Avoid calculating all outstanding requests to improve performance.
Why are the changes needed?
Follow the comment (#34711 (review)) , we can implement a "has outstanding requests" method in the response handler that doesn't even need to get a count，let's do this with PR.
Does this PR introduce any user-facing change?
No
How was this patch tested?
Exist unittests.