[SPARK-1112, 2156] (1.0 edition) Use correct akka frame size and overhead amounts.#1172
[SPARK-1112, 2156] (1.0 edition) Use correct akka frame size and overhead amounts.#1172pwendell wants to merge 2 commits into
Conversation
…head amounts. SPARK-1112: This is a more conservative version of apache#1132 that doesn't change around the actor system initialization on the executor. Instead we just directly read the current frame size limit from the ActorSystem. SPARK-2156: This uses the same fixe as in apache#1132.
|
@mengxr - do you mind reviewing this? |
|
Merged build triggered. |
|
Merged build started. |
|
Merged build finished. |
|
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15998/ |
|
Jenkins, retest it please. |
|
LGTM. I will merge it to branch-1.0 if Jenkins is happy. |
There was a problem hiding this comment.
Slightly confused on this -- won't this make LocalBackend and MesosExecutorBackend never use the BlockManager for results, causing them to have the undesired behavior?
There was a problem hiding this comment.
The MesosExecutorBackend sends results through mesos, not akka. The LocalBackend sends a message to an actor within the same actor system... which I assumed won't go over TCP.
There was a problem hiding this comment.
I see. So the only real change is that in certain cases the LocalBackend will no longer use the BlockManager for returning results. Sounds fine.
There was a problem hiding this comment.
So that change actually alters the expectations of the unit tests, so I went ahead and just enforced the limit in the LocalBackend anwyays.
|
Jenkins, retest this please. |
|
Merged build triggered. |
|
Merged build started. |
|
Merged build finished. |
|
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16007/ |
|
Jenkins, retest this please. |
|
Merged build triggered. |
|
Merged build started. |
|
Merged build finished. |
|
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16010/ |
|
I think these are failing because our tests assume that in local mode we enforce the frame size limit (which we actually don't need to). I'll make the appropriate adjustments in a bit. |
|
Jenkins, retest this please. |
|
Merged build triggered. |
|
Merged build started. |
|
Merged build finished. All automated tests passed. |
|
All automated tests passed. |
|
@aarondav mind taking a final pass and merging this? |
|
Absolutely. LGTM, merging into branch-1.0. |
…head amounts. SPARK-1112: This is a more conservative version of #1132 that doesn't change around the actor system initialization on the executor. Instead we just directly read the current frame size limit from the ActorSystem. SPARK-2156: This uses the same fixe as in #1132. Author: Patrick Wendell <pwendell@gmail.com> Closes #1172 from pwendell/akka-10-fix and squashes the following commits: d56297e [Patrick Wendell] Set limit in LocalBackend to preserve test expectations 9f5ed19 [Patrick Wendell] [SPARK-1112, 2156] (1.0 edition) Use correct akka frame size and overhead amounts.
|
You may have to close this manually, @pwendell, I'm not sure github will close it if it's not in master. |
|
Thanks, closed. |
4 participants
SPARK-1112: This is a more conservative version of #1132 that doesn't change
around the actor system initialization on the executor. Instead we just
directly read the current frame size limit from the ActorSystem.
SPARK-2156: This uses the same fixe as in #1132.