[SPARK-2109] Setting SPARK_MEM for bin/pyspark does not work.#1050
[SPARK-2109] Setting SPARK_MEM for bin/pyspark does not work.#1050ScrapCodes wants to merge 2 commits into
Conversation
|
Merged build triggered. |
|
Merged build started. |
|
Merged build finished. All automated tests passed. |
|
All automated tests passed. |
|
LGTM |
|
This looks ok, but note that this isn't the only thing that can cause pyspark to not work. e.g. "SPARK_PRINT_LAUNCH_COMMAND=1 bin/pyspark" also fails for the same reason. Perhaps a more long-term solution would be to not use stdout for communicating the port number... |
|
Merged build triggered. |
|
Merged build started. |
|
@vanzin You are right, for now I have done the audit of all options available. Do you have an alternative? I did not look deeply enough. Will see if there is something simple can be done as an alternative. |
|
Merged build finished. All automated tests passed. |
|
All automated tests passed. |
|
@ScrapCodes there are alternatives if you're willing to write more code. e.g., write the port to a separate file, or write the port with some specific formatting and look for it in the output instead of relying on the first line from the child process. |
|
Also, your latest fixes use the block EOF sometimes, and multiple lines of echo sometimes. It would be good if we could keep this consistent. |
|
Build triggered. |
|
Build started. |
|
Hey @andrewor14, I sort of avoided EOF blocks for nested ifs, since that can look a bit less nicer because of no indentation. |
|
Build finished. |
|
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15754/ |
There was a problem hiding this comment.
Indentation is weird here
There was a problem hiding this comment.
If I indent the contents of this EOF block, then those indentation will become part of the string that gets printed on the screen. I can still do it, if you think ? But I think some people will object to that.
|
Hi @ScrapCodes. There are a couple of weird indentations, but other than that this looks good. |
|
Hi @andrewor14, added a comment about them. Not sure if we can fix them. |
|
@ScrapCodes: in that case, wouldn't it be better to just use echo everywhere? Yeah, it's a little more verbose, but doesn't suffer from the indentation issue. (It would be probably be cleaner to have an "error" function somewhere that does the correct redirection, but that would mean having a common script with utility functions that is sources from every script...) |
|
Yeah, I personally prefer echoes. The indentation thing is a little hard to track sometimes, especially because this is bash. |
|
Build triggered. |
|
Build started. |
|
Build finished. |
|
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15846/ |
|
Jenkins, retest this please. |
|
Build triggered. |
|
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15847/ |
|
You are getting test failures because the patch doesn't merge cleanly. |
|
Merged build triggered. |
|
Merged build started. |
|
Merged build finished. |
|
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16026/ |
|
I just checked it merges cleanly and still these test failures. |
|
Jenkins, retest this please. |
|
Merged build triggered. |
|
Merged build started. |
|
Merged build finished. |
|
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16028/ |
|
Jenkins, test this please |
|
Merged build triggered. |
|
Merged build started. |
|
Merged build finished. All automated tests passed. |
|
All automated tests passed. |
|
LGTM |
Trivial fix. Author: Prashant Sharma <prashant.s@imaginea.com> Closes #1050 from ScrapCodes/SPARK-2109/pyspark-script-bug and squashes the following commits: 77072b9 [Prashant Sharma] Changed echos to redirect to STDERR. 13f48a0 [Prashant Sharma] [SPARK-2109] Setting SPARK_MEM for bin/pyspark does not work. (cherry picked from commit 731f683) Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Trivial fix. Author: Prashant Sharma <prashant.s@imaginea.com> Closes #1050 from ScrapCodes/SPARK-2109/pyspark-script-bug and squashes the following commits: 77072b9 [Prashant Sharma] Changed echos to redirect to STDERR. 13f48a0 [Prashant Sharma] [SPARK-2109] Setting SPARK_MEM for bin/pyspark does not work. (cherry picked from commit 731f683) Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Trivial fix. Author: Prashant Sharma <prashant.s@imaginea.com> Closes apache#1050 from ScrapCodes/SPARK-2109/pyspark-script-bug and squashes the following commits: 77072b9 [Prashant Sharma] Changed echos to redirect to STDERR. 13f48a0 [Prashant Sharma] [SPARK-2109] Setting SPARK_MEM for bin/pyspark does not work.
5 participants
Trivial fix.