[SPARK-8844][SPARKR] head/collect is broken in SparkR.#7419
Conversation
|
Test build #37361 has finished for PR 7419 at commit
|
There was a problem hiding this comment.
So fundamentally it looks like this we want to create a vector with mode = colType if the column is empty. But does it matter if we get the colType wrong ? i.e. what happens if we just use the default mode (which I think is logical)
|
cc @davies |
|
@sun-rui Any update on this ? |
|
@shivaram , as you pointed out, I come with a simpler fix. I realized that simply creating an empty vector using vector() without mode is OK when there is no row in columns. Because vector() creates a zero-length vector of type "logical", and "logical" is of virutally lowest precedence in the R's type hierarchy, so it is supposed that "logical" type won't affect later coercion in operations on a data.frame of 0 row, for example, rbind(). |
|
Jenkins, retest this please |
|
Thanks @sun-rui -- this fix is simple and looks good. Couple of minor things
|
|
Test build #40969 has finished for PR 7419 at commit
|
|
@shivaram, fixed. |
|
LGTM |
|
Test build #40978 has finished for PR 7419 at commit
|
|
LGTM. Merging this |
This is a WIP patch for SPARK-8844 for collecting reviews. This bug is about reading an empty DataFrame. in readCol(), lapply(1:numRows, function(x) { does not take into consideration the case where numRows = 0. Will add unit test case. Author: Sun Rui <rui.sun@intel.com> Closes #7419 from sun-rui/SPARK-8844. (cherry picked from commit 5f9ce73) Signed-off-by: Shivaram Venkataraman <shivaram@cs.berkeley.edu>
This is a WIP patch for SPARK-8844 for collecting reviews.
This bug is about reading an empty DataFrame. in readCol(),
lapply(1:numRows, function(x) {
does not take into consideration the case where numRows = 0.
Will add unit test case.
Author: Sun Rui <rui.sun@intel.com>
Closes apache#7419 from sun-rui/SPARK-8844.
This is a WIP patch for SPARK-8844 for collecting reviews.
This bug is about reading an empty DataFrame. in readCol(),
lapply(1:numRows, function(x) {
does not take into consideration the case where numRows = 0.
Will add unit test case.