[SPARK-25117][R] Add EXEPT ALL and INTERSECT ALL support in R#22107
[SPARK-25117][R] Add EXEPT ALL and INTERSECT ALL support in R#22107dilipbiswal wants to merge 6 commits into
Conversation
|
Test build #94760 has finished for PR 22107 at commit
|
|
cc @felixcheung |
|
Test build #94764 has finished for PR 22107 at commit
|
dilipbiswal
changed the title
| #' intersectAllDF <- intersectAll(df1, df2) | ||
| #' } | ||
| #' @rdname intersectAll | ||
| #' @note intersectAll since 2.4 |
|
Test build #94777 has finished for PR 22107 at commit
|
| list("a", 1), | ||
| list("a", 1), | ||
| list("b", 3), | ||
| list("c", 4)), |
There was a problem hiding this comment.
nit:
list(list("a", 1), list("a", 1), list("a", 1),
list("a", 1), list("b", 3), list("c", 4)),
| schema = c("a", "b")) | ||
| df2 <- createDataFrame( | ||
| list(list("a", 1), list("a", 1), list("b", 3)), | ||
| schema = c("a", "b")) |
There was a problem hiding this comment.
nit:
df2 <- createDataFrame(list(list("a", 1), list("a", 1), list("b", 3)), schema = c("a", "b"))
| stringsAsFactors = FALSE) | ||
| except_all_expected <- data.frame("a" = c("a", "a", "c"), "b" = c(1, 1, 4), | ||
| stringsAsFactors = FALSE) | ||
| intersect_all_df <- arrange(intersectAll(df1, df2), df1$a) |
There was a problem hiding this comment.
Strictly, the naming rule is intersectAllDf or intersect.all.df (see #17590 (comment))
|
Seems fine. |
|
Test build #94789 has finished for PR 22107 at commit
|
|
retest this please |
|
Test build #94791 has finished for PR 22107 at commit
|
| #' df2 <- read.json(path2) | ||
| #' exceptAllDF <- exceptAll(df1, df2) | ||
| #' } | ||
| #' @rdname exceptAll |
There was a problem hiding this comment.
this is a bug in except there should only be one @rdname for each
There was a problem hiding this comment.
@felixcheung Thanks .. Did you want the original function except fixed at part of this ?
| #' df2 <- read.json(path2) | ||
| #' intersectAllDF <- intersectAll(df1, df2) | ||
| #' } | ||
| #' @rdname intersectAll |
| function(x, y) { | ||
| intersected <- callJMethod(x@sdf, "intersectAll", y@sdf) | ||
| dataFrame(intersected) | ||
| }) |
There was a problem hiding this comment.
add extra empty line after code
| excepted <- callJMethod(x@sdf, "exceptAll", y@sdf) | ||
| dataFrame(excepted) | ||
| }) | ||
|
|
There was a problem hiding this comment.
nit: remove one of the two empty lines
|
Test build #94844 has finished for PR 22107 at commit
|
|
@felixcheung I have incorporated the comments. |
|
merged to master |
|
Thank you very much @HyukjinKwon @felixcheung |
6 participants
What changes were proposed in this pull request?
SPARK-21274 added support for EXCEPT ALL and INTERSECT ALL. This PR adds the support in R.
How was this patch tested?
Added test in test_sparkSQL.R