Skip to content

[SPARK-32647][INFRA] Report SparkR test results with JUnit reporter#29456

Closed
HyukjinKwon wants to merge 5 commits into
apache:masterfrom
HyukjinKwon:sparkr-junit
Closed

[SPARK-32647][INFRA] Report SparkR test results with JUnit reporter#29456
HyukjinKwon wants to merge 5 commits into
apache:masterfrom
HyukjinKwon:sparkr-junit

Conversation

@HyukjinKwon

@HyukjinKwon HyukjinKwon commented Aug 18, 2020 •
edited
Loading

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

This PR proposes to generate JUnit XML test report in SparkR tests that can be leveraged in both Jenkins and GitHub Actions.

GitHub Actions

Screen Shot 2020-08-18 at 12 42 46 PM

Jenkins

Screen Shot 2020-08-18 at 2 03 42 PM

NOTE that while I am here, I am switching back the console reporter from "progress" to "summary". Currently non-ascii codes are broken in Jenkins console and switching it to "summary" can work around it.
"summary" is the default format used in testthat 1.x.

Why are the changes needed?

To check the test failures more easily.

Does this PR introduce any user-facing change?

No, dev-only

How was this patch tested?

It is tested in GitHub Actions at https://github.com/HyukjinKwon/spark/pull/23/checks?check_run_id=996586446
In case of Jenkins, https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127525/testReport/

@HyukjinKwon HyukjinKwon marked this pull request as draft August 18, 2020 03:45
@SparkQA

SparkQA commented Aug 18, 2020

Copy link
Copy Markdown

Test build #127525 has finished for PR 29456 at commit 8fb651a.

  • This patch fails SparkR unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@HyukjinKwon HyukjinKwon changed the title [WIP][SPARK-32647][INFRA] Report SparkR test results with JUnit reporter [SPARK-32647][INFRA] Report SparkR test results with JUnit reporter Aug 18, 2020
@HyukjinKwon
Revert "Add test failures"
5bca181 
This reverts commit 8fb651a.
@HyukjinKwon HyukjinKwon marked this pull request as ready for review August 18, 2020 05:17
@HyukjinKwon

HyukjinKwon commented Aug 18, 2020

Copy link
Copy Markdown
Member Author

cc @dongjoon-hyun, @viirya, @felixcheung, @shivaram, @gengliangwang FYI

@HyukjinKwon

HyukjinKwon commented Aug 18, 2020 •
edited
Loading

Copy link
Copy Markdown
Member Author

Note that PySpark, Scala and Java are already doing this. SparkR is only the one which currently does not leverage JUnit XML reports. This PR completes and consistently supports it in all languages with the consistent directory names (target/test-reports).

Jenkins and GitHub Actions are already configured to look up these directories to report in test results.

viirya
viirya reviewed Aug 18, 2020
View reviewed changes
Comment thread R/pkg/tests/run-all.R
dir.create("target/test-reports", showWarnings = FALSE)
reporter <- MultiReporter$new(list(
SummaryReporter$new(),
JunitReporter$new(file = "target/test-reports/test-results.xml")

@viirya viirya Aug 18, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it only supported by testthat 2.0+?

@HyukjinKwon HyukjinKwon Aug 18, 2020 •
edited
Loading

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks supported in testthat 1.x but testthat 1.x support here is supposed to be removed soon at SPARK-30663. I didn't bother test as Jenkins and GitHub Actions use testthat 2+ anyway.

viirya
viirya reviewed Aug 18, 2020
View reviewed changes
Comment thread R/pkg/tests/run-all.R
Comment on lines +73 to +74
SummaryReporter$new(),
JunitReporter$new(file = "target/test-reports/test-results.xml")

@viirya viirya Aug 18, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need both SummaryReporter and JunitReporter?

@HyukjinKwon HyukjinKwon Aug 18, 2020

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup, otherwise, the console output is not shown. I manually tested.

It was previously ProgressReporter (from testthat 2+) by default. It shows the console output such as:

��� |  0       | SparkSQL functions
��� |  1       | SparkSQL functions
��� |  2       | SparkSQL functions
...

in Jenkins (ever since we switched the default system encoding if I am not mistaken).

Now it follows the default format SummaryReporter (in testthat 1.x). It shows the console output as below:

functions in client.R: .....
test functions in sparkR.R: ..............................................
include R packages: ..
JVM API: ..
MLlib classification algorithms, except for tree-based algorithms: ...........................................................................

There are no differences in the report format when some tests are failed.

@HyukjinKwon

HyukjinKwon commented Aug 18, 2020

Copy link
Copy Markdown
Member Author

oh, cc @zero323 too.

@SparkQA

SparkQA commented Aug 18, 2020

Copy link
Copy Markdown

Test build #127533 has finished for PR 29456 at commit 5bca181.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@HyukjinKwon

HyukjinKwon commented Aug 18, 2020

Copy link
Copy Markdown
Member Author

Thanks guys. Let me merge this after I cherry-pick Github Actions to other branches (at #29460)

@SparkQA

SparkQA commented Aug 18, 2020

Copy link
Copy Markdown

Test build #127544 has finished for PR 29456 at commit 603268e.

  • This patch fails due to an unknown error code, -9.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA

SparkQA commented Aug 18, 2020

Copy link
Copy Markdown

Test build #127557 has finished for PR 29456 at commit 1fe2806.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@HyukjinKwon

HyukjinKwon commented Aug 18, 2020

Copy link
Copy Markdown
Member Author

Merged to master.

HyukjinKwon added a commit that referenced this pull request Aug 19, 2020
@HyukjinKwon
[SPARK-32647][INFRA] Report SparkR test results with JUnit reporter
b3a971a 
This PR proposes to generate JUnit XML test report in SparkR tests that can be leveraged in both Jenkins and GitHub Actions.

**GitHub Actions**

![Screen Shot 2020-08-18 at 12 42 46 PM](https://user-images.githubusercontent.com/6477701/90467934-55b85b00-e150-11ea-863c-c8415e764ddb.png)

**Jenkins**

![Screen Shot 2020-08-18 at 2 03 42 PM](https://user-images.githubusercontent.com/6477701/90472509-a5505400-e15b-11ea-9165-777ec9b96eaa.png)

NOTE that while I am here, I am switching back the console reporter from "progress" to "summary". Currently non-ascii codes are broken in Jenkins console and switching it to "summary" can work around it.
"summary" is the default format used in testthat 1.x.

To check the test failures more easily.

No, dev-only

It is tested in GitHub Actions at https://github.com/HyukjinKwon/spark/pull/23/checks?check_run_id=996586446
In case of Jenkins, https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127525/testReport/

Closes #29456 from HyukjinKwon/sparkr-junit.

Authored-by: HyukjinKwon <gurwls223@apache.org>
Signed-off-by: HyukjinKwon <gurwls223@apache.org>
@HyukjinKwon

HyukjinKwon commented Aug 19, 2020 •
edited
Loading

Copy link
Copy Markdown
Member Author

I merged into branch-3.0 and branch-2.4 as well. cc @zhengruifeng and @ScrapCodes . this is test-only so should be fine.

HyukjinKwon added a commit that referenced this pull request Aug 19, 2020
@HyukjinKwon
[SPARK-32647][INFRA] Report SparkR test results with JUnit reporter
ff8a663 
This PR proposes to generate JUnit XML test report in SparkR tests that can be leveraged in both Jenkins and GitHub Actions.

**GitHub Actions**

![Screen Shot 2020-08-18 at 12 42 46 PM](https://user-images.githubusercontent.com/6477701/90467934-55b85b00-e150-11ea-863c-c8415e764ddb.png)

**Jenkins**

![Screen Shot 2020-08-18 at 2 03 42 PM](https://user-images.githubusercontent.com/6477701/90472509-a5505400-e15b-11ea-9165-777ec9b96eaa.png)

NOTE that while I am here, I am switching back the console reporter from "progress" to "summary". Currently non-ascii codes are broken in Jenkins console and switching it to "summary" can work around it.
"summary" is the default format used in testthat 1.x.

To check the test failures more easily.

No, dev-only

It is tested in GitHub Actions at https://github.com/HyukjinKwon/spark/pull/23/checks?check_run_id=996586446
In case of Jenkins, https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127525/testReport/

Closes #29456 from HyukjinKwon/sparkr-junit.

Authored-by: HyukjinKwon <gurwls223@apache.org>
Signed-off-by: HyukjinKwon <gurwls223@apache.org>
@felixcheung

felixcheung commented Aug 23, 2020 via email

Copy link
Copy Markdown
Member

@HyukjinKwon HyukjinKwon deleted the sparkr-junit branch December 7, 2020 02:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@viirya viirya viirya approved these changes
@gengliangwang gengliangwang gengliangwang approved these changes

Assignees

No one assigned

Labels

INFRA R SQL

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@HyukjinKwon @SparkQA @felixcheung @viirya @gengliangwang