Use API instead of XML for result parsing for android by jan-goral · Pull Request #666 · Flank/flank

jan-goral · 2020-03-13T17:03:51Z

Fixes #618, Fixes #545

Checklist

Documented
Unit tested
release_notes.md updated

pawelpasterz · 2020-03-13T22:01:17Z

+            // it is making api calls under the hood
+            testExecution.createTestExecutionData()
+        }
+    }.map { deferred ->


I think we can use .awaitAll() instead, just for simplification

bootstraponline · 2020-03-13T23:45:42Z

+    args: IArgs
+): JUnitTestResult = GcTestMatrix.refresh(
+    // Android uses server side sharding, it results only one matrix, so we can simply get first.
+    matrices.map.values.first().matrixId,


iOS doesn't use server side sharding right now. Is it possible to use the API for both Android & iOS?

Edit: I see that processXmlFromApi is only used for Android right now. That makes sense.

bootstraponline · 2020-03-13T23:46:27Z

+import java.io.File
+import kotlin.system.exitProcess
+
+object TmpV2 {


we should probably delete TmpV2 and turn this into a proper JUnit test. What do you think?

Also probably delete the original tmp

Of Course I will do that. This is only draft of implementation that I tested manually and looks correct. I now need to update unit tests, cleanup a little, and It will be ready for review.

bootstraponline · 2020-03-14T05:55:20Z

+package ftl.reports.api
+
+import com.google.api.services.toolresults.model.Duration
+import java.util.*


I think we have a lint rule about no wildcard imports.

bootstraponline · 2020-03-14T20:45:46Z

+        println(" - OK")
+
+        print("writing api results to file")
+        File(JUNIT_REPORT_FILE).writeText(jUnitTestResult.xmlToString())


To test this, run an Android app with multiple tests using server side sharding that has flaky tests. Use num-flaky-test-attempts set to a non-zero value such as 2. Then compare the JUnit XML file that FTL is creating vs the one we create via the API.

I was testing exactly that case. I know there is a difference between old and new JUnitReport.xml.
Old:

<?xml version='1.0' encoding='UTF-8' ?> <testsuites> <testsuite name="NexusLowRes-28-en-portrait#" tests="3" failures="0" errors="0" skipped="0" time="3.103" timestamp="2020-03-10T17:12:24" hostname="localhost"> <testcase name="test0" classname="com.example.test_app.InstrumentedTest" time="1.161"> <webLink></webLink> </testcase> <testcase name="test1" classname="com.example.test_app.InstrumentedTest" time="1.034"> <webLink></webLink> </testcase> <testcase name="test2" classname="com.example.test_app.InstrumentedTest" time="0.908"> <webLink></webLink> </testcase> </testsuite> </testsuites>

New:

<?xml version='1.0' encoding='UTF-8' ?> <testsuites> <testsuite name="NexusLowRes-28-en-portrait" tests="1" failures="0" errors="0" skipped="0" time="1.161" timestamp="2020-03-10T17:07:54.422Z"> <testcase name="test0" classname="com.example.test_app.InstrumentedTest" time="1.161"/> </testsuite> <testsuite name="NexusLowRes-28-en-portrait" tests="2" failures="0" errors="0" skipped="0" time="1.942" timestamp="2020-03-10T17:07:54.422Z"> <testcase name="test1" classname="com.example.test_app.InstrumentedTest" time="1.034"/> <testcase name="test2" classname="com.example.test_app.InstrumentedTest" time="0.908"/> </testsuite> </testsuites>

In new JUnitReport suites reflects shards. I can fix it easily, but that was first what I achieved. I guess that maybe is better to expose some info about shards in JUnitReport. If you don't like that I will fix it.

The timestamp is different yes. Old timestamp is simply timestamp of first of downloaded reports. New one is not correct I will fix it.

I need to also fix weblinks.

makes sense!

bootstraponline · 2020-03-14T22:31:26Z

When using Android Test Orchestrator, we'll want to verify the API test times match the times returned in the FTL XML file.

#557

codecov-io · 2020-03-23T05:22:58Z

Codecov Report

Merging #666 into master will decrease coverage by 0.99%.
The diff coverage is 65.95%.

@@             Coverage Diff              @@
##             master     #666      +/-   ##
============================================
- Coverage     77.10%   76.10%   -1.00%     
- Complexity      598      615      +17     
============================================
  Files            99      109      +10     
  Lines          2367     2528     +161     
  Branches        337      359      +22     
============================================
+ Hits           1825     1924      +99     
- Misses          335      381      +46     
- Partials        207      223      +16

pawelpasterz

Great job! That is one big PR with lot of work. I am really impressed.

The only thing I can complain is multiple usage of runBlocking. Personally I don't like when this is used as kind of bridge between synchronous and asynchronous code (when the first one is invoked inside a coroutine). But TBH I don't know if we have better solution right now.

Again, great job!

pawelpasterz · 2020-03-23T18:28:33Z

+    return listOf(map["Model"], map["Version"], map["Locale"], map["Orientation"]).joinToString("-")
+}
+
+private fun List<JUnitTestCase>.sumTime() = this


I think this is redundant here but I guess it was left intentionally for sake of readability?

That's right, intentionally for readability. I usually use this for extension functions with expression body. Thats allow clear chain.

pawelpasterz · 2020-03-23T18:35:39Z

+    .getTestExecutions()
+    .createJUnitTestResult()
+
+internal fun refreshTestMatrices(


I think we can make it private

pawelpasterz · 2020-03-23T18:35:51Z

+    }.awaitAll()
+}
+
+internal fun List<TestMatrix>.getTestExecutions(): List<TestExecution> = this


I think we can make it private

pawelpasterz · 2020-03-23T19:00:02Z

+        skipped = if (testCase.status == "skipped") null else "absent"
+    ).apply {
+        if (errors != null || failures != null) {
+            webLink = getWebLink(toolResultsStep, testCase.testCaseId)


pawelpasterz · 2020-03-23T19:12:35Z

+
+private fun TestExecution.createTestExecutionData(): TestExecutionData {
+    val response: ListTestCasesResponse = GcToolResults.listTestCases(toolResultsStep)
+    val step: Step = GcToolResults.getStepResult(toolResultsStep)


Both GcToolResults.listTestCases and GcToolResults.getStepResult are making api calls and seems that they don't depend on each other. Do you think we could make async call for each?

Right, I will fix that.

pawelpasterz · 2020-03-23T19:33:10Z

+
+fun Long.formatUtcDate() = utcDateFormat.format(this)!!
+
+var TestCase.flaky: Boolean by mutableMapProperty { false }


pawelpasterz · 2020-03-23T19:35:14Z

@@ -0,0 +1,34 @@
+package ftl.reports.api


We already have Utils.kt file, can we rename this one to sth like ApiUtils or somehow different? Just for sake of clarity.

Looks like you don't care about packages :). I guess adding part of package as prefix for file is redundant.

Yeah, true. Sometimes I am kind of bitchy :D ¯_(ツ)_/¯

Easy. You are doing your job. It is always better to express your opinion than to do nothing. There are no ultimate ways, everything we do is based on consensus.

pawelpasterz · 2020-03-23T19:40:53Z

+internal val xmlPrettyWriter = xmlMapper.writerWithDefaultPrettyPrinter()

 private fun xmlBytes(path: Path): ByteArray {
    if (!path.toFile().exists()) RuntimeException("$path doesn't exist!")


I am totally aware it is not part of your PR (so feel free to skip this comment). I think there is throw missing

Good checkup. I can fix it in this PR, no problem.

pawelpasterz · 2020-03-23T19:45:07Z


    @JacksonXmlProperty(isAttribute = true)
-    val hostname: String, // String.
+    val hostname: String? = null, // String.


Just for my understanding, are we ok with possible null here? Previous implementation did not predicted such value. Thanks!

I have asked @bootstraponline about that, he said #545 (comment). So I decide to change it to nullable here, because I didn't find hostname in API

Got it, thanks for explanation

pawelpasterz · 2020-03-23T19:51:40Z

-        Paths.get(args.localResultDir, FtlConstants.configFileName(args))
-    } else {
+    val configFilePath = if (args.useLocalResultDir())
+        Paths.get(args.localResultDir, FtlConstants.configFileName(args)) else


TBH I don't like it when I need to read whole line to find out there is else at the end. But it is matter of personal preferences :)

TBH I don't like if else and prefer using when but our detect config forces to use if else in cases like this. Anyway I like that kind of formatting cause of clear indentation same for two similar parts.

* Fix coroutines errors after rebase

jan-goral self-assigned this Mar 13, 2020

pawelpasterz reviewed Mar 13, 2020

View reviewed changes

bootstraponline reviewed Mar 13, 2020

View reviewed changes

bootstraponline reviewed Mar 14, 2020

View reviewed changes

jan-goral force-pushed the 545-junit-result-from-api branch 3 times, most recently from 7d905dc to 15d9f39 Compare March 23, 2020 05:18

jan-goral force-pushed the 545-junit-result-from-api branch from 80ecae9 to 25e5867 Compare March 23, 2020 09:42

jan-goral marked this pull request as ready for review March 23, 2020 11:29

bootstraponline previously approved these changes Mar 23, 2020

View reviewed changes

pawelpasterz previously approved these changes Mar 23, 2020

View reviewed changes

jan-goral dismissed stale reviews from pawelpasterz and bootstraponline via 3d5e8bf March 24, 2020 06:33

pawelpasterz self-requested a review March 24, 2020 06:35

pawelpasterz previously approved these changes Mar 24, 2020

View reviewed changes

jan-goral dismissed pawelpasterz’s stale review via e7bd0b0 March 24, 2020 08:51

jan-goral added 10 commits March 24, 2020 09:54

Use API instead of XML for result parsing for android

762de61

CR fixes

260cd77

Fix failing tests

b3985af

Add web link to JUnitTestCase generated from api

b473fa2

* Fix coroutines errors after rebase

Fix timestamp for JUnitTestSuite generated from api

84bfcf2

Fix failed test case filtering for flaky tests

a5610f4

Add additional info about flaky tests

3536005

Add support for multiple matrices

f8c505d

Add legacy-junit-result flag

9d3b4a3

Add tests for createJUnitTestCases & createTestSuitOverviewData

24bbe8e

jan-goral added 7 commits March 24, 2020 09:59

Add test for createJUnitTestSuites

7fb7c1f

Add test for prepareForJUnitResult

33899b6

Cleanup

2c1ce19

Cleanup

5428e82

Cleanup

af78de7

CR fixes

dc48a10

Update release notes

169c7b5

jan-goral force-pushed the 545-junit-result-from-api branch from e7bd0b0 to 169c7b5 Compare March 24, 2020 08:59

pawelpasterz approved these changes Mar 24, 2020

View reviewed changes

jan-goral merged commit f114c83 into master Mar 24, 2020

jan-goral deleted the 545-junit-result-from-api branch March 24, 2020 09:36

jan-goral mentioned this pull request Apr 22, 2020

com.fasterxml.jackson.module.kotlin.MissingKotlinParameterException in v8.1.0 #735

Closed

RainNapper mentioned this pull request Jun 24, 2020

Web link missing from successful test results #867

Closed


		fun Long.formatUtcDate() = utcDateFormat.format(this)!!

		var TestCase.flaky: Boolean by mutableMapProperty { false }

Conversation

jan-goral commented Mar 13, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bootstraponline Mar 13, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jan-goral Mar 16, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bootstraponline commented Mar 14, 2020

Uh oh!

codecov-io commented Mar 23, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

pawelpasterz left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jan-goral Mar 24, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jan-goral Mar 24, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jan-goral Mar 24, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jan-goral Mar 24, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

jan-goral commented Mar 13, 2020 •

edited

Loading

bootstraponline Mar 13, 2020 •

edited

Loading

jan-goral Mar 16, 2020 •

edited

Loading

codecov-io commented Mar 23, 2020 •

edited

Loading

pawelpasterz left a comment •

edited

Loading

jan-goral Mar 24, 2020 •

edited

Loading

jan-goral Mar 24, 2020 •

edited

Loading

jan-goral Mar 24, 2020 •

edited

Loading

jan-goral Mar 24, 2020 •

edited

Loading