Added printing outcome details by piotradamczyk5 · Pull Request #862 · Flank/flank

piotradamczyk5 · 2020-06-22T16:12:55Z

Fixes #829

Test Plan

How do we know the code works?

Result table displayed correct outcome details according to outcome status

Checklist

Documented
Unit tested
release_notes.md updated

bootstraponline · 2020-06-23T16:48:25Z

Let's make sure to test with gcloud CLI to ensure the table/outcomes match.

piotradamczyk5 · 2020-06-23T17:05:36Z

Let's make sure to test with gcloud CLI to ensure the table/outcomes match.

The implementation is the same as for gcloud (from the code)
What I changed is just replacing current testing device with matrix id (gcloud does not have option to run on multiple matrix) after consulting with other team members

bootstraponline · 2020-06-23T18:46:59Z

Makes sense.

adamfilipow92 · 2020-06-24T13:11:13Z

When you cancel test on pending status OutcomeDetailFormatter Crash here is stack trace, i try cancel on running status and stack trace is the same


java.lang.IllegalStateException: infrastructureFailure must not be null
	at ftl.util.OutcomeDetailsFormatterKt.formatOutcomeDetails(OutcomeDetailsFormatter.kt:57)
	at ftl.util.OutcomeDetailsFormatterKt.getDetails(OutcomeDetailsFormatter.kt:20)
	at ftl.json.SavedMatrix.updateOutcomeDetails(SavedMatrix.kt:127)
	at ftl.json.SavedMatrix.updateFinishedMatrixData(SavedMatrix.kt:109)
	at ftl.json.SavedMatrix.finished(SavedMatrix.kt:93)
	at ftl.json.SavedMatrix.update(SavedMatrix.kt:74)
	at ftl.json.MatrixMapKt.update(MatrixMap.kt:47)
	at ftl.run.NewTestRunKt$newTestRun$2$2.invokeSuspend(NewTestRun.kt:24)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.ResumeModeKt.resumeUninterceptedMode(ResumeMode.kt:45)
	at kotlinx.coroutines.internal.ScopeCoroutine.afterCompletionInternal(Scopes.kt:32)
	at kotlinx.coroutines.JobSupport.completeStateFinalization(JobSupport.kt:310)
	at kotlinx.coroutines.JobSupport.tryFinalizeSimpleState(JobSupport.kt:276)
	at kotlinx.coroutines.JobSupport.tryMakeCompleting(JobSupport.kt:807)
	at kotlinx.coroutines.JobSupport.makeCompletingOnce$kotlinx_coroutines_core(JobSupport.kt:787)
	at kotlinx.coroutines.AbstractCoroutine.resumeWith(AbstractCoroutine.kt:111)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46)
	at kotlinx.coroutines.ResumeModeKt.resumeUninterceptedMode(ResumeMode.kt:45)
	at kotlinx.coroutines.internal.ScopeCoroutine.afterCompletionInternal(Scopes.kt:32)
	at kotlinx.coroutines.JobSupport.completeStateFinalization(JobSupport.kt:310)
	at kotlinx.coroutines.JobSupport.tryFinalizeFinishingState(JobSupport.kt:236)
	at kotlinx.coroutines.JobSupport.tryMakeCompletingSlowPath(JobSupport.kt:849)
	at kotlinx.coroutines.JobSupport.tryMakeCompleting(JobSupport.kt:811)
	at kotlinx.coroutines.JobSupport.makeCompletingOnce$kotlinx_coroutines_core(JobSupport.kt:787)
	at kotlinx.coroutines.AbstractCoroutine.resumeWith(AbstractCoroutine.kt:111)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46)
	at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:241)
	at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:270)
	at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:79)
	at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:54)
	at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
	at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:36)
	at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
	at ftl.cli.firebase.test.android.AndroidRunCommand.run(AndroidRunCommand.kt:48)
	at picocli.CommandLine.executeUserObject(CommandLine.java:1769)
	at picocli.CommandLine.access$900(CommandLine.java:145)
	at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2150)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2144)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2108)
	at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:1975)
	at picocli.CommandLine.execute(CommandLine.java:1904)
	at DebugKt$main$1.invoke(Debug.kt:19)
	at DebugKt$main$1.invoke(Debug.kt)
	at ftl.util.Utils.withGlobalExceptionHandling(Utils.kt:130)
	at DebugKt.main(Debug.kt:18)
	at DebugKt.main(Debug.kt)

Process finished with exit code 1

Fixed

adamfilipow92

👍

jan-goral · 2020-06-24T14:02:52Z

We have to check this PR very carefully. There is a lot of conditions and we don't want to get wrong output on some of them.

pawelpasterz · 2020-06-24T13:26:49Z

+        matrix.testExecutions.createTestExecutionDataListAsync()
+            .map {
+                FinishedTestMatrixData(
+                    stepOutcome = GcToolResults.getExecutionResult(it.testExecution).outcome,


outcome might be null (it's java accessor)

Right. Anyway this is not correct way to obtain flay test outcome, we have to fix it

pawelpasterz · 2020-06-24T13:29:44Z

+    private fun updateFinishedMatrixData(matrix: TestMatrix) {
+        matrix.testExecutions.createTestExecutionDataListAsync()
+            .map {
+                FinishedTestMatrixData(


I am not fully convinced with this approach. First we map list to create list of FinishedTestMatrixData objects and with next step we destruct all of them and use its values.
I might be missing something here.

Right it would be better to have function instead of data class:

private fun updateFinishedMatrixData(matrix: TestMatrix) = matrix.testExecutions.createTestExecutionDataListAsync().forEach { updateAllOutcomes( stepOutcome = GcToolResults.getExecutionResult(it.testExecution).outcome, isVirtualDevice = AndroidCatalog.isVirtualDevice( it.testExecution.environment.androidDevice, matrix.projectId.orEmpty() ), testSuiteOverviewData = it.createTestSuitOverviewData(), billableMinutes = it.step.testExecutionStep?.testTiming?.testProcessDuration?.seconds ?.let { testTimeSeconds -> Billing.billableMinutes(testTimeSeconds) } ) } private fun updateAllOutcomes( stepOutcome: Outcome, isVirtualDevice: Boolean, testSuiteOverviewData: TestSuiteOverviewData?, billableMinutes: Long? ) { updateOutcome(stepOutcome) updateOutcomeDetails(stepOutcome, testSuiteOverviewData) billableMinutes?.let { updateBillableMinutes(it, isVirtualDevice) } }

I change the proposed name from updateAllOutcomes to updatedFinishedInfo, cause we update single outcome, details and minutes not all outcomes

We need to check this PR more carefully

jan-goral · 2020-06-24T17:39:43Z

+}
+
+private fun getSuccessOutcomeDetails(
+    testSuiteOverviewData: TestSuiteOverviewData,


If test TestSuiteOverviewData will be receiver this method, will looks:

private fun TestSuiteOverviewData.getSuccessOutcomeDetails( otherNativeCrash: Boolean ) = StringBuilder("$successCount test cases passed").apply { if (skipped > 0) append(skippedMessage(skipped)) if (flakes > 0) append(flakesMessage(flakes)) if (otherNativeCrash) append(NATIVE_CRASH_MESSAGE) }.toString()

and getDetails also will be shorter:

fun Outcome.getDetails( testSuiteOverviewData: TestSuiteOverviewData? ) = when (summary) { success, flaky -> testSuiteOverviewData?.getSuccessOutcomeDetails(successDetail?.otherNativeCrash ?: false) failure -> failureDetail.getFailureOutcomeDetails(testSuiteOverviewData) inconclusive -> inconclusiveDetail.formatOutcomeDetails() skipped -> skippedDetail.formatOutcomeDetails() unset -> "unset" else -> "unknown" }

jan-goral

The code looks great. But I noticed that flank in difference to gcloud is treating flaky outcome as success. That was introduced long time ago https://github.com/Flank/flank/pull/554/files#diff-0863ffce1d9b2a9c25d2c95e3d4c6022R104. @bootstraponline What was the reason? IMO flaky tests are worst than failed.

bootstraponline · 2020-06-26T04:48:25Z

But I noticed that flank in difference to gcloud is treating flaky outcome as success. That was introduced long time ago https://github.com/Flank/flank/pull/554/files#diff-0863ffce1d9b2a9c25d2c95e3d4c6022R104. @bootstraponline What was the reason? IMO flaky tests are worst than failed.

What happens if we treat flaky tests as flaky? For Flank, I think we need to consider the run to be successful for flaky tests. A flaky test is one that passed at least once. Flank should never error when all tests passed, even if they're flaky and passed on retry.

There's a larger roadmap item around metrics and a system to manage flakiness called guardian. I'll share more on that soon.

jan-goral · 2020-06-26T11:08:59Z

We have bug with displaying proper outcome. I commented line which treats flaky as success

    private fun updateOutcome(stepOutcome: Outcome?) {
        println("next outcome: ${stepOutcome?.summary}")
        outcome = when {
            stepOutcome == null -> unset
            // the matrix outcome is failure if any step fails
            // if the matrix outcome is already set to failure then we can ignore the other step outcomes.
            // inconclusive is treated as a failure
            outcome == failure || outcome == inconclusive -> return
//            stepOutcome.summary == flaky -> success
            else -> stepOutcome.summary
        }
    }

And the summary outcome is success

AndroidArgs
    gcloud:
      results-bucket: test-lab-v9cn46bb990nx-kz69ymd4nm9aq
      results-dir: 2020-06-26_09-32-15.508000_WMjM
      record-video: false
      timeout: 15m
      async: false
      client-details: 
      network-profile: null
      results-history-name: null
      # Android gcloud
      app: /Users/janek/projects/flank-project/flank/test_runner/src/test/kotlin/ftl/fixtures/tmp/apk/app-debug.apk
      test: /Users/janek/projects/flank-project/flank/test_runner/src/test/kotlin/ftl/fixtures/tmp/apk/app-multiple-flaky-debug-androidTest.apk
      additional-apks: 
      auto-google-login: false
      use-orchestrator: false
      directories-to-pull:
        - /sdcard/
      other-files:
      performance-metrics: false
      num-uniform-shards: null
      test-runner-class: null
      test-targets:
      robo-directives:
      robo-script: null
      device:
        - model: NexusLowRes
          version: 28
          locale: en
          orientation: portrait
      num-flaky-test-attempts: 3    flank:
      max-test-shards: 4
      shard-time: -1
      num-test-runs: 1
      smart-flank-gcs-path: 
      smart-flank-disable-upload: false
      files-to-download:
        - .*/sdcard/[^/]+\.ec$
      test-targets-always-run:
      disable-sharding: false
      project: flank-open-source
      local-result-dir: results
      full-junit-result: true
      # Android Flank Yml
      keep-file-path: false
      additional-app-test-apks:
      run-timeout: -1
      legacy-junit-result: false
      ignore-failed-tests: false
      output-style: singleRunTests Smart Flank cache hit: 0% (0 / 9)
  Shard times: 240s, 240s, 240s, 360s  Uploading app-debug.apk .
  Uploading app-multiple-flaky-debug-androidTest.apk .
  9 tests / 4 shards  1 matrix ids created in 0m 7s
  https://console.developers.google.com/storage/browser/test-lab-v9cn46bb990nx-kz69ymd4nm9aq/2020-06-26_09-32-15.508000_WMjMMatrices webLink
  matrix-3pu5ro3me2p6f https://console.firebase.google.com/project/flank-open-source/testlab/histories/bh.da0c237aaa33732/matrices/4938200577475745476  5m 37s Test executions status: FINISHED:7
  5m 37s matrix-3pu5ro3me2p6f FINISHEDFetchArtifacts
  .......
  Updating matrix fileCostReport
  Virtual devices
    $0.12 for 7m  Uploading CostReport.txt .
MatrixResultsReport
  1 / 1 (100.00%)
More details are available at [https://console.firebase.google.com/project/flank-open-source/testlab/histories/bh.da0c237aaa33732/matrices/4938200577475745476]
┌─────────┬──────────────────────┬─────────────────────┐
│ OUTCOME │      MATRIX ID       │    TEST DETAILS     │
├─────────┼──────────────────────┼─────────────────────┤
│ success │ matrix-3pu5ro3me2p6f │ 4 test cases passed │
└─────────┴──────────────────────┴─────────────────────┘
  Uploading MatrixResultsReport.txt .
  Uploading JUnitReport.xml .
  Uploading FullJUnitReport.xml .Process finished with exit code 0

bootstraponline · 2020-06-26T15:29:06Z

We have bug with displaying proper outcome. I commented line which treats flaky as success

If the exit code of Flank is still 0 when the outcome is corrected, and the Jenkins JUnit XML plugin still parses the results as successful then I think we can remove this flaky = success line.

adamfilipow92 · 2020-06-30T15:51:59Z

 enum class SystemOutColor(val ansiCode: String) {
    DEFAULT("\u001B[0m"),
    RED("\u001B[31m"),
+    BLUE("\u001B[34m"),


I known you add only blue color but probably ansiCode can be private.

cannot be cause it is used later in printing informations

adamfilipow92 · 2020-06-30T15:54:43Z

+    TableColumn(OUTCOME_DETAILS_COLUMN_HEADER, mapNotNull { it.outcomeDetails })
 )

 private fun getOutcomeColor(outcome: String): SystemOutColor {


Its only suggestion but, maybe something like:

private fun getOutcomeColor(outcome: String): SystemOutColor = // inconclusive is treated as a failure, flaky as a success when (outcome) { failure -> SystemOutColor.RED success -> SystemOutColor.GREEN flaky -> SystemOutColor.BLUE else -> SystemOutColor.DEFAULT }

changed 👍 and removed deprecated comment

piotradamczyk5 · 2020-06-30T16:27:32Z

+## Possible outputs
+Numbers are representing `OUTCOME` column, points are representing `TEST DETAILS` column.
+1. `success | flaky`
+    * `${1} test cases passed | ${2} skipped | ${3} flakes | (Native crash) | ---`


Suggested change

* `${1} test cases passed | ${2} skipped | ${3} flakes | (Native crash) | ---`

* `${1} test cases passed | ${2} skipped | ${3} flakes | (Native crash) | --- | Robo test`

bootstraponline · 2020-06-30T17:15:56Z

Let's embed the uml diagram in the markdown file.

Here's an example:

http://www.plantuml.com/plantuml/proxy?cache=no&fmt=svg&src=https://raw.githubusercontent.com/software-platform/monorepo/master/metrics/web/docs/diagrams/create_base_widget_activity_diagram.puml

adamfilipow92

👍

piotradamczyk5 self-assigned this Jun 22, 2020

piotradamczyk5 force-pushed the #829-added-outcome-details branch from a70adf1 to 5f9c388 Compare June 23, 2020 14:48

piotradamczyk5 marked this pull request as ready for review June 23, 2020 14:49

piotradamczyk5 requested review from adamfilipow92, bootstraponline, jan-goral and pawelpasterz June 23, 2020 14:50

piotradamczyk5 force-pushed the #829-added-outcome-details branch from ea423a5 to 252771d Compare June 24, 2020 13:23

adamfilipow92 previously approved these changes Jun 24, 2020

View reviewed changes

pawelpasterz reviewed Jun 24, 2020

View reviewed changes

jan-goral self-assigned this Jun 24, 2020

jan-goral reviewed Jun 24, 2020

View reviewed changes

jan-goral reviewed Jun 25, 2020

View reviewed changes

piotradamczyk5 force-pushed the #829-added-outcome-details branch from 742d2df to b5fb892 Compare June 25, 2020 18:00

piotradamczyk5 force-pushed the #829-added-outcome-details branch from d54c30a to bae8463 Compare June 29, 2020 18:35

adamfilipow92 reviewed Jun 30, 2020

View reviewed changes

piotradamczyk5 commented Jun 30, 2020

View reviewed changes

adamfilipow92 previously approved these changes Jun 30, 2020

View reviewed changes

pawelpasterz reviewed Jul 1, 2020

View reviewed changes

Piotr Adamczyk and others added 21 commits July 2, 2020 15:20

#829 Added printing outcome details

195efa7

#829 update tests

957bcfc

#829 update release_notes.md

a6e0a39

#829 fix OutcomeDetailsFormatter.kt

0cc62e3

#829 Fixing pr comments

8d3335e

#829 Handle unknown failure

943c48f

Simplify

9c170d2

Fix test

2d2d8a8

Add plant uml diagram for results_summary.py

4e0e266

Move OutcomeDetailsFormatter.kt to same packet as SavedMatrix.kt

2fadd9d

Fix

8102b71

#829 Display outcome details based on shards sum

1949804

Add summary_output.md doc

e513201

#829 Show robo test information

ccbbe90

#829 Fixed flaky outcome

5ba7230

#829 Fixed tests and change flaky outcome logic

418e1f9

Fix total count

3ca25de

Fix total count test

d6e49ac

#829 small refactor

1d58a1a

#829 PR fixes

45b4629

#829 PR fixes

d911dc5

piotradamczyk5 dismissed adamfilipow92’s stale review via d911dc5 July 2, 2020 13:51

piotradamczyk5 force-pushed the #829-added-outcome-details branch from 71d0bad to d911dc5 Compare July 2, 2020 13:51

piotradamczyk5 requested review from adamfilipow92, jan-goral and pawelpasterz July 2, 2020 13:57

pawelpasterz approved these changes Jul 2, 2020

View reviewed changes

piotradamczyk5 merged commit e1daf1a into master Jul 2, 2020

piotradamczyk5 deleted the #829-added-outcome-details branch July 2, 2020 18:09

	* `${1} test cases passed \| ${2} skipped \| ${3} flakes \| (Native crash) \| ---`
	* `${1} test cases passed \| ${2} skipped \| ${3} flakes \| (Native crash) \| --- \| Robo test`

Conversation

piotradamczyk5 commented Jun 22, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Plan

Checklist

Uh oh!

bootstraponline commented Jun 23, 2020

Uh oh!

piotradamczyk5 commented Jun 23, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bootstraponline commented Jun 23, 2020

Uh oh!

adamfilipow92 commented Jun 24, 2020 • edited by piotradamczyk5 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adamfilipow92 left a comment

Choose a reason for hiding this comment

Uh oh!

jan-goral commented Jun 24, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jan-goral Jun 24, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jan-goral left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bootstraponline commented Jun 26, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jan-goral commented Jun 26, 2020

Uh oh!

bootstraponline commented Jun 26, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

piotradamczyk5 Jun 30, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bootstraponline commented Jun 30, 2020

Uh oh!

adamfilipow92 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

piotradamczyk5 commented Jun 22, 2020 •

edited

Loading

piotradamczyk5 commented Jun 23, 2020 •

edited

Loading

adamfilipow92 commented Jun 24, 2020 •

edited by piotradamczyk5

Loading

jan-goral Jun 24, 2020 •

edited

Loading

jan-goral left a comment •

edited

Loading

bootstraponline commented Jun 26, 2020 •

edited

Loading

bootstraponline commented Jun 26, 2020 •

edited

Loading

piotradamczyk5 Jun 30, 2020 •

edited

Loading