#870 Added --directories-to-pull validation and avoid making request with empty toolStepResult

[piotradamczyk5]

Fixes #870

Test Plan

How do we know the code works?

NPE with missing toolStepResult was caused because we try to get results for an invalid matrix. Matrix was invalid because FTL rejects the request with invalid --directories-to-pull value.
To avoid this issue correct validation of this parameter was introduced, as well as filtering getting results for empty toolStepResult.
Gcloud policies were applied to --directories-to-pull:

• the path must start with /sdcard or /data/local/tmp
• the path must contain only characters a-zA-Z0-9_-./+

In order to test this issue, please use --directories-to-pull value which does not meet this 2 criteria.

Checklist

• Documented
• Unit tested
• release_notes.md updated

[adamfilipow92]

I noticed some differnces on master and your branche.

We have multiple-success yml

gcloud:
  app: ./src/test/kotlin/ftl/fixtures/tmp/apk/app-debug.apk
  test: ./src/test/kotlin/ftl/fixtures/tmp/apk/app-multiple-success-debug-androidTest.apk
  environment-variables: 
    coverage: true
    coverageFilePath: /sdcard/
    clearPackageData: true
  directories-to-pull: 
    - /sdcard/
  use-orchestrator: false

flank:
  disable-sharding: false
  max-test-shards: 2
  files-to-download:
    - .*/sdcard/[^/]+\.ec$

1. When I run this config on master its working
2. When I run this config with your changes its failing with message

Invalid value for [directories-to-pull]: Invalid path [/sdcard/].
Path must be absolute paths under /sdcard or /data/local/tmp (for example, --directories-to-pull /sdcard/tempDir1,/data/local/tmp/tempDir2).
Path names are restricted to the characters [a-zA-Z0-9_-./+]. 

Process finished with exit code 2

FIXED

[jan-goral]

1. I wonder why FTL is returning null for toolResultsStep?
2. Can we reproduce this error?
3. Its safe to skip some tests execution and get correct results?
4. Maybe flank should display proper warning about some unexpected behaviour?

[piotradamczyk5]

1. I wonder why FTL is returning null for toolResultsStep?
2. Can we reproduce this error?
3. Its safe to skip some tests execution and get correct results?
4. Maybe flank should display proper warning about some unexpected behaviour?

AD 1. If matrix is invalid FTL returns null as toolResultsStep
AD 2. This is more to avoid crashing flank in the future, so far, for now, I do not know how to reproduce this issue
AD 3. This is safe because tests results are based on a request made with toolResultsStep which will not succeed without toolResultsStep
AD 4. Ideally, we should not trigger INVALID matrix request, but for now, we do not know which settings will cause matrix invalid (see point 2)

[jan-goral]

There will be no table output if flank throw FTLError before MatrixResultsReport?

[piotradamczyk5]

FTLError is thrown after printing table

listOf(
    CostReport,
    MatrixResultsReport
).map {
    it.run(matrices, testSuite, printToStdout = true, args = args)  >>>>>>>>>>>>> PRINTING TABLE
}
if (!matrices.allSuccessful()) {
    listOf(
        HtmlErrorReport
    ).map { it.run(matrices, testSuite, printToStdout = false, args = args) }
}
JUnitReport.run(matrices, testSuite?.apply {
    if (ignoredTestCases.isNotEmpty()) {
        testsuites?.add(ignoredTestCases.toJunitTestsResults())
    }
}, printToStdout = false, args = args)
when {
    args.fullJUnitResult -> processFullJunitResult(args, matrices, testShardChunks)
    args.useLegacyJUnitResult -> processJunitXml(testSuite, args, testShardChunks)
}
matrices.validateMatrices(args.ignoreFailedTests) >>>>>>>>>>> FTLError