[tests-only] [full-ci] Report passed-on-retry to RocketChat

[phil-davis]

Description

1. Process comments for passedOnRetry in a separate file (so that we can easily have them separate to any other comments to be posted to a GitHub PR)

2. coming...

How Has This Been Tested?

CI

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Technical debt
• Tests

Checklist:

• Code changes
• Unit tests added
• Acceptance tests added
• Documentation ticket raised:

[ownclouders]

Results for oC10CreateDelete https://drone.owncloud.com/owncloud/web/16920/7/1
The following scenarios passed on retry:

• webUIDeleteFilesFolders/deleteFilesFolders.feature:59

[ownclouders]

Results for oCISSharingBasic https://drone.owncloud.com/owncloud/web/16920/47/1
The following scenarios passed on retry:

• webUIRestrictSharing/restrictSharing.feature:49

[ownclouders]

Results for oCISSharingBasic https://drone.owncloud.com/owncloud/web/16930/47/1
The following scenarios passed on retry:

• webUIRestrictSharing/restrictSharing.feature:49

[phil-davis]

With the first commit, I didn't break the existing behavior. "Passed on retry" scenarios are still being reported here as GitHub comments - good.

[ownclouders]

Results for oCISSharingBasic https://drone.owncloud.com/owncloud/web/16945/47/1
The following scenarios passed on retry:

• webUIRestrictSharing/restrictSharing.feature:49

[ownclouders]

Results for oCISSharingBasic https://drone.owncloud.com/owncloud/web/17027/47/1
The following scenarios passed on retry:

• webUIRestrictSharing/restrictSharing.feature:49

[ownclouders]

Results for oCISSharingBasic https://drone.owncloud.com/owncloud/web/17029/47/1
The following scenarios passed on retry:

• webUIRestrictSharing/restrictSharing.feature:49

[ownclouders]

Results for oCISSharingBasic https://drone.owncloud.com/owncloud/web/17030/47/1
The following scenarios passed on retry:

• webUIRestrictSharing/restrictSharing.feature:49

[ownclouders]

Results for oC10Basic https://drone.owncloud.com/owncloud/web/17030/6/1
The following scenarios passed on retry:

• webUIPreview/imageMediaViewer.feature:35

[ownclouders]

Results for oCISSharingBasic https://drone.owncloud.com/owncloud/web/17031/47/1
The following scenarios passed on retry:

• webUIRestrictSharing/restrictSharing.feature:49

[phil-davis]

It reports nicely to RocketChat when there is a passed-on-retry scenario.
Now I need to work out how to bypass reporting when there is no problem to report, without crashing the build.

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
No Duplication information

[ownclouders]

Results for oC10iPhone1 https://drone.owncloud.com/owncloud/web/17032/43/1
The following scenarios passed on retry:

• webUIDeleteFilesFolders/deleteFilesFolders.feature:59

[ownclouders]

Results for oCISSharingBasic https://drone.owncloud.com/owncloud/web/17032/47/1
The following scenarios passed on retry:

• webUIRestrictSharing/restrictSharing.feature:49

[phil-davis]

I tried having an empty file by default for doing the report to RocketChat. That stops things from failing, but I get lots of empty messages spamming the builds channel!

Now to think of a way to avoid the notify-passedOnRetry when there is nothing to notify.

[pascalwengerter]

@phil-davis what's the state here? .drone.star has changed quite a bit so probably needs a rebase, I'm not sure how valuable the report in Rocketchat is though. I'm mostly using my email for notifications (in the case of retries it's notifications about GH comments, but still). Also, few people from the QA team and hardly anyone from the web team is in the respective channel (#builds)

[phil-davis]

@pascalwengerter for nightly runs we want to know somewhere about tests that passed on retry. That nightly CI is a chance to regularly find out if tests are flaky, and that the full test suite is still passing.

Another thought is to not enable "test retry" in the nightly run. Whatever test scenarios fail, let them cause the pipeline to fail, and let all pipelines run to completion. That will report the nightly build to rocketchat idfit fails - and then we can easily look through the drone output of pipelines that fail. That might be a cleaner approach anyway.

I do need to rebase this!

Also, I look at the builds channel every day and we look into any nightly CI that fails.

[phil-davis]

See #5979 for a different approach.