Resolve Flaky CI builds

[martincostello]

Our AppVeyor CI build has been flaky for a while now, which causes unnecessary complication to making changes, and with different permissions to the GitHub repository makes being able to do things like retry a build difficult to achieve despite write access to the repo.

As a stop-gap measure, I added a Windows CI using GitHub Actions in #979 to give an alternative view of whether the project is building successfully.

The question is, what should we do to push forward and get CI stable again so that contributors can be confident in changes they submit?

Two possible approaches include:

1. Fix the AppVeyor CI and update the permissions so those with write access to the repo can do things like retry builds.
2. Move away from AppVeyor, and use GitHub Actions instead.

Cards on the table, I'm in favour of item 2 because I use Actions exclusively for my own open source projects and at work, and I'm a big fan of having one CI/CD system, rather than multiples. It also has the benefit of allowing us to build and test on Linux, macOS and Windows in a single CI/CD system.

I am however happy to go with the majority position if that is to keep the status quo of AppVeyor.

If we decided to move to GitHub Actions, the following things are missing from the basic one I've set up already:

1. The build doesn't yet work for macOS and Linux;
2. The created NuGet packages are missing the appropriate SemVer;
3. The build isn't currently triggered by tags (and doesn't do any tagging either).
4. Something like GitHubActionsTestLogger needs adding to show test results in the build logs.
5. NuGet packages aren't published anywhere.

[martincostello]

Discussed with @joelhulen and @SimonCropp, and (probably tomorrow) I will:

• Remove the AppVeyor CI
• Investigate fixing the build to include Linux and macOS
• Investigate adding codecov coverage metrics to give us a coverage baseline for the future
• Investigate how we could create a workflow for making publishing easier while still keeping the existing gates on what gets released and who can do so

[joelhulen]

100% in agreement to move away from AppVeyor in favor of GitHub Actions. We implemented AppVeyor before GitHub Actions was a thing. Plus, AppVeyor gave us free build minutes each month since this is an Open Source project. Since then, GitHub Actions has become a viable (and, in many ways, better) alternative. Moving to GH Actions will also open the door to implementing policies like code coverage, improving deployment options, etc.

Thanks, Martin!

[martincostello]

@joelhulen I've deleted the AppVeyor config, but it's still trying to build things. I guess there's something you need to turn off somewhere in the AppVeyor website.

[joelhulen]

@martincostello I'll look into it now.

[joelhulen]

@martincostello I've deleted the Polly project from AppVeyor.

As for GitHub, we want to create two deployments: GitHub releases and nuget.org. I can help set up the NuGet piece when you're ready.

[joelhulen]

As for the releases, those need to be gated so that an admin needs to approve the deployments.

[martincostello]

Cheers Joel - I'm working on the CI now in #995, then I'll sort out xplat, then once that's done look a the publish/release flow.

[martincostello]

The CI is now working correctly with respect to:

• Versioning
• Test results
• Deterministic builds
• Code coverage (we're currently at 71% in codecov.io)

Next up I'll look at sorting out the Linux and macOS builds.