Official build publish order problem: downstream update PRs created before assets are ready

[dagood]

We have an ordering issue with the custom publish step:

https://dev.azure.com/dnceng/internal/_build/results?buildId=316758&view=results

Promoting the build to the BAR channel causes an auto-update PR in Core-SDK. The custom publish step uploads to dotnetcli. The Core-SDK PR's validation can fail because it depends on the dotnetcli endpoint.

Ideas:

1. Some way to inject my stage between the Arcade publish stage and Arcade publish validation stage.
2. A way to inject my own jobs into the Arcade publish stage.
   • This is planned, but I can't find the issue.
3. Wait for blob storage publish to be implemented by the Arcade publish stage and use that in Core-Setup.
   • The other custom publish steps should be fine to keep after build promotion because Core-SDK doesn't depend on them.
   • I'd rather not rely on this method without lead-in time. I don't know much about the implementation.

/cc @JohnTortugo @mmitche @MichaelSimons

[dagood]

Remembered that the "publish build to BAR" job is actually outside of post-build, so I believe we can add something between them:

https://github.com/dotnet/core-setup/blob/7ccd18d805ec46de2bc87fecdc0c3f780f933cc1/azure-pipelines.yml#L166-L177

So, another possibility:

4. Split out the custom blob storage publish and put it between publish-build-assets.yml and post-build.yml.
   • Don't include dotnet/versions or the dotnetcli/dotnetclichecksums Channel publish this early, because package publish hasn't happened yet.
   • Requires no Arcade changes and has optimal ordering.

[dagood]

I gave 4 a stab: https://dev.azure.com/dnceng/internal/_build/results?buildId=317885&view=results.

[JohnTortugo]

@dagood is this a blocking issue?

@markwilkie

[dagood]

It isn't blocking. When Core-SDK hits it, the upgrade PR can be be retried until the artifacts are available in blob storage and then it succeeds. (@mmitche would have more context than me on how painful that is right now.)

[mmitche]

I'm seeing some cases where it doesn't seem to be able to be retried:

https://dev.azure.com/dnceng/public/_build/results?buildId=318032

[dagood]

I'm seeing some cases where it doesn't seem to be able to be retried:

https://dev.azure.com/dnceng/public/_build/results?buildId=318032

For future readers, this is a separate issue where custom publishing wasn't defined for the 3.0 Release BAR channel. @mmitche is fixing with e.g. PR dotnet/core-setup#7785.

[dagood]

Complete: fixed in master dotnet/core-setup#7779 and release/3.0 dotnet/core-setup#7796, and I tentatively opened dotnet/core-setup#7797 for release/3.0-preview9 to be merged if needed.

[dagood]

We also need to add publishUsingPipelines: true to the publish build to BAR job, or else the build is promoted way early, before any publish happens at all.

See dotnet/core-setup#7797 (comment).

[dagood]

Added publishUsingPipelines: true with dotnet/core-setup#7807 and dotnet/core-setup#7808.