Reduce Maven Transfer Failures in Pipeline Runs

[alzimmermsft]

PR validation and daily testing pipelines occasionally fail when Maven fails to transfer dependency or build artifacts. An investigation should be done to determine if it is possible to configure Maven to retry these transfers or if we are able to cache artifacts in a system that can more reliably transfer artifacts.

[mitchdenny]

I have a new hypothesis to test with this. Apparently when builds are running on a VM within a scale set, the range of outbound ports that the VM can use is shared between the VMs, it is possible that from time to time there is so much load within the scale set that outbound ports get blocked.

It is just a hypothesis at this point, but the good news is that with the 1ES hosted agents coming online we actually have less contention for the outbound ports. So if this is a contributing factor, the issue might go away.

Switching to 1ES hosted agents for Java is currently in progress. Currently blocked on a configuration issue with the image which is being investigated.

[weshaggard]

We should also attempt to try and enable the Caching of the m2 repository for all the different jobs we run. We currently have it partially enabled for some jobs but even those are not currently working because our dev versioning is invalidating the cache key because it updates the pom files.

We should also update to using the released cache task following the pattern at https://docs.microsoft.com/en-us/azure/devops/pipelines/release/caching?view=azure-devops#maven.

As part of the caching work we want to be sure to not cache things we build in the repo so we should delete anything under "com/azure" and "com/microsoft/azure" in the cache folder before the cache gets uploaded.

[weshaggard]

Fixed with #21797