[BUG] Connection reset by peer is not handled well by retry policy

[kasobol-msft]

We observe Connection reset by peer being propagated as an error in tests and reported by customers.
Looking for holistic solution for this problem in the core.

References:

Customer reported: #21066

[somanshreddy]

@kasobol-msft A similar issue seen here #15215

CC: @rickle-msft @alzimmermsft

[alzimmermsft]

Thanks for submitting this issue @kasobol-msft.

Just finished a rough investigation on this. It appears that when the expected response body of a request is void/Void the networking layer won't eagerly read the response body (as there isn't expected to be one). But, later in the call stack when converting the response headers, and body but there is none here, into the returned response there is an attempt to consume/fast-forward the response body. This is where the issue lies as doing this can result in a state when the remote closes/closed the connection triggering an exception to be thrown. Unfortunately, at this point the response has egressed past the retry policy, resulting in the exception being thrown to the caller instead of being captured and retried.

[kasobol-msft]

There seems to be few missing cases that we should cover:

• buffered upload
• download
• pageable operations

https://dev.azure.com/azure-sdk/internal/_build/results?buildId=900469&view=ms.vss-test-web.build-test-results-tab&runId=19372912&resultId=103744&paneView=debug

[kasobol-msft]

https://dev.azure.com/azure-sdk/internal/_build/results?buildId=938985&view=ms.vss-test-web.build-test-results-tab&runId=20123708&resultId=101687&paneView=debug

[kasobol-msft]

Looks like there are still some gaps https://dev.azure.com/azure-sdk/internal/_build/results?buildId=954096&view=ms.vss-test-web.build-test-results-tab&runId=20421522&resultId=100738&paneView=debug