[HTTP/3] Test bug (flaky) - premature connection close - need some synchro on app level

[CarnaViire]

I've seen it (rarely) happening on GetAsync_LargeHeader_Success, but it is most probably a generic HTTP/3 test issue (I suspect LoopbackServer)

System.Net.Http.Functional.Tests.SocketsHttpHandlerTest_HttpClientHandlerTest_Headers_Http3_MsQuic.GetAsync_LargeHeader_Success(headerName: "Content-Security-Policy", headerValueLength: 4618) [FAIL]
        System.Net.Http.HttpRequestException : An error occurred while sending the request.
        ---- System.Net.Quic.QuicOperationAbortedException : Operation aborted.
        Stack Trace:
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http3RequestStream.cs(290,0): at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http3RequestStream.cs(298,0): at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http3Connection.cs(214,0): at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\HttpConnectionPool.cs(795,0): at System.Net.Http.HttpConnectionPool.TrySendUsingHttp3Async(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)      
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\HttpConnectionPool.cs(871,0): at System.Net.Http.HttpConnectionPool.DetermineVersionAndSendAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\HttpConnectionPool.cs(901,0): at System.Net.Http.HttpConnectionPool.SendAndProcessAltSvcAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)   
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\HttpConnectionPool.cs(920,0): at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\RedirectHandler.cs(30,0): at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\HttpClient.cs(527,0): at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\tests\FunctionalTests\HttpClientHandlerTest.Headers.cs(223,0): at System.Net.Http.Functional.Tests.HttpClientHandlerTest_Headers.<>c__DisplayClass8_0.<<GetAsync_LargeHeader_Success>b__1>d.MoveNext()
          --- End of stack trace from previous location ---
          C:\Users\knatalia\dev\git\runtime\src\libraries\Common\tests\System\Threading\Tasks\TaskTimeoutExtensions.cs(63,0): at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks)
          C:\Users\knatalia\dev\git\runtime\src\libraries\Common\tests\System\Threading\Tasks\TaskTimeoutExtensions.cs(82,0): at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks)
          C:\Users\knatalia\dev\git\runtime\src\libraries\Common\tests\System\Net\Http\GenericLoopbackServer.cs(38,0): at System.Net.Test.Common.LoopbackServerFactory.<>c__DisplayClass5_0.<<CreateClientAndServerAsync>b__0>d.MoveNext()
          --- End of stack trace from previous location ---
          C:\Users\knatalia\dev\git\runtime\src\libraries\Common\tests\System\Net\Http\Http3LoopbackServer.cs(100,0): at System.Net.Test.Common.Http3LoopbackServerFactory.CreateServerAsync(Func`3 funcAsync, Int32 millisecondsTimeout, GenericLoopbackOptions options)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\tests\FunctionalTests\HttpClientHandlerTest.Headers.cs(220,0): at System.Net.Http.Functional.Tests.HttpClientHandlerTest_Headers.GetAsync_LargeHeader_Success(String headerName, Int32 headerValueLength)
          --- End of stack trace from previous location ---
          ----- Inner Stack Trace -----
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Quic\src\System\Net\Quic\Implementations\MsQuic\MsQuicStream.cs(1386,0): at System.Net.Quic.Implementations.MsQuic.MsQuicStream.HandleEventConnectionClose(State state)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Quic\src\System\Net\Quic\Implementations\MsQuic\MsQuicStream.cs(965,0): at System.Net.Quic.Implementations.MsQuic.MsQuicStream.HandleEventShutdownComplete(State state, StreamEvent& evt)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Quic\src\System\Net\Quic\Implementations\MsQuic\MsQuicStream.cs(813,0): at System.Net.Quic.Implementations.MsQuic.MsQuicStream.HandleEvent(State state, StreamEvent& evt)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Quic\src\System\Net\Quic\Implementations\MsQuic\MsQuicStream.cs(772,0): at System.Net.Quic.Implementations.MsQuic.MsQuicStream.NativeCallbackHandler(IntPtr stream, IntPtr context, StreamEvent& streamEvent)
          --- End of stack trace from previous location ---
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Quic\src\System\Net\Quic\Implementations\MsQuic\Internal\ResettableCompletionSource.cs(57,0): at System.Net.Quic.Implementations.MsQuic.Internal.ResettableCompletionSource`1.GetResult(Int16 token)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http3RequestStream.cs(796,0): at System.Net.Http.Http3RequestStream.ReadHeadersAsync(Int64 headersLength, CancellationToken cancellationToken)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http3RequestStream.cs(322,0): at System.Net.Http.Http3RequestStream.ReadResponseAsync(CancellationToken cancellationToken)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http3RequestStream.cs(201,0): at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken)

Callstack shows MsQuicStream.HandleEventConnectionClose which means connection was closed while stream was in flight reading the response.

This brings me to thing we've discussed today:

Important thing I've remembered: it is not enough to wait until SHUTDOWN_COMPLETE event on stream to be sure it is safe to close the connection: it means shutdown is acked on transport level, but on app level peer might still be in the middle of reading. Long time ago we had a huge email thread on this topic with Nick and aligned that such synchronization should be done on app level. That's why in RunClientServer test method there was additional synchronization done via semaphores.

This exact issue manifests itself in our HTTP/3 tests, so we need to add some kind of app-level synchro there (or think of another solution).

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

I've seen it (rarely) happening on GetAsync_LargeHeader_Success, but it is most probably a generic HTTP/3 test issue (I suspect LoopbackServer)

System.Net.Http.Functional.Tests.SocketsHttpHandlerTest_HttpClientHandlerTest_Headers_Http3_MsQuic.GetAsync_LargeHeader_Success(headerName: "Content-Security-Policy", headerValueLength: 4618) [FAIL]
        System.Net.Http.HttpRequestException : An error occurred while sending the request.
        ---- System.Net.Quic.QuicOperationAbortedException : Operation aborted.
        Stack Trace:
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http3RequestStream.cs(290,0): at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http3RequestStream.cs(298,0): at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http3Connection.cs(214,0): at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\HttpConnectionPool.cs(795,0): at System.Net.Http.HttpConnectionPool.TrySendUsingHttp3Async(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)      
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\HttpConnectionPool.cs(871,0): at System.Net.Http.HttpConnectionPool.DetermineVersionAndSendAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\HttpConnectionPool.cs(901,0): at System.Net.Http.HttpConnectionPool.SendAndProcessAltSvcAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)   
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\HttpConnectionPool.cs(920,0): at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\RedirectHandler.cs(30,0): at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\HttpClient.cs(527,0): at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\tests\FunctionalTests\HttpClientHandlerTest.Headers.cs(223,0): at System.Net.Http.Functional.Tests.HttpClientHandlerTest_Headers.<>c__DisplayClass8_0.<<GetAsync_LargeHeader_Success>b__1>d.MoveNext()
          --- End of stack trace from previous location ---
          C:\Users\knatalia\dev\git\runtime\src\libraries\Common\tests\System\Threading\Tasks\TaskTimeoutExtensions.cs(63,0): at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks)
          C:\Users\knatalia\dev\git\runtime\src\libraries\Common\tests\System\Threading\Tasks\TaskTimeoutExtensions.cs(82,0): at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks)
          C:\Users\knatalia\dev\git\runtime\src\libraries\Common\tests\System\Net\Http\GenericLoopbackServer.cs(38,0): at System.Net.Test.Common.LoopbackServerFactory.<>c__DisplayClass5_0.<<CreateClientAndServerAsync>b__0>d.MoveNext()
          --- End of stack trace from previous location ---
          C:\Users\knatalia\dev\git\runtime\src\libraries\Common\tests\System\Net\Http\Http3LoopbackServer.cs(100,0): at System.Net.Test.Common.Http3LoopbackServerFactory.CreateServerAsync(Func`3 funcAsync, Int32 millisecondsTimeout, GenericLoopbackOptions options)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\tests\FunctionalTests\HttpClientHandlerTest.Headers.cs(220,0): at System.Net.Http.Functional.Tests.HttpClientHandlerTest_Headers.GetAsync_LargeHeader_Success(String headerName, Int32 headerValueLength)
          --- End of stack trace from previous location ---
          ----- Inner Stack Trace -----
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Quic\src\System\Net\Quic\Implementations\MsQuic\MsQuicStream.cs(1386,0): at System.Net.Quic.Implementations.MsQuic.MsQuicStream.HandleEventConnectionClose(State state)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Quic\src\System\Net\Quic\Implementations\MsQuic\MsQuicStream.cs(965,0): at System.Net.Quic.Implementations.MsQuic.MsQuicStream.HandleEventShutdownComplete(State state, StreamEvent& evt)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Quic\src\System\Net\Quic\Implementations\MsQuic\MsQuicStream.cs(813,0): at System.Net.Quic.Implementations.MsQuic.MsQuicStream.HandleEvent(State state, StreamEvent& evt)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Quic\src\System\Net\Quic\Implementations\MsQuic\MsQuicStream.cs(772,0): at System.Net.Quic.Implementations.MsQuic.MsQuicStream.NativeCallbackHandler(IntPtr stream, IntPtr context, StreamEvent& streamEvent)
          --- End of stack trace from previous location ---
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Quic\src\System\Net\Quic\Implementations\MsQuic\Internal\ResettableCompletionSource.cs(57,0): at System.Net.Quic.Implementations.MsQuic.Internal.ResettableCompletionSource`1.GetResult(Int16 token)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http3RequestStream.cs(796,0): at System.Net.Http.Http3RequestStream.ReadHeadersAsync(Int64 headersLength, CancellationToken cancellationToken)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http3RequestStream.cs(322,0): at System.Net.Http.Http3RequestStream.ReadResponseAsync(CancellationToken cancellationToken)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http3RequestStream.cs(201,0): at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken)

Callstack shows MsQuicStream.HandleEventConnectionClose which means connection was closed while stream was in flight reading the response.

This brings me to thing we've discussed today:

Important thing I've remembered: it is not enough to wait until SHUTDOWN_COMPLETE event on stream to be sure it is safe to close the connection: it means shutdown is acked on transport level, but on app level peer might still be in the middle of reading. Long time ago we had a huge email thread on this topic with Nick and aligned that such synchronization should be done on app level. That's why in RunClientServer test method there was additional synchronization done via semaphores.

This exact issue manifests itself in our HTTP/3 tests, so we need to add some kind of app-level synchro there (or think of another solution).

Author:	CarnaViire
Assignees:	-
Labels:	area-System.Net.Http, untriaged
Milestone:	-

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

I've seen it (rarely) happening on GetAsync_LargeHeader_Success, but it is most probably a generic HTTP/3 test issue (I suspect LoopbackServer)

System.Net.Http.Functional.Tests.SocketsHttpHandlerTest_HttpClientHandlerTest_Headers_Http3_MsQuic.GetAsync_LargeHeader_Success(headerName: "Content-Security-Policy", headerValueLength: 4618) [FAIL]
        System.Net.Http.HttpRequestException : An error occurred while sending the request.
        ---- System.Net.Quic.QuicOperationAbortedException : Operation aborted.
        Stack Trace:
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http3RequestStream.cs(290,0): at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http3RequestStream.cs(298,0): at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http3Connection.cs(214,0): at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\HttpConnectionPool.cs(795,0): at System.Net.Http.HttpConnectionPool.TrySendUsingHttp3Async(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)      
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\HttpConnectionPool.cs(871,0): at System.Net.Http.HttpConnectionPool.DetermineVersionAndSendAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\HttpConnectionPool.cs(901,0): at System.Net.Http.HttpConnectionPool.SendAndProcessAltSvcAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)   
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\HttpConnectionPool.cs(920,0): at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\RedirectHandler.cs(30,0): at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\HttpClient.cs(527,0): at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\tests\FunctionalTests\HttpClientHandlerTest.Headers.cs(223,0): at System.Net.Http.Functional.Tests.HttpClientHandlerTest_Headers.<>c__DisplayClass8_0.<<GetAsync_LargeHeader_Success>b__1>d.MoveNext()
          --- End of stack trace from previous location ---
          C:\Users\knatalia\dev\git\runtime\src\libraries\Common\tests\System\Threading\Tasks\TaskTimeoutExtensions.cs(63,0): at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks)
          C:\Users\knatalia\dev\git\runtime\src\libraries\Common\tests\System\Threading\Tasks\TaskTimeoutExtensions.cs(82,0): at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks)
          C:\Users\knatalia\dev\git\runtime\src\libraries\Common\tests\System\Net\Http\GenericLoopbackServer.cs(38,0): at System.Net.Test.Common.LoopbackServerFactory.<>c__DisplayClass5_0.<<CreateClientAndServerAsync>b__0>d.MoveNext()
          --- End of stack trace from previous location ---
          C:\Users\knatalia\dev\git\runtime\src\libraries\Common\tests\System\Net\Http\Http3LoopbackServer.cs(100,0): at System.Net.Test.Common.Http3LoopbackServerFactory.CreateServerAsync(Func`3 funcAsync, Int32 millisecondsTimeout, GenericLoopbackOptions options)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\tests\FunctionalTests\HttpClientHandlerTest.Headers.cs(220,0): at System.Net.Http.Functional.Tests.HttpClientHandlerTest_Headers.GetAsync_LargeHeader_Success(String headerName, Int32 headerValueLength)
          --- End of stack trace from previous location ---
          ----- Inner Stack Trace -----
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Quic\src\System\Net\Quic\Implementations\MsQuic\MsQuicStream.cs(1386,0): at System.Net.Quic.Implementations.MsQuic.MsQuicStream.HandleEventConnectionClose(State state)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Quic\src\System\Net\Quic\Implementations\MsQuic\MsQuicStream.cs(965,0): at System.Net.Quic.Implementations.MsQuic.MsQuicStream.HandleEventShutdownComplete(State state, StreamEvent& evt)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Quic\src\System\Net\Quic\Implementations\MsQuic\MsQuicStream.cs(813,0): at System.Net.Quic.Implementations.MsQuic.MsQuicStream.HandleEvent(State state, StreamEvent& evt)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Quic\src\System\Net\Quic\Implementations\MsQuic\MsQuicStream.cs(772,0): at System.Net.Quic.Implementations.MsQuic.MsQuicStream.NativeCallbackHandler(IntPtr stream, IntPtr context, StreamEvent& streamEvent)
          --- End of stack trace from previous location ---
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Quic\src\System\Net\Quic\Implementations\MsQuic\Internal\ResettableCompletionSource.cs(57,0): at System.Net.Quic.Implementations.MsQuic.Internal.ResettableCompletionSource`1.GetResult(Int16 token)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http3RequestStream.cs(796,0): at System.Net.Http.Http3RequestStream.ReadHeadersAsync(Int64 headersLength, CancellationToken cancellationToken)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http3RequestStream.cs(322,0): at System.Net.Http.Http3RequestStream.ReadResponseAsync(CancellationToken cancellationToken)
          C:\Users\knatalia\dev\git\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http3RequestStream.cs(201,0): at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken)

Callstack shows MsQuicStream.HandleEventConnectionClose which means connection was closed while stream was in flight reading the response.

This brings me to thing we've discussed today:

Important thing I've remembered: it is not enough to wait until SHUTDOWN_COMPLETE event on stream to be sure it is safe to close the connection: it means shutdown is acked on transport level, but on app level peer might still be in the middle of reading. Long time ago we had a huge email thread on this topic with Nick and aligned that such synchronization should be done on app level. That's why in RunClientServer test method there was additional synchronization done via semaphores.

This exact issue manifests itself in our HTTP/3 tests, so we need to add some kind of app-level synchro there (or think of another solution).

Author:	CarnaViire
Assignees:	-
Labels:	area-System.Net.Http, area-System.Net.Quic, untriaged
Milestone:	-

[karelz]

Failures 5/20-7/20 (incl. PRs):

Day	Failures
7/12	3x PR
7/13	6x PR
7/14	3x PR
7/15	1x PR
7/20	1x PR

[geoffkizer]

I suspect this is related to #55025

The loopback server doesn't ever close the connection today. But it could be getting finalized, which seems like it could manifest as above.

If the loopback server waited for graceful shutdown, it would solve that.

[geoffkizer]

I tested with my loopback fix and it seems to fix this issue.