HttpStress, HTTP/3: content mismatch in POST Duplex Slow

[antonfirsov]

The following issue has appeared two times in the CI stress runs against main in the POST Duplex Slow operation, first noticed on 09/13. At this point I cannot exclude the possibility that the issue also affects the release/7.0 code.

https://dev.azure.com/dnceng-public/public/_build/results?buildId=14692&view=logs&j=0da5d1d9-276d-5173-c4c4-9d4d4ed14fdb&t=1451f5f3-0108-5a08-5b92-e984b2a85bbd&l=634

 System.Exception: Expected response content "X4Fzl4GA06N7jZWOoUNPatAQ8KM4YRNVQAQDvZ5f1VN5asSKJr5djlietpqRhBbYjsgZ8JSEOiN5V5pGr2lhMkleRlJ6jexptpLiBtB0OqE1Od8ozaZfQtCz627NzMVkkXBgj75yrkJWVkGocV694IP0cMc8lo01KUGqSS4sUbnfZfTLmJkUGMfS8A3MUwC3kIB1UQMAQEeTUjQxtGMS3VlxoohHOhNXOkvHUSrMxZ41iBxOk6TDakzEWDCgi4liq65ko9TQPzUURjFdqIWIE9welgkGFBdsnLBb6DZad2obFVmnyZpPYTUg", got "X4Fzl4GA06N7jZWOoUNPatAQ8KM4YRNVQAQDvZ5f1VN5asSKJr5djlietpqRhBbYjsgZ8JSEOiN5V5pGr2lhMkleRlJ6jexptpLiBtB0OqE1Od8ozaZfQtCz627NzMVkkXBgj75yrJWVkGocV694IP0cMc8lo01KUGqSS4sUbnfZfTLmJkUGMfS8A3MUwC3kIB1UQMAQEeTUjQxtGMS3VlxoohHOhNXOkvHUSrMxZ41iBxOk6TDakzEWDCgi4liq65ko9TQPzUURjFdqIWIE9welgkGFBdsnLBb6DZad2obFVmnyZpPYTUg".
client_1  |  Diverging at index 137. server checksum does not match client value.
client_1  |    at HttpStress.ClientOperations.ValidateContent(String expectedContent, String actualContent, String details) in /app/ClientOperations.cs:line 494
client_1  |    at HttpStress.ClientOperations.<>c.<<get_Operations>b__1_8>d.MoveNext() in /app/ClientOperations.cs:line 384
client_1  | --- End of stack trace from previous location ---
client_1  |    at HttpStress.StressClient.<>c__DisplayClass17_0.<<StartCore>g__RunWorker|0>d.MoveNext() in /app/StressClient.cs:line 224
client_1  | 
client_1  | 	 8: POST Duplex Slow          Fail: 1	Timestamps: 14:43:07.6682381, Duration: 00:00:00.0612812, Cancelled: False
client_1  | 	    TOTAL                     Fail: 1

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

The following issue has appeared two times in the CI stress runs against main in the POST Duplex Slow operation, first noticed on 09/13. At this point I cannot exclude the possibility that the issue also affects the release/7.0 code.

https://dev.azure.com/dnceng-public/public/_build/results?buildId=14692&view=logs&j=0da5d1d9-276d-5173-c4c4-9d4d4ed14fdb&t=1451f5f3-0108-5a08-5b92-e984b2a85bbd&l=634

 System.Exception: Expected response content "X4Fzl4GA06N7jZWOoUNPatAQ8KM4YRNVQAQDvZ5f1VN5asSKJr5djlietpqRhBbYjsgZ8JSEOiN5V5pGr2lhMkleRlJ6jexptpLiBtB0OqE1Od8ozaZfQtCz627NzMVkkXBgj75yrkJWVkGocV694IP0cMc8lo01KUGqSS4sUbnfZfTLmJkUGMfS8A3MUwC3kIB1UQMAQEeTUjQxtGMS3VlxoohHOhNXOkvHUSrMxZ41iBxOk6TDakzEWDCgi4liq65ko9TQPzUURjFdqIWIE9welgkGFBdsnLBb6DZad2obFVmnyZpPYTUg", got "X4Fzl4GA06N7jZWOoUNPatAQ8KM4YRNVQAQDvZ5f1VN5asSKJr5djlietpqRhBbYjsgZ8JSEOiN5V5pGr2lhMkleRlJ6jexptpLiBtB0OqE1Od8ozaZfQtCz627NzMVkkXBgj75yrJWVkGocV694IP0cMc8lo01KUGqSS4sUbnfZfTLmJkUGMfS8A3MUwC3kIB1UQMAQEeTUjQxtGMS3VlxoohHOhNXOkvHUSrMxZ41iBxOk6TDakzEWDCgi4liq65ko9TQPzUURjFdqIWIE9welgkGFBdsnLBb6DZad2obFVmnyZpPYTUg".
client_1  |  Diverging at index 137. server checksum does not match client value.
client_1  |    at HttpStress.ClientOperations.ValidateContent(String expectedContent, String actualContent, String details) in /app/ClientOperations.cs:line 494
client_1  |    at HttpStress.ClientOperations.<>c.<<get_Operations>b__1_8>d.MoveNext() in /app/ClientOperations.cs:line 384
client_1  | --- End of stack trace from previous location ---
client_1  |    at HttpStress.StressClient.<>c__DisplayClass17_0.<<StartCore>g__RunWorker|0>d.MoveNext() in /app/StressClient.cs:line 224
client_1  | 
client_1  | 	 8: POST Duplex Slow          Fail: 1	Timestamps: 14:43:07.6682381, Duration: 00:00:00.0612812, Cancelled: False
client_1  | 	    TOTAL                     Fail: 1

Author:	antonfirsov
Assignees:	-
Labels:	area-System.Net.Http
Milestone:	-

[karelz]

Triage: To our best knowledge, it should be addressed in main and latest 7.0 by copying buffer in HTTP/3. We should not see it.
It is low frequency, so let's keep it in 8.0 - once we understand the impact there, we can consider servicing if it is impactful.

[ManickaP]

The content is diverging at 137 position of 312, so in the middle. And it's diverging in one character. Also the test is sending the data one byte at a time, so one of the bytes got lost somewhere. This might be a bug in our code as well as in msquic. We should try to reproduce locally and get logs.

[ManickaP]

Another hit on Windows in https://dev.azure.com/dnceng-public/public/_build/results?buildId=254920&view=logs&j=0bc77094-9fcd-5c38-f6e4-27d2ae131589&t=4fd93899-d50b-5dc1-f9eb-66ab1b883c22:

httpstress-client-1  | Failure Type 1/1:
httpstress-client-1  | System.Exception: Expected response content "VIGhAOYBM7psBozVtNFKJzX9QQvf5xVCSeeF6tfVxSd6k3eySIX6nWJWl7GkSs5OccPXAjPzijMqMQONlxDnQPPvK8Uzdv35MEU7d7oitXO49T9KityVfIyCDUC1FQMmfHT8wHke8Uts1Llrz9HkwYQ3FHu9unTTBbOsz0NQ9KMAP5BsiL159bq7qPsfNajK0QgQwtrymqbcBFe962RYlOJZlG1a7i3Ya19yPlvoU3II0EwaljZK9AZc66ffrHq4m2196fkK6c4i51ZWKRpO8iPPSXsI6FxCCEWyCmU5yEEW6rW5X8LQSSv4dbhZnSEqerVrVbKYwkQuUsJb1mkyuXkkJRKb5vHVzNsELeawnnTi11ZlH6X1iGw77lY3GiS1Tuu6z4GMCnA732iIsYJzNFnjRl7Z6Wg4KSF66TlCqBou6o343FBiPTdD7AeExgFf75mbnbBOm69BQsdch2WrLyfqp1zwpWF9w8n12SHR4mvIKRmEX1XuAOK9PvmlcFZGrzFH5baFl5mY3MWNFkOnzKzXodb6gVopDfRKBU9vRH94bl62POWrrmGW0hTR0UZFGxLOvBBkbSMhBuw3XYgHzxyROuKU1hvwitoeQi7DDlFOQY", got "VIGhAOYBM7psBozVtNFKJzX9QQvf5xVCSeeF6tfVxSd6k3eySIX6nWJWl7GkSs5OccPXAjPzijMqMQONlxDnQPPvK8Uzdv35MEU7d7oitXO49T9KityVfIyCDUC1FQMmfHT8wHke8Uts1Llrz9HkwYQ3FHu9unTTBbOsz0NQ9KMAP5BsiL159bq7qPs;NajK0QgQwtrymqbcBFe962RYlOJZlG1a7i3Ya19yPlvoU3II0EwaljZK9AZc66ffrHq4m2196fkK6c4i51ZWKRpO8iPPSXsI6FxCCEWyCmU5yEEW6rW5X8LQSSv4dbhZnSEqerVrVbKYwkQuUsJb1mkyuXkkJRKb5vHVzNsELeawnnTi11ZlH6X1iGw77lY3GiS1Tuu6z4GMCnA732iIsYJzNFnjRl7Z6Wg4KSF66TlCqBou6o343FBiPTdD7AeExgFf75mbnbBOm69BQsdch2WrLyfqp1zwpWF9w8n12SHR4mvIKRmEX1XuAOK9PvmlcFZGrzFH5baFl5mY3MWNFkOnzKzXodb6gVopDfRKBU9vRH94bl62POWrrmGW0hTR0UZFGxLOvBBkbSMhBuw3XYgHzxyROuKU1hvwitoeQi7DDlFOQY".
httpstress-client-1  |  Diverging at index 187. server checksum does not match client value.
httpstress-client-1  |    at HttpStress.ClientOperations.ValidateContent(String expectedContent, String actualContent, String details) in C:\app\ClientOperations.cs:line 494
httpstress-client-1  |    at HttpStress.ClientOperations.<>c.<<get_Operations>b__1_8>d.MoveNext() in C:\app\ClientOperations.cs:line 377
httpstress-client-1  | --- End of stack trace from previous location ---
httpstress-client-1  |    at HttpStress.StressClient.<>c__DisplayClass17_0.<<StartCore>g__RunWorker|0>d.MoveNext() in C:\app\StressClient.cs:line 204
httpstress-client-1  | 
httpstress-client-1  | 	 8: POST Duplex Slow          Fail: 1	Timestamps: 19:39:37.3417219, Duration: 00:00:00.0690585, Cancelled: False
httpstress-client-1  | 	    TOTAL                     Fail: 1

In this case, we're not missing any bytes, but ; is received instead of f.