[Regression] Test failed: System.Net.Http.Functional.Tests.HttpClientHandler_DangerousAcceptAllCertificatesValidator_Test / SetDelegate_ConnectionSucceeds

[stephentoub]

Types of failures

System.Net.Http.HttpRequestException : An error occurred while sending the request.
---- System.IO.IOException : The decryption operation failed, see inner exception.
-------- Interop+OpenSsl+SslException : Decrypt failed with OpenSSL error - SSL_ERROR_SSL.
------------ System.Security.Cryptography.CryptographicException : Error occurred during a cryptographic operation.
at System.Net.Http.HttpConnection.<SendAsync>d__38.MoveNext() in /root/corefx-1259674/src/System.Net.Http/src/System/Net/Http/Managed/HttpConnection.cs:line 438
--- End of stack trace from previous location where exception was thrown ---
at System.Net.Http.HttpConnectionHandler.<SendAsyncWithAwaitedConnection>d__4.MoveNext() in /root/corefx-1259674/src/System.Net.Http/src/System/Net/Http/Managed/HttpConnectionHandler.cs:line 43
--- End of stack trace from previous location where exception was thrown ---
at System.Net.Http.CookieHandler.<SendAsync>d__3.MoveNext() in /root/corefx-1259674/src/System.Net.Http/src/System/Net/Http/Managed/CookieHandler.cs:line 31
--- End of stack trace from previous location where exception was thrown ---
at System.Net.Http.AuthenticateAndRedirectHandler.<SendAsync>d__7.MoveNext() in /root/corefx-1259674/src/System.Net.Http/src/System/Net/Http/Managed/AuthenticateAndRedirectHandler.cs:line 62
--- End of stack trace from previous location where exception was thrown ---
at System.Net.Http.HttpClient.<FinishSendAsyncBuffered>d__62.MoveNext() in /root/corefx-1259674/src/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 488
--- End of stack trace from previous location where exception was thrown ---
at System.Net.Http.Functional.Tests.HttpClientHandler_DangerousAcceptAllCertificatesValidator_Test.<>c__DisplayClass3_1.<<SetDelegate_ConnectionSucceeds>b__0>d.MoveNext() in /root/corefx-1259674/src/System.Net.Http/tests/FunctionalTests/HttpClientHandlerTest.AcceptAllCerts.cs:line 54
--- End of stack trace from previous location where exception was thrown ---
at System.Net.Test.Common.LoopbackServer.<>c__DisplayClass3_0.<CreateServerAsync>b__0(Task t) in /root/corefx-1259674/src/Common/tests/System/Net/Http/LoopbackServer.cs:line 68
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state) in /root/coreclr/src/mscorlib/shared/System/Threading/ExecutionContext.cs:line 151
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot) in /root/coreclr/src/mscorlib/src/System/Threading/Tasks/Task.cs:line 2440
--- End of stack trace from previous location where exception was thrown ---
at System.Net.Http.Functional.Tests.HttpClientHandler_DangerousAcceptAllCertificatesValidator_Test.<SetDelegate_ConnectionSucceeds>d__3.MoveNext() in /root/corefx-1259674/src/System.Net.Http/tests/FunctionalTests/HttpClientHandlerTest.AcceptAllCerts.cs:line 52
--- End of stack trace from previous location where exception was thrown ---
--- End of stack trace from previous location where exception was thrown ---
--- End of stack trace from previous location where exception was thrown ---
----- Inner Stack Trace -----
at System.Net.Security.SslStreamInternal.<ReadAsyncInternal>d__32`1.MoveNext() in /root/corefx-1259674/src/System.Net.Security/src/System/Net/Security/SslStreamInternal.cs:line 285
--- End of stack trace from previous location where exception was thrown ---
at System.Net.Http.HttpConnection.<FillAsync>d__64.MoveNext() in /root/corefx-1259674/src/System.Net.Http/src/System/Net/Http/Managed/HttpConnection.cs:line 958
--- End of stack trace from previous location where exception was thrown ---
at System.Net.Http.HttpConnection.<ReadNextLineAsync>d__62.MoveNext() in /root/corefx-1259674/src/System.Net.Http/src/System/Net/Http/Managed/HttpConnection.cs:line 867
--- End of stack trace from previous location where exception was thrown ---
at System.Net.Http.HttpConnection.<SendAsync>d__38.MoveNext() in /root/corefx-1259674/src/System.Net.Http/src/System/Net/Http/Managed/HttpConnection.cs:line 313
----- Inner Stack Trace -----
at Interop.OpenSsl.Decrypt(SafeSslHandle context, Byte[] outBuffer, Int32 offset, Int32 count, SslErrorCode& errorCode) in /root/corefx-1259674/src/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.OpenSsl.cs:line 269
at System.Net.Security.SslStreamPal.EncryptDecryptHelper(SafeDeleteContext securityContext, ReadOnlyMemory`1 input, Int32 offset, Int32 size, Boolean encrypt, Byte[]& output, Int32& resultSize) in /root/corefx-1259674/src/System.Net.Security/src/System/Net/Security/SslStreamPal.Unix.cs:line 207
----- Inner Stack Trace -----

History of failures

Lots of failures in CI:

• 12/2-12/18 - 58 failures (avg 3+/day) - see https://github.com/dotnet/corefx/issues/25676#issuecomment-353659053 for details on first failures on 12/2
• 12/19-1/5 - 44 failures (avg 2.5/day)
• 1/6-1/14 - 44 failures (avg 5/day)

Day	Build	OS	Details
1/3	20180103.03	Fedora25	[ManagedHandler] link

Original report

Related to dotnet/corefx#25646? cc: @Drawaes

[karelz]

@stephentoub it is ManagedHandler test. Do you know if it affects non-ManagedHandler tests as well?

[stephentoub]

Do you know if it affects non-ManagedHandler tests as well?

The issue is unrelated to ManagedHandler.

[Drawaes]

The issue is occurring because the Curl implementation and SslStream's use of OpenSsl seem to be interacting. They both share all data structures, which is horrible but is what it is ....

[karelz]

This is failing CI pretty badly: 58 failures 12/2-12/18 (avg 3+/day)

First failures on 12/2:

• b6394f5b4c31cb07e14e6c3b5575eacfb265c5e4
• ec0c6df968ba315114a317d25bfbd9ba5a322bc1
• 5e70fee9f1339db71f723caac7a85d55346c9a09

We need to identify the breaking change and roll it back ASAP (or disable test if the change was really good and the test is bad) - @Priya91 can you please help?

[Priya91]

Looks like it is this commit that can cause this failure of the 3, dotnet/corefx@ec0c6df

The other two are not product changes or test changes that can affect this test. Will submit a PR to rollback the 2nd commit.

[stephentoub]

@Drawaes, did you make progress on identifying the issue here? That was a very important fix; I'm loathe to roll it back if we can fix the problem instead.

[Drawaes]

Well I did some tidy up in this PR
dotnet/corefx#25680

about 2 weeks ago I marked it as no longer WIP. With those changes I couldn't get it to fail after running the outerloop over and over (I got bored at about 20 in a row on Ubuntu).

But as it's intermittent I would say the fix needs to go in and see if the CI settles down after it. In theory that PR should see the same behaviour as old, but keep the "don't lock the error queue for each call" behaviour.

[Drawaes]

The above PR is still waiting. Without that in and some time (due to intermittent failure) we can't 100% tell if that fixes it.

[Priya91]

The proposed PR to fix this issue is merged, hence will close it as fixed. If issue still repros please re-open this issue with relevant information.