SSLStream Fixing GC Hole - #24799
Conversation
|
@dotnet-bot Test Outerloop Linux x64 Release Build |
|
@dotnet-bottest Outerloop Linux x64 Debug Build |
|
fixes #24775 #24722 |
|
Failing tests are consistent ALPN on Debian 90 No crashes or segfaults I can see anymore. |
|
/cc @danmosemsft |
|
cc @janvorli |
| fixed (byte* sp = server) | ||
| { | ||
| return Interop.Ssl.SslSelectNextProto(out outp, out outlen, (IntPtr)sp, (uint)server.Length, inp, inlen) == Interop.Ssl.OPENSSL_NPN_NEGOTIATED ? | ||
| return Interop.Ssl.SslSelectNextProto(out outp, out outlen, (IntPtr)protocols.AddrOfPinnedObject(), (uint)server.Length, inp, inlen) == Interop.Ssl.OPENSSL_NPN_NEGOTIATED ? |
There was a problem hiding this comment.
Is the issue then that SslSelectNextProto uses the passed in buffer beyond the end of the synchronous call to the method?
There was a problem hiding this comment.
Actually not exactly that... its that you pass it back out with the "out IntPtr"
There was a problem hiding this comment.
I see. The docs say:
The out value will point into either server or client, so it should be copied immediately.
This change ensures the server input is appropriately immovable. Is that already true for the client input?
There was a problem hiding this comment.
Yes because the client input comes from OpenSSL this method is only called on the server side. As we don't store this but retrieve the value later from OpenSSL itself the lifetime of the client buffer is fine.
| } | ||
|
|
||
| GCHandle protocols = GCHandle.FromIntPtr(arg); | ||
| if (!protocols.IsAllocated || protocols.Target == null) |
There was a problem hiding this comment.
In what situation would it be freed? If there's a race condition where it could be freed before this callback, is it possible it could be freed during?
There was a problem hiding this comment.
Well not anymore, before there was, I could remove that code in theory. I actually put it in, in my first round of trying to find the issue, thought it was worth leaving in for now. A Debug.Assert might suffice, which was my original idea.
| SslGetAlpnSelected(ssl, out protocol, out len); | ||
|
|
||
| if (len == 0) | ||
| if (len < 1) |
There was a problem hiding this comment.
who knows its coming from unmanaged code, I was merely adding extra safety. It might be worth me explaining the hole
There was a problem hiding this comment.
@Priya91, CryptoNative_SslGet0AlpnSelected should be initializing len always; any of our shims that are used with out vars should initialize the value to the default. Otherwise the calling C# code using out might end up with garbage in the value yet C# will allow the value to be used.
There was a problem hiding this comment.
@Drawaes, I don't think this len < 1 is correct or the right fix. The way the native shim function is currently coded, if for example HAVE_OPENSSL_ALPN isn't defined or if !API_EXISTS(SSL_get0_alpn_selected), the P/Invoke is a nop and len won't be modified, leaving it at whatever garbage was in len from the stack here, which means len could be anything, including a garbage value >=1. The right fix would seem to be to ensure that CryptoNative_SslGet0AlpnSelected initializes len to 0 if it's not calling SSL_get0_alpn_selected, and then this call site can return to a check for == 0.
There was a problem hiding this comment.
@Drawaes Will you be making this change in this PR?
|
The issue was this Unmanaged code calls the callback -> callback fixes the alpn buffer and calls the unmanaged select ALPN method -> this method just returns a POINTER into the server buffer and length.... Finally the Managed callback returns this pointer ... this means that the unmanaged code now has a interior pointer into a managed buffer that is no longer fixed. |
| if (_sslAuthenticationOptions.AlpnProtocolsHandle.IsAllocated) | ||
| { | ||
| _sslAuthenticationOptions.AlpnProtocolsHandle.Free(); | ||
| } |
There was a problem hiding this comment.
Is the Close method that contains the remaining Free always going to be called? e.g. is it called by both a Dispose/Close method and a finalizer?
There was a problem hiding this comment.
I believe its part of an Handle that is always called from my scanning of it.
There was a problem hiding this comment.
I'll be honest the pinning could be made tighter I suspect. Need to be careful though because I am not 100% the original supported renegotiation .... (Which there are no tests to catch). Because what happened if you freed the GCHandle, then called renegotiate and the callback was called? There was no catch if that was unallocated. However now it will last for the lifetime of the context.
There was a problem hiding this comment.
I just double checked that close is always called from SslStream.Dispose(disposing true) which comes from AuthenticatedStream which comes from Stream. I can't actually see a finalizer in that chain. So it might be missing finalisation. A better option might be to just hook into the SslContext handle as that already runs a finalizer... or to just add one to SslStream.
I can look at adding this to the SslContext as the callback is registered with that so freeing it just after the context is freed will ensure it's never run with an unpinned buffer.
|
I suspect that the Debian failure is due to it doing something different for selection that "Standard" OpenSSL, I would suggest that I write a matching method so that its consistent, its better .net does the same thing across distros... happy to do it in another PR ? |
| outp = IntPtr.Zero; | ||
| outlen = 0; | ||
|
|
||
| if (arg == IntPtr.Zero) |
There was a problem hiding this comment.
In what situation will it be null? Could you add a comment?
There was a problem hiding this comment.
I might remove that check now. I have just moved the GCHandle to be inside the SslCTX handle and tied directly to it's lifetime, so now this method should never be called after the handle is freed. I would perhaps change these to debug.asserts "just in case" but will remove the actual check.
| { | ||
| get { return handle == IntPtr.Zero; } | ||
| } | ||
| public GCHandle AlpnHandle { get => _alpnHandle; set => _alpnHandle = value; } |
There was a problem hiding this comment.
Could just be:
public GCHandle AlpnHandle { get; set; }and then you wouldn't need to explicitly define the field.
There was a problem hiding this comment.
its a struct right? what happens if you free the struct from the get? isn't a copy that now isn't set to free?
I have been caught out on GCHandle before with Readonly, in that the copy updates the "Free" but the original doesn't thus causing a double free and an exception.
(I could be wrong as well :) but I assume that the Get will cause a copy).
There was a problem hiding this comment.
I don't understand the questions. The code:
public GCHandle AlpnHandle { get; set; }is functionally identical to:
private GCHandle _alpnHandle;
public GCHandle AlpnHandle { get => _alpnHandle; set => _alpnHandle = value; }The only difference is what the name of the field is, as the compiler will generate that field for you in the first case.
There was a problem hiding this comment.
if(_alpnHandle.IsAllocated)
{
_alpnHandle.Free();
}There was a problem hiding this comment.
If I do that, and then call IsAllocated straight after and its a property, wont isallocated still be true?
There was a problem hiding this comment.
Ah, you're concerned about a subsequent call to ReleaseHandle seeing AlpnHandle.IsAllocated as true and trying to free it again? Yes, that could happen, so if that's the concern, then yeah, you could stick with what you have. Though ReleaseHandle should not be called multiple times.
There was a problem hiding this comment.
Your call, I don't mind either way
stephentoub
left a comment
There was a problem hiding this comment.
Thanks for tracking this down and fixing it, @Drawaes!
| SslGetAlpnSelected(ssl, out protocol, out len); | ||
|
|
||
| if (len < 1) | ||
| if (len == 0) |
There was a problem hiding this comment.
What about the native change to go along with this?
There was a problem hiding this comment.
Your too quick, I was using Github to transfer files to my unix box to make the change :P
|
Fixed the check for < 0 and added a *len = 0; to the native method. |
|
|
||
| extern "C" void CryptoNative_SslGet0AlpnSelected(SSL* ssl, const uint8_t** protocol, uint32_t* len) | ||
| { | ||
| *len = 0; |
There was a problem hiding this comment.
Maybe make it:
#ifdef HAVE_OPENSSL_ALPN
if (API_EXISTS(SSL_get0_alpn_selected))
{
SSL_get0_alpn_selected(ssl, protocol, len);
}
else
#endif
{
*protocol = NULL;
*len = 0;
}
}?
There was a problem hiding this comment.
(protocol should be initialized, too)
| fixed (byte* sp = server) | ||
| { | ||
| return Interop.Ssl.SslSelectNextProto(out outp, out outlen, (IntPtr)sp, (uint)server.Length, inp, inlen) == Interop.Ssl.OPENSSL_NPN_NEGOTIATED ? | ||
| return Interop.Ssl.SslSelectNextProto(out outp, out outlen, (IntPtr)protocols.AddrOfPinnedObject(), (uint)server.Length, inp, inlen) == Interop.Ssl.OPENSSL_NPN_NEGOTIATED ? |
There was a problem hiding this comment.
I may be confusing it with another API, but doesn't AddrOfPinnedObject return an IntPtr? I'm wondering what the cast is for.
There was a problem hiding this comment.
Nit: @Drawaes, the cast is back? I'd thought you'd removed it, but it appears to have been reverted?
There was a problem hiding this comment.
It is my bad, I was resetting/dropping test changes to figure out what was going on and I killed that as I had merged locally that change into a cleanup commit. I am reverting that.
| { | ||
| Interop.Ssl.SslCtxDestroy(handle); | ||
| SetHandle(IntPtr.Zero); | ||
| if(_alpnHandle.IsAllocated) |
|
@dotnet-bot test Outerloop Linux x64 Debug Build please |
|
@Drawaes, this assert is getting hit: |
|
Yeah just seen that (why I love a good assert). I will check it after I am home from work/dinner ;) |
|
This is a bit of a change, I am happy to back it out if you want. But after re-reading the code we can completely avoid the pin. This means that we don't hold that ugly pin for the lifetime of the connection which will cause fragmentation and bad things in the GC'd heap. Instead what we do is the matching in c# (also means we completely avoid the conversion to array of the protocol list). It also means that we need no pinning, as we return a pointer into the client buffer if a match exists. We also then avoid the issue that we saw earlier that one distro had different matching so we have a consistent match for all of .net. Also we remove the OpenSSL Surface area we hit. |
|
Can we do it as a follow-up PR? I'd like to get the crashes fixed asap. |
| [Fact] | ||
| [PlatformSpecific(~TestPlatforms.OSX)] | ||
| [PlatformSpecific(TestPlatforms.Linux)] | ||
| public void SslStream_StreamToStream_Alpn_NonMatchingProtocols_Fail() |
There was a problem hiding this comment.
@Drawaes Why are you excluding this test from Windows? This should run on all OSes except OSX.
There was a problem hiding this comment.
because now it doesn't work, I am trying to get the maximum to pass,
this test as far as I can tell never did anything on any platform before because the tasks were not awaited.
| Assert.ThrowsAsync<AuthenticationException>(() => { return client.AuthenticateAsClientAsync(clientOptions, CancellationToken.None); }); | ||
| Assert.ThrowsAsync<AuthenticationException>(() => { return server.AuthenticateAsServerAsync(serverOptions, CancellationToken.None); }); | ||
| Assert.True(DoHandshakeWithOptions(client, server, clientOptions, serverOptions)); | ||
| Assert.Equal(default, client.NegotiatedApplicationProtocol); |
There was a problem hiding this comment.
This behavior is wrong, if the client and server fail to match alpn, it should fail the handshake as per the rfc. Why did you change this behavior?
There was a problem hiding this comment.
I didn't change the behaviour, see above I don't believe the test was ever working.
There was a problem hiding this comment.
once I figure out what is going on, I will just disable the tests with an issue.
There was a problem hiding this comment.
Uhh, that's a bug, it should be async () => await
There was a problem hiding this comment.
I addressed this in the cancellation PR #24849 You can undo these changes here.
There was a problem hiding this comment.
Are you sure that fixes it, don't you need to await the result of ThrowAsync from memory the code is basically
public async static Task<T> ThrowsAsync<T>(Func<Task> testCode) where T : Exception
{
try
{
await testCode();
Assert.Throws<T>(() => { }); // Use xUnit's default behavior.
}
catch (T exception)
{
return exception;
}
return null;
}
without the
await Assert.ThrowAsync(
it still won't wait for the assert? Or am I wrong on that?
There was a problem hiding this comment.
i've been trying to get the implementation of ThrowsAsync, thanks for getting it. Yeah it looks like the throwsasync needs to be awaited.
There was a problem hiding this comment.
so when i do, the windows tests hang, and the linux tests fail.I will disable with an issue and check them in.
There was a problem hiding this comment.
Just letting them run with async to make sure that need to be disabled on everything or just one OS. Then I will tidy up with an [Issue] tag
Priya91
left a comment
There was a problem hiding this comment.
You can either file bug for that non matching protocols test case and move on with this PR, or address that here.
Putting back the init code in the cpp file for the out params (lost in a reset)
| Task t2 = Assert.ThrowsAsync<InvalidOperationException>(() => server.AuthenticateAsServerAsync(serverOptions, CancellationToken.None)); | ||
|
|
||
| await Task.WhenAll(t1, t2); | ||
| } |
There was a problem hiding this comment.
if t1, t2 don't complete this will result in hang no, should this be Assert.True(Task.WaitAll(t1,t2, timeout)), similarly below as well. There is a passingtesttimeout value in this project, you could use that.
There was a problem hiding this comment.
no because then it will pass, as it won't wait for the asserts. There is a timeout already on the xunit tests, they don't hang for ever
There was a problem hiding this comment.
It wont pass if waitall completed becoz of timeout, as it will return false, and assert.true will throw. If not this can cause long test times for the xunit timeout to be reached. Per test timeout is better than global one.
There was a problem hiding this comment.
the only way to do this would be,
Task delay = Task.Delay(sometimeout);
Task passingTask = Task.WhenAny(delay, Task.WhenAll(t1,t2));
Assert.NotEqual(passingTask, delay);
There was a problem hiding this comment.
Why wouldn't task.waitall with timeout work?
There was a problem hiding this comment.
Changed to wait all, seems odd still, you know that you can use a .json file in the solution folder to configure a timeout for all tests in a project rather than writing the code for each method.
Anyway updated to this as it seems the pattern currently used
|
Disabled test with active issue #24853 |
|
I debugged the non-matching protocol test failure on windows, it is throwing authenticationexception as expected, but the exception gets thrown as timeout aggregrate exception. |
|
Yeah I believe the timeout comes from the virtual network. So it's hanging but you are seeing the virtual network stream exit. |
| { | ||
| _sslAuthenticationOptions.AlpnProtocolsHandle.Free(); | ||
| } | ||
|
|
There was a problem hiding this comment.
Nit: unnecessary blank line (in addition to the missing space after the if above
There was a problem hiding this comment.
Yeah sorry was late, thought you meant a new line ... I have fixed.
|
@dotnet-bot test OSX x64 Debug Build please |
|
@Drawaes, doesn't the SslStream_StreamToStream_Alpn_Success test still need an [ActiveIssue] on it? |
|
Yes I just ran out of time last night to sort it out (LDN timezone) should be sorted now. I launched a new issue for it as it's a different issue I suspect (I think it's that version of debians matching without looking or checking ;) ) |
Ah, sorry, I saw you pushed a commit after my comments and figured you'd just missed that one. |
|
Don't code and not build when between meetings |
|
The Linux failure is https://github.com/dotnet/corefx/issues/24869 and is unrelated. I'm going to go ahead and merge this to unblock the rest of the branch. @Drawaes, thanks for getting this fixed. |
* Fixing GC Hole * Moved back to original pin location * Added Finializer * Make tests async * Reacting to review Putting back the init code in the cpp file for the out params (lost in a reset) * Added active issue on the failure test * React to review * Added Active Issue for Linux success tests
* Fixing GC Hole * Moved back to original pin location * Added Finializer * Make tests async * Reacting to review Putting back the init code in the cpp file for the out params (lost in a reset) * Added active issue on the failure test * React to review * Added Active Issue for Linux success tests Commit migrated from dotnet/corefx@2bd69b9
No description provided.