SafeSocketHandle.CloseAsIs hanging in finalizer thread

[jaredpar]

This issue comes from investigating test failures on this Roslyn PR dotnet/roslyn#46510

At the conclusion of running the unit tests for VBCSCompiler server the xUnit process will refuse to exit. The xUnit output will indicate that the tests have completed running but the process itself will not exit. Attaching the debugger to the xUnit process and there are two threads of note that are still running:

GC Finalizer

System.Private.CoreLib.dll!System.Threading.SpinWait.SpinOnceCore(int sleep1Threshold) (Unknown Source:0)
System.Net.Sockets.dll!System.Net.Sockets.SafeSocketHandle.CloseAsIs(bool abortive) (Unknown Source:0)
System.Net.Sockets.dll!System.Net.Sockets.Socket.Dispose(bool disposing) (Unknown Source:0)
System.Net.Sockets.dll!System.Net.Sockets.Socket.~Socket() (Unknown Source:0)
[Native to Managed Transition] (Unknown Source:0)

.NET Sockets

System.Net.Sockets.dll!System.Net.Sockets.SocketAsyncEngine.EventLoop() (Unknown Source:0)
System.Net.Sockets.dll!System.Net.Sockets.SocketAsyncEngine..ctor.AnonymousMethod__14_0(object s) (Unknown Source:0)
System.Private.CoreLib.dll!System.Threading.ThreadHelper.ThreadStart(object obj) (Unknown Source:0)
[Native to Managed Transition] (Unknown Source:0)

The VBCSCompiler server makes heavy use of named pipes. Looking through the Socket on the finalizer thread I can confirm it's a Unix Domain socket related to the named pipes the compiler is creating (the path in the end point matches the paths we create in the tests).

Unfortunately after a day of debugging I have not been able to narrow this problem down any further:

1. The problem only repros when running the entire test assembly. I ran each test class in the assembly and none of them individually reproduce the issue. Have to run them as a group.
2. Went through a cycle of causing the hang, identifying the test which created the socket that was hung in the finalizer, disable that test, re-run the assembly. Did this for about five different tests and it had no impact on the hang.
3. Thought this may be related to WaitForConnectionAsync does not respond to cancellation #40289 so I tried aggressively calling NamedPipeServerStream.Dispose on any instance that was hung in a WaitForConnectionAsync call.

None of these has had any impact though. I've also been unsuccessful in constructing a more concise repro. 😦

More than happy to provide any info to make tracking this down easier.

Repro Information:

• Runtime: .NET 5 Preview 7
• OS: Ubuntu 18.04, Ubuntu 18.04 via WSL2, OSX

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

[stephentoub]

cc: @tmds

[jaredpar]

Did my best to narrow this down to a smaller problem. Created a branch where you can see the delta between our unit tests passing and hanging. This commit shows that it's simply enabling one new test to run on Linux that causes this hang.

To repro this do the following:

> git clone https://github.com/jaredpar/roslyn -o jaredpar
> git checkout -B repro jaredpar/repro/pipe-hang
> cd src/Compilers/Server/VBCSCompilerServerTests
> dotnet build
> dotnet msbuild /t:Test

Couple notes:

1. About 1/2 of the time this will cause the tests to fail, the other half it will hang in the finalizer
2. Running the new test alone is not sufficient to demonstrate the hang, have to run the entire suite (ensure why).

A brief description of what this particular test is doing:

• Opening four NamedPipeServerStream instances on the same pipe name
• Creates one NamedPipeClientStream instance that fully connects
• Dispose three of the NamedPipeServerStream instances, does not dispose the connected one
• Disposes the NamedPipeClientStream
• That means after the test completes one of the NamedPipeServerStream instances remains undisposed and it appears this is the one that ends up stuck in the finalizer thread.

[tmds]

System.Private.CoreLib.dll!System.Threading.SpinWait.SpinOnceCore(int sleep1Threshold) (Unknown Source:0)
System.Net.Sockets.dll!System.Net.Sockets.SafeSocketHandle.CloseAsIs(bool abortive) (Unknown Source:0)

Since .NET 5, SafeHandles are ref'ed for on-going Socket operations (instead of calling DangerousGetHandle). If we are spinning here, there is a on-going operation against the Socket that keeps the handle from being released. It would be interesting to identify the operation that is on-going on some other thread. It may help put together a reproducer.

Aborting the on-going Socket operations is attempted by the TryUnblockSocket. If the Socket doesn't have CLOEXEC this method no-ops. This method is best-effort. There are tests that verify abortive behavior for UDP/TCP, but no tests for domain sockets.

runtime/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SafeSocketHandle.Unix.cs

Lines 196 to 235 in be197ad

	private unsafe bool TryUnblockSocket(bool abortive)
	{
	// Calling 'close' on a socket that has pending blocking calls (e.g. recv, send, accept, ...)
	// may block indefinitely. This is a best-effort attempt to not get blocked and make those operations return.
	// We need to ensure we keep the expected TCP behavior that is observed by the socket peer (FIN vs RST close).
	// What we do here isn't specified by POSIX and doesn't work on all OSes.
	// On Linux this works well.
	// On OSX, TCP connections will be closed with a FIN close instead of an abortive RST close.
	// And, pending TCP connect operations and UDP receive are not abortable.
	// Unless we're doing an abortive close, don't touch sockets which don't have the CLOEXEC flag set.
	// These may be shared with other processes and we want to avoid disconnecting them.
	if (!abortive)
	{
	int fdFlags = Interop.Sys.Fcntl.GetFD(handle);
	if (fdFlags == 0)
	{
	return false;
	}
	}
	int type = 0;
	int optLen = sizeof(int);
	Interop.Error err = Interop.Sys.GetSockOpt(handle, SocketOptionLevel.Socket, SocketOptionName.Type, (byte*)&type, &optLen);
	if (err == Interop.Error.SUCCESS)
	{
	// For TCP (SocketType.Stream), perform an abortive close.
	// Unless the user requested a normal close using Socket.Shutdown.
	if (type == (int)SocketType.Stream && !_hasShutdownSend)
	{
	Interop.Sys.Disconnect(handle);
	}
	else
	{
	Interop.Sys.Shutdown(handle, SocketShutdown.Both);
	}
	}
	return true;
	}

Note that async Socket operations always nicely abort because there are no on-going Socket operations that keep the SafeHandle ref'ed.

[antonfirsov]

It would be interesting to identify the operation that is on-going on some other thread.

I'm currently trying to figure that out, but my gut feel is that spinning in the finalizer thread is a bad idea in any case, and we should probably rethink the way CloseAsIs works, regardless of the outcome of the investigation.

[jaredpar]

@tmds

It would be interesting to identify the operation that is on-going on some other thread. It may help put together a reproducer.

Really wish I could've found a simpler repro here that would more clearly identify the operation occurring. Spent several hours on this but was unable to do so.

More than happy to help out with any questions on the current repro.