Unable to benchmark with Mono runtime on Linux

[Windows10CE]

Benchmarking using Mono as a runtime on Linux seems to be broken since 0.13.3, with an error message saying "The Roslyn toolchain is only supported on .NET Framework". This leads me to believe that #2120 is the cause for the issue. Simply downgrading to 0.13.2 allows the benchmark to run as expected, and is correctly reported as using the Mono runtime.

Minimal repro (just needs the .NET 7 SDK to build and Mono to run): BDNMinimalRepro.zip

[AndreyAkinshin]

@Windows10CE thanks for the bug report! Indeed, the Mono support on Linux has been broken since v0.13.3. If we remove the "The Roslyn toolchain is only supported on .NET Framework" validation, we will discover that the actual problem is caused by anonymous pipes that were introduced in v0.13.3:

Unhandled Exception:
System.IO.IOException: Invalid handle to path "[Unknown]"
  at System.IO.FileStream.Init (Microsoft.Win32.SafeHandles.SafeFileHandle safeHandle, System.IO.FileAccess access, System.Boolean ownsHandle, System.Int32 bufferSize, System.Boolean isAsync, System.Boolean isConsoleWrapper) [0x0006a] in <de882a77e7c14f8ba5d298093dde82b2>:0 
  at System.IO.FileStream..ctor (Microsoft.Win32.SafeHandles.SafeFileHandle handle, System.IO.FileAccess access, System.Int32 bufferSize, System.Boolean isAsync) [0x00011] in <de882a77e7c14f8ba5d298093dde82b2>:0 
  at System.IO.FileStream..ctor (Microsoft.Win32.SafeHandles.SafeFileHandle handle, System.IO.FileAccess access, System.Int32 bufferSize) [0x00000] in <de882a77e7c14f8ba5d298093dde82b2>:0 
  at (wrapper remoting-invoke-with-check) System.IO.FileStream..ctor(Microsoft.Win32.SafeHandles.SafeFileHandle,System.IO.FileAccess,int)
  at BenchmarkDotNet.Engines.AnonymousPipesHost.OpenAnonymousPipe (System.String fileHandle, System.IO.FileAccess access) [0x00011] in <752c4abba0be4396bb787413596bc67e>:0 
  at BenchmarkDotNet.Engines.AnonymousPipesHost..ctor (System.String writHandle, System.String readHandle) [0x00006] in <752c4abba0be4396bb787413596bc67e>:0 
  at BenchmarkDotNet.Autogenerated.UniqueProgramName.AfterAssemblyLoadingAttached (System.String[] args) [0x0000c] in <f86452cd291f4212bc49ff000eb478bc>:0 
  at BenchmarkDotNet.Autogenerated.UniqueProgramName.Main (System.String[] args) [0x00000] in <f86452cd291f4212bc49ff000eb478bc>:0 
[ERROR] FATAL UNHANDLED EXCEPTION: System.IO.IOException: Invalid handle to path "[Unknown]"
  at System.IO.FileStream.Init (Microsoft.Win32.SafeHandles.SafeFileHandle safeHandle, System.IO.FileAccess access, System.Boolean ownsHandle, System.Int32 bufferSize, System.Boolean isAsync, System.Boolean isConsoleWrapper) [0x0006a] in <de882a77e7c14f8ba5d298093dde82b2>:0 
  at System.IO.FileStream..ctor (Microsoft.Win32.SafeHandles.SafeFileHandle handle, System.IO.FileAccess access, System.Int32 bufferSize, System.Boolean isAsync) [0x00011] in <de882a77e7c14f8ba5d298093dde82b2>:0 
  at System.IO.FileStream..ctor (Microsoft.Win32.SafeHandles.SafeFileHandle handle, System.IO.FileAccess access, System.Int32 bufferSize) [0x00000] in <de882a77e7c14f8ba5d298093dde82b2>:0 
  at (wrapper remoting-invoke-with-check) System.IO.FileStream..ctor(Microsoft.Win32.SafeHandles.SafeFileHandle,System.IO.FileAccess,int)
  at BenchmarkDotNet.Engines.AnonymousPipesHost.OpenAnonymousPipe (System.String fileHandle, System.IO.FileAccess access) [0x00011] in <752c4abba0be4396bb787413596bc67e>:0 
  at BenchmarkDotNet.Engines.AnonymousPipesHost..ctor (System.String writHandle, System.String readHandle) [0x00006] in <752c4abba0be4396bb787413596bc67e>:0 
  at BenchmarkDotNet.Autogenerated.UniqueProgramName.AfterAssemblyLoadingAttached (System.String[] args) [0x0000c] in <f86452cd291f4212bc49ff000eb478bc>:0 
  at BenchmarkDotNet.Autogenerated.UniqueProgramName.Main (System.String[] args) [0x00000] in <f86452cd291f4212bc49ff000eb478bc>:0 

I tried various ways to set up a simple AnonymousPipeServerStream-based demo for communication between two processes using Mono 6.12.0.182 on Linux, but it seems this API is broken on the runtime level (or it has some unobvious usage limitations).

The only solution I have in mind is to recover the old communication behavior based on the console output parsing as a backup workaround for the Linux+Mono case. However, it would force us to support two communication protocols and will lead to the increased complexity of the BenchmarkDotNet execution toolchain.

@adamsitnik do you have any ideas of how to make AnonymousPipeServerStream work on Linux+Mono?

[timcassell]

I'm not sure if this mono issue is helpful here. mono/mono#18100

[adamsitnik]

@timcassell it's very useful! It shows that it's not a bug on BDN, but Mono side (Mono closes the pipe file descriptor and there is nothing we can do about it)

However, it would force us to support two communication protocols and will lead to the increased complexity of the BenchmarkDotNet execution toolchain.

I don't think it's worth it. 0.13.3 was released 8 months ago and we have received only one bug report so far.

We could address that by using NoAcknowledgementConsoleHost for the old Mono on Linux, which is already used for WASM:

BenchmarkDotNet/src/BenchmarkDotNet/Templates/BenchmarkProgram.txt

Line 31 in e0c667f

host = new BenchmarkDotNet.Engines.NoAcknowledgementConsoleHost();

It would simply disable the possibility to respond to "events", but since we don't support many diagnosers for old Mono anyway, it should not be an issue.

[timcassell]

Mono is now legacy technology, and has been taken over by Wine. It's not worth it for us to try to fix any issues specific to Mono going forward. Users wanting to run benchmarks on legacy Mono can do so on Windows, or using BenchmarkDotNet v0.13.2 or older, or by running in-process.

[timcassell]

The IPC mechanism was changed from anonymous pipes to TCP in #2958 for other reasons, but it should also fix this issue.