Crossgen2 failure: tracing/eventpipe/reverse/reverse/reverse.sh

[v-haren]

failed in job: runtime-coreclr crossgen2 20200420.1

Error message

rm: /private/tmp/helix/working/B20C0A0D/w/BBD209FA/e/tracing/eventpipe/reverse/reverse/reverse.dll.rsp: No such file or directory

Return code: 1
Raw output file: /private/tmp/helix/working/B20C0A0D/w/BBD209FA/e/tracing/eventpipe/Reports/tracing.eventpipe/reverse/reverse/reverse.output.txt
Raw output:
BEGIN EXECUTION
in takeLock
Response file: /private/tmp/helix/working/B20C0A0D/w/BBD209FA/e/tracing/eventpipe/reverse/reverse/reverse.dll.rsp
--inputbubble
/private/tmp/helix/working/B20C0A0D/w/BBD209FA/e/tracing/eventpipe/reverse/reverse/IL/reverse.dll
-r:/private/tmp/helix/working/B20C0A0D/w/BBD209FA/e/tracing/eventpipe/reverse/reverse/IL/.dll
-o:/private/tmp/helix/working/B20C0A0D/w/BBD209FA/e/tracing/eventpipe/reverse/reverse/reverse.dll
-r:/tmp/helix/working/B20C0A0D/p/System..dll
-r:/tmp/helix/working/B20C0A0D/p/Microsoft.*.dll
-r:/tmp/helix/working/B20C0A0D/p/mscorlib.dll
--targetarch=x64
-O
Running CrossGen2: /tmp/helix/working/B20C0A0D/p/corerun /tmp/helix/working/B20C0A0D/p/crossgen2/crossgen2.dll @/private/tmp/helix/working/B20C0A0D/w/BBD209FA/e/tracing/eventpipe/reverse/reverse/reverse.dll.rsp 
Warning: Method [reverse]Tracing.Tests.ReverseValidation.ReverseValidation+<>c__DisplayClass8_1+<<TEST_ReverseConnectionCanRecycleWhileTracing>b__1>d.MoveNext() was not compiled because: [TEMPORARY EXCEPTION MESSAGE] FileLoadErrorGeneric: netstandard
Warning: Method [reverse]Tracing.Tests.ReverseValidation.ReverseValidation+<>c__DisplayClass6_0.<TEST_ServerWorksIfClientDoesntAccept>b__1() was not compiled because: [TEMPORARY EXCEPTION MESSAGE] FileLoadErrorGeneric: netstandard
Warning: Method [reverse]Tracing.Tests.ReverseValidation.ReverseValidation+<>c+<<TEST_StandardConnectionStillWorksIfReverseConnectionIsBroken>b__9_0>d.MoveNext() was not compiled because: [TEMPORARY EXCEPTION MESSAGE] FileLoadErrorGeneric: netstandard
Warning: Method [reverse]Tracing.Tests.ReverseValidation.ReverseValidation+<>c__DisplayClass5_2.<TEST_CanConnectServerAndClientAtSameTime>b__3() was not compiled because: [TEMPORARY EXCEPTION MESSAGE] FileLoadErrorGeneric: netstandard
Warning: Method [reverse]Tracing.Tests.ReverseValidation.ReverseValidation+<>c+<<TEST_ServerIsResilientToNoBufferAgent>b__7_0>d.MoveNext() was not compiled because: [TEMPORARY EXCEPTION MESSAGE] FileLoadErrorGeneric: netstandard
Warning: Method [reverse]Tracing.Tests.ReverseValidation.ReverseValidation+<>c+<<TEST_ServerWorksIfClientDoesntAccept>b__6_0>d.MoveNext() was not compiled because: [TEMPORARY EXCEPTION MESSAGE] FileLoadErrorGeneric: netstandard
Warning: Method [reverse]Tracing.Tests.ReverseValidation.ReverseValidation+<>c__DisplayClass5_1+<<TEST_CanConnectServerAndClientAtSameTime>b__2>d.MoveNext() was not compiled because: [TEMPORARY EXCEPTION MESSAGE] FileLoadErrorGeneric: netstandard
Warning: Method [reverse]Tracing.Tests.ReverseValidation.ReverseValidation+<>c__DisplayClass9_0.<TEST_StandardConnectionStillWorksIfReverseConnectionIsBroken>b__1() was not compiled because: [TEMPORARY EXCEPTION MESSAGE] FileLoadErrorGeneric: netstandard
Warning: Method [reverse]Tracing.Tests.ReverseValidation.ReverseValidation+<>c__DisplayClass8_2.<TEST_ReverseConnectionCanRecycleWhileTracing>b__3() was not compiled because: [TEMPORARY EXCEPTION MESSAGE] FileLoadErrorGeneric: netstandard
Warning: Method [reverse]Tracing.Tests.ReverseValidation.ReverseValidation+<>c__DisplayClass7_0.<TEST_ServerIsResilientToNoBufferAgent>b__1() was not compiled because: [TEMPORARY EXCEPTION MESSAGE] FileLoadErrorGeneric: netstandard
Emitting R2R PE file: /private/tmp/helix/working/B20C0A0D/w/BBD209FA/e/tracing/eventpipe/reverse/reverse/reverse.dll
in ReleaseLock
/tmp/helix/working/B20C0A0D/p/corerun reverse.dll ''
 0.0s: ::== Running test: TEST_RuntimeIsResilientToServerClosing
 0.0s: Server name is '/var/folders/52/pswj9r6x25vf6kvtw66zgbhw0000gy/T/DOTNET_TRACE_TESTS_pc220qgn.qvj'
 0.1s: running sub-process: /private/tmp/helix/working/B


Stack trace
   at tracing_eventpipe._reverse_reverse_reverse_._reverse_reverse_reverse_sh() in /Users/runner/runners/2.166.3/work/1/s/artifacts/tests/coreclr/OSX.x64.Checked/TestWrappers/tracing.eventpipe/tracing.eventpipe.XUnitWrapper.cs:line 334

[v-haren]

failed again in job: runtime-coreclr outerloop 20200510.5

failed test: tracing/eventpipe/reverse/reverse/reverse.sh

Error message

Return code: 1
Raw output file: /private/tmp/helix/working/B018091C/w/B9410A0E/e/tracing/eventpipe/Reports/tracing.eventpipe/reverse/reverse/reverse.output.txt
Raw output:
BEGIN EXECUTION
/tmp/helix/working/B018091C/p/corerun reverse.dll ''
 0.0s: ::== Running test: TEST_RuntimeIsResilientToServerClosing
 0.1s: Server name is '/var/folders/19/2fnchf3x2vng_n9fx9ktyhk80000gy/T/DOTNET_TRACE_TESTS_grgoqgj1.5ab'
 0.3s: running sub-process: /private/tmp/helix/working/B018091C/p/corerun /private/tmp/helix/working/B018091C/w/B9410A0E/e/tracing/eventpipe/reverse/reverse/reverse.dll 0
 0.5s: subprocess started: True
 0.5s: subprocess PID: 35537
 1.4s: Waiting for connection
 1.7s: Got a connection
 1.9s: { Magic=System.Byte[]; ClrInstanceId=1b8c4e20-56dc-9db6-84e2-0f779dd9213d; ProcessId=35537; Unused=0; }
 1.9s: Waiting for connection
 1.9s: Got a connection
 1.9s: { Magic=System.Byte[]; ClrInstanceId=1b8c4e20-56dc-9db6-84e2-0f779dd9213d; ProcessId=35537; Unused=0; }
 1.9s: Waiting for connection
 1.9s: Got a connection
 1.9s: { Magic=System.Byte[]; ClrInstanceId=1b8c4e20-56dc-9db6-84e2-0f779dd9213d; ProcessId=35537; Unused=0; }
 1.9s: Waiting for connection
 2.0s: Got a connection
 2.0s: { Magic=System.Byte[]; ClrInstanceId=1b8c4e20-56dc-9db6-84e2-0f779dd9213d; ProcessId=35537; Unused=0; }
 2.0s: Test passed: True
 2.0s: 
 2.0s: ::== Running test: TEST_RuntimeConnectsToExistingServer
 2.0s: Waiting for connection
 2.0s: Server name is /var/folders/19/2fnchf3x2vng_n9fx9ktyhk80000gy/T/DOTNET_TRACE_TESTS_qwtwvpmw.2mm
 2.0s: running sub-process: /private/tmp/helix/working/B018091C/p/corerun /private/tmp/helix/working/B018091C/w/B9410A0E/e/tracing/eventpipe/reverse/reverse/reverse.dll 0
 2.0s: subprocess started: True
 2.0s: subprocess PID: 35545
 2.1s: Got a connection
 2.2s: { Magic=System.Byte[]; ClrInstanceId=ea33427d-0635-1cc6-8b82-2c8ca0a2cf6d; ProcessId=35545; Unused=0; }
 2.2s: Test passed: True
 2.2s: 
 2.2s: ::== Running test: TEST_CanConnectServerAndClientAtSameTime
 2.2s: Server name is '/var/folders/19/2fnchf3x2vng_n9fx9ktyhk80000gy/T/DOTNET_TRACE_TESTS_a3aie4ue.4x5'
 2.2s: running sub-process: /private/tmp/helix/working/B018091C/p/corerun /private/tmp/helix/working/B018091C/w/B9410A0E/e/tracing/eventpipe/reverse/reverse/reverse.dll 0
 2.2s: subprocess started: True
 2.2s: subprocess PID: 35546
 2.3s: Waiting for reverse connection
 2.3s: Got reverse connection
 2.3s: { Magic=System.Byte[]; ClrInstanceId=daa64d90-baf0-18d1-b6f6-0606be76842c; ProcessId=35546; Unused=0; }
 2.4s: Starting EventPipeSession over standard connection
 2.5s: Started EventPipeSession over standard connection with session id: 0x7fa78fd477a0
 3.0s: Stopping EventPipeSession over standard connection
 3.3s: Stopped EventPipeSession over standard connection
 3.3s: Test passed: True
 3.3s: 
 3.3s: ::== Running test: TEST_ServerWorksIfClientDoesntAccept
 3.3s: Server name is '/var/folders/19/2fnchf3x2vng_n9fx9ktyhk80000gy/T/DOTNET_TRACE_TESTS_kpjhecj5.ab2'
 3.3s: running sub-process: /private/tmp/helix/working/B018091C/p/corerun /private/tmp/helix/working/B018091C/w/B9410A0E/e/tracing/eventpipe/reverse/reverse/reverse.dll 0
 3.4s: subprocess started: True
 3.4s: subprocess PID: 35547
 3.4s: Starting EventPipeSession over standard connection
 3.7s: System.Net.Internals.SocketExceptionFactory+ExtendedSocketException (61): Connection refused /var/folders/19/2fnchf3x2vng_n9fx9ktyhk80000gy/T/dotnet-diagnostic-35547-1589046231-socket
 at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress) in //src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.cs:line 4442
 at System.Net.Sockets.Socket.Connect(EndPoint remoteEP) in //src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.cs:line 901
 at Microsoft.Diagnostics.Tools.RuntimeClient.DiagnosticsIpc.IpcClient.GetTransport(Int32 processI


Stack trace
   at tracing_eventpipe._reverse_reverse_reverse_._reverse_reverse_reverse_sh() in /Users/runner/runners/2.166.4/work/1/s/artifacts/tests/coreclr/OSX.x64.Checked/TestWrappers/tracing.eventpipe/tracing.eventpipe.XUnitWrapper.cs:line 722

[am11]

Happened again OSX build; console log.

[safern]

Happened again in: https://dev.azure.com/dnceng/public/_build/results?buildId=661003&view=logs&j=9eaa4d7e-225e-55b8-2a77-e1ac7e76d448&t=f70528f6-0b93-5c49-dfc7-de8b842ee9d6&l=53

https://helix.dot.net/api/2019-06-17/jobs/22a3504d-c12f-4136-a2ac-357c4e620e92/workitems/PayloadGroup0/console

[mangod9]

I am guessing the CI failures are not related to crossgen2, and possibly a different issue which should be assigned to the Tracing team.

[safern]

@mangod9 do you know who the tracing team would be so that we can loop them in?

[mangod9]

@tommcdon ^^

[tommcdon]

@josalem