[ARM] Intermittent segfaults in JIT/Methodical/cctor/misc/threads1_cs_r

[mskvortsov]

One of the possible stack traces that gdb shows after loading a core dump:

$ gdb clr-debug/corerun core
Reading symbols from clr-debug/corerun...done.
[New LWP 22222]
[New LWP 22214]
[New LWP 22216]
[New LWP 22218]
[New LWP 22220]
[New LWP 22215]
[New LWP 22219]
[New LWP 22223]
[New LWP 22221]
[New LWP 22217]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
Core was generated by `clr-debug/corerun tests-release/JIT/Methodical/cctor/misc/threads1_cs_r/threads'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00000000 in ?? ()
[Current thread is 1 (Thread 0xadbff450 (LWP 22222))]
(gdb) bt
#0  0x00000000 in ?? ()
dotnet/coreclr#1  0xb650a34c in DomainLocalModule::GetPrecomputedNonGCStaticsBasePointer (this=0xb1c2a594) at /home/mskvortsov/git/coreclr/src/vm/appdomain.hpp:234
dotnet/coreclr#2  0xb648c48c in CallDescrWorker (pCallDescrData=0xadbfe4c0) at /home/mskvortsov/git/coreclr/src/vm/callhelpers.cpp:135
dotnet/coreclr#3  0xb648c32e in CallDescrWorkerWithHandler (pCallDescrData=0xadbfe4c0, fCriticalCall=0) at /home/mskvortsov/git/coreclr/src/vm/callhelpers.cpp:78
dotnet/coreclr#4  0xb648d374 in MethodDescCallSite::CallTargetWorker (this=0xadbfe62c, pArguments=0xadbfe6a0, pReturnValue=0x0, cbReturnValue=0) at /home/mskvortsov/git/coreclr/src/vm/callhelpers.cpp:645
dotnet/coreclr#5  0xb637b4ee in MethodDescCallSite::Call (this=0xadbfe62c, pArguments=0xadbfe6a0) at /home/mskvortsov/git/coreclr/src/vm/callhelpers.h:433
dotnet/coreclr#6  0xb649a438 in ThreadNative::KickOffThread_Worker (ptr=0xadbfeab8) at /home/mskvortsov/git/coreclr/src/vm/comsynchronizable.cpp:257
dotnet/coreclr#7  0xb644b07c in ManagedThreadBase_DispatchInner (pCallState=0xadbfe93c) at /home/mskvortsov/git/coreclr/src/vm/threads.cpp:9187
dotnet/coreclr#8  0xb644e8a8 in ManagedThreadBase_DispatchMiddle (pCallState=0xadbfe93c) at /home/mskvortsov/git/coreclr/src/vm/threads.cpp:9238
dotnet/coreclr#9  0xb644e760 in ManagedThreadBase_DispatchOuter(ManagedThreadCallState*)::$_6::operator()(ManagedThreadBase_DispatchOuter(ManagedThreadCallState*)::TryArgs*) const::{lambda(Param*)#1}::operator()(Param*) const (this=0xadbfe884, pParam=0xadbfe8e8) at /home/mskvortsov/git/coreclr/src/vm/threads.cpp:9476
dotnet/coreclr#10 0xb644e620 in ManagedThreadBase_DispatchOuter(ManagedThreadCallState*)::$_6::operator()(ManagedThreadBase_DispatchOuter(ManagedThreadCallState*)::TryArgs*) const (this=0xadbfe8d0, pArgs=0xadbfe8d8) at /home/mskvortsov/git/coreclr/src/vm/threads.cpp:9478
dotnet/coreclr#11 0xb644adb0 in ManagedThreadBase_DispatchOuter (pCallState=0xadbfe93c) at /home/mskvortsov/git/coreclr/src/vm/threads.cpp:9515
dotnet/coreclr#12 0xb644aee6 in ManagedThreadBase_FullTransitionWithAD (pAppDomain=..., pTarget=0xb649a231 <ThreadNative::KickOffThread_Worker(void*)>, args=0xadbfeab8, filterType=ManagedThread) at /home/mskvortsov/git/coreclr/src/vm/threads.cpp:9536
dotnet/coreclr#13 0xb644ae6e in ManagedThreadBase::KickOff (pAppDomain=..., pTarget=0xb649a231 <ThreadNative::KickOffThread_Worker(void*)>, args=0xadbfeab8) at /home/mskvortsov/git/coreclr/src/vm/threads.cpp:9571
dotnet/coreclr#14 0xb649a998 in ThreadNative::KickOffThread (pass=0x1063f0) at /home/mskvortsov/git/coreclr/src/vm/comsynchronizable.cpp:376
dotnet/coreclr#15 0xb6441e1c in Thread::intermediateThreadProc (arg=0xd4ec0) at /home/mskvortsov/git/coreclr/src/vm/threads.cpp:2584
dotnet/coreclr#16 0xb69a2f02 in CorUnix::CPalThread::ThreadEntry (pvParam=0x10c048) at /home/mskvortsov/git/coreclr/src/pal/src/thread/thread.cpp:1749
dotnet/coreclr#17 0xb6f295b4 in start_thread (arg=0x0) at pthread_create.c:335
dotnet/coreclr#18 0xb6d1caac in ?? () at ../sysdeps/unix/sysv/linux/arm/clone.S:89 from /lib/arm-linux-gnueabihf/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

[mskvortsov]

@dotnet/arm32-contrib Can someone reproduce this? I use a script like this:

$ cat repro.sh
#!/bin/sh
ulimit -c unlimited
#export COMPlus_AltJitAssertOnNYI=0
#export COMPlus_AltJit=*
try=0
while true
do
  try=$((try+1))
  clr-debug/corerun tests-release/JIT/Methodical/cctor/misc/threads1_cs_r/threads1_cs_r.exe >/dev/null 2>&1
  if [ $? = 100 ]
  then
    echo -n .
  else
    echo
    echo Failed on a try \#$try
    exit
  fi
done
$ ./repro.sh 
......................................
Failed on a try dotnet/runtime#3867
$ gdb clr-debug/corerun core

[mskvortsov]

And here is another kind of a stack trace I get:

$ ./repro.sh 
............................
Failed on a try dotnet/runtime#3861
$ gdb clr-debug/corerun core
Reading symbols from clr-debug/corerun...done.
[New LWP 26190]
[New LWP 26183]
[New LWP 26187]
[New LWP 26185]
[New LWP 26184]
[New LWP 26186]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
Core was generated by `clr-debug/corerun tests-release/JIT/Methodical/cctor/misc/threads1_cs_r/threads'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  access_mem (as=0xb61bd130 <local_addr_space>, addr=4257793536, val=0xae3f918c, write=0, arg=0xae3fd224) at arm/Ginit.c:86
86	arm/Ginit.c: No such file or directory.
[Current thread is 1 (Thread 0xae3ff450 (LWP 26190))]
(gdb) bt
#0  access_mem (as=0xb61bd130 <local_addr_space>, addr=4257793536, val=0xae3f918c, write=0, arg=0xae3fd224) at arm/Ginit.c:86
dotnet/coreclr#1  0xb61a05b4 in dwarf_get (c=0xae3f9224, c=0xae3f9224, val=0xae3f918c, loc=...) at ../include/tdep-arm/libunwind_i.h:203
dotnet/coreclr#2  _Uarm_step (cursor=0xae3f9224) at arm/Gstep.c:233
dotnet/coreclr#3  0xb687e2fa in PAL_VirtualUnwind (context=0xae3fd640, contextPointers=0x0) at /home/mskvortsov/git/coreclr/src/pal/src/exception/seh-unwind.cpp:309
dotnet/coreclr#4  0xb654df5e in UnwindManagedExceptionPass1 (ex=..., frameContext=0xae3fd640) at /home/mskvortsov/git/coreclr/src/vm/exceptionhandling.cpp:4663
dotnet/coreclr#5  0xb654e3be in DispatchManagedException (ex=..., isHardwareException=true) at /home/mskvortsov/git/coreclr/src/vm/exceptionhandling.cpp:4752
dotnet/coreclr#6  0xb6546074 in HandleHardwareException (ex=0xae3fdb60) at /home/mskvortsov/git/coreclr/src/vm/exceptionhandling.cpp:5243
dotnet/coreclr#7  0xb687dfce in SEHProcessException (exception=0xae3fdb60) at /home/mskvortsov/git/coreclr/src/pal/src/exception/seh.cpp:283
dotnet/coreclr#8  0xb6880036 in common_signal_handler (code=11, siginfo=0xae503e40, sigcontext=0xae503ec0, numParams=2) at /home/mskvortsov/git/coreclr/src/pal/src/exception/signal.cpp:819
dotnet/coreclr#9  0xb687fe2c in signal_handler_worker (code=11, siginfo=0xae503e40, context=0xae503ec0, returnPoint=0xae503a50) at /home/mskvortsov/git/coreclr/src/pal/src/exception/signal.cpp:414
dotnet/coreclr#10 0xb6930b8e in CallSignalHandlerWrapper0 () at /home/mskvortsov/git/coreclr/src/pal/src/arch/arm/callsignalhandlerwrapper.S:31

[janvorli]

@mskvortsov regarding the 2nd dump, are you sure the failure point is in the thread whose stack you've dumped? My experience is that when you load a dump with SIGSEGV, the faulting thread is often different from the thread 1 that it shows by default. I usually have to do bt for all threads and then pinpoint the one that have caused the problem by looking at the current instruction's operands and checking if the related memory address is valid or not.

[mskvortsov]

@janvorli I didn't know the first thread gdb shows may differ from the failing one, thanks for pointing this out!

I have checked and can confirm the 2nd dump is indeed from the failing thread:

Program terminated with signal SIGSEGV, Segmentation fault.
#0  access_mem (as=0xb61bd130 <local_addr_space>, addr=4257793536, val=0xae3f918c, write=0, arg=0xae3fd224) at arm/Ginit.c:86
86	arm/Ginit.c: No such file or directory.
[Current thread is 1 (Thread 0xae3ff450 (LWP 26190))]
(gdb) info threads
  Id   Target Id         Frame 
* 1    Thread 0xae3ff450 (LWP 26190) access_mem (as=0xb61bd130 <local_addr_space>, addr=4257793536, val=0xae3f918c, write=0, arg=0xae3fd224) at arm/Ginit.c:86
  2    Thread 0xb6f12000 (LWP 26183) __libc_do_syscall () at ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46
  3    Thread 0xb43ff450 (LWP 26187) __libc_do_syscall () at ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46
  4    Thread 0xb57bf450 (LWP 26185) __libc_do_syscall () at ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46
  5    Thread 0xb613c450 (LWP 26184) 0xb6c9da50 in poll () at ../sysdeps/unix/syscall-template.S:84
  6    Thread 0xb4dff450 (LWP 26186) 0xb6eb7e4e in open () at ../sysdeps/unix/syscall-template.S:84
(gdb) disas
Dump of assembler code for function access_mem:
   0xb619faf0 <+0>:	cbnz	r3, 0xb619fafa <access_mem+10>
=> 0xb619faf2 <+2>:	ldr	r3, [r1, #0]
   0xb619faf4 <+4>:	movs	r0, #0
   0xb619faf6 <+6>:	str	r3, [r2, #0]
   0xb619faf8 <+8>:	bx	lr
   0xb619fafa <+10>:	ldr	r3, [r2, #0]
   0xb619fafc <+12>:	movs	r0, #0
   0xb619fafe <+14>:	str	r3, [r1, #0]
   0xb619fb00 <+16>:	bx	lr
End of assembler dump.
(gdb) p/x $r1 
$1 = 0xfdc8c600
(gdb) p/x *$r1
Cannot access memory at address 0xfdc8c600
(gdb)

[hqueue]

@mskvortsov Unfortunately, I couldn't reproduce this one using your repro.sh on Rpi3 with last week CoreCLR. (commit a2ecf15) until more than 100 trials(maybe).

$ ./repro.sh
..................................................................................................................................................................................................................................................................................................................

My rpi3 use Ubuntu mate (16.04) with glibc 2.23 and libunwind8(1.1-4.1) where access_mem is defined.

$ dpkg -l |grep libunwind8
ii  libunwind8                            1.1-4.1                                    armhf        library to determine the call-chain of a program - runtime
$ dpkg -l |grep libc6
ii  libc6:armhf                           2.23-0ubuntu9                              armhf        GNU C Library: Shared libraries
ii  libc6-dbg:armhf                       2.23-0ubuntu9                              armhf        GNU C Library: detached debugging symbols
ii  libc6-dev:armhf                       2.23-0ubuntu9                              armhf        GNU C Library: Development Libraries and Header Files

[alpencolt]

@janvorli I've reproduced this issue and it looks that crash occurred on Thread 1:

(gdb) i threads
  Id   Target Id         Frame 
* 1    Thread 0xaf7fd450 (LWP 17145) 0xb636aa68 in MethodTable::GetFlag (this=0x1000400, flag=MethodTable::enum_flag_HasComponentSize) at /home/alexander/src/coreclr/src/vm/methodtable.h:3979
  2    Thread 0xaedff450 (LWP 17146) __libc_do_syscall () at ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46
  3    Thread 0xadbff450 (LWP 17148) __libc_do_syscall () at ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46
  4    Thread 0xad1ff450 (LWP 17149) __libc_do_syscall () at ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46
  5    Thread 0xb619a450 (LWP 17141) 0xb6cf7a50 in poll () at ../sysdeps/unix/syscall-template.S:84
  6    Thread 0xb4dff450 (LWP 17143) 0xb6f11e4e in open () at ../sysdeps/unix/syscall-template.S:84
  7    Thread 0xb45ff450 (LWP 17144) __libc_do_syscall () at ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46
  8    Thread 0xb57ff450 (LWP 17142) __libc_do_syscall () at ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46
  9    Thread 0xae3ff450 (LWP 17147) __libc_do_syscall () at ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46
  10   Thread 0xb6f6c000 (LWP 17140) __libc_do_syscall () at ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46

Almost all other threads is waiting on:

#0  __libc_do_syscall () at ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46
dotnet/coreclr#1  0xb6f0edba in __pthread_cond_wait (cond=0xfd5e8, mutex=0xfd5d0) at pthread_cond_wait.c:186
dotnet/coreclr#2  0xb6972b4e in CorUnix::CPalSynchronizationManager::ThreadNativeWait (ptnwdNativeWaitData=0xfd5d0, dwTimeout=4294967295, ptwrWakeupReason=0xadbfcd58, pdwSignaledObject=0xadbfcd54)
    at /home/alexander/src/coreclr/src/pal/src/synchmgr/synchmanager.cpp:475
dotnet/coreclr#3  0xb6972036 in CorUnix::CPalSynchronizationManager::BlockThread (this=0x32330, pthrCurrent=0xfd438, dwTimeout=4294967295, fAlertable=false, fIsSleep=false, ptwrWakeupReason=0xadbfd074, pdwSignaledObject=0xadbfd09c)
    at /home/alexander/src/coreclr/src/pal/src/synchmgr/synchmanager.cpp:298
dotnet/coreclr#4  0xb697f920 in CorUnix::InternalWaitForMultipleObjectsEx (pThread=0xfd438, nCount=1, lpHandles=0xadbfd1a0, bWaitAll=0, dwMilliseconds=4294967295, bAlertable=0) at /home/alexander/src/coreclr/src/pal/src/synchmgr/wait.cpp:561
dotnet/coreclr#5  0xb6980128 in WaitForSingleObjectEx (hHandle=0xa8, dwMilliseconds=4294967295, bAlertable=0) at /home/alexander/src/coreclr/src/pal/src/synchmgr/wait.cpp:96
dotnet/coreclr#6  0xb65750c2 in CLREventWaitHelper2 (handle=0xa8, dwMilliseconds=4294967295, alertable=0) at /home/alexander/src/coreclr/src/vm/synch.cpp:385
dotnet/coreclr#7  0xb6574fb4 in CLREventWaitHelper(void*, unsigned int, int)::$_1::operator()(CLREventWaitHelper(void*, unsigned int, int)::Param*) const (this=0xadbfd254, pParam=0xadbfd25c) at /home/alexander/src/coreclr/src/vm/synch.cpp:411
dotnet/coreclr#8  0xb6574a32 in CLREventWaitHelper (handle=0xa8, dwMilliseconds=4294967295, alertable=0) at /home/alexander/src/coreclr/src/vm/synch.cpp:413
dotnet/coreclr#9  0xb657499a in CLREventBase::WaitEx (this=0x7c588, dwMilliseconds=4294967295, mode=WaitMode_None, syncState=0x0) at /home/alexander/src/coreclr/src/vm/synch.cpp:483
dotnet/coreclr#10 0xb6574844 in CLREventBase::Wait (this=0x7c588, dwMilliseconds=4294967295, alertable=0, syncState=0x0) at /home/alexander/src/coreclr/src/vm/synch.cpp:426
dotnet/coreclr#11 0xb6739772 in GCEvent::Impl::Wait (this=0x7c588, timeout=4294967295, alertable=false) at /home/alexander/src/coreclr/src/vm/gcenv.os.cpp:769
dotnet/coreclr#12 0xb6738f7c in GCEvent::Wait (this=0xb6c5cf78 <WKS::gc_heap::gc_done_event>, timeout=4294967295, alertable=false) at /home/alexander/src/coreclr/src/vm/gcenv.os.cpp:847
dotnet/coreclr#13 0xb6600116 in WKS::gc_heap::wait_for_gc_done (timeOut=-1) at /home/alexander/src/coreclr/src/gc/gc.cpp:10216
dotnet/coreclr#14 0xb660dcf4 in WKS::gc_heap::try_allocate_more_space (acontext=0xf63e8, size=16, gen_number=0) at /home/alexander/src/coreclr/src/gc/gc.cpp:13094
dotnet/coreclr#15 0xb660dec4 in WKS::gc_heap::allocate_more_space (acontext=0xf63e8, size=16, alloc_generation_number=0) at /home/alexander/src/coreclr/src/gc/gc.cpp:13465
dotnet/coreclr#16 0xb66363d8 in WKS::gc_heap::allocate (jsize=16, acontext=0xf63e8) at /home/alexander/src/coreclr/src/gc/gc.cpp:13496
dotnet/coreclr#17 0xb662e3a4 in WKS::GCHeap::Alloc (this=0x45af8, context=0xf63e8, size=16, flags=2) at /home/alexander/src/coreclr/src/gc/gc.cpp:34381
dotnet/coreclr#18 0xb64d0272 in Alloc (size=16, bFinalize=0, bContainsPointers=16777216) at /home/alexander/src/coreclr/src/vm/gchelpers.cpp:241
dotnet/coreclr#19 0xb64cec56 in AllocateArrayEx (arrayType=..., pArgs=0xadbfd834, dwNumArgs=1, bAllocateInLargeHeap=0, bDontSetAppDomain=0) at /home/alexander/src/coreclr/src/vm/gchelpers.cpp:583
dotnet/coreclr#20 0xb64cf876 in AllocateObjectArray (cElements=1, elementType=..., bAllocateInLargeHeap=0) at /home/alexander/src/coreclr/src/vm/gchelpers.cpp:979
dotnet/coreclr#21 0xb6647d8c in ThreadStaticHandleBucket::ThreadStaticHandleBucket (this=0xad2006f8, pNext=0x0, Size=1, pDomain=0x5add0) at /home/alexander/src/coreclr/src/vm/appdomain.cpp:615
dotnet/coreclr#22 0xb6647ef8 in ThreadStaticHandleTable::AllocateHandles (this=0xad2006d8, nRequested=1) at /home/alexander/src/coreclr/src/vm/appdomain.cpp:701
dotnet/coreclr#23 0xb6443ba4 in ThreadLocalBlock::AllocateStaticFieldObjRefPtrs (this=0xad200628, nRequested=1, ppLazyAllocate=0xad2006b0) at /home/alexander/src/coreclr/src/vm/threadstatics.cpp:335
dotnet/coreclr#24 0xb6443aca in ThreadLocalBlock::AllocateThreadStaticHandles (this=0xad200628, pModule=0xb595fb28, pThreadLocalModule=0xad2006a8) at /home/alexander/src/coreclr/src/vm/threadstatics.cpp:303
dotnet/runtime#3858 0xb644420a in ThreadStatics::AllocateAndInitTLM (index=..., pThreadLocalBlock=0xad200628, pModule=0xb595fb28) at /home/alexander/src/coreclr/src/vm/threadstatics.cpp:587
dotnet/runtime#3859 0xb64443ae in ThreadStatics::GetTLM (index=..., pModule=0xb595fb28) at /home/alexander/src/coreclr/src/vm/threadstatics.cpp:617
dotnet/coreclr#27 0xb6444416 in ThreadStatics::GetTLM (pMT=0xb5960e94) at /home/alexander/src/coreclr/src/vm/threadstatics.cpp:626
dotnet/runtime#3860 0xb64e6c80 in JIT_GetGCThreadStaticBase_Helper (pMT=0xb5960e94) at /home/alexander/src/coreclr/src/vm/jithelpers.cpp:1829
dotnet/runtime#3861 0xb64e714c in JIT_GetSharedGCThreadStaticBase (moduleDomainID=3046506992, dwClassDomainID=5) at /home/alexander/src/coreclr/src/vm/jithelpers.cpp:1929
dotnet/runtime#3862 0xaf83d41a in ?? ()

It look crash occurred on garbage collecting. I've got the SIGSEGV on:

GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.04) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabihf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from src/corerun...done.
[New LWP 4531]
[New LWP 4534]
[New LWP 4533]
[New LWP 4532]
[New LWP 4528]
[New LWP 4526]
[New LWP 4530]
[New LWP 4529]
[New LWP 4527]
[New LWP 4535]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
Core was generated by `src/corerun ../mskvortsov/tests-release/JIT/Methodical/cctor/misc/threads1_cs_r'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0xb63bea68 in MethodTable::GetFlag (this=0x1000400, flag=MethodTable::enum_flag_HasComponentSize) at /home/alexander/src/coreclr/src/vm/methodtable.h:3979
3979	/home/alexander/src/coreclr/src/vm/methodtable.h: No such file or directory.
[Current thread is 1 (Thread 0xaf84d450 (LWP 4531))]
(gdb) bt
#0  0xb63bea68 in MethodTable::GetFlag (this=0x1000400, flag=MethodTable::enum_flag_HasComponentSize) at /home/alexander/src/coreclr/src/vm/methodtable.h:3979
dotnet/coreclr#1  0xb63cb8b0 in MethodTable::HasComponentSize (this=0x1000400) at /home/alexander/src/coreclr/src/vm/methodtable.h:1819
dotnet/coreclr#2  0xb6442d88 in MethodTable::GetComponentSize (this=0x1000400) at /home/alexander/src/coreclr/src/vm/methodtable.h:1841
dotnet/coreclr#3  0xb643d756 in MethodTable::SanityCheck (this=0x1000400) at /home/alexander/src/coreclr/src/vm/methodtable.cpp:7580
dotnet/coreclr#4  0xb64404ce in MethodTable::Validate (this=0x1000400) at /home/alexander/src/coreclr/src/vm/methodtable.cpp:9627
dotnet/coreclr#5  0xb6449102 in Object::ValidateInner (this=0xb1553654, bDeep=1, bVerifyNextHeader=1, bVerifySyncBlock=1) at /home/alexander/src/coreclr/src/vm/object.cpp:1733
dotnet/coreclr#6  0xb644864c in Object::Validate (this=0xb1553654, bDeep=1, bVerifyNextHeader=1, bVerifySyncBlock=1) at /home/alexander/src/coreclr/src/vm/object.cpp:1709
dotnet/coreclr#7  0xb660766e in GcInfoDecoder::ReportStackSlotToGC (this=0xaf84a1a4, spOffset=-32, spBase=GC_FRAMEREG_REL, gcFlags=0, pRD=0xaf84aaa0, flags=0, pCallBack=0xb6521829 <GcEnumObject(void*, OBJECTREF*, unsigned int)>, hCallBack=0xaf84b0c8)
    at /home/alexander/src/coreclr/src/vm/gcinfodecoder.cpp:1821
dotnet/coreclr#8  0xb6607e20 in GcInfoDecoder::ReportSlotToGC (this=0xaf84a1a4, slotDecoder=..., slotIndex=10, pRD=0xaf84aaa0, reportScratchSlots=false, inputFlags=0, pCallBack=0xb6521829 <GcEnumObject(void*, OBJECTREF*, unsigned int)>, 
    hCallBack=0xaf84b0c8) at /home/alexander/src/coreclr/src/inc/gcinfodecoder.h:665
dotnet/coreclr#9  0xb66062fa in GcInfoDecoder::EnumerateLiveSlots (this=0xaf84a1a4, pRD=0xaf84aaa0, reportScratchSlots=false, inputFlags=0, pCallBack=0xb6521829 <GcEnumObject(void*, OBJECTREF*, unsigned int)>, hCallBack=0xaf84b0c8)
    at /home/alexander/src/coreclr/src/vm/gcinfodecoder.cpp:934
dotnet/coreclr#10 0xb63cd790 in EECodeManager::EnumGcRefs (this=0x60528, pRD=0xaf84aaa0, pCodeInfo=0xaf84a958, flags=0, pCallBack=0xb6521829 <GcEnumObject(void*, OBJECTREF*, unsigned int)>, hCallBack=0xaf84b0c8, relOffsetOverride=4294967295)
    at /home/alexander/src/coreclr/src/vm/eetwain.cpp:5062
dotnet/coreclr#11 0xb6521d80 in GcStackCrawlCallBack (pCF=0xaf84a738, pData=0xaf84b0c8) at /home/alexander/src/coreclr/src/vm/gcenv.ee.common.cpp:280
dotnet/coreclr#12 0xb646dcc0 in Thread::MakeStackwalkerCallback (this=0xf51b8, pCF=0xaf84a738, pCallback=0xb65219a5 <GcStackCrawlCallBack(CrawlFrame*, void*)>, pData=0xaf84b0c8, uFramesProcessed=2)
    at /home/alexander/src/coreclr/src/vm/stackwalk.cpp:868
dotnet/coreclr#13 0xb646de62 in Thread::StackWalkFramesEx (this=0xf51b8, pRD=0xaf84aaa0, pCallback=0xb65219a5 <GcStackCrawlCallBack(CrawlFrame*, void*)>, pData=0xaf84b0c8, flags=34048, pStartFrame=0x0)
    at /home/alexander/src/coreclr/src/vm/stackwalk.cpp:949
dotnet/coreclr#14 0xb646e6e6 in Thread::StackWalkFrames (this=0xf51b8, pCallback=0xb65219a5 <GcStackCrawlCallBack(CrawlFrame*, void*)>, pData=0xaf84b0c8, flags=34048, pStartFrame=0x0) at /home/alexander/src/coreclr/src/vm/stackwalk.cpp:1032
dotnet/coreclr#15 0xb678a884 in ScanStackRoots (pThread=0xf51b8, fn=0xb6671fb1 <WKS::GCHeap::Promote(Object**, ScanContext*, unsigned int)>, sc=0xaf84b228) at /home/alexander/src/coreclr/src/vm/gcenv.ee.cpp:149
dotnet/coreclr#16 0xb678a604 in GCToEEInterface::GcScanRoots (fn=0xb6671fb1 <WKS::GCHeap::Promote(Object**, ScanContext*, unsigned int)>, condemned=0, max_gen=2, sc=0xaf84b228) at /home/alexander/src/coreclr/src/vm/gcenv.ee.cpp:178
dotnet/coreclr#17 0xb67c1d26 in GCScan::GcScanRoots (fn=0xb6671fb1 <WKS::GCHeap::Promote(Object**, ScanContext*, unsigned int)>, condemned=0, max_gen=2, sc=0xaf84b228) at /home/alexander/src/coreclr/src/gc/gcscan.cpp:155
dotnet/coreclr#18 0xb66666f6 in WKS::gc_heap::mark_phase (condemned_gen_number=0, mark_only_p=0) at /home/alexander/src/coreclr/src/gc/gc.cpp:19553
dotnet/coreclr#19 0xb6664550 in WKS::gc_heap::gc1 () at /home/alexander/src/coreclr/src/gc/gc.cpp:15367
dotnet/coreclr#20 0xb666ca9a in WKS::gc_heap::garbage_collect (n=0) at /home/alexander/src/coreclr/src/gc/gc.cpp:16915
dotnet/coreclr#21 0xb6660a1c in WKS::GCHeap::GarbageCollectGeneration (this=0x45b40, gen=0, reason=reason_alloc_soh) at /home/alexander/src/coreclr/src/gc/gc.cpp:35039
dotnet/coreclr#22 0xb6661dac in WKS::gc_heap::try_allocate_more_space (acontext=0xd5d70, size=16, gen_number=0) at /home/alexander/src/coreclr/src/gc/gc.cpp:13169
dotnet/coreclr#23 0xb6661ec4 in WKS::gc_heap::allocate_more_space (acontext=0xd5d70, size=16, alloc_generation_number=0) at /home/alexander/src/coreclr/src/gc/gc.cpp:13465
dotnet/coreclr#24 0xb668a3d8 in WKS::gc_heap::allocate (jsize=16, acontext=0xd5d70) at /home/alexander/src/coreclr/src/gc/gc.cpp:13496
dotnet/runtime#3858 0xb66823a4 in WKS::GCHeap::Alloc (this=0x45b40, context=0xd5d70, size=16, flags=2) at /home/alexander/src/coreclr/src/gc/gc.cpp:34381
dotnet/runtime#3859 0xb6524272 in Alloc (size=16, bFinalize=0, bContainsPointers=16777216) at /home/alexander/src/coreclr/src/vm/gchelpers.cpp:241
dotnet/coreclr#27 0xb6522c56 in AllocateArrayEx (arrayType=..., pArgs=0xaf84b96c, dwNumArgs=1, bAllocateInLargeHeap=0, bDontSetAppDomain=0) at /home/alexander/src/coreclr/src/vm/gchelpers.cpp:583
dotnet/runtime#3860 0xb6523876 in AllocateObjectArray (cElements=1, elementType=..., bAllocateInLargeHeap=0) at /home/alexander/src/coreclr/src/vm/gchelpers.cpp:979
dotnet/runtime#3861 0xb669bd8c in ThreadStaticHandleBucket::ThreadStaticHandleBucket (this=0xaef006b0, pNext=0x0, Size=1, pDomain=0x5ae00) at /home/alexander/src/coreclr/src/vm/appdomain.cpp:615
dotnet/runtime#3862 0xb669bef8 in ThreadStaticHandleTable::AllocateHandles (this=0xaef00fa0, nRequested=1) at /home/alexander/src/coreclr/src/vm/appdomain.cpp:701
dotnet/runtime#3863 0xb6497ba4 in ThreadLocalBlock::AllocateStaticFieldObjRefPtrs (this=0xaef082a0, nRequested=1, ppLazyAllocate=0xaef005d8) at /home/alexander/src/coreclr/src/vm/threadstatics.cpp:335
dotnet/coreclr#32 0xb6497aca in ThreadLocalBlock::AllocateThreadStaticHandles (this=0xaef082a0, pModule=0xb59b3b28, pThreadLocalModule=0xaef005d0) at /home/alexander/src/coreclr/src/vm/threadstatics.cpp:303
dotnet/runtime#3864 0xb649820a in ThreadStatics::AllocateAndInitTLM (index=..., pThreadLocalBlock=0xaef082a0, pModule=0xb59b3b28) at /home/alexander/src/coreclr/src/vm/threadstatics.cpp:587
dotnet/runtime#3865 0xb64983ae in ThreadStatics::GetTLM (index=..., pModule=0xb59b3b28) at /home/alexander/src/coreclr/src/vm/threadstatics.cpp:617
dotnet/coreclr#35 0xb6498416 in ThreadStatics::GetTLM (pMT=0xb59b4e94) at /home/alexander/src/coreclr/src/vm/threadstatics.cpp:626
dotnet/coreclr#36 0xb653ac80 in JIT_GetGCThreadStaticBase_Helper (pMT=0xb59b4e94) at /home/alexander/src/coreclr/src/vm/jithelpers.cpp:1829
dotnet/runtime#3866 0xb653b14c in JIT_GetSharedGCThreadStaticBase (moduleDomainID=3046851056, dwClassDomainID=5) at /home/alexander/src/coreclr/src/vm/jithelpers.cpp:1929
dotnet/coreclr#38 0xaf88d41a in ?? ()

[alpencolt]

We use Exynos 5422 which has 4 A7 and 4 A15 cores. The funniest thing is that when I run dotnet only on one type of core (e.g only on A7 or A15) by using taskset bug cannot be reproduced. But if I use mixed mode (like taskset -c 2-5 corerun ...) crash occurs.

[alpencolt]

Guys from Mono faced problem caused different cache size in big.LITTLE architectire:
http://www.mono-project.com/news/2016/09/12/arm64-icache/

I've checked source code and __clear_cach is used in FlushInstructionCache() only, I'm not sure that it's our case.
Bug cannot be reproduced on Samsung Z3 (btw it has only 2 cores with identical architectures.).

[lemmaa]

@alpencolt , Can you please make a chance to apply the patch, https://github.com/mono/mono/pull/3549/files, and share the results here? It is not urgent but it seems to be worth try if you agree.

[alpencolt]

@lemmaa I will try after closing current issues. Do you have any considerations where flush_icache() may occur?

[alpencolt]

I've got following exception on the latest master:

FailFast: 

   at System.Diagnostics.Debug.Assert(Boolean condition, String message, String detailMessage)
   at System.Threading.ExecutionContext.Restore(Thread currentThread, ExecutionContext executionContext)

   at System.Environment.FailFast(System.String, System.Exception)
   at System.Diagnostics.Debug.ShowAssertDialog(System.String, System.String, System.String)
   at System.Diagnostics.Debug.Assert(Boolean, System.String, System.String)
   at System.Threading.ExecutionContext.Restore(System.Threading.Thread, System.Threading.ExecutionContext)
Aborted (core dumped)

This assertion failed:

internal static void Restore(Thread currentThread, ExecutionContext executionContext)
{
    Debug.Assert(currentThread == Thread.CurrentThread);
    ...
}

          

[alpencolt]

Debug version failed with the same exception as #7825. Back trace:

#0  __libc_do_syscall () at ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:47
dotnet/coreclr#1  0xb6c25648 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
dotnet/coreclr#2  0xb6c2634a in __GI_abort () at abort.c:89
dotnet/coreclr#3  0xb6921484 in PROCAbort () at /home/alexander/src/coreclr/src/pal/src/thread/process.cpp:3046
dotnet/coreclr#4  0xb691e188 in PROCEndProcess (hProcess=0xffffff01, uExitCode=123456789, bTerminateUnconditionally=1) at /home/alexander/src/coreclr/src/pal/src/thread/process.cpp:1394
dotnet/coreclr#5  0xb691e2c6 in TerminateProcess (hProcess=0xffffff01, uExitCode=123456789) at /home/alexander/src/coreclr/src/pal/src/thread/process.cpp:1310
dotnet/coreclr#6  0xb62d963e in TerminateOnAssert () at /home/alexander/src/coreclr/src/utilcode/debug.cpp:183
dotnet/coreclr#7  0xb62d9e76 in _DbgBreakCheck (szFile=0xb69bc6a8 "/home/alexander/src/coreclr/src/vm/object.cpp", iLine=1738, szExpr=0xb69bca15 "!CREATE_CHECK_STRING(pMT)", fConstrained=0) at /home/alexander/src/coreclr/src/utilcode/debug.cpp:436
dotnet/coreclr#8  0xb62da254 in _DbgBreakCheckNoThrow (szFile=0xb69bc6a8 "/home/alexander/src/coreclr/src/vm/object.cpp", iLine=1738, szExpr=0xb69bca15 "!CREATE_CHECK_STRING(pMT)", fConstrained=0)
    at /home/alexander/src/coreclr/src/utilcode/debug.cpp:548
dotnet/coreclr#9  0xb62da5ae in DbgAssertDialog (szFile=0xb69bc6a8 "/home/alexander/src/coreclr/src/vm/object.cpp", iLine=1738, szExpr=0xb69bca15 "!CREATE_CHECK_STRING(pMT)") at /home/alexander/src/coreclr/src/utilcode/debug.cpp:735
dotnet/coreclr#10 0xb6389d66 in Object::ValidateInner (this=0xadafddf8, bDeep=1, bVerifyNextHeader=1, bVerifySyncBlock=1) at /home/alexander/src/coreclr/src/vm/object.cpp:1738
dotnet/coreclr#11 0xb6389288 in Object::Validate (this=0xadafddf8, bDeep=1, bVerifyNextHeader=1, bVerifySyncBlock=1) at /home/alexander/src/coreclr/src/vm/object.cpp:1709
dotnet/coreclr#12 0xb654e148 in GcInfoDecoder::ReportStackSlotToGC (this=0xadaf90bc, spOffset=-16, spBase=GC_FRAMEREG_REL, gcFlags=0, pRD=0xadaf99b8, flags=2, pCallBack=0xb6466d3d <GcEnumObject(void*, OBJECTREF*, unsigned int)>, hCallBack=0xadaf9fe0)
    at /home/alexander/src/coreclr/src/vm/gcinfodecoder.cpp:1826
dotnet/coreclr#13 0xb654e8fc in GcInfoDecoder::ReportSlotToGC (this=0xadaf90bc, slotDecoder=..., slotIndex=13, pRD=0xadaf99b8, reportScratchSlots=false, inputFlags=2, pCallBack=0xb6466d3d <GcEnumObject(void*, OBJECTREF*, unsigned int)>, 
    hCallBack=0xadaf9fe0) at /home/alexander/src/coreclr/src/inc/gcinfodecoder.h:665
dotnet/coreclr#14 0xb654cdea in GcInfoDecoder::EnumerateLiveSlots (this=0xadaf90bc, pRD=0xadaf99b8, reportScratchSlots=false, inputFlags=2, pCallBack=0xb6466d3d <GcEnumObject(void*, OBJECTREF*, unsigned int)>, hCallBack=0xadaf9fe0)
    at /home/alexander/src/coreclr/src/vm/gcinfodecoder.cpp:934
dotnet/coreclr#15 0xb630db54 in EECodeManager::EnumGcRefs (this=0x61ff0, pRD=0xadaf99b8, pCodeInfo=0xadaf9870, flags=2, pCallBack=0xb6466d3d <GcEnumObject(void*, OBJECTREF*, unsigned int)>, hCallBack=0xadaf9fe0, relOffsetOverride=4294967295)
    at /home/alexander/src/coreclr/src/vm/eetwain.cpp:5062
dotnet/coreclr#16 0xb6467294 in GcStackCrawlCallBack (pCF=0xadaf9650, pData=0xadaf9fe0) at /home/alexander/src/coreclr/src/vm/gcenv.ee.common.cpp:280
dotnet/coreclr#17 0xb63b01c8 in Thread::MakeStackwalkerCallback (this=0xe7dc8, pCF=0xadaf9650, pCallback=0xb6466eb9 <GcStackCrawlCallBack(CrawlFrame*, void*)>, pData=0xadaf9fe0, uFramesProcessed=52)
    at /home/alexander/src/coreclr/src/vm/stackwalk.cpp:864
dotnet/coreclr#18 0xb63b036a in Thread::StackWalkFramesEx (this=0xe7dc8, pRD=0xadaf99b8, pCallback=0xb6466eb9 <GcStackCrawlCallBack(CrawlFrame*, void*)>, pData=0xadaf9fe0, flags=34048, pStartFrame=0x0)
    at /home/alexander/src/coreclr/src/vm/stackwalk.cpp:945
dotnet/coreclr#19 0xb63b0bee in Thread::StackWalkFrames (this=0xe7dc8, pCallback=0xb6466eb9 <GcStackCrawlCallBack(CrawlFrame*, void*)>, pData=0xadaf9fe0, flags=34048, pStartFrame=0x0) at /home/alexander/src/coreclr/src/vm/stackwalk.cpp:1028
dotnet/coreclr#20 0xb66cc9d4 in ScanStackRoots (pThread=0xe7dc8, fn=0xb65b8bb9 <WKS::GCHeap::Promote(Object**, ScanContext*, unsigned int)>, sc=0xadafa140) at /home/alexander/src/coreclr/src/vm/gcenv.ee.cpp:149
dotnet/coreclr#21 0xb66cc754 in GCToEEInterface::GcScanRoots (fn=0xb65b8bb9 <WKS::GCHeap::Promote(Object**, ScanContext*, unsigned int)>, condemned=0, max_gen=2, sc=0xadafa140) at /home/alexander/src/coreclr/src/vm/gcenv.ee.cpp:178
dotnet/coreclr#22 0xb670d01e in GCScan::GcScanRoots (fn=0xb65b8bb9 <WKS::GCHeap::Promote(Object**, ScanContext*, unsigned int)>, condemned=0, max_gen=2, sc=0xadafa140) at /home/alexander/src/coreclr/src/gc/gcscan.cpp:155
dotnet/coreclr#23 0xb65ad2fe in WKS::gc_heap::mark_phase (condemned_gen_number=0, mark_only_p=0) at /home/alexander/src/coreclr/src/gc/gc.cpp:19585
dotnet/coreclr#24 0xb65ab154 in WKS::gc_heap::gc1 () at /home/alexander/src/coreclr/src/gc/gc.cpp:15396
dotnet/runtime#3858 0xb65b36a2 in WKS::gc_heap::garbage_collect (n=0) at /home/alexander/src/coreclr/src/gc/gc.cpp:16947
dotnet/runtime#3859 0xb65a7620 in WKS::GCHeap::GarbageCollectGeneration (this=0x470b8, gen=0, reason=reason_alloc_soh) at /home/alexander/src/coreclr/src/gc/gc.cpp:35074
dotnet/coreclr#27 0xb65a89b0 in WKS::gc_heap::try_allocate_more_space (acontext=0xe7e08, size=20, gen_number=0) at /home/alexander/src/coreclr/src/gc/gc.cpp:13198
dotnet/runtime#3860 0xb65a8ac8 in WKS::gc_heap::allocate_more_space (acontext=0xe7e08, size=20, alloc_generation_number=0) at /home/alexander/src/coreclr/src/gc/gc.cpp:13494
dotnet/runtime#3861 0xb65d1000 in WKS::gc_heap::allocate (jsize=20, acontext=0xe7e08) at /home/alexander/src/coreclr/src/gc/gc.cpp:13525
dotnet/runtime#3862 0xb65c8fc4 in WKS::GCHeap::Alloc (this=0x470b8, context=0xe7e08, size=20, flags=0) at /home/alexander/src/coreclr/src/gc/gc.cpp:34416
dotnet/runtime#3863 0xb6469816 in Alloc (size=20, bFinalize=0, bContainsPointers=0) at /home/alexander/src/coreclr/src/vm/gchelpers.cpp:241
dotnet/coreclr#32 0xb64689ac in FastAllocatePrimitiveArray (pMT=0xb3c371e0, cElements=8, bAllocateInLargeHeap=0) at /home/alexander/src/coreclr/src/vm/gchelpers.cpp:824
dotnet/runtime#3864 0xb64836d0 in JIT_NewArr1 (arrayMT=0xb3c371e0, size=8) at /home/alexander/src/coreclr/src/vm/jithelpers.cpp:3228
dotnet/runtime#3865 0xab5b4d1e in ?? ()

After few times test were passed it failed with this stack. So as you can see there is garbage collection on JIT_NewArr1() calling. The reason is that GetGCSafeMethodTable() returns 0.
Also I've found that before crash StackWalker iterates over big amount of frames and there is one frame with Frame::FRAME_ATTR_EXCEPTION attribute.

[alpencolt]

Crash on debug is another error dotnet/coreclr#14238. Root cause is the same - assertion in System.Threading.ExecutionContext.Restore()