.NET 9 crashes on Windows arm64 with attached profiler and tiered compilation.

[kalikin]

Description

Our profiler's attach test crashes in .NET 9 (and 10) on Windows arm64. Maybe it crashes on other platforms, but on Windows arm64 almost always. I just realized that #100024 was similar, but using regular RequestReJIT doesn't fix it now, although it may crash less with it on .NET 9, but not on .NET 10.

I've opened a crash dump in WinDbg and it appears to be NPE in tieredcompilation.cpp:
coreclr!TieredCompilationManager::GetInitialOptimizationTier+0x1b(MethodDesc * pMethodDesc = 0x0)

Reproduction Steps

For example:

• Attach profiler using attach API and get eligible methods with EnumJITedFunctions
• Call RequestReJITWithInliners for these methods from a profiler thread

Expected behavior

No crash is expected. The profiler API should return an error result if the error is caused by incorrect usage or an internal issue within the API.

Actual behavior

Here's the WinDbg stack from the crash dump:

[0x0]   ntdll!#NtWaitForSingleObject+0x14   0x27bc87b800   0x7ffcc6584d94   
[0x1]   KERNELBASE!WaitForSingleObjectEx+0x84()   0x27bc87b810   0x7ffcc66c8afc   
[0x2]   KERNELBASE!$ientry_thunk$cdecl$i8$i8+0x24   0x27bc87b8a0   0x7ffc65c06770   
[0x3]   coreclr!LaunchCreateDump+0xe0(wchar_t * lpCommandLine = 0x1d9791f3900 : "--- memory read error at address 0x000001d9`791f3900 ---", wchar_t * lpCommandLine = 0x2 : "--- memory read error at address 0x00000000`00000002 ---")   0x27bc87b950   0x7ffc65c02e3c   
[0x4]   coreclr!CreateCrashDumpIfEnabled+0x80(bool stackoverflow = false)   0x27bc87ba90   0x7ffc65c09e3b   
[0x5]   coreclr!WatsonLastChance+0x193(Thread * pThread = 0x1d979290b00, _EXCEPTION_POINTERS * pExceptionInfo = 0x27bc87cfa8, TypeOfReportedError tore)   0x27bc87bae0   0x7ffc65c46751   
[0x6]   coreclr!EEPolicy::LogFatalError+0x691(unsigned int exitCode = 0x80131506, unsigned int exitCode = 0x80131506, unsigned int exitCode = 0x1953d0, unsigned __int64 address = 0x0, wchar_t * pszMessage = 0x0, wchar_t * pszMessage = 0x7ffc6875eb20 : "潴???", _EXCEPTION_POINTERS * pExceptionInfo = 0x27bc87cfa8, _EXCEPTION_POINTERS * pExceptionInfo = 0x27bc87cfa8, _EXCEPTION_POINTERS * pExceptionInfo = 0x27bc87d020, wchar_t * errorSource = 0x0, wchar_t * argExceptionString = 0x0)   0x27bc87bb50   0x7ffc65c4594b   
[0x7]   coreclr!EEPolicy::HandleFatalError+0x133(unsigned int exitCode = 0x80131506, unsigned __int64 address = 0x7ffc65bdf66d, wchar_t * pszMessage = 0x0, _EXCEPTION_POINTERS * pExceptionInfo = 0x27bc87cfa8, wchar_t * errorSource = 0x0, wchar_t * errorSource = 0x0, wchar_t * argExceptionString = 0x0, wchar_t * argExceptionString = 0x0)   0x27bc87c810   0x7ffc65bdf68c   
[0x8]   coreclr!CLRVectoredExceptionHandlerPhase3+0xa83b4()   (Inline Function)   (Inline Function)   
[0x9]   coreclr!CLRVectoredExceptionHandlerPhase2+0xa83d8(_EXCEPTION_POINTERS * pExceptionInfo = 0x27bc87cfa8)   0x27bc87ce10   0x7ffc65b37289   
[0xa]   coreclr!CLRVectoredExceptionHandler+0x199(_EXCEPTION_POINTERS * pExceptionInfo = 0x27bc87cfa8)   0x27bc87ce70   0x7ffc65b370ad   
[0xb]   coreclr!CLRVectoredExceptionHandlerShim+0xdd(_EXCEPTION_POINTERS * pExceptionInfo = 0x27bc87cfa8)   0x27bc87cef0   0x7ffccad52fac   
[0xc]   ntdll!$iexit_thunk$cdecl$i8$i8+0x1c   0x27bc87cf40   0x7ffccac611d0   
[0xd]   ntdll!#RtlpCallVectoredHandlers+0x140   0x27bc87cf70   0x7ffccacf6338   
[0xe]   ntdll!#RtlDispatchException+0x78   0x27bc87d000   0x7ffccac157e0   
[0xf]   ntdll!KiUserExceptionDispatcher_DetourReturn+0x10   0x27bc87d790   0x7ffc65a970a3   
[0x10]   coreclr!TieredCompilationManager::GetInitialOptimizationTier+0x1b(MethodDesc * pMethodDesc = 0x0)   0x27bc87e0b0   0x7ffc65ac7a9d   
[0x11]   coreclr!NativeCodeVersion::IsFinalTier+0x9()   0x27bc87e100   0x7ffc65ac78ef   
[0x12]   coreclr!CEEJitInfo::getHelperFtn+0x18f(CorInfoHelpFunc ftnNum = 1954968, void * * ppIndirection = 0x27bc87e228)   0x27bc87e130   0x7ffc686831f4   
[0x13]   clrjit!Lowering::LowerDirectCall+0xf4(GenTreeCall * call = 0x1d9001dd498)   0x27bc87e1c0   0x7ffc686835bf   
[0x14]   clrjit!Lowering::LowerCall+0x25f(GenTree * node = 0x1d9001dd498)   0x27bc87e220   0x7ffc68684270   
[0x15]   clrjit!Lowering::LowerNode+0xf0(GenTree * node = 0x1d9001dd498, GenTree * node = 0x1d9001dd498)   0x27bc87e310   0x7ffc68683f8b   
[0x16]   clrjit!Lowering::LowerBlock+0x19()   (Inline Function)   (Inline Function)   
[0x17]   clrjit!Lowering::DoPhase+0x8b()   0x27bc87e560   0x7ffc686520f9   
[0x18]   clrjit!Phase::Run+0x20()   (Inline Function)   (Inline Function)   
[0x19]   clrjit!Compiler::compCompile+0x14e9(void * * methodCodePtr = 0x27bc87ed00, unsigned int * methodCodeSize = 0x27bc87edb0 : 0x98, compileFlags = <unavailable>)   0x27bc87e5c0   0x7ffc686816f0   
[0x1a]   clrjit!Compiler::compCompileHelper+0x360(CORINFO_MODULE_STRUCT_ * classPtr = 0x1d9001ceeb8, compHnd = <unavailable>, CORINFO_METHOD_INFO * methodInfo = 0x27bc87ef90, void * * methodCodePtr = 0x27bc87ed00, unsigned int * methodCodeSize = 0x27bc87edb0 : 0x98, JitFlags * compileFlags = 0x27bc87ed20, JitFlags * compileFlags = 0x27bc87ed20)   0x27bc87e960   0x7ffc6868301d   
[0x1b]   clrjit!Compiler::compCompile+0x24d(CORINFO_MODULE_STRUCT_ * classPtr = 0x7ffc05e64000, CORINFO_MODULE_STRUCT_ * classPtr = 0x27bc87ef90, void * * methodCodePtr = 0x27bc87ed00, void * * methodCodePtr = 0x27bc87ef90, unsigned int * methodCodeSize = 0x27bc87edb0 : 0x98, unsigned int * methodCodeSize = 0x48 : Unable to read memory at Address 0x48, JitFlags * compileFlags = 0x27bc87ed20)   0x27bc87ea40   0x7ffc68683ddb   
[0x1c]   clrjit!jitNativeCode+0x24b(CORINFO_METHOD_STRUCT_ * methodHnd = 0x7ffc06095db8, CORINFO_MODULE_STRUCT_ * classPtr = 0x7ffc05e64000, ICorJitInfo * compHnd = 0x27bc87f0a0, ICorJitInfo * compHnd = 0x27bc87f0a0, CORINFO_METHOD_INFO * methodInfo = 0x27bc87ef90, CORINFO_METHOD_INFO * methodInfo = 0x27bc87ef90, void * * methodCodePtr = 0x27bc87ed00, unsigned int * methodCodeSize = 0x27bc87edb0 : 0x98, JitFlags * compileFlags = 0x27bc87ed20, inlineInfoPtr = <unavailable>)   0x27bc87eb30   0x7ffc686af406   
[0x1d]   clrjit!CILJit::compileMethod+0xa6(ICorJitInfo * compHnd = 0x27bc87f0a0, CORINFO_METHOD_INFO * methodInfo = 0x27bc87ef90, unsigned int flags = 0x1, unsigned char * * entryAddress = 0x27bc87edc0, unsigned char * * entryAddress = 0x27bc87edc0, unsigned int * nativeSizeOfCode = 0x27bc87edb0 : 0x98, unsigned int * nativeSizeOfCode = 0x27bc87edb0 : 0x98)   0x27bc87ecc0   0x7ffc65a9ebb3   
[0x1e]   coreclr!invokeCompileMethodHelper+0x77()   (Inline Function)   (Inline Function)   
[0x1f]   coreclr!invokeCompileMethod+0xa1()   (Inline Function)   (Inline Function)   
[0x20]   coreclr!UnsafeJitFunction+0x5a3(config = <unavailable>, COR_ILMETHOD_DECODER * ILHeader = 0x27bc87f438, CORJIT_FLAGS * pJitFlags = 0x27bc87f328, pSizeOfCode = <unavailable>)   0x27bc87ed60   0x7ffc65a4f259   
[0x21]   coreclr!MethodDesc::JitCompileCodeLocked+0xe1(PrepareCodeConfig * pConfig = 0x27bc87f690, PrepareCodeConfig * pConfig = 0x27bc87f690, PrepareCodeConfig * pConfig = 0x7ffc00000000, COR_ILMETHOD_DECODER * pilHeader = 0x1d90046fc00, ListLockEntryBase<NativeCodeVersion> * pEntry = 0x1d906745450, ListLockEntryBase<NativeCodeVersion> * pEntry = 0x1d906745450, ListLockEntryBase<NativeCodeVersion> * pEntry = 0x27bc87f3d0, pSizeOfCode = <unavailable>)   0x27bc87f270   0x7ffc65a4f0d4   
[0x22]   coreclr!MethodDesc::JitCompileCodeLockedEventWrapper+0x16c(PrepareCodeConfig * pConfig = 0x27bc87f690, ListLockEntryBase<NativeCodeVersion> * pEntry = 0x1d906745450, ListLockEntryBase<NativeCodeVersion> * pEntry = 0x1d906745450)   0x27bc87f390   0x7ffc65a4edd1   
[0x23]   coreclr!MethodDesc::JitCompileCode+0x2f1(PrepareCodeConfig * pConfig = 0x27bc87f690)   0x27bc87f4c0   0x7ffc65a9602e   
[0x24]   coreclr!MethodDesc::PrepareILBasedCode+0xca(PrepareCodeConfig * pConfig = 0x27bc87f690)   0x27bc87f560   0x7ffc65a95db5   
[0x25]   coreclr!MethodDesc::PrepareCode+0x10()   (Inline Function)   (Inline Function)   
[0x26]   coreclr!TieredCompilationManager::CompileCodeVersion+0xc5(NativeCodeVersion * nativeCodeVersion = 0x64)   0x27bc87f5f0   0x7ffc65a9568d   
[0x27]   coreclr!TieredCompilationManager::OptimizeMethod+0x1d()   (Inline Function)   (Inline Function)   
[0x28]   coreclr!TieredCompilationManager::DoBackgroundWork+0x16d(unsigned __int64 * workDurationTicksRef = 0x27bc87f8d8 : 0x17700, unsigned __int64 minWorkDurationTicks = 0x17700, unsigned __int64 maxWorkDurationTicks = 0x124f80)   0x27bc87f710   0x7ffc65b1865f   
[0x29]   coreclr!TieredCompilationManager::BackgroundWorkerStart+0xcb()   0x27bc87f870   0x7ffc65b18555   
[0x2a]   coreclr!TieredCompilationManager::BackgroundWorkerBootstrapper1+0x55(void * __formal = 0x1d9792915d0)   0x27bc87f8c0   0x7ffc65abef35   
[0x2b]   coreclr!ManagedThreadBase_DispatchInner+0xd()   (Inline Function)   (Inline Function)   
[0x2c]   coreclr!ManagedThreadBase_DispatchMiddle+0x79(ManagedThreadCallState * pCallState = 0x0)   0x27bc87f900   0x7ffc65abee5d   
[0x2d]   coreclr!ManagedThreadBase_DispatchOuter+0x8d(pCallState = <unavailable>)   0x27bc87f9b0   0x7ffc65b4c7ea   
[0x2e]   coreclr!ManagedThreadBase_FullTransition+0x24()   (Inline Function)   (Inline Function)   
[0x2f]   coreclr!ManagedThreadBase::KickOff+0x24()   (Inline Function)   (Inline Function)   
[0x30]   coreclr!TieredCompilationManager::BackgroundWorkerBootstrapper0+0x3a(void * args = 0x1d979290b00)   0x27bc87fa20   0x7ffcc94709fc   
[0x31]   kernel32!$iexit_thunk$cdecl$i8$i8+0x1c   0x27bc87fa70   0x7ffcc9407bb0   
[0x32]   kernel32!#BaseThreadInitThunk+0x30   0x27bc87faa0   0x7ffccac5c4c8   

Regression?

.NET 7 and 8 didn't crash.

Known Workarounds

Disabling tiered compilation fixes the issue:

DOTNET_TieredCompilation=0

Configuration

• All .NET 9 builds and .NET 10 on Windows arm64 can crash.
• .NET runtime architecture doesn't matter - x64, x86, arm64 can crash.
• The Windows 11 arm64 runs as Parallels VM on Apple Mac M1.

Other information

No response

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

[JulieLeeMSFT]

@kunalspathak, PTAL.

[AndyAyersMS]

Probably not a codegen issue?

[kunalspathak]

Agree, I don't think this is a codegen issue.

[kunalspathak]

@mangod9

[dotnet-policy-service]

Tagging subscribers to this area: @mangod9
See info in area-owners.md if you want to be subscribed.