JIT crash caused by late GetType() optimizations

[jkotas]

Repro:

Compile and JIT w/ optimizations

using System;
using System.Runtime.CompilerServices;

sealed class Program
{
    [MethodImpl(MethodImplOptions.NoInlining)]
    public object M()
    {
        return GetType2() == typeof(Program) ? base.ToString() : ToString();
    }

    Type GetType2() => GetType();

    static void Main(string[] args)
    {
        new Program().M();
    }
}

Result:

Assert failure(PID 23416 [0x00005b78], Thread: 15036 [0x3abc]): Assertion failed 'OperIs(GT_NOP, GT_CALL, GT_COMMA) || OperIsCompare() || OperIsLong() || OperIsSIMD() || OperIsHWIntrinsic()' in 'Program:M():System.Object:this' during 'Lowering nodeinfo' (IL size 37)

Release build hits AV later

More context: dotnet/corert#6729 (comment)

[jkotas]

Stacktrace:

#0  0x00007fff24c45c9a in CodeGen::genCodeForTreeNode(GenTree*)
dotnet/corert#1  0x00007fff24b2b2f7 in CodeGen::genGenerateMachineCode() ()
dotnet/corert#2  0x00007fff24b2638c in CodeGen::genGenerateCode(void**, unsigned int*) ()
dotnet/corert#3  0x00007fff24b41ae7 in Compiler::compCompileHelper(CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*) ()
dotnet/corert#4  0x00007fff24b45890 in jitNativeCode(CORINFO_METHOD_STRUCT_*, CORINFO_MODULE_STRUCT_*, ICorJitInfo*, CORINFO_METHOD_INFO*, void**, unsigned int*, JitFlags*, void*) ()
dotnet/corert#5  0x00007fff24b4ad67 in CILJit::compileMethod(ICorJitInfo*, CORINFO_METHOD_INFO*, unsigned int, unsigned char**, unsigned int*) ()
dotnet/corert#6  0x00007fff24df6721 in JitCompileMethod (ppException=0x7fff277fd698,

[jkotas]

Crashing on Windows as well

[jkotas]

The problem is that late GetType() optimizations produce IR that goes straight to lowering, but that lowering is not able to handle.

[jkotas]

cc @dotnet/jit-contrib

[EgorBo]

Ah, I guess that was me in #32790 🙁

[AndyAyersMS]

The problematic IR is a comma under a JTRUE:

Optimizing compare of obj.GetType() and type-from-handle to compare method table pointer
GenTreeNode creates assertion:
               [000041] ---X--------              *  NULLCHECK byte  
In BB01 New Local Constant Assertion: V00 != null index=#01, mask=0000000000000001

fgMorphTree BB01, STMT00001 (after)
[000007] ---XG+------              *  JTRUE     void  
[000042] ---X-+------              \--*  COMMA     int   
[000041] ---X-+------                 +--*  NULLCHECK byte  
[000000] -----+------                 |  \--*  LCL_VAR   ref    V00 this         
[000040] -----+------                 \--*  CNS_INT   int    1

You might imagine something would clean this up, but nothing does.

Related: see workarounds to avoid creating this kind of IR in optVNConstantPropOnJTrue.

[AndyAyersMS]

@EgorBo worked up some ideas on how to fix this in fgFoldConditional -- see here.

[EgorBo]

Ok, working on test cases