Execute x64 CoreCLR tests dlls on arm32

[gbalykov]

Previously, CoreCLR tests dlls, built for x64, could be executed on arm32, but now there is an error:

Unhandled exception. System.BadImageFormatException: Could not load file or assembly '/home/owner/BitTest.dll'. An attempt was made to load a program with an incorrect format

Could you, please, describe what has changed?

cc @alpencolt

[jkotas]

cc @trylek ?

[trylek]

@gbalykov - can you please clarify whether you're using AOT (Crossgen-compiled) or MSIL code when running the tests? I believe that, when Crossgen compiles MSIL, it checks the architecture word in the PE header and refuses to compile in case of mismatch i.e. it shouldn't let you compile MSIL claiming to be targeting x64 to native code targeting arm32. AFAIK CoreCLR tests are run via corerun.exe - I'm not sure if it's internally checking the architecture word in the PE header but at the very least I'm not aware of any recent changes to the tool. Also, are you observing this in a local or lab run? If it's a lab run, could you please share a link? We're currently tracking several issues blocking arm runs in the consolidated runtime repo.

[gbalykov]

Simpler example is to try to launch x86 tests dlls on x64 runtime, result is the same as for x86/x64 tests dlls on armel.

z@z:~/Dev/runtime/artifacts/bin/runtime/netcoreapp5.0-Linux-Release-x64$ cp ../../../tests/coreclr/Linux.x64.Release/JIT/Directed/BitTest/BitTest/BitTest.dll .
z@z:~/Dev/runtime/artifacts/bin/runtime/netcoreapp5.0-Linux-Release-x64$ ./corerun BitTest.dll 
PASSED
z@dagon:~/Dev/runtime/artifacts/bin/runtime/netcoreapp5.0-Linux-Release-x64$ cp ../../../tests/coreclr/Linux.x86.Release/JIT/Directed/BitTest/BitTest/BitTest.dll .
z@dagon:~/Dev/runtime/artifacts/bin/runtime/netcoreapp5.0-Linux-Release-x64$ ./corerun BitTest.dll 
Unhandled exception. System.BadImageFormatException: Could not load file or assembly '/home/z/Dev/runtime/artifacts/bin/runtime/netcoreapp5.0-Linux-Release-x64/BitTest.dll'. An attempt was made to load a program with an incorrect format.

File name: '/home/z/Dev/runtime/artifacts/bin/runtime/netcoreapp5.0-Linux-Release-x64/BitTest.dll'
Aborted (core dumped)

I've built runtime using next commands on Ubuntu 18.04:

./src/coreclr/build.sh -x64 -Release -clang9 -skipnuget -msbuildonunsupportedplatform
./libraries.sh --arch x64 --configuration Release --clang9
./build-test.sh release x64 clang9 priority1 -msbuildonunsupportedplatform

./src/coreclr/build.sh cross -x86 -Release -clang9 -skipnuget -msbuildonunsupportedplatform
./libraries.sh --arch x86 --configuration Release --cross --clang9
./build-test.sh release x86 clang9 priority1 -msbuildonunsupportedplatform

After running ildasm on these two dlls diff is only in the next lines:

1. x64

.module BitTest.dll
// MVID: {fdff248d-eecd-447a-934b-558cb16486bb}
.imagebase 0x0000000140000000
.file alignment 0x00000200
.stackreserve 0x0000000000400000
.subsystem 0x0003       // WINDOWS_CUI
.corflags 0x00000001    //  ILONLY
// Image base: 0x00007F4C9702A000

2. x86

.module BitTest.dll
// MVID: {3e7b7547-2dfd-41a9-9653-a55fdb18c236}
.imagebase 0x00400000
.file alignment 0x00000200
.stackreserve 0x00100000
.subsystem 0x0003       // WINDOWS_CUI
.corflags 0x00000003    //  ILONLY 32BITREQUIRED
// Image base: 0x00007F1B594DA000

On previous versions we were able to execute tests, which were built for x86, on armel and vice versa.

For example, I can successfully launch CoreCLR 3.0 armel test dll (BitTest.exe) on the latest x64 dotnet/runtime. ildasm shows diff of that old armel built BitTest.exe with newest x64 built BitTest.dll only in these lines:

3. armel

.module BitTest.exe
// MVID: {4d309ffd-657b-4296-b387-64943be74b44}
.imagebase 0x00400000
.file alignment 0x00000200
.stackreserve 0x00100000
.subsystem 0x0003       // WINDOWS_CUI
.corflags 0x00000001    //  ILONLY
// Image base: 0x00007F5DC4BC2000

So I wonder, what has changed.

[BruceForstall]

It looks like the test build process on x86 now sets 32BITREQUIRED. Maybe it didn't before?

cc @jkoritzinsky

[jkoritzinsky]

It looks like the armel example is from before we switched to using SDK-style projects. The switch to using the SDK may have changed how some things were set. I’ll take a look.

[gbalykov]

@jkoritzinsky Do you have any update on this?

[gbalykov]

I've checked the difference between old and new dlls.

If I try to launch crossgen2.dll built for arm on x64, I get to the next lines in assembly.cpp and ERROR_BAD_FORMAT is thrown:

if (!IsValidArchitecture(kAssemblyArchitecture))
{
    // Assembly image can't be executed on this platform
    IF_FAIL_GO(HRESULT_FROM_WIN32(ERROR_BAD_FORMAT));
}

Here kAssemblyArchitecture is peARM, which is not equal to host arch peAMD64 or peMSIL, so the error is thrown.

If I try to launch old tests dll (CLR 3.0) built for armel on x64, I get to the same lines with kAssemblyArchitecture = peMSIL and error is not thrown.

This difference between dlls comes from FindNTHeaders()->FileHeader.Machine, which returns IMAGE_FILE_MACHINE_ARMNT now instead of IMAGE_FILE_MACHINE_I386 (see PEDecoder::GetPEKindAndMachine).

Both dlls are peILonly, so IMAGE_FILE_MACHINE_ARMNT seems to be incorrect and sdk should generate dlls with IMAGE_FILE_MACHINE_I386. Am I right? Where should this be changed in sdk?

[gbalykov]

After #25761 Platform is set equal to BuildArch instead of AnyCPU, which makes dll become platform specific, as I mentioned above.

runtime/src/coreclr/tests/dir.sdkbuild.props

Line 12 in b52c779

<Platform>$(BuildArch)</Platform>

Why is this change required?

[jkotas]

It should not be required, for majority of test projects at least.

[gbalykov]

It should not be required, for majority of test projects at least.

Is the same true for crossgen2? I.e. can it be built for one arch and used on another?

[jkotas]

The managed binaries of crossgen2 driver are platform neutral. The unmanaged JIT binaries loaded by crossgen2 are not. So that makes crossgen2 to be platform specific as a whole.

[gbalykov]

I was wondering about managed part, so that's great to hear, thank you. There is the same problem currently with incorrect format of crossgen2.dll, I'm investigating.