[BUG] ld.lld: error: Invalid record (Producer: 'LLVM12.0.5git' Reader: 'LLVM 12.0.5git')

[DoDoENT]

Description

Happens when LTO is enabled for armeabi-v7a ABI with NDK r23 - the same code works correctly with NDK r22b.

I still haven't been able to create a minimum reproducible sample (I'll update this issue if I manage to do so), but @lmglmg and I observed the same issue also with armv7 slice when building the same code for iOS using Xcode 13 (it also uses the LLVM 12-based backend).

I'm posting this here in case if someone has already seen this issue and found a workaround.

Environment Details

• NDK Version: r23
• Build system: cmake
• Host OS: MacOS
• ABI: armeabi-v7a
• NDK API level: 16
• Device API level: 16

[DoDoENT]

It appears that this has occasionally happened even with earlier versions of LLVM, but not on our codebase...

[DoDoENT]

On iOS we get Invalid record (Producer: 'APPLE_1_1300.0.29.3_0' Reader: 'LLVM APPLE_1_1300.0.29.3_0') for architecture armv7.

My current hunch is that it's somehow related to combining -fenable-matrix and -flto, as all our other projects that don't use clang built-in matrices work correctly. Disabling -flto makes the build work, however, disabling -fenable-matrix is not so simple in our codebase (we are trying to see this, also making min reproducible sample).

Note that arm64 works correctly.

[DanAlbert]

I haven't heard of that error before. Maybe @pirama-arumuga-nainar has?

Given that it's a toolchain crash that was introduced in r23, it'd also be worth trying your code with the latest canary once we actually move it to a new toolchain. Leaving this open as a reminder to notify once we've done that (otherwise we'll close until you can share a repro case, but that's just for tracking reasons; it doesn't mean we won't fix it).

[pirama-arumuga-nainar]

This can happens when LLVM IR (produced during LTO) from newer clang is passed to older linker. If this may be the case, I'd suggest to bisect/reduce the set of files having LTO to find a problematic library.

Beyond this suggestion, would need a repro to investigate.

[DoDoENT]

This can happens when LLVM IR (produced during LTO) from newer clang is passed to older linker. If this may be the case, I'd suggest to bisect/reduce the set of files having LTO to find a problematic library.

Wouldn't, in that case, be a different error, i.e. different versions of producer and consumer? Anyway, in our case, we build all our code from the source using the same compiler, so this is definitely not the cause for us.

Yesterday, I've narrowed it down to two source files that use matrix multiplications assisted by clangs native matrix support (-fenable-matrix) - after I remove those two files from the project, the linker succeeds. However, if I simply set -fno-lto on those two files, but keep them compiling, I get the same linker error 🤷‍♂️. Only by completely disabling the LTO on this project (but not on the libraries that it gets linked to), the linker succeeds.

Unfortunately, as I said earlier, we have this same problem also with Xcode 13, but disabling LTO doesn't help there - it just turns the linker error into the compiler error on one single source file (getting Invalid record (Producer: 'APPLE_1_1300.0.29.3_0' Reader: 'LLVM APPLE_1_1300.0.29.3_0') for architecture armv7 as compile error has never happened to us before, however, by analyzing this source file we may find out what triggers the bug and, hopefully, make a minimum repro case - fortunately, this time both NDK r23 and Apple used same/similar commit from upstream LLVM with slightly different behaviour, which may or may not help us in tracking down this bug). Stay tuned!

[psiha]

The problem is with __builtin_matrix_column_major_load and __builtin_matrix_column_major_store when used in template contexts - the errors vary from ICEs (in debug builds with NDK r22) to the above error in release build with r23. We're unable to produce a minimal repro as of yet...

[DoDoENT]

It also happens for x86. So, it appears that it may be a 32-bit-specific issue, not armv7-specific.

[DanAlbert]

I don't think the presubmit builds are downloadable by non googlers, but https://android-review.googlesource.com/c/platform/ndk/+/1839724 updates to a newer LLVM. I've only lightly tested it but all our tests build fine and run correctly on KitKat (just what I have on hand atm; it's going to be a week or so before I have all my usual devices available and I didn't want to delay getting you a build to test).

Shortly after that's submitted a build will start on https://ci.android.com/builds/branches/aosp-master-ndk/grid that you can try. If that fixes the problem then you can save yourself the effort of reducing a repro case. If it doesn't, we're probably picking up one more compiler update before we ship r24, so we'll want to try that too.

(bonus points: it sounds like if we do confirm that the new build fixes the problem you'll probably get the iOS fix in the next version of xcode too)

[DoDoENT]

I don't think the presubmit builds are downloadable by non googlers, but https://android-review.googlesource.com/c/platform/ndk/+/1839724 updates to a newer LLVM.

If you can smuggle a macOS (or even Linux) build to me, I could test it with our codebase and give you feedback. Also, emscripten 2.0.27 no longer has ICE with that code (it's based on LLVM 14), so this may be a good indication that the newer LLVM no longer has the problem. I've tested that yesterday when I was applying @psiha's workaround in our codebase - I've had to check all combinations (NDK r23 debug/release all ABIs (this is how I discovered that problem is also present for x86) and Emscripten 2.0.27 debug/release (we still haven't updated to later version of emscripten, but from their changelog, they haven't touched the compiler since 2.0.27)).

[DanAlbert]

If you can smuggle a macOS (or even Linux) build to me, I could test it with our codebase and give you feedback

That'd be paragraph 2 :) It'll have builds for macos/linux/windows after it's submitted: https://ci.android.com/builds/branches/aosp-master-ndk/grid (all our canary builds, if you haven't seen that before: https://android.googlesource.com/platform/ndk/+/master/docs/ContinuousBuilds.md)

emscripten 2.0.27 no longer has ICE with that code (it's based on LLVM 14), so this may be a good indication that the newer LLVM no longer has the problem

The build I just picked up is LLVM 13, but the build we'll actually ship for r24 will be LLVM 14 I believe. That does sound promising. I think we're only a few weeks away from having the LLVM 14 based toolchain in the canary build (but @stephenhines or @pirama-arumuga-nainar will know better).

[DanAlbert]

https://ci.android.com/builds/branches/aosp-master-ndk/grid?head=7778403&tail=7778403 is the build that has the newer LLVM. lmk if it fixes the problem. If not, like I said, we have another update coming in the near future you can also try.

[DoDoENT]

Unfortunately, it also happens with the canary r24 build. Except that now the error is ld.lld: error: Invalid record (Producer: 'LLVM13.0.2git' Reader: 'LLVM 13.0.2git')

Let me know when you have the build based on LLVM 14 so I can try again.

[DanAlbert]

Will do. Thanks for checking.