[BUG] Broken crash stack traces with NDK r22 and SDK 29

[pharscoet]

Description

I am seeing a strange issue where after upgrading code to use NDK r22, the crash stack traces in the log are bogus or truncated.
And it seems to happen when using a min SDK version of 29 and not with earlier versions (I haven't tried 30). NDK r21 with min SDK 29 does not have the issue.

I am able to reproduce the issue with the native-activity app from the NDK samples.
That sample app has ndkVersion set to 22.1.7171670.
I added a packagingOptions.doNotStrip '**' to the build.gradle file so the symbols are not stripped and an assert(false); in main.cpp:engine_draw_frame() to cause a crash.

When run, the assert produces the stack trace:

09-25 00:02:43.208 13587 13587 F DEBUG   : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
09-25 00:02:43.208 13587 13587 F DEBUG   : Build fingerprint: 'samsung/x1qsqx/x1q:10/QP1A.190711.020/G981USQE1ATG1:userdebug/test-keys'
09-25 00:02:43.208 13587 13587 F DEBUG   : Revision: '13'
09-25 00:02:43.208 13587 13587 F DEBUG   : ABI: 'arm64'
09-25 00:02:43.208 13587 13587 F DEBUG   : Timestamp: 2021-09-25 00:02:43+0000
09-25 00:02:43.208 13587 13587 F DEBUG   : pid: 13525, tid: 13580, name: Thread-8  >>> com.example.native_activity <<<
09-25 00:02:43.208 13587 13587 F DEBUG   : uid: 10586
09-25 00:02:43.208 13587 13587 F DEBUG   : signal 6 (SIGABRT), code -1 (SI_QUEUE), fault addr --------
09-25 00:02:43.208 13587 13587 F DEBUG   : Abort message: '/home/p.harscoet/src/external/ndk-samples/native-activity/app/src/main/cpp/main.cpp:171: void engine_draw_frame(struct engine *): assertion "false" failed'
09-25 00:02:43.208 13587 13587 F DEBUG   :     x0  0000000000000000  x1  000000000000350c  x2  0000000000000006  x3  00000074ea51ba10
09-25 00:02:43.208 13587 13587 F DEBUG   :     x4  0000800000000000  x5  0000800000000000  x6  0000800000000000  x7  0000000000000018
09-25 00:02:43.208 13587 13587 F DEBUG   :     x8  00000000000000f0  x9  fc919477fc66a6df  x10 0000000000000001  x11 0000000000000000
09-25 00:02:43.208 13587 13587 F DEBUG   :     x12 fffffff0fffffbdf  x13 00000000614e6723  x14 000a36f96e853e42  x15 0000343a9094ffc3
09-25 00:02:43.208 13587 13587 F DEBUG   :     x16 00000075840d28c0  x17 00000075840aefe0  x18 0000007490ec2000  x19 00000000000034d5
09-25 00:02:43.208 13587 13587 F DEBUG   :     x20 000000000000350c  x21 00000000ffffffff  x22 00000074ea51c060  x23 00000074ea51bdd8
09-25 00:02:43.208 13587 13587 F DEBUG   :     x24 00000074ea51bd50  x25 00000074ea51bd50  x26 00000074ea51c020  x27 0000007585a2a020
09-25 00:02:43.208 13587 13587 F DEBUG   :     x28 0000007fdb86c8c0  x29 00000074ea51bab0
09-25 00:02:43.208 13587 13587 F DEBUG   :     sp  00000074ea51b9f0  lr  000000758406027c  pc  00000075840602a8
09-25 00:02:43.208 13587 13587 F DEBUG   : 
09-25 00:02:43.208 13587 13587 F DEBUG   : backtrace:
09-25 00:02:43.209 13587 13587 F DEBUG   :       #00 pc 00000000000832a8  /apex/com.android.runtime/lib64/bionic/libc.so (abort+160) (BuildId: c2f522895fa97ff8c517cefefec04171)
09-25 00:02:43.209 13587 13587 F DEBUG   :       #01 pc 00000000000839e8  /apex/com.android.runtime/lib64/bionic/libc.so (__assert2+36) (BuildId: c2f522895fa97ff8c517cefefec04171)
09-25 00:02:43.209 13587 13587 F DEBUG   :       #02 pc 0000000000016ee4  /data/app/com.example.native_activity-m-19VjUkyzX_CJh6Nm3jxw==/base.apk!libnative-activity.so (offset 0x2c000) (engine_draw_frame(engine*)+68) (BuildId: 7c55057c211ad669f0056966fb673de79aa68fd8)
09-25 00:02:43.209 13587 13587 F DEBUG   :       #03 pc 0000000000016cc4  /data/app/com.example.native_activity-m-19VjUkyzX_CJh6Nm3jxw==/base.apk!libnative-activity.so (offset 0x2c000) (engine_handle_cmd(android_app*, int)+188) (BuildId: 7c55057c211ad669f0056966fb673de79aa68fd8)
09-25 00:02:43.209 13587 13587 F DEBUG   :       #04 pc 0000000000018ddc  /data/app/com.example.native_activity-m-19VjUkyzX_CJh6Nm3jxw==/base.apk!libnative-activity.so (offset 0x2c000) (process_cmd+72) (BuildId: 7c55057c211ad669f0056966fb673de79aa68fd8)
09-25 00:02:43.209 13587 13587 F DEBUG   :       #05 pc 0000000000016ae8  /data/app/com.example.native_activity-m-19VjUkyzX_CJh6Nm3jxw==/base.apk!libnative-activity.so (offset 0x2c000) (android_main+376) (BuildId: 7c55057c211ad669f0056966fb673de79aa68fd8)
09-25 00:02:43.209 13587 13587 F DEBUG   :       #06 pc 0000000000018d78  /data/app/com.example.native_activity-m-19VjUkyzX_CJh6Nm3jxw==/base.apk!libnative-activity.so (offset 0x2c000) (android_app_entry+288) (BuildId: 7c55057c211ad669f0056966fb673de79aa68fd8)
09-25 00:02:43.209 13587 13587 F DEBUG   :       #07 pc 00000000000e6f10  /apex/com.android.runtime/lib64/bionic/libc.so (__pthread_start(void*)+36) (BuildId: c2f522895fa97ff8c517cefefec04171)
09-25 00:02:43.209 13587 13587 F DEBUG   :       #08 pc 00000000000850c8  /apex/com.android.runtime/lib64/bionic/libc.so (__start_thread+64) (BuildId: c2f522895fa97ff8c517cefefec04171)

which is correct.

Now, if I change the minSdkVersion to 29 in build.gradle and I run it again, the new stack trace becomes:

09-25 00:03:43.940 14006 14006 F DEBUG   : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
09-25 00:03:43.940 14006 14006 F DEBUG   : Build fingerprint: 'samsung/x1qsqx/x1q:10/QP1A.190711.020/G981USQE1ATG1:userdebug/test-keys'
09-25 00:03:43.940 14006 14006 F DEBUG   : Revision: '13'
09-25 00:03:43.940 14006 14006 F DEBUG   : ABI: 'arm64'
09-25 00:03:43.941 14006 14006 F DEBUG   : Timestamp: 2021-09-25 00:03:43+0000
09-25 00:03:43.941 14006 14006 F DEBUG   : pid: 13955, tid: 13997, name: Thread-8  >>> com.example.native_activity <<<
09-25 00:03:43.941 14006 14006 F DEBUG   : uid: 10586
09-25 00:03:43.941 14006 14006 F DEBUG   : signal 6 (SIGABRT), code -1 (SI_QUEUE), fault addr --------
09-25 00:03:43.941 14006 14006 F DEBUG   : Abort message: '/home/p.harscoet/src/external/ndk-samples/native-activity/app/src/main/cpp/main.cpp:171: void engine_draw_frame(struct engine *): assertion "false" failed'
09-25 00:03:43.941 14006 14006 F DEBUG   :     x0  0000000000000000  x1  00000000000036ad  x2  0000000000000006  x3  00000074ea51ba10
09-25 00:03:43.941 14006 14006 F DEBUG   :     x4  0000800000000000  x5  0000800000000000  x6  0000800000000000  x7  0000000000000018
09-25 00:03:43.941 14006 14006 F DEBUG   :     x8  00000000000000f0  x9  fc919477fc66a6df  x10 0000000000000001  x11 0000000000000000
09-25 00:03:43.941 14006 14006 F DEBUG   :     x12 fffffff0fffffbdf  x13 00000000614e675f  x14 00359b5f575ac642  x15 000055dea014c8a5
09-25 00:03:43.941 14006 14006 F DEBUG   :     x16 00000075840d28c0  x17 00000075840aefe0  x18 0000007490a00000  x19 0000000000003683
09-25 00:03:43.941 14006 14006 F DEBUG   :     x20 00000000000036ad  x21 00000000ffffffff  x22 00000074ea51c060  x23 00000074ea51bdd8
09-25 00:03:43.941 14006 14006 F DEBUG   :     x24 00000074ea51bd50  x25 00000074ea51bd50  x26 00000074ea51c020  x27 0000007585a2a020
09-25 00:03:43.941 14006 14006 F DEBUG   :     x28 0000007fdb86c8c0  x29 00000074ea51bab0
09-25 00:03:43.941 14006 14006 F DEBUG   :     sp  00000074ea51b9f0  lr  000000758406027c  pc  00000075840602a8
09-25 00:03:43.941 14006 14006 F DEBUG   : 
09-25 00:03:43.941 14006 14006 F DEBUG   : backtrace:
09-25 00:03:43.941 14006 14006 F DEBUG   :       #00 pc 00000000000832a8  /apex/com.android.runtime/lib64/bionic/libc.so (abort+160) (BuildId: c2f522895fa97ff8c517cefefec04171)
09-25 00:03:43.941 14006 14006 F DEBUG   :       #01 pc 00000000000839e8  /apex/com.android.runtime/lib64/bionic/libc.so (__assert2+36) (BuildId: c2f522895fa97ff8c517cefefec04171)
09-25 00:03:43.942 14006 14006 F DEBUG   :       #02 pc 0000000000016f1c  /data/app/com.example.native_activity-qD7yK_my8queF7hezbbAfA==/base.apk!libnative-activity.so (offset 0x2c000) (BuildId: 5c54360fd3b54ce346c314db3122af123693b9c8)

it is now truncated and not useful. With other code I have seen the trace containing symbols completely unrelated to the crash.

If leaving minSdkVersion at 29 and changing ndkVersion to 21.4.7075529, the correct stack trace is produced.

Environment Details

Not all of these will be relevant to every bug, but please provide as much
information as you can.

• NDK Version: 22.1.7171670
• Build system: CMake
• Host OS: Linux (Ubuntu 18.04)
• ABI: arm64
• NDK API level: 29
• Device API level: 29

[DanAlbert]

r22 is no longer supported. Do you see the same issue with NDK r23?

[pharscoet]

r22 is no longer supported. Do you see the same issue with NDK r23?

Yes, having the same issue with r23.

[DanAlbert]

Thanks for confirming. Not a regression from r22 but is from the previous LTS so our rules here are a little hazy. I'd be in favor of a cherry-pick to r23, but it depends on the scope of the fix. For now triaging to r24.

[pharscoet]

Thanks for confirming. Not a regression from r22 but is from the previous LTS so our rules here are a little hazy. I'd be in favor of a cherry-pick to r23, but it depends on the scope of the fix. For now triaging to r24.

Ok thanks. I noticed the issue on a Galaxy S20 with Android 10 and I just tried an S21 with Android 11 and I'm not seeing the problem there so it could depend on the target device's version too.

[DanAlbert]

That reminds me of #1196 (comment), but I don't think it's actually related (that issue was fixed in r22 after all).

[pharscoet]

That reminds me of #1196 (comment), but I don't think it's actually related (that issue was fixed in r22 after all).

It does sound like a similar type of issue.

[pharscoet]

That reminds me of #1196 (comment), but I don't think it's actually related (that issue was fixed in r22 after all).

Adding -Wl,--no-rosegment with r23 as mentioned in that comment fixes this issue. It's also happening on a somewhat older device (S10 with Android 10) and not on a more recent one and your explanation in that other comment about the unwinder in older devices now being about to understand the output from LLD could explain it too.

[DanAlbert]

Huh. I saw earlier that we do have a test for this, but maybe the test was only for ndk-build? I'll have to look again tomorrow. Thanks for confirming. Really didn't expect that to be the answer :(

You're just using whatever version of CMake and AGO are the defaults for that sample, right? Only changed the NDK version?

[pharscoet]

You're just using whatever version of CMake and AGO are the defaults for that sample, right? Only changed the NDK version?

The sample uses NDK r22 by default and I only need to change the min SDK version to 29 to see the issue. Changing the NDK version to r21 does not show the issue even with the min SDK version still at 29. I didn't change the CMake version or any other version.

[DanAlbert]

Hmm. It sounds like we got the API level for the min version off by one then, but I checked and https://android-review.googlesource.com/c/platform/system/core/+/777288/ was in Q (29). I wonder if Samsung for some reason reverted that patch (I can't imagine why they would, so that seems really unlikely to me), or if that was only a partial fix and we just need to raise that to 30.

[pharscoet]

Hmm. It sounds like we got the API level for the min version off by one then, but I checked and https://android-review.googlesource.com/c/platform/system/core/+/777288/ was in Q (29). I wonder if Samsung for some reason reverted that patch (I can't imagine why they would, so that seems really unlikely to me), or if that was only a partial fix and we just need to raise that to 30.

I was able to reproduce the same issue with the sample native-activity on a Pixel 3a with Android 10 so it does not appear to be something Samsung changed.

[DanAlbert]

Excellent, thanks for checking that (was on my list but you beat me to it). Sounds like all we need to do then is just change the < 29 to < 30 then and the bug is fixed. We can include that in r24 for sure, and probably r23c as well.

[pharscoet]

Excellent, thanks for checking that (was on my list but you beat me to it). Sounds like all we need to do then is just change the < 29 to < 30 then and the bug is fixed. We can include that in r24 for sure, and probably r23c as well.

Hopefully it's as simple as that but I don't know enough about the issue to confirm. Thanks for looking into it :-)