[BUG] CMake's find_library returns incorrect libdl

[neobrain]

Description

In CMake projects, find_library picks up the wrong libdl.a: It returns ndk/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/lib/arm-linux-androideabi/libdl.a instead of ndk/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/lib/arm-linux-androideabi/24/libdl.so. The former is just a stub implementation (dlopen() always returns NULL), so it seems preferable to use the latter.

Notably, this breaks SDL2 for me, since it relies on dlopen to do anything interesting on Android.

Example CMakeLists.txt:

cmake_minimum_required(VERSION 3.13)
project(dlopen_test)
find_library(DLPATH dl)

# cmake -DCMAKE_TOOLCHAIN_FILE=$ANDROID_NDK_HOME/build/cmake/android.toolchain.cmake -DANDROID_PLATFORM_LEVEL=24 -DANDROID_PLATFORM=android-24 && cat CMakeCache.txt  | grep DLPATH

I found that reverting the change to CMAKE_SYSTEM_LIBRARY_PATH in 12d79d64f29f5f392183a1d1f763dbdfcebe7919 (i.e. prefixing that variable with the sysroot path again) resolves the issue for me, but I'm not sure this is a proper fix.

Environment Details

• NDK Version: 19.1.5304403 (also tried copying android.toolchain.cmake from git over my NDK)
• Build system: cmake
• Host OS: Linux, openSUSE Tumbleweed
• ABI: Any (e.g. arm64-v8a)
• NDK API level: Any (e.g. 24)

[alexcohn]

AFAIK, there are no stub static libraries in NDK. For me, android-ndk-r19b/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/lib/arm-linux-androideabi/24/libdl.a and android-ndk-r19b/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/lib/arm-linux-androideabi/libdl.a are binary the same.

Normally, CMake will use the shared libdl.so to link your project, and does not rely on find_library(… dl). And yes, NDK ships with a stub for libdl.so, as well as other system libs. This should not cause trouble for you, because this stub should not be part of your deliverables. The real libdl is waiting for your app in /system/lib/.

[neobrain]

@alexcohn Hey, thanks for the quick reply (and sorry I couldn't reply any earlier!)

As you're saying, CMake usually picks up libdl automatically without using find_library. SDL2 itself, however, uses find_library to link against libdl (see https://hg.libsdl.org/SDL/file/3477a301a5a8/CMakeLists.txt#l884). It's this find_library call that returns …/arm-linux-androideabi/libdl.a rather than the shared library …/arm-linux-androideabi/24/libdl.so. (Note that indeed you're right and the two libdl.a files are identical - this was a typo in my top post. The second path should've referred to the shared lib.)

Does this clarify the issue?

[neobrain]

I ran CMake through a debugger and set a breakpoint in the find_library handler. Apparently the list of paths it searches in is:

(gdb) p SearchPaths
$8 = std::vector of length 14, capacity 16 = {
  "/opt/android/sdk/ndk-bundle/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/local/",
  "/opt/android/sdk/ndk-bundle/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/lib/aarch64-linux-android/",
  "/opt/android/sdk/ndk-bundle/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/lib/",
  "/opt/android/sdk/ndk-bundle/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/",
  "/opt/android/sdk/ndk-bundle/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/lib/aarch64-linux-android/24/",
  "/usr/local/lib64/", "/usr/local/lib/", "/usr/local/", "/usr/lib64/", "/usr/lib/", "/usr/", "/lib64/", "/lib/", "/opt/"
}

Notice the path that contains the desired libdl.so (.../aarch64-linux-android/24) is at the back at the list, preceded by .../aarch64-linux-android/ (which contains libdl.a).

I haven't been able to figure out where specifically those paths come from, but maybe that gives us some clues as to what's wrong?

[alexcohn]

If you must use find_library() for libdl, one solution would be to use

find_library(DLPATH NAMES libdl.so dl)

This way, CMake will first try to find the shared library, and fall back to whatever matches the target platform if libdl.so cannot be found.

[neobrain]

Cool, I'll make sure to see if that adjusted find_library call will do too.

As for the static library, are you sure libdl.a is not a stub of sorts, though? Running objdump -D libdl.a, this is what I get:

0000000000000000 <dlopen>:
   0:   aa1f03e0        mov     x0, xzr
   4:   d65f03c0        ret

Am I missing some hidden purpose of this placeholder implementation?

[enh]

no, that's expected: Android has never supported the dl* functions from static binaries. they just fail at runtime. you never want libdl.a unless you're trying to build something where you know you don't actually need the dl* functionality but aren't able to actually cut it out of the source.

[alexcohn]

Android has never supported the dl* functions from static binaries. they just fail at runtime.
🤦‍♂️

[neobrain]

@enh Hah, interesting. Thanks for shedding some light on that :)

I can now confirm that adding NAMES libdl.so to the find_library call works. I'll check with the SDL developers to see if this change can be upstreamed.

This works on my end - feel free to close this issue at your discretion. From what you're saying it seems this is a necessary evil that we can't really do anything about in the NDK.

Thanks for your help!

[DanAlbert]

We'll leave it open. That workaround shouldn't be necessary.

[Zingam]

If the workaround should not be necessary what would be the proper solution?

[DanAlbert]

I'm not sure yet because I haven't had a chance to look deeper. I suspect this is a bug that we'll need to fix in the NDK or CMake. The workaround is what you'll need to rely on for now, but I want to leave the bug open so we can fix it properly later.

[DanAlbert]

It seems the issue is that CMake searches CMAKE_SYSROOT first and CMAKE_SYSTEM_LIBRARY_PATH after. CMake doesn't natively understand the NDK sysroot layout (or perhaps it does now, but has the inverted preference for API-generic as opposed to API-specific libraries).

I tried a handful of tweaks and the only way I could get it to prefer the shared library was to stop setting CMAKE_SYSROOT. That might have other search order effects, and if we remove that we'll break compatibility with some versions of Studio (they identify Android targets based on --sysroot pointing to an NDK, rather than based on the Clang target name like they do now). I'll have to dig up when we made that change in AGP to see if it's reasonable to break compatibility. Triaging to r22 to figure that out.

If that doesn't work out, this will need to be fixed in CMake proper. We should teach it about the NDK sysroot layout anyway, but that'll happen naturally as part of #463.