<unknown>:0: error: __ia32_compat_sys_sysctl changed binding to STB_GLOBAL

[nathanchance]

While building kernel/common's android-4.19-stable branch (ARCH=x86_64 gki_defconfig):

$ make -skj"$(nproc)" LLVM=1 gki_defconfig bzImage
...
<unknown>:0: error: __ia32_compat_sys_sysctl changed binding to STB_GLOBAL
LLVM ERROR: Error parsing inline asm

PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace.
Stack dump:
0.      Program arguments: ld.lld -m elf_x86_64 -z max-page-size=0x200000 -plugin-opt=-code-model=kernel -plugin-opt=-stack-alignment=8 -mllvm -import-instr-limit=5 -r -o vmlinux.o -T .tmp_lto.lds --whole-archive built-in.a --no-whole-archive --start-group lib/lib.a arch/x86/lib/lib.a --end-group
1.      Running pass 'Function Pass Manager' on module 'ld-temp.o'.
2.      Running pass 'X86 Assembly Printer' on function '@__ia32_compat_sys_sysctl'
 #0 0x000000000388e6ac llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/home/nathan/cbl/toolchains/llvm-20210126-1943-145898740764ff2ff90f7b74e6bb0bb88422afa1/bin/lld+0x388e6ac)
 #1 0x0000000003768ad5 (/home/nathan/cbl/toolchains/llvm-20210126-1943-145898740764ff2ff90f7b74e6bb0bb88422afa1/bin/lld+0x3768ad5)
 #2 0x00007f0a78b5f3c0 __restore_rt (/lib/x86_64-linux-gnu/libpthread.so.0+0x153c0)
 #3 0x00007f0a7862418b raise (/lib/x86_64-linux-gnu/libc.so.6+0x4618b)
 #4 0x00007f0a78603859 abort (/lib/x86_64-linux-gnu/libc.so.6+0x25859)
 #5 0x0000000002c12b52 (/home/nathan/cbl/toolchains/llvm-20210126-1943-145898740764ff2ff90f7b74e6bb0bb88422afa1/bin/lld+0x2c12b52)
 #6 0x00000000025dafb5 (/home/nathan/cbl/toolchains/llvm-20210126-1943-145898740764ff2ff90f7b74e6bb0bb88422afa1/bin/lld+0x25dafb5)
 #7 0x0000000002c12734 llvm::AsmPrinter::emitInlineAsm(llvm::StringRef, llvm::MCSubtargetInfo const&, llvm::MCTargetOptions const&, llvm::MDNode const*, llvm::InlineAsm::AsmDialect) const (/home/nathan/cbl/toolchains/llvm-20210126-1943-145898740764ff2ff90f7b74e6bb0bb88422afa1/bin/lld+0x2c12734)
 #8 0x0000000002c1119f llvm::AsmPrinter::emitInlineAsm(llvm::MachineInstr const*) const (/home/nathan/cbl/toolchains/llvm-20210126-1943-145898740764ff2ff90f7b74e6bb0bb88422afa1/bin/lld+0x2c1119f)
 #9 0x0000000002c0858a llvm::AsmPrinter::emitFunctionBody() (/home/nathan/cbl/toolchains/llvm-20210126-1943-145898740764ff2ff90f7b74e6bb0bb88422afa1/bin/lld+0x2c0858a)
#10 0x0000000003d36b20 (/home/nathan/cbl/toolchains/llvm-20210126-1943-145898740764ff2ff90f7b74e6bb0bb88422afa1/bin/lld+0x3d36b20)
#11 0x00000000024d0bc7 llvm::MachineFunctionPass::runOnFunction(llvm::Function&) (/home/nathan/cbl/toolchains/llvm-20210126-1943-145898740764ff2ff90f7b74e6bb0bb88422afa1/bin/lld+0x24d0bc7)
#12 0x00000000024db96a llvm::FPPassManager::runOnFunction(llvm::Function&) (/home/nathan/cbl/toolchains/llvm-20210126-1943-145898740764ff2ff90f7b74e6bb0bb88422afa1/bin/lld+0x24db96a)
#13 0x00000000024db6cd llvm::FPPassManager::runOnModule(llvm::Module&) (/home/nathan/cbl/toolchains/llvm-20210126-1943-145898740764ff2ff90f7b74e6bb0bb88422afa1/bin/lld+0x24db6cd)
#14 0x00000000024dabe3 llvm::legacy::PassManagerImpl::run(llvm::Module&) (/home/nathan/cbl/toolchains/llvm-20210126-1943-145898740764ff2ff90f7b74e6bb0bb88422afa1/bin/lld+0x24dabe3)
#15 0x0000000004043545 (/home/nathan/cbl/toolchains/llvm-20210126-1943-145898740764ff2ff90f7b74e6bb0bb88422afa1/bin/lld+0x4043545)
#16 0x000000000402946d llvm::lto::backend(llvm::lto::Config const&, std::function<std::unique_ptr<llvm::lto::NativeObjectStream, std::default_delete<llvm::lto::NativeObjectStream> > (unsigned int)>, unsigned int, std::unique_ptr<llvm::Module, std::default_delete<llvm::Module> >, llvm::ModuleSummaryIndex&) (/home/nathan/cbl/toolchains/llvm-20210126-1943-145898740764ff2ff90f7b74e6bb0bb88422afa1/bin/lld+0x402946d)
#17 0x0000000004027acd llvm::lto::LTO::runRegularLTO(std::function<std::unique_ptr<llvm::lto::NativeObjectStream, std::default_delete<llvm::lto::NativeObjectStream> > (unsigned int)>) (/home/nathan/cbl/toolchains/llvm-20210126-1943-145898740764ff2ff90f7b74e6bb0bb88422afa1/bin/lld+0x4027acd)
#18 0x00000000040271d8 llvm::lto::LTO::run(std::function<std::unique_ptr<llvm::lto::NativeObjectStream, std::default_delete<llvm::lto::NativeObjectStream> > (unsigned int)>, std::function<std::function<std::unique_ptr<llvm::lto::NativeObjectStream, std::default_delete<llvm::lto::NativeObjectStream> > (unsigned int)> (unsigned int, llvm::StringRef)>) (/home/nathan/cbl/toolchains/llvm-20210126-1943-145898740764ff2ff90f7b74e6bb0bb88422afa1/bin/lld+0x40271d8)
#19 0x0000000004c28874 lld::elf::BitcodeCompiler::compile() (/home/nathan/cbl/toolchains/llvm-20210126-1943-145898740764ff2ff90f7b74e6bb0bb88422afa1/bin/lld+0x4c28874)
#20 0x0000000004bba6b1 void lld::elf::LinkerDriver::compileBitcodeFiles<llvm::object::ELFType<(llvm::support::endianness)1, true> >() (/home/nathan/cbl/toolchains/llvm-20210126-1943-145898740764ff2ff90f7b74e6bb0bb88422afa1/bin/lld+0x4bba6b1)
#21 0x0000000004ba9e99 void lld::elf::LinkerDriver::link<llvm::object::ELFType<(llvm::support::endianness)1, true> >(llvm::opt::InputArgList&) (/home/nathan/cbl/toolchains/llvm-20210126-1943-145898740764ff2ff90f7b74e6bb0bb88422afa1/bin/lld+0x4ba9e99)
#22 0x0000000004b9499f lld::elf::LinkerDriver::linkerMain(llvm::ArrayRef<char const*>) (/home/nathan/cbl/toolchains/llvm-20210126-1943-145898740764ff2ff90f7b74e6bb0bb88422afa1/bin/lld+0x4b9499f)
#23 0x0000000004b91673 lld::elf::link(llvm::ArrayRef<char const*>, bool, llvm::raw_ostream&, llvm::raw_ostream&) (/home/nathan/cbl/toolchains/llvm-20210126-1943-145898740764ff2ff90f7b74e6bb0bb88422afa1/bin/lld+0x4b91673)
#24 0x0000000004ab6e9b (/home/nathan/cbl/toolchains/llvm-20210126-1943-145898740764ff2ff90f7b74e6bb0bb88422afa1/bin/lld+0x4ab6e9b)
#25 0x0000000004ab643f main (/home/nathan/cbl/toolchains/llvm-20210126-1943-145898740764ff2ff90f7b74e6bb0bb88422afa1/bin/lld+0x4ab643f)
#26 0x00007f0a786050b3 __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x270b3)
#27 0x0000000004ab602e _start (/home/nathan/cbl/toolchains/llvm-20210126-1943-145898740764ff2ff90f7b74e6bb0bb88422afa1/bin/lld+0x4ab602e)
scripts/link-vmlinux.sh: line 101: 986749 Aborted                 ${LD} ${KBUILD_LDFLAGS} -r -o ${1} $(lto_lds) ${objects}
make[1]: *** [Makefile:1157: vmlinux] Error 134

This showed up during a personal CI run: https://github.com/nathanchance/continuous-integration2/runs/1778992951

I am testing right now to see if this is an LLVM 12 regression.

[nathanchance]

Bad: llvm/llvm-project@50bff64
Good: llvm/llvm-project@22d10b8

Bisecting now.

[nickdesaulniers]

reminiscent of ##1190?

[nathanchance]

Hmmm indeed, that is what the bisection converged on:

$ git bisect log
# bad: [50bff64158e9db3f91dd2d611a14707e8d173163] [SLP][Test] Add test for PR46983
# good: [22d10b8ab44f703b72b8316a9b3b8adc623ca73f] [CostModel] remove cost-kind predicate for vector reduction costs
git bisect start '50bff64158e9db3f91dd2d611a14707e8d173163' '22d10b8ab44f703b72b8316a9b3b8adc623ca73f'
# bad: [935ca5a1a7bec8d0f5e8aaf0c5589916b6dc48aa] [examples] Fix Kaleidoscope examples after OrcJIT break-up / remote TPC commit.
git bisect bad 935ca5a1a7bec8d0f5e8aaf0c5589916b6dc48aa
# bad: [6c337945c8954bc7aeb265ea5c70e455a951bd8a] [openmp][openacc][NFC] Simplify access and validation of DirectiveBase information
git bisect bad 6c337945c8954bc7aeb265ea5c70e455a951bd8a
# good: [f52c1b53310a1715b85b358a5c3de0e1f7c2e227] [VE] Add missing symbolic branch patterns
git bisect good f52c1b53310a1715b85b358a5c3de0e1f7c2e227
# bad: [b231396122f131ffaff5fc4ba36685a7df554aaa] [Sema] Diagnose annotating `if constexpr` with a likelihood attribute
git bisect bad b231396122f131ffaff5fc4ba36685a7df554aaa
# skip: [bbdbd020d2c2f315ed1545b23c23ec6ff1abc022] Address ABI issues introduced with CXCursor_CXXAddrspaceCastExpr
git bisect skip bbdbd020d2c2f315ed1545b23c23ec6ff1abc022
# skip: [ef22d500f7505d02a3861a9acf7eaacb375c7652] [NFCI][SCEV] getPtrToIntExpr(): use SCEVRewriteVisitor<> for ptrtoint cast sinking
git bisect skip ef22d500f7505d02a3861a9acf7eaacb375c7652
# bad: [b7e7131af2dd7bdb03fa42a3bc1b4bc72ab95ce1] [MC] Add SMLoc to MCStreamer::emitSymbolAttribute and report changed binding warnings/errors for ELF
git bisect bad b7e7131af2dd7bdb03fa42a3bc1b4bc72ab95ce1
# bad: [9bb9b737c5573cf3850230bc4db8dac7be0e1e85] Remove HAVE_VCS_VERSION_INC, not needed
git bisect bad 9bb9b737c5573cf3850230bc4db8dac7be0e1e85
# bad: [93ebefb9ee75143625d701250e69e701df4a7702] [X86] Add PR46393 test case
git bisect bad 93ebefb9ee75143625d701250e69e701df4a7702
# good: [7b8de9fc6bce01daed49d3f6a8bd71a25e8c717f] [ARM] Add IT block generation test
git bisect good 7b8de9fc6bce01daed49d3f6a8bd71a25e8c717f
# good: [81f7b96ed0a2295e0b82ca185019370ac8e1895e] [sanitizer] Print errno for report file open failure
git bisect good 81f7b96ed0a2295e0b82ca185019370ac8e1895e
# bad: [8f8b5e5587c3e204f21db975a0a76d4462ce3c57] [MC] Error for .globl/.local which change the symbol binding and warn for .weak
git bisect bad 8f8b5e5587c3e204f21db975a0a76d4462ce3c57
# good: [58de4b205310d18614eabdcbaa1772e9fc090df3] [AMDGPU] Use pseudo instructions for readlane/writelane
git bisect good 58de4b205310d18614eabdcbaa1772e9fc090df3
# first bad commit: [8f8b5e5587c3e204f21db975a0a76d4462ce3c57] [MC] Error for .globl/.local which change the symbol binding and warn for .weak

Guess there must be some assembler directives in the spaghetti that is __ia32_compat_sys_sysctl, which is generated from madness in arch/x86/include/asm/syscall_wrapper.h.

There could be something missing between 4.19 and 5.4 though.

[MaskRay]

This is due to .weak foo; ... .globl foo

Note: newer integrated assembler gives a location (instead of <unknown>:0) for the place .globl is used.

[nickdesaulniers]

There could be something missing between 4.19 and 5.4 though.

Probably something that changed how __ia32_compat_sys_sysctl was defined in assembler.

[nathanchance]

Note: newer integrated assembler gives a location (instead of <unknown>:0) for the place .globl is used.

How much newer? I see that exact error message at llvm/llvm-project@6f3d322 but this is during LTO so that could impact it I suppose.

Probably something that changed how __ia32_compat_sys_sysctl was defined in assembler.

Tracking that down is going to be rather painful but I will see if I can take a whack at it later this week.

[MaskRay]

Note: newer integrated assembler gives a location (instead of <unknown>:0) for the place .globl is used.

How much newer? I see that exact error message at llvm/llvm-project@6f3d322 but this is during LTO so that could impact it I suppose.

https://reviews.llvm.org/D90511 (2020-11). Some LTO code paths may not pass through the location. I'll check whether they can be improved after you track it down:)

Probably something that changed how __ia32_compat_sys_sysctl was defined in assembler.

Tracking that down is going to be rather painful but I will see if I can take a whack at it later this week.

[nickdesaulniers]

I can reproduce.

I'm curious whether this is:

1. a config in android-4.19-stable that's not enabled in defconfig or tested in stable/4.19.y
2. an out of tree patch in Android (hence why we haven't observed this in stable/4.19.y)

cc @samitolvanen in case
commit 2d96fcb ("ANDROID: x86: use the correct function type in SYSCALL_DEFINE0")
commit 1693025 ("ANDROID: x86: fix function types in COND_SYSCALL")
are a case of 2. I'll test reverts. (The first patch landed in stable/4.19.y as 0aeb658).

[nathanchance]

I think it is LTO related, as I do not think that it is reproducible when that is turned off.

[samitolvanen]

The syscall patches were upstreamed later. There might have been some changes in the final versions, perhaps worth checking. However, the main difference between 4.19 and later versions is that we still use full LTO on x86_64 with 4.19.

[nickdesaulniers]

It's 1693025. It will have to get reverted or fixed for an AOSP toolchain update.