Using -fPIE/-pie on macOS results in many spurious warnings

[rgommers]

When building a Python extension module on macOS (both x86-64 and arm64) with clang or clang++, you will see this warning for each extension (example log from a conda-forge SciPy build):

ld: warning: -pie being ignored. It is only used when linking a main executable

When you build a package like SciPy, that's O(100) warnings.

The problem isn't actually limited to macOS I think, I just don't see it on Linux because the compilers package pulls in GCC rather than Clang.

I have read https://conda-forge.org/docs/user/faq.html#faq-compiler-required-options, but I don't think adding -fPIE is one of the options that qualifies here - because it is not a flag that is valid for building Python extensions nor other types of binaries that aren't a main executable. Therefore I don't think it should be added by default.

I can imagine that the choice was made to only do this in Clang because macOS seemingly preferring position independent executables (https://en.wikipedia.org/wiki/Position-independent_code#Position-independent_executables) and warning on non-PIE binaries being uploaded to the App Store. That said, this should be dealt with in the build options for packages that are main executables though, and not for every compiler invocation.

[rgommers]

Apologies, wrong repo. Moving this to https://github.com/conda-forge/clang-compiler-activation-feedstock, closing.

[isuruf]

I have read https://conda-forge.org/docs/user/faq.html#faq-compiler-required-options, but I don't think adding -fPIE is one of the options that qualifies here - because it is not a flag that is valid for building Python extensions nor other types of binaries that aren't a main executable. Therefore I don't think it should be added by default.

Agreed that it is only for main executables. However, there's no way to add it just for main executable just yet.

The problem isn't actually limited to macOS I think, I just don't see it on Linux because the compilers package pulls in GCC rather than Clang.

GCC has a configure time option --enable-pie which does the same thing but correctly does it for main executables only.

I can imagine that the choice was made to only do this in Clang because macOS seemingly preferring position independent executables (https://en.wikipedia.org/wiki/Position-independent_code#Position-independent_executables) and warning on non-PIE binaries being uploaded to the App Store.

No, this is done as a security measure. Almost all linux distros do this. For eg: https://wiki.ubuntu.com/SecurityTeam/PIE

That said, this should be dealt with in the build options for packages that are main executables though, and not for every compiler invocation.

As it's a security measure, we should enable it for all conda packages.

Clang also has a cmake build option, but it is only enabled for Linux (https://github.com/llvm/llvm-project/search?q=CLANG_DEFAULT_PIE_ON_LINUX). We should do the same for Darwin as well and then we can just remove these flags.

[rgommers]

Clang also has a cmake build option, but it is only enabled for Linux (https://github.com/llvm/llvm-project/search?q=CLANG_DEFAULT_PIE_ON_LINUX). We should do the same for Darwin as well and then we can just remove these flags.

That would be good indeed. Either way, if this remains the default then the actual fix is to add -no-pie to each build system as a default flag for Python extensions, right?

[rgommers]

I tried to add this for SciPy:

if cc.get_id() == 'clang'
  # Unlike GCC, Clang does not yet have a way to add `-fPIE` and `-pie` only to
  # main executables. See https://github.com/conda-forge/clang-compiler-activation-feedstock/issues/90
  add_project_link_arguments(cc.get_supported_link_arguments('-no-pie'), language: ['c', 'cpp']) 
endif

It doesn't work though, because LDFLAGS gets added to the right of flags from build scripts so they take precedence. The result is:

[93/833] Linking target scipy/stats/_qmc_cy.cpython-39-darwin.so
clang-11: warning: argument unused during compilation: '-no-pie' [-Wunused-command-line-argument]
ld: warning: -pie being ignored. It is only used when linking a main executable

So it looks like there is no way to work around this right now.

Clang also has a cmake build option, but it is only enabled for Linux (https://github.com/llvm/llvm-project/search?q=CLANG_DEFAULT_PIE_ON_LINUX). We should do the same for Darwin as well and then we can just remove these flags.

I looked into this. Rather than the option only being enabled for Linux, it is still default off and only available as an option for Linux, since Nov 2021: llvm/llvm-project#13782. There is no discussion on the issue or corresponding PR about adding an option for macOS, so I think this is not something that can easily be enabled for macOS in this feedstock, right? It would need to be contributed upstream first.

[isuruf]

You need -Wl,-no-pie instead of just -no-pie.

Rather than the option only being enabled for Linux, it is still default off and only available as an option for Linux, since Nov 2021: llvm/llvm-project#13782.

It is default ON with LLVM>=15.

There is no discussion on the issue or corresponding PR about adding an option for macOS, so I think this is not something that can easily be enabled for macOS in this feedstock, right? It would need to be contributed upstream first.

Should be easy to add a patch in the feedstock.

[rgommers]

You need -Wl,-no-pie instead of just -no-pie.

It doesn't look like it:

Compiler for C supports link arguments -no-pie: YES
Compiler for C supports link arguments -Wl,-no-pie: NO

It's also plain -pie in the activate script, and the man page says it's -no-pie.

Should be easy to add a patch in the feedstock.

Okay thanks, good to know. I'll give it a try if I find some time. I haven't touched Clang builds before, so it may take a while.

[rgommers]

It's also plain -pie in the activate script

There's actually both. It seems to have to do with LDFLAGS vs LDFLAGS_LD. The one bit of info I could find on that (from this Gitter thread) seems to say that it has to do with the differences between ld64 and ld.

There is something there I don't quite understand. The generated build.ninja indeed has -Wl,-pie, so I think you are right it should be -Wl,-no-pie as well - but the linker doesn't understand the latter when testing with get_supported_link_arguments.

EDIT: the was due to an incomplete edit of activate_clang_osx-arm64.sh. Final verdict:

• for the Clang linker, it should be -no-pie.
• the warning clang-11: warning: argument unused during compilation: '-no-pie' shows up because that's already the default behaviour,
• it is not possible to do anything in SciPy's meson.build to override what's being put in LDFLAGS_LD_USED & co by the activation script, because the env var takes precedence.
• To get rid of the warnings now for local development, delete -fPIE and -pie from the activation scripts.