Flags in compiler activation vs. out-of-conda-forge usage

[h-vetinari]

Part of the success of conda-forge is that many upstream projects have adopted (or at least support) development environments based on conda-forge packages. Almost inevitably, this ends up pulling in compilers as well - either through the explicit compilers package, or through other means (like directly depending on gcc_linux-64 or clang_osx-64).

This runs into the question whether this is something we can or want to support. The status quo following Hyrum's law is of course that they're being used in several ways.

This "off-label" compiler use then runs into issues, like that the flags we inject into our compiler activation create problems for development. Anything from -DNDEBUG / -O2, etc. etc., see:

• Don't add -O flags to [C/CXX/F]FLAGS outside of conda-build ctng-compiler-activation-feedstock#80
• Activating the gcc compiler should not set the install prefix ctng-compiler-activation-feedstock#77
• Inclusion of CXXFLAGS from x86_64 & aarch64 Linux variants causes build error ctng-compiler-activation-feedstock#74
• -fvisibility-inlines-hidden breaks libprotobuf (and potentially other software) ctng-compiler-activation-feedstock#73
• Should LD_FLAGS be the same across all linuxen? ctng-compiler-activation-feedstock#72
• Add -fno-merge-constants to default flags? ctng-compiler-activation-feedstock#63
• GCC 9.3 compilers should not include -fopenmp or -std=c++17 by default ctng-compiler-activation-feedstock#42
• Using -fPIE/-pie on macOS results in many spurious warnings clang-compiler-activation-feedstock#90
• CMAKE_ARGS should contain -DCMAKE_BUILD_TYPE=None, else builds picks up -O3 instead -O2 #1859
• [DOC] Compiler flags and where they come from #893

Latest comment to this effect:

@rgommers: Polluting CFLAGS & co gives so many problems, isn't there an issue/plan somewhere to get rid of unnecessary flags and insert the desired ones for package building only inside conda-build/mamba-build?

In this same release cycle we also already had to add this warning:

Assuming we want to make our support explicit (for the usage of our compilers outside of conda-forge), we could potentially have both an internal & external compiler activation; the conda-forge internal one would add all the necessary flags (as now), and the external one would be reduced to a minimum.

Thoughts?

[WillAyd]

Wanted to comment here as I also have struggled with the changes made to environment variables like CFLAGS in the past. The issues this poses range from mildly annoying (ex: having to clean and rebuild your environment when you realize you have the wrong flags) to outright confusing (ex: running benchmarking tools like ASV with conda and having no idea why you get wildly different timings).

For users that aren't yet that experienced with how environment variables affect the build system, I think this adds a layer of confusion that makes the compilers seem more mysterious than they actually are.

Before I found this I also posted a similar question on SO, as the mechanics of how to prevent this from happening were not that obvious to me:

https://stackoverflow.com/questions/77651168/prevent-conda-from-changing-cflags-cppflags

Not sure if this is considered too heavy-handed but maybe even a compilers-dev package that doesn't mess with the environment like this would be a net improvement?

[rgommers]

Not sure if this is considered too heavy-handed but maybe even a compilers-dev package that doesn't mess with the environment like this would be a net improvement?

That seems like a good alternative to changing the activation scripts for the compilers package to me. Either way, we need a single "install compilers-package" that just works as expected for local development. Which means no flags added to any env vars beyond what's essential, which is probably only -isystem path/to/env for all relevant env vars, plus -stdlib=libc++ for CXXFLAGS.

[WillAyd]

Another advantage of a debug compiler package is that gdb could be compiled with the appropriate --with-python flag that enables extending the debugger. This is something that would make things like using cygdb for Cython somewhat more tenable.

See also cython/cython#5186 (comment)