static libraries can be picked up instead of system libraries when -l is used

[andrewrk]

Extracted from #14647.

Zig Version

0.11.0-dev.2148+b3af5d076

Steps to Reproduce and Observed Behavior

The test case from 4efeeaa passes.

In particular, test_exe.linkSystemLibrary("a"); // force linking liba.a as -la.

Expected Behavior

I expect -l to link dynamic libraries, and static libraries to be put on the command line as positional arguments. I don't want the linker to decide whether to link statically or dynamically. It's too late at that point. I want the frontend to decide whether a library is linked statically or dynamically.

There should never be any ambiguity. -l should always dynamically link, and positional arguments are already unambiguous.

[gshanemiller]

Fixing this might fix #14939 .... I hope so

[kubkon]

Is this change only affecting the build system and how it resolves the libraries for consumption by the linker, or would you have the linker enforce this behaviour as well? In particular, doing this

$ file libfoo.a
libfoo.a: current ar archive random library
$ clang main.c -lfoo -L.

is valid and common on macOS. If we enforce the rule such that -lfoo only ever resolves to dynamic libraries, we risk zig cc not being a drop-in replacement for clang on macOS for example. Another example,

$ ls A
libfoo.dylib
$ ls B
libfoo.a
$ clang main.c -lfoo -LA -LB
$ otool -lv a.out | grep libfoo
name libfoo.dylib

But flipping the dirs around and we get

$ clang main.c -lfoo -LB -LA
$ otool -lv a.out | grep libfoo

This behaviour can be overridden at the linker level on macOS via -search_dylibs_first flag

$ clang main.c -lfoo -LB -LA -Wl,-search_dylibs_first
$ otool -lv a.out | grep libfoo
name libfoo.dylib

The default however is for the linker to search for a dylib in a dir and then for a static lib before proceeding to the next search dir. Please note that the MachO linker currently matches the behaviour of the system linker and I think we should strive to do that for each platform to offer smooth sailing when using zig cc as a drop-in replacement. Or so I thought, but perhaps we should try going against the grain and see how it pans out.

[gshanemiller]

I think we need to put aside OS specializations, and focus instead on the known, defined behavior in clang.

For example, consider this link. As Andrew writes above and this link seems to support, the -l idiom means link to a dynamic library ... more or less case closed. Archive libraries don't typically arise with -l. Maybe Zig should just do that.

In the alternative zig build files could delegate this problem to pkg-config. Zig has some support for pkg-config, however, the Zig doc is littered with phrases like:

linkSystemLibraryNeededPkgConfigOnly: This links against a system library, exclusively using pkg-config to find the library. Prefer to use linkSystemLibraryNeeded instead.

I find this suggestion unhelpful. In C/C++ build hell, using pkg-config with an explicit argument of -static when static libs are desired (and omitting when shared libraries are desired) is known, general solution with good support in industry.

Ideally, this issue would:

• nail down -l means shared libs period (and then do it). My bug cannot link against shared system libraries -AND- ZIG task is bad #14939 seems to not honor this, which hopefully this issue will fix
• fully support pkg-config

Try building, for example, DPDK and DPDK apps without pkg-config. Good luck!

Acknowledgement: @kubon remarks pkg-config is linux only. I was not aware. So maybe pkg-config is secondary, and enforcing defined, known clang behavior is primary.

Link library confusion, for example, does not arise with CMAKE. And that's because CMAKE is a conceptually simple pass-through to the tool-chain. While CMAKE can come up with default flags for a release (or debug) build, there's straightforward ways to just tell the tool chain what to do by standing on knowledge of the tool chain's defined behavior. And that's because programmers always have access to the string holding the final, bonafide command the tool chain will run. CMAKE's approach is:

• I can give you defaults through a build target e.g. release mode
• I get some defaults externally e.g. pkg-config
• You can always override and define CC/LD defines, CC/LD include paths, etc etc etc so it's totally clear what's going on

[kubkon]

I think we need to put aside OS specializations, and focus instead on the known, defined behavior in clang.

For example, consider https://jonwillia.ms/2018/02/02/static-linking. As Andrew writes above and this link seems to support, the -l idiom means link to a dynamic library ... more or less case closed. Archive libraries don't typically arise with -l. Maybe Zig should just do that.

In the alternative zig build files could delegate this problem to pkg-config. Zig supports pkg-config, however, the Zig doc is littered with phrases like:

linkSystemLibraryNeededPkgConfigOnly: This links against a system library, exclusively using pkg-config to find the library. Prefer to use linkSystemLibraryNeeded instead.

I find this needlessly pejorative. In C/C++ build hell, using pkg-config with an explicit argument of -static when static libs are desired (and omitting when shared libraries are desired) is known, perfectly good solution.

Ideally, this issue would:

* nail down that -l means shared libs period (and then do it)

* fully support pkg-config.

Try building, for example, DPDK without pkg-config. Good luck.

I very much like consistency across targets. I merely warn against potential hurdles we might face if we force a non-drop-in replacement option. Also, pkg-config is a Linux-specific thing and we care about all targets including macOS and Windows.

[andrewrk]

C compiler drop-in replacement logic belongs in the CLI code, not in the linker.

[kubkon]

C compiler drop-in replacement logic belongs in the CLI code, not in the linker.

What about the linker's drop-in behavior then?

[andrewrk]

andy@ark ~> zig ld
info: Usage: zig [command] [options]

Commands:

  build            Build project from build.zig
  init-exe         Initialize a `zig build` application in the cwd
  init-lib         Initialize a `zig build` library in the cwd

  ast-check        Look for simple compile errors in any set of files
  build-exe        Create executable from source or object files
  build-lib        Create library from source or object files
  build-obj        Create object from source or object files
  fmt              Reformat Zig source into canonical form
  run              Create executable and run immediately
  test             Create and run a test build
  translate-c      Convert C code to Zig code

  ar               Use Zig as a drop-in archiver
  cc               Use Zig as a drop-in C compiler
  c++              Use Zig as a drop-in C++ compiler
  dlltool          Use Zig as a drop-in dlltool.exe
  lib              Use Zig as a drop-in lib.exe
  ranlib           Use Zig as a drop-in ranlib
  objcopy          Use Zig as a drop-in objcopy

  env              Print lib path, std path, cache directory, and version
  help             Print this help and exit
  libc             Display native libc paths file or validate one
  targets          List available compilation targets
  version          Print version number and exit
  zen              Print Zen of Zig and exit

General Options:

  -h, --help       Print command-specific usage

error: unknown command: ld

Not sure what you are talking about. Can you elaborate?

[kubkon]

#8755

[gshanemiller]

C compiler drop-in replacement logic belongs in the CLI code, not in the linker.

Well ... absolutely ... the CLI is the boundary between the programmer and Zig build. (Tangentially build.zig is a player too). But there's a lot of blue sky after that. The challenge is to telescope defined behavior at that high level of abstraction with users who are 100.0% familiar all day long with how gcc/clang works on the same command line.