CPPFLAGS & LDFLAGS from environment are not passed and replaced your ones

[Perlover]

Issue and Steps to Reproduce

[UPDATE] How to reproduce: #1454 (comment)

If user has CPPFLAGS & LDFLAGS environments as set and exported (these variables consist flags for gcc & linker) then your Makefile redefines its and configure libraries (external/*) gets incorrect LDFLAGS and CPPFLAGS variables.

Your Makefile:

https://github.com/ElementsProject/lightning/blob/master/Makefile#L163
https://github.com/ElementsProject/lightning/blob/master/Makefile#L171

For example: I cannot compile your c-lightning in CentOS 6.* from user bitcoin because i have my own gcc & autotools programs in $HOME. I have as defined these variables but your make's recipe commands get your ones not my ones. It's incorrect behaviour.

The make docs:

When make runs a recipe, variables defined in the makefile are placed into
the environment of each shell. This allows you to pass values to sub-make
invocations (see Recursive Use of make). By default, only variables that
came from the environment or the command line are passed to recursive
invocations. You can use the export directive to pass other variables. See
Communicating Variables to a Sub-make, for full details. 

So if i have CPPFLAGS & LDFLAGS are defined in my environment all recipes in your Makefiles and sub-Makefiles will get your CPPFLAGS & LDFLAGS variables not my ones. And i cannot compile your c-lightning (because gcc and other tools should look in directories which i defined in these variables).

Correct behaviour:

I'm sure you should consider the value of these variables from outside. I checked Makefiles created by third-party programs (bitcoin core for example) which i compilied successfully. The configure script consides these variables and to add own options not replace.

Conclusion

May be to use += operation for these flags not =. I will try to make patch.

[Perlover]

I made patch but it didn't help to me
Now i restore CPPFLAGS & LDFLAGS from user but i noticed that it's not main problem

My patch was:

diff --git a/Makefile b/Makefile
index 7248739..6d8d624 100644
--- a/Makefile
+++ b/Makefile
@@ -31,8 +31,8 @@ COVFLAGS = --coverage
 endif
 
 ifeq ($(PIE),1)
-PIE_CFLAGS=-fPIE -fPIC
-PIE_LDFLAGS=-pie
+PIE_CFLAGS := -fPIE -fPIC
+PIE_LDFLAGS := -pie
 endif
 
 ifneq ($(NO_COMPAT),1)
@@ -160,7 +160,19 @@ WIRE_GEN := tools/generate-wire.py
 
 ALL_PROGRAMS =
 
-CPPFLAGS = -DBINTOPKGLIBEXECDIR='"'$(shell sh tools/rel.sh $(bindir) $(pkglibexecdir))'"'
+# Installation directories
+prefix := /usr/local
+exec_prefix := $(prefix)
+bindir := $(exec_prefix)/bin
+libexecdir := $(exec_prefix)/libexec
+pkglibexecdir := $(libexecdir)/$(PKGNAME)
+datadir := $(prefix)/share
+docdir := $(datadir)/doc/$(PKGNAME)
+mandir := $(datadir)/man
+man1dir := $(mandir)/man1
+man7dir := $(mandir)/man7
+
+CPPFLAGS := $(CPPFLAGS) -DBINTOPKGLIBEXECDIR='"'$(shell sh tools/rel.sh $(bindir) $(pkglibexecdir))'"'
 CWARNFLAGS := -Werror -Wall -Wundef -Wmissing-prototypes -Wmissing-declarations -Wstrict-prototypes -Wold-style-definition
 CDEBUGFLAGS := -std=gnu11 -g -fstack-protector
 CFLAGS = $(CPPFLAGS) $(CWARNFLAGS) $(CDEBUGFLAGS) -I $(CCANDIR) $(EXTERNAL_INCLUDE_FLAGS) -I . -I/usr/local/include $(FEATURES) $(COVFLAGS) $(DEV_CFLAGS) -DSHACHAIN_BITS=48 -DJSMN_PARENT_LINKS $(PIE_CFLAGS) $(COMPAT_CFLAGS)
@@ -168,7 +180,7 @@ CFLAGS = $(CPPFLAGS) $(CWARNFLAGS) $(CDEBUGFLAGS) -I $(CCANDIR) $(EXTERNAL_INCLU
 # We can get configurator to run a different compile cmd to cross-configure.
 CONFIGURATOR_CC := $(CC)
 
-LDFLAGS = $(PIE_LDFLAGS)
+LDFLAGS := $(LDFLAGS) $(PIE_LDFLAGS)
 LDLIBS = -L/usr/local/lib -lm -lgmp -lsqlite3 $(COVFLAGS)
 
 default: all-programs all-test-programs
@@ -355,18 +367,6 @@ update-mocks/%: %
 unittest/%: %
    $(VALGRIND) $(VALGRIND_TEST_ARGS) $*
 
-# Installation directories
-prefix = /usr/local
-exec_prefix = $(prefix)
-bindir = $(exec_prefix)/bin
-libexecdir = $(exec_prefix)/libexec
-pkglibexecdir = $(libexecdir)/$(PKGNAME)
-datadir = $(prefix)/share
-docdir = $(datadir)/doc/$(PKGNAME)
-mandir = $(datadir)/man
-man1dir = $(mandir)/man1
-man7dir = $(mandir)/man7
-
 # Commands
 MKDIR_P = mkdir -p
 INSTALL = install

But i have same problem:

if git submodule status external/libbacktrace | grep -q '^[-+]'; then rm -rf external/libbacktrace; git submodule update --init external/libbacktrace; fi
cd external/libbacktrace-build && ../libbacktrace/configure CC="cc" --enable-static=yes --host="" --build="" --enable-shared=no --prefix=/ --libdir=/ && make
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking target system type... x86_64-unknown-linux-gnu
....
configure: creating ./config.status
./config.status: command substitution: line 577: unexpected EOF while looking for matching `"'
./config.status: command substitution: line 578: syntax error: unexpected end of file
./config.status: line 576: -e: command not found
./config.status: line 692: -e: command not found
./config.status: line 692: -e: command not found
./config.status: line 692: -e: command not found
./config.status: line 692: -e: command not found
./config.status: line 692: -e: command not found
./config.status: line 692: -e: command not found
./config.status: line 692: -e: command not found
./config.status: line 692: -e: command not found
./config.status: line 692: -e: command not found
./config.status: line 704: -e: command not found
./config.status: line 704: -e: command not found
./config.status: line 704: -e: command not found
./config.status: line 704: -e: command not found
config.status: creating Makefile
config.status: creating backtrace-supported.h
config.status: creating config.h
config.status: config.h is unchanged
config.status: executing libtool commands
config.status: executing default commands
make[1]: Entering directory `/home/bitcoin/tmp/lightning/external/libbacktrace-build'
make  all-am
make[2]: Entering directory `/home/bitcoin/tmp/lightning/external/libbacktrace-build'
/bin/sh ./libtool --tag=CC   --mode=compile cc -DHAVE_CONFIG_H -I. -I../libbacktrace   -I/home/bitcoin/include -DBINTOPKGLIBEXECDIR='"'../libexec/c-lightning'"' -DBINTOPKGLIBEXECDIR='"'../libexec/c-lightning'"' -funwind-tables -frandom-seed=atomic.lo -W -Wall -Wwrite-strings -Wstrict-prototypes -Wmissing-prototypes -Wold-style-definition -Wmissing-format-attribute -Wcast-qual  -g -O2 -c -o atomic.lo ../libbacktrace/atomic.c
./libtool: line 496: CDPATH: command not found
libtool: Version mismatch error.  This is libtool 2.2.4, but the
libtool: definition of this LT_INIT comes from an older release.
libtool: You should recreate aclocal.m4 with macros from libtool 2.2.4
libtool: and run autoconf again.
make[2]: *** [atomic.lo] Ошибка 63
make[2]: Leaving directory `/home/bitcoin/tmp/lightning/external/libbacktrace-build'
make[1]: *** [all] Ошибка 2
make[1]: Leaving directory `/home/bitcoin/tmp/lightning/external/libbacktrace-build'
make: *** [external/libbacktrace.a] Ошибка 2

It's because the libtool in external/libbacktrace-build has:

lt_unset=

And then:

$lt_unset CDPATH

So the bash sees the command CDPATH instead unset CDPATH

But! If i do manualy after:

cd external/libbacktrace-build && ../libbacktrace/configure CC="cc" --enable-static=yes --host="" --build="" --enable-shared=no --prefix=/ --libdir=/

Everything is ok. (the lt_unset=unset in libtool)

I reseacrhed the problem. When i run by mannualy i have $CPPFLAGS as:

-I/home/bitcoin/include

But if i log environments from make i see the $CPPFLAGS as (after my patch):

-I/home/bitcoin/include -DBINTOPKGLIBEXECDIR='"'../libexec/c-lightning'"'

Before my patch i had it as a following (because CPPFLAGS was defined in shell environment and the make passed it to recipe's commands):

-DBINTOPKGLIBEXECDIR='"'../libexec/c-lightning'"'

So i see the strange bug: if the BINTOPKGLIBEXECDIR macro is defined through CPPFLAGS then the libbacktrace cannot make correct libtool script by configure
I think it related with quotes in macro...

I am thinking now how can i fix it...

[Perlover]

Wow! I can write here how you can easy to reproduce it :)

CPPFLAGS='' make

[Perlover]

After researching the c-lightning's make files i see that LDFLAGS is not used only definition. The CPPFLAGS is used only one place:

lightning/Makefile

Line 166 in 7767b68

CFLAGS = $(CPPFLAGS) $(CWARNFLAGS) $(CDEBUGFLAGS) -I $(CCANDIR) $(EXTERNAL_INCLUDE_FLAGS) -I . -I/usr/local/include $(FEATURES) $(COVFLAGS) $(DEV_CFLAGS) -DSHACHAIN_BITS=48 -DJSMN_PARENT_LINKS $(PIE_CFLAGS) $(COMPAT_CFLAGS)

For example:

$ grep LDFLAGS external/Makefile bitcoin/Makefile common/Makefile wire/Makefile wallet/Makefile hsmd/Makefile gossipd/Makefile openingd/Makefile channeld/Makefile closingd/Makefile onchaind/Makefile lightningd/Makefile cli/Makefile doc/Makefile devtools/Makefile bitcoin/test/Makefile common/test/Makefile wire/test/Makefile wallet/test/Makefile gossipd/test/Makefile channeld/test/Makefile onchaind/test/Makefile lightningd/test/Makefile cli/test/Makefile Makefile

---

Makefile:PIE_LDFLAGS=-pie
Makefile:LDFLAGS = $(PIE_LDFLAGS)

---

$ grep CPPFLAGS external/Makefile bitcoin/Makefile common/Makefile wire/Makefile wallet/Makefile hsmd/Makefile gossipd/Makefile openingd/Makefile channeld/Makefile closingd/Makefile onchaind/Makefile lightningd/Makefile cli/Makefile doc/Makefile devtools/Makefile bitcoin/test/Makefile common/test/Makefile wire/test/Makefile wallet/test/Makefile gossipd/test/Makefile channeld/test/Makefile onchaind/test/Makefile lightningd/test/Makefile cli/test/Makefile Makefile

---

Makefile:CPPFLAGS = -DBINTOPKGLIBEXECDIR='"'$(shell sh tools/rel.sh $(bindir) $(pkglibexecdir))'"'
Makefile:CFLAGS = $(CPPFLAGS) $(CWARNFLAGS) $(CDEBUGFLAGS) -I $(CCANDIR) $(EXTERNAL_INCLUDE_FLAGS) -I . -I/usr/local/include $(FEATURES) $(COVFLAGS) $(DEV_CFLAGS) -DSHACHAIN_BITS=48 -DJSMN_PARENT_LINKS $(PIE_CFLAGS) $(COMPAT_CFLAGS)

---

[ZmnSCPxj]

After researching the c-lightning's make files i see that LDFLAGS is not used only definition.

Welcome, to the nasty world of make.

Try running the command: make -p -f/dev/null.

You will see that one of the lines is:

LINK.o = $(CC) $(LDFLAGS) $(TARGET_ARCH)

(or something similar anyway)

Now try to grep LINK.o -R *.

So yes, LDFLAGS is in fact used, not just defined. make does a lot of very magical things.

[Perlover]

Try running the command: make -p -f/dev/null.

UPDATE: Yes, i see now. It's implicit rules. Now i understand that it's used.

LINK.m = $(OBJC) $(OBJCFLAGS) $(CPPFLAGS) $(LDFLAGS) $(TARGET_ARCH)
LINK.p = $(PC) $(PFLAGS) $(CPPFLAGS) $(LDFLAGS) $(TARGET_ARCH)
LINK.cc = $(CXX) $(CXXFLAGS) $(CPPFLAGS) $(LDFLAGS) $(TARGET_ARCH)
LINK.r = $(FC) $(FFLAGS) $(RFLAGS) $(LDFLAGS) $(TARGET_ARCH)
LINK.S = $(CC) $(ASFLAGS) $(CPPFLAGS) $(LDFLAGS) $(TARGET_MACH)
LINK.c = $(CC) $(CFLAGS) $(CPPFLAGS) $(LDFLAGS) $(TARGET_ARCH)
LINK.s = $(CC) $(ASFLAGS) $(LDFLAGS) $(TARGET_MACH)
LINK.F = $(FC) $(FFLAGS) $(CPPFLAGS) $(LDFLAGS) $(TARGET_ARCH)
LINK.f = $(FC) $(FFLAGS) $(LDFLAGS) $(TARGET_ARCH)
LINK.o = $(CC) $(LDFLAGS) $(TARGET_ARCH)

[Perlover]

I apllied only commit 62d25464d2180702b2116201eb6b6c30bee2d9ff and now it works so i don't have a problem.

Thanks!

I wait this patch in master branch ;-)

[Perlover]

Now i even installed by this command :)

make -j28 prefix=$HOME && make prefix=$HOME install

[rustyrussell]

No. Make doesn't take vars from the environment by default. If you want this, you do CFLAGS=$CFLAGS etc on the make command line. I'm surprised you don't have to do that for other projects?

Overriding the implicit rules is a backwards step here.

I have some work on a ./configure step, which would give you a place to set CFLAGS etc. more conveniently.

[Perlover]

No. Make doesn't take vars from the environment by default.

https://www.gnu.org/software/make/manual/make.html#Variables-from-the-Environment

Variables in make can come from the environment in which make is run. Every environment variable that make sees when it starts up is transformed into a make variable with the same name and value

---

If you want this, you do CFLAGS=$CFLAGS etc on the make command line.\

I cannot use it because makefiles can run other scripts (configure of third-party libraries) and gcc and ld tools will not get paths of include & lib dirs.

P.S. There problem was from CPPFLAGS not from CFLAGS

[Perlover]

I googled now these veriables (CPPFLAGS, CFLAGS and LDFLAGS) and i understand that these veriables only from make for implicit rules. I had these environments in my shell and i thought that it helped to me to build programs (CPPFLAGS has include directories in $HOME and LDFLAGS has lib directories same in $HOME). May be it's true because make will get variables from environment if same variables is not initialized in Makefiles. May be it's my mistake to use theirs as environment variables but it difinitely caused the problem (reproduce as CPPFLAGS='' make). I will use other variables for gcc and ld for sucessfully building of third-party makefiles.