toolchain/43132: make(1) improper variable expansion

[dholland%eecs.harvard.edu@localhost]

>Number:         43132
>Category:       toolchain
>Synopsis:       make(1) improper variable expansion
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    toolchain-manager
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Tue Apr 06 06:50:00 +0000 2010
>Originator:     David A. Holland
>Release:        NetBSD 5.99.24 (20100327)
>Organization:
>Environment:
System: NetBSD tanaqui 5.99.22 NetBSD 5.99.22 (TANAQUI) #31: Tue Dec 8 22:53:35 
EST 2009 dholland@tanaqui:/usr/src/sys/arch/i386/compile/TANAQUI i386
Architecture: i386
Machine: i386
>Description:

Make expands some variables at parse time and some variables at run
time, based on internal flags. This is well and good, if perhaps
sometimes aggravating and/or confusing to beginners; but it means that
some lines of the makefile are variable-expanded more than once:
   - recipe lines are expanded only at runtime
   - directive and assignment lines are expanded only at parse time
   - rule lines are expanded at both times.

This leads to inconsistencies if $$ appears on a rule line.
Specifically, $$ followed by a variable name, as in $$(FOO), is
expanded with the runtime value of the variable. $$(FOO) is turned
into $(FOO) by the first variable expansion pass, and then the second
runtime pass turns that into the post-parse value of FOO. However,
this works only on rule lines.

For variables that are not expanded at all until runtime, like
$(.TARGET), writing $$ instead of $ does not change the behavior; in
the $ case the variable is not expanded on the first pass, and in the
$$ case the firts pass does nothing but turn the $$ into $ where it's
seen by the second pass.

Note that the observed behavior is also inconsistent with the expected
behavior of variable expansion, which is that expansion continues
until every variable that can be expanded has been expanded.

>How-To-Repeat:

(1)
   FOO=
   all: $(FOO)
        @echo foo
   FOO=bar

   % make
   foo
   % 

(2)
   FOO=
   all: $$(FOO)
        @echo foo
   FOO=bar

   % make
   make: don't know how to make bar. Stop
   % 

(3)
   FOO=
   all: $$$$(FOO)
        @echo foo
   FOO=bar

   % make
   make: don't know how to make $(FOO). Stop
   % 

(4)
   all: $(.TARGET).foo
           @echo foo

   % make
   make: don't know how to make all.foo. Stop
   % 

(5)
   all: $$(.TARGET).foo
           @echo foo

   % make
   make: don't know how to make all.foo. Stop
   % 

(6)
   all: $$$$(.TARGET).foo
           @echo foo

   % make
   make: don't know how to make $(.TARGET).foo. Stop
   % 

and
   FOO=bar
   all: $$(FOO)
           @echo '$$(FOO)'
   FOO=

   % make
   $(FOO)
   % 

(which shows that the rule line and recipe line are being handled
differently.)

>Fix:

In the long term, probably parse trees. For now, maybe substitute $$
back in on the first pass on rule lines? That'd be gross but it ought
to work.

Note that there perhaps ought to be some way to get the run-time value
on a rule line (and the parse-time value in a recipe too, probably)
but it should absolutely not be done by exposing this quirk of the
eval behavior. make has too many odd eval quirks already.

You can also sort of already get the run-time value in a recipe by
misusing for loops, like this:
   FOO=bar
   all:
   .for FU in $(FOO)
           @echo $(FU)
   .endfor
   FOO=baz

   % make
   bar
   % 

but this is gross and only works for values that are single words.
(The :tW modifier doesn't help.) I think the right approach is to
allow the user to explicitly tag variables as loop-scope or
target-scope. (This would also allow working loop-local variables,
which is currently a thorny mess.)