toolchain/49085: make(1): several parsing errors

[Jarmo Jaakkola <jarmo.jaakkola%roskakori.fi@localhost>]

>Number:         49085
>Category:       toolchain
>Synopsis:       some expansions and line continuations are parsed incorrectly
>Confidential:   no
>Severity:       non-critical
>Priority:       medium
>Responsible:    toolchain-manager
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Thu Aug 07 16:05:00 +0000 2014
>Originator:     Jarmo Jaakkola <jarmo.jaakkola%roskakori.fi@localhost>
>Release:        NetBSD 6.1.2_PATCH
>Organization:
>Environment:
System: NetBSD kotoisa.roskakori.fi 6.1.2_PATCH NetBSD 6.1.2_PATCH (KOTOISA) 
#5: Mon Jan 20 17:01:44 EET 2014 
jammuli%kotoisa.roskakori.fi@localhost:/usr/src/sys/arch/amd64/compile/KOTOISA 
amd64
Architecture: x86_64
Machine: amd64
>Description:

I have found several parsing bugs in make(1).  Here's a rundown of them:

Issue 1: variable modifiers don't work with D and F forms of local variables
Issue 2: $(?D) and $(?F) expand to null string
Issue 3: line continuations in commands are handled incorrectly
Issue 4: lib(member) as the last target before ':' results in syntax error

I have prepared a patch to fix all of these and will submit it
as a follow-up.



Issue 1: variable modifiers don't work with D and F forms of local variables
========

When variable modifiers are used with the POSIX specified D and F forms
of local variables (e.g. $(@D:.c=.y)), the modifier is not processed
and the colon is treated as the closing parentheses/brace.  The only
variable modifier allowed by posix is the string replacement shown in
the example, but it affects all modifiers equally.

--<Start>---------------------------------------------------------------
$ cat >Makefile1 <<EOF
dir.d/target.t:
        @echo 'Expected: dir.d/target.t dir target.T'
        @echo 'Actual:   \$(@) \$(@D:.d=) \${@F:.t=.T}'
EOF
$ make -rf Makefile1
Expected: dir.d/target.t dir target.T
Actual:   dir.d/target.t dir.d.d=) target.t.t=.T}
--<End>-----------------------------------------------------------------

The problem is a premature exit from the var.c:Var_Parse() function after
a D or F modified version of a variable is detected and the following
character (assumed to be the closing parentheses/brace) is eaten.  This
bypasses the modifier processing that occurs later in the function.



Issue 2: $(?D) and $(?F) expand to null string
========

All (POSIX) local variables should be available as D and F variants, but
$(?) isn't.

--<Start>---------------------------------------------------------------
$ cat >Makefile2 <<EOF
target: dir/source1 source2
        @echo 'Expected: dir/source1 source2 dir . source1 source2'
        @echo 'Actual:   \$(?) \$(?D) \$(?F)'

dir/source1 source2:
EOF
$ make -rf Makefile2
Expected: dir/source1 source2 dir . source1 source2
Actual:   dir/source1 source2  
--<End>-----------------------------------------------------------------

Again the problem is with the var.c:Var_Parse() function.  The strchr()
check for local variable names does not include '?'.



Issue 3: line continuations in commands are handled incorrectly
========

All line continuations and the whitespace on the following line
is collapsed into one space.  Command lines should be treated
differently according to POSIX.

Quoting POSIX:
    When an escaped <newline> (one preceded by a backslash) is found
    anywhere in the makefile except in a command line, it shall be
    replaced, along with any leading white space on the following line,
    with a single <space>.  When an escaped <newline> is found in a
    command line in a makefile, the command line shall contain the
    backslash, the <newline>, and the next line, except that the first
    character of the next line shall not be included if it is a <tab>.

--<Start>---------------------------------------------------------------
$ cat >Makefile3 <<EOF
target:
        @printf 'Expected:\na\nb\nc\n'
        @echo 'Actual:'
        @echo 'aXbXc' | sed -e 's/X/\\
        /g'
EOF
$ make -rf Makefile3
Expected:
a
b
c
Actual:
a b c
--<End>-----------------------------------------------------------------

The problem is in parse.c:ParseGetLine().  The handling of escaped
newlines does not take into account command lines.



Issue 4: lib(member) as the last target before ':' results in syntax error
========

When an archive member target ("lib(member)") is specified as the last
target on a dependency line, parsing of the line fails.

--<Start>---------------------------------------------------------------
$ cat >Makefile4 <<EOF
lib: lib(foo.o)
        ar -s \$(@)

# No it isn't missing, its right there
#         v
lib(foo.o): foo.c
        cc -c -o \$(%) foo.c
        ar -rc \$(@) \$(%)
        rm -f \$(%)

foo.c:
        echo 'int foo = 1;' >\$(@)

# Expected:
# echo 'int foo = 1;' >foo.c
# cc -c -o foo.o foo.c
# ar -rc lib foo.o
# rm -f foo.o
# ar -s lib
EOF
$ make -rf Makefile4
make: "/some/dir/Makefile4" line 6: Missing dependency operator
make: Fatal errors encountered -- cannot continue
make: stopped in /some/dir
--<End>-----------------------------------------------------------------

The parsing error goes away when a dummy target is inserted between
the archive member and the dependency operator.  Even after
the workaround this makefile won't work because the value of $(@) is
incorrect for archive member targets.  I have another PR prepared for
that one and will post its number as a follow-up.

The problem is in parse.c:ParseDoDependency().  Arch_ParseArchive()
advances line (start of next target name) instead of cp (current
position).  The rest of the loop is based on cp with line being reset
to cp as the last statement in the loop.  The rest of the function is
based on cp too, assuming that cp points at the dependency operator
after the loop finishes.  One would think that this would result in an
infinite loop, but this does not happen because the loop condition
happens to be based on line instead of cp.


>How-To-Repeat:

Inlined into the description.

>Fix:

I will post a patch as a follow-up.