Changing make(1) behavior: overriding suffix rule selection

[Jarmo Jaakkola <jarmo.jaakkola+netbsd-toolchain%roskakori.fi@localhost>]

Recent changes cause make to behave differently when an inferred target
(one which is made by a suffix rule) has explicit dependencies whose
basenames could be used as the implied source.  This breaks some
existing makefiles and people want the feature back.  To my understanding
the breakage caused by this change in NetBSD's tree has been fixed, so
we're talking about 3rd party makefiles at this point.

While the behavior is historical, it can also lead to problems if
the user is expecting POSIX compliant or gmake compatible behavior.
A consensus on how make should work in this case is needed.

Here is a simplified makefile and its usage demonstrating the case.
Assume that files foo.c and foo.S both exist.

    .SUFFIXES: .c .S .o
    .c.o: ; $(CC) ...
    .S.o: ; $(AS) ...
    foo.o: foo.S

    $ make foo.o


Historical behavior
===================

While .SUFFIXES ordering says that foo.c should be the implied source,
the existance of foo.S as an explicit dependency issues an override
because .S is an active suffix and the .S.o rule exists.
Therefore foo.o is made by assembling foo.S.

Those who have followed the discussion earlier may want to bring up
.PATH.  It is completely irrelevant.  The only check done by make was
if a dependency had the proper basename.  If it had, it was made
the implied source and searched for just as any other target / source
would: if it is not in $(.CURDIR), check .PATH.  The effect of
"overriding the .PATH search" to choose "dir/foo.c" instead of "foo.c"
happened because "dir/foo.c" is found from $(.CURDIR).


New behavior
============

Make pays no heed to the dependencies and does what .SUFFIXES tells
it to.  foo.o is made by compiling foo.c.  This is POSIX compliant
and compatible with gmake.


Comparing old and new
=====================

Historical behavior is, well, historical. The new behavior changes
the semantics and requires changes into makefiles.

The historical behavior is unexpected for people accustomed to POSIX
or gmake behavior, unlike the new behavior.

With the old behavior application of suffix rules can be controlled on
a target by target basis, but this is not portable.  With the new
behavior the closest one can get to the old behavior is to remove
the explicit dependency and hide the file you don't want with
.INVISIBLE (not portable).  Other, portable approaches are:
 1) Have the files of one type in dir1 and the other in dir2, then use
    dir1/foo.o or dir2/foo.o depending on which you want. [1]
 2) Re-order .SUFFIXES (only works if you don't need target-by-target
    control).
 3) Use inference where possible, explicit rules when not.  Re-order
    .SUFFIXES so the amount of explicit rules is minimized. [2]
These should work with the old behavior too.

With historical behavior dependencies on inferred targets have different
semantics based on their names and currently active .SUFFIXES.  Consider
how the dependency is written when foo.c includes foo.h, assuming that
there is no rule .h.o (as there usually isn't):
    foo.o: foo.h
And with historical behavior this is what you have to write if
the file includes foo.S and the rule .S.o exists [3]:
    foo.o: foo.c foo.S
With new behavior or old behavior when there is no rule:
    foo.o: foo.S

The historical behavior was not documented (at least in the manual
page), so it could be argued that makefiles relying on it did so at
their own risk.


Re-implementing overrides
=========================

1) Is the old behavior or the new behavior the default?

    Making the old behavior the default allows people to go on as if
    nothing has changed.  It also allows people to go on writing their
    makefiles to depend on this feature, building more resistance to
    changing it later on.

    How do you select between them?  .POSIX special target is supposed
    to select strict POSIX mode, but this would also mean that many
    other features should be disabled too.  And implementing that is a
    huge undertaking.

    A command line switch might be nice but if there will be multiple
    toggleable features, it will get unmaintainable quite fast.

2) Will the old behavior be deprecated?

3) Are there warnings?

    If the new behavior is the default, a warning would be nice if a
    dependency which would have triggered the old behavior is found.

    Likewise for the old behavior.  A warning when the rule is selected
    based on a dependency instead of .SUFFIXES would be good.

    Should there be way to silence these warnings?
        .SHUTUP: suffix-override

4) Are all dependencies eligible to become the implied source?

    Consider:
        foo.o: gazillion dependencies ... foo.S ... bazillion more

    Historically, foo.S will still be the implied source.  From
    usability point of view it might be better that only the first
    dependency is eligible.  If the overriding source is "hidden" like
    that, it was probably generated automatically and has a higher
    change of being unintentional.

5) Are all overrides allowed or only those which select one of
   the eligible ones?

    .PATH: dir
    foo.o: dir2/foo.c

    Historically, this will make foo.o from either ./dir2/foo.c or
    ./dir/dir2/foo.c, using the .c.o rule.  Normally neither file
    would be eligible to become the implied source, only either
    ./foo.c or dir/foo.c.


If I did not include some comparison or behavior choice from the previous
discussion, it is completely unintentional.  I merely forgot it, so please
bring it up.  Forcing of inferred dependencies was not forgotten, it is
not related to this behavior, so there will be another email about that.


Footnotes:
[1]
    .if use_type_1
    exe: dir1/foo.o
    .else
    exe: dir2/foo.o
    .fi

[2]
    .SUFFIXES:
    .if prefer_asm
    .SUFFIXES: .S .c
    .else
    .SUFFIXES: .c .S
    .fi
    exe: $(C_OBJS) $(ASM_OBJS)

  way 1:
    ASSEMBLE = $(AS) ...
    .S.o: ; $(ASSEMBLE)
    ASM_OBJS += ...
    $(ASM_OBJS): $(*).S ; $(ASSEMBLE)

  way 2:
    ASM_OBJS += ...
    $(ASM_OBJS): $(*).S
    .S.o $(ASM_OBJS): ; $(AS) ...

There was an objection to this technique earlier because it would seem
like this introduces new variables to system.mk.  This was not intended,
this merely follows the nice note from the POSIX standard:
    "The best way to provide portable makefiles is to include all of
    the rules needed in the makefile itself."
That is, the suffix rule is in the users makefile.

[3]  An opinion was voiced that this is a contrived example.  Yours truly
does not share this opinion.  Perhaps the assembly file contains code
that is compiled standalone when another configuration is chosen.
Perhaps it is not assembly code at all, but a generated piece of code:
    foo.o: generated/foo.c
Perhaps we're not talking about C code at all as make has many other
uses than building C programs.  The possibilities of generating
an equivalent scenario are limitless.

-- 
Jarmo Jaakkola