Re: make: avoid breaking backwards compatability

[Jarmo Jaakkola <jarmo.jaakkola+netbsd-toolchain%roskakori.fi@localhost>]

On Wed, Sep 03, 2014 at 07:11:58PM +0000, David Holland wrote:
> On Wed, Sep 03, 2014 at 08:50:44PM +0300, Jarmo Jaakkola wrote:
>  > > foo.o: subdir/foo.c
>  > 
>  > No, this works exactly as suffix rules should work, in my opinion.  The
>  > problem here is not that subdir/foo.c is overridden, it is that you're
>  > expecting some random extra dependency to become the implied source in
>  > a suffix rule.
> 
> I think you misunderstand the way .PATH and pattern rule searches are
> supposed to work.
> 
> If I write a .c.o rule, and ask for a .o file, make is supposed to
> find the corresponding .c file in some directory. So if I'm trying to
> build foo.o, it's looking for foo.c. If "subdir/foo.c" is already on
> the list of sources, that's not some random extra dependency; that's
> the file named foo.c that it's looking for. So it's found it and it
> doesn't need to search any further.

Yes, a corresponding file, which is a file with the same path, but
a different suffix.  So if you want subdir/foo.c to be the file used,
you either request subdir/foo.o, add subdir to .PATH and request for
foo.o _without_ there being foo.c in the current directory or write
an explicit rule.  There are three perfectly good ways to achieve
this, why would anyone need a fourth where make has to guess what they
mean?  If you're going through the trouble of writing the supposedly
implied dependency explicitly, you might as well write the whole rule.

And subdir/foo.c is exactly an "extra dependency".  There are two kinds
of dependency lines: those which have associated commands (explicit
rules) and those which do not (extra dependencies).  The semantics of
an extra dependency, in my opinion, is
    "use whatever rule you otherwise have for this target, but add these
    sources as dependencies too"
not
    "use whatever rule you otherwise have, except use some completely
    different one if under some other configuration one of these sources
    could have been the implied source, because I'm a lazy bugger and
    can't be bothered to do this correctly".
And in this case the "whatever rule" is an inference rule, where
the implied source is foo.c.

So, why should subdir/foo.c be the file we're looking for?  I have never
seen it mentioned anywhere that a listed dependency should affect
the implied source.  I must admit that I have experience only from
NetBSD's make, gmake and reading the POSIX standard.

Admittedly, gmake is a different implementation, but they do it "my way"
too: the listed extra dependencies have no bearing on the implicit
source.  The implied source is found primarily from the current directory
and secondarily from VPATH.  For the following example, assume foo.c,
dir/foo.c and dir2/foo.c exist.

    foo.o: dir/foo.c

    $ gmake foo.o
    cc    -c -o foo.o foo.c

    VPATH = dir2/foo.c
    foo.o: dir/foo.c

    $ gmake foo.o
    cc    -c -o foo.o foo.c

When done without foo.c, you get these two:
    gmake: Nothing to be done for `foo.o'.
    cc    -c -o foo.o dir2/foo.c

As you can see, dir/foo.c is purely an extra dependency, it is not
eligible to become the implied source.

And also admittedly, we're not slaves to POSIX, but this is exactly
the same as the previously mentioned issue 10.  In that issue
the suffix of the source is also changed by the extra dependency, but
it is still basically the same thing.  It just happens to change
the entire rule, not just the implied source!

    .SUFFIXES: .a .b .c
    .a.c: ; ...
    .b.c: ; ...
    test.c: test.b
    test.a:
    test.b:

.a.c should be used, not .b.c.  This is completely within the realm
of POSIX, and I think it would have been specified that the dependencies
must affect the implied rule / source selection process, if it really
should work like that.

Also note that there is no way of implementing the previous
functionality without issue 10 coming back.  Not unless you do some
special-case-to-end-all-special-cases kludge, and it still wouldn't
be waterproof.

This feature is also quite hazardous in my opinion: so you just added
foo.y as a dependency on foo.z, which is supposed to be made from foo.x.
Did you remember that the rule .y.z exists, because make surely does.
Oh you didn't add it?  Well, somefile did...  Not too far fetched for
a bigger project in my opinion.

> If there's more than one foo.c in the sources list, it takes the
> first, the same as if there's more than one foo.c on .PATH. And it
> does this before it searches .PATH at all, including the current
> directory, as one would expect.

I would not expect it to look at the listed dependencies, because
they're extra dependencies.  Why should the inference rule search look
at them?

And actually, it did not do it before looking at the path.
It searched and found foo.c, then it started looking at the dependency
list, noticed subdir/foo.c and made that one the implied source, but
_did_not_ remove foo.c from the list of dependencies.  And any target
whose filename part was foo.X (where there is a rule .X.o), not just
foo.c, would have been accepted.  And this was done even if subdir was
not in .PATH.

>  > Let me turn the tables on your example.  Let us assume that make allows
>  > the explicitly listed dependencies to become the implied source, instead
>  > of the one found by regular suffix rule search.  Let us also assume that
>  > the file subdir/foo.c contains something that is included by foo.c, so
>  > foo.o naturally depends on it.  How will I then write the Makefile?
>  > Exactly like you did, of course:
>  > 
>  >     foo.o: subdir/foo.c
>  > 
>  > Running make would compile foo.o from subdir/foo.c, which is obviously
>  > incorrect and completely unexpected, because foo.c exists.  Therefore,
>  > the extra dependencies should not affect the suffix rule selection.
>  > (I'd like to add that the overriding behavior was not documented in any
>  > way, I only found out about it after I read the source)
> 
> No, this is exactly what it used to do before it got broken, and this
> is exactly what the makefile asked for.

No it did not, not according to any interpretation of suffix rules I
have encountered, as I explained above.

> ...basically you're saying that pattern rules no longer work for these
> cases.

Yes.  But I am also saying that a minefield that was set up to trap
the unaware has been cleared.  Instead of make (undocumentedly) making
guesses from the dependencies, you now have complete confidence
on what will happen [Yes, the "not documented" aspect could've
been fixed].  No dependency, possibly added "behind your back", will
change what the implied source is.

> Note that I've been saying for years that .PATH and pattern rules for
> non-local files are bad and that makefiles shouldn't be written in
> this style;

I have to agree here.  However, this whole thing actually had nothing
to do with .PATH.  The problem here is that there seems to be a desire
for injecting a special case into suffix rule processing: if some listed
dependency could have become the implied source with a proper .PATH,
that source must be selected regardless of the actual setting of
.PATH or any other source found using the correct algorithm.

> This portion of the recent changes needs to be reverted.

If this actually turns out to be the consensus, I can reimplement this
feature on top of the new functionality.

-- 
Jarmo Jaakkola