Re: make: avoid breaking backwards compatability

[David Holland <dholland-tech%netbsd.org@localhost>]

On Thu, Sep 04, 2014 at 02:47:40AM +0300, Jarmo Jaakkola wrote:
 > >  > > foo.o: subdir/foo.c
 > >  > 
 > >  > No, this works exactly as suffix rules should work, in my opinion.  The
 > >  > problem here is not that subdir/foo.c is overridden, it is that you're
 > >  > expecting some random extra dependency to become the implied source in
 > >  > a suffix rule.
 > > 
 > > I think you misunderstand the way .PATH and pattern rule searches are
 > > supposed to work.
 > > 
 > > If I write a .c.o rule, and ask for a .o file, make is supposed to
 > > find the corresponding .c file in some directory. So if I'm trying to
 > > build foo.o, it's looking for foo.c. If "subdir/foo.c" is already on
 > > the list of sources, that's not some random extra dependency; that's
 > > the file named foo.c that it's looking for. So it's found it and it
 > > doesn't need to search any further.
 > 
 > Yes, a corresponding file, which is a file with the same path, but
 > a different suffix.

No, it's a file with the same basename (in potentially any directory)
and a different suffix.

That's the point of .PATH. This is something that (IIRC) POSIX doesn't
have at all, even as VPATH, so it's no surprise that the description
of suffix rules in POSIX doesn't support the feature or include the
hooks wanted to use it successfully.

You might find it helpful to look at the (admittedly confusing)
documentation and behavior of the :P variable modifier. Before the
changes. (I suspect that this is now broken too, in at least some
cases.) This will give a better idea about how make thinks about files
that were found via .PATH.

 > So if you want subdir/foo.c to be the file used,
 > you either request subdir/foo.o, add subdir to .PATH and request for
 > foo.o _without_ there being foo.c in the current directory or write
 > an explicit rule.  There are three perfectly good ways to achieve
 > this, why would anyone need a fourth where make has to guess what they
 > mean?  If you're going through the trouble of writing the supposedly
 > implied dependency explicitly, you might as well write the whole rule.

Those are not perfectly good alternatives: the first isn't equivalent,
the second can create problems if foo.c isn't the only such file, the
third requires duplication of logic.

There is no need for make to guess what was meant; it's very clear. It
just isn't the behavior you expected.

 > And subdir/foo.c is exactly an "extra dependency".  There are two kinds
 > of dependency lines: those which have associated commands (explicit
 > rules) and those which do not (extra dependencies).  The semantics of
 > an extra dependency, in my opinion, is
 >     "use whatever rule you otherwise have for this target, but add these
 >     sources as dependencies too"
 > not
 >     "use whatever rule you otherwise have, except use some completely
 >     different one if under some other configuration one of these sources
 >     could have been the implied source, because I'm a lazy bugger and
 >     can't be bothered to do this correctly".
 > And in this case the "whatever rule" is an inference rule, where
 > the implied source is foo.c.

While it's possible to distinguish those two kinds of dependency
lines, the sources listed by them are not distinguished. There's
nothing magical about the one (or ones) that have commands attached.

Try

   abc: def
   abc: ghi
        echo $(.ALLSRC)
   abc: jkl
   def ghi jkl: ;

It prints
   def ghi jkl

not
   ghi def jkl

or anything else. The ordering is significant. This is essential to be
able to use .WAIT (something else POSIX doesn't have) sanely.

AIUI, if you use $< in this make will now pick out "ghi" instead of
"def"; I don't particularly like this behavior, but I'll agree that
it's consistent with what people mostly expect. (What does it do if
you use :: rules?)

 > So, why should subdir/foo.c be the file we're looking for?  I have never
 > seen it mentioned anywhere that a listed dependency should affect
 > the implied source.  I must admit that I have experience only from
 > NetBSD's make, gmake and reading the POSIX standard.

Because it's looking for sources. If a matching source is already
provided, why should it search for another one? Then it'll have two
and it has to guess, or in this case assume, which one you meant to be
the "real" one.

As to why it would be provided, one reason (as we found when the build
broke repeatedly) is to indicate specifically which of several
potential matching sources to use. Otherwise there's no way to do this.

 > Admittedly, gmake is a different implementation, but they do it "my way"
 > too: the listed extra dependencies have no bearing on the implicit
 > source.  The implied source is found primarily from the current directory
 > and secondarily from VPATH.  For the following example, assume foo.c,
 > dir/foo.c and dir2/foo.c exist.
 > 
 >     foo.o: dir/foo.c
 > 
 >     $ gmake foo.o
 >     cc    -c -o foo.o foo.c
 > 
 >     VPATH = dir2/foo.c
 >     foo.o: dir/foo.c
 > 
 >     $ gmake foo.o
 >     cc    -c -o foo.o foo.c
 > 
 > When done without foo.c, you get these two:
 >     gmake: Nothing to be done for `foo.o'.
 >     cc    -c -o foo.o dir2/foo.c
 > 
 > As you can see, dir/foo.c is purely an extra dependency, it is not
 > eligible to become the implied source.

...whatever that means. There are just sources and targets, you know...
one of the sources is $(.IMPSRC) but it's not otherwise special.

 > And also admittedly, we're not slaves to POSIX, but this is exactly
 > the same as the previously mentioned issue 10.  In that issue
 > the suffix of the source is also changed by the extra dependency, but
 > it is still basically the same thing.  It just happens to change
 > the entire rule, not just the implied source!

No, it doesn't "change" the suffix of the source, it chooses a
different rule to use.

 > 
 >     .SUFFIXES: .a .b .c
 >     .a.c: ; ...
 >     .b.c: ; ...
 >     test.c: test.b
 >     test.a:
 >     test.b:
 > 
 > .a.c should be used, not .b.c.  This is completely within the realm
 > of POSIX, and I think it would have been specified that the dependencies
 > must affect the implied rule / source selection process, if it really
 > should work like that.

Why? POSIX doesn't have .PATH and there's no notion of the source of a
suffix rule being anywhere but the current directory. So of course it
doesn't have the apparatus to support it.

 > Also note that there is no way of implementing the previous
 > functionality without issue 10 coming back.  Not unless you do some
 > special-case-to-end-all-special-cases kludge, and it still wouldn't
 > be waterproof.
 >
 > This feature is also quite hazardous in my opinion: so you just added
 > foo.y as a dependency on foo.z, which is supposed to be made from foo.x.
 > Did you remember that the rule .y.z exists, because make surely does.
 > Oh you didn't add it?  Well, somefile did...  Not too far fetched for
 > a bigger project in my opinion.

...you realize that this is a complete nonissue as no portable
makefile can depend on the order the builtin suffix rules are defined?
(Not to mention a completely contrived example...)

 > > If there's more than one foo.c in the sources list, it takes the
 > > first, the same as if there's more than one foo.c on .PATH. And it
 > > does this before it searches .PATH at all, including the current
 > > directory, as one would expect.
 > 
 > I would not expect it to look at the listed dependencies, because
 > they're extra dependencies.  Why should the inference rule search look
 > at them?

Because they're sources and it's looking for sources?

 > And actually, it did not do it before looking at the path.
 > It searched and found foo.c, then it started looking at the dependency
 > list, noticed subdir/foo.c and made that one the implied source, but
 > _did_not_ remove foo.c from the list of dependencies.

That would be a bug.

 > And any target
 > whose filename part was foo.X (where there is a rule .X.o), not just
 > foo.c, would have been accepted.  And this was done even if subdir was
 > not in .PATH.

Target? I think you mean source?

It should also not match foo.S in the sources if processing a .c.o
rule, only a .S.o rule. If it was doing that, that would be a bug too.
It should only match in the .S.o rule.

...some testing with a make from before the changes shows that it does
exactly that. Given

     .SUFFIXES:
     .SUFFIXES: .c .S .o

     all: foo.o

     .c.o:
             echo compile C: $<
     .S.o:
             echo assemble: $<

     foo.o: foo.S

if both foo.c and foo.S exist, the old make assembles foo.S and the
new one compiles foo.c. Which, bluntly, is exactly what one wants --
there is no non-contrived case where it's reasonable for the 'extra
dependency' to mean that I wanted to compile foo.c and I know that
foo.c includes foo.S. If I'd written an 'extra dependency' on a .h
file, there wouldn't be a .h.o rule to try to compile the header file
on its own, because that isn't a sensible thing to do.

 > >  > [...]
 > > 
 > > No, this is exactly what it used to do before it got broken, and this
 > > is exactly what the makefile asked for.
 > 
 > No it did not, not according to any interpretation of suffix rules I
 > have encountered, as I explained above.

You have encountered it now; this is how I understand BSD make to have
been intended to work.

 > > ...basically you're saying that pattern rules no longer work for these
 > > cases.
 > 
 > Yes.  But I am also saying that a minefield that was set up to trap
 > the unaware has been cleared.  Instead of make (undocumentedly) making
 > guesses from the dependencies, you now have complete confidence
 > on what will happen [Yes, the "not documented" aspect could've
 > been fixed].  No dependency, possibly added "behind your back", will
 > change what the implied source is.

No, instead what you've done is castrate the functionality so it's no
longer possible to explicitly indicate what source to use with a
suffix rule. Instead make splatches ahead under the assumption that
what you wrote is not what you intended, ignores what you said, and
does something you didn't want. And there's no way to prevent it.

I don't think that this is a step forward. Nor is it 'correct'.

 > > Note that I've been saying for years that .PATH and pattern rules for
 > > non-local files are bad and that makefiles shouldn't be written in
 > > this style;
 > 
 > I have to agree here.  However, this whole thing actually had nothing
 > to do with .PATH.

It has a lot to do with .PATH because the point at which this behavior
becomes needed is when .PATH contains multiple candidate sources and
you want to explicitly specify which one to use. As I said above,
without .PATH it's unnecessary because there can be only one possible
$(.IMPSRC) for any given suffix rule and any given target.

However, applying this logic only for sources found from .PATH would
be horribly messy and confusing, so that isn't a reasonable choice.

 > > This portion of the recent changes needs to be reverted.
 > 
 > If this actually turns out to be the consensus, I can reimplement this
 > feature on top of the new functionality.

That doesn't necessarily seem like a good plan.

-- 
David A. Holland
dholland%netbsd.org@localhost