bin/51344: make(1) should have a way to cope with transient intermediate files

[dholland%eecs.harvard.edu@localhost]

>Number:         51344
>Category:       bin
>Synopsis:       make(1) should have a way to cope with transient intermediate files
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    bin-bug-people
>State:          open
>Class:          change-request
>Submitter-Id:   net
>Arrival-Date:   Mon Jul 18 22:30:00 +0000 2016
>Originator:     David A. Holland
>Release:        NetBSD 7.99.30
>Organization:
>Environment:
System: NetBSD macaran 7.99.30 NetBSD 7.99.30 (MACARAN) #39: Tue Jun 7 15:26:15 EDT 2016 dholland@macaran:/usr/src/sys/arch/amd64/compile/MACARAN amd64
Architecture: x86_64
Machine: amd64
>Description:

Suppose you have a source file of some kind, and a multi-stage build
process that constructs one or more layers of intermediate files
before generating final output.

For example, something like

  foo.a -> foo1.b -> foo1.c -> foo1a.d
                            -> foo1b.d
        -> foo2.b -> foo2.c -> foo2a.d
                            -> foo2b.d
        -> foo3.b -> foo3.c -> foo3a.d
                            -> foo3b.d

Suppose also that the intermediate files are both large and expensive
to generate, so that you don't want to build any one of them more than
once at any given time but also don't want to keep them around from
one day to the next.

Currently there is no way to do this in make. Your options are:
   (1) Write one big recipe that builds all the output files at once;
       but this sucks because you throw away all your parallelism.
   (2) Write a recipe that builds one of the final outputs by running
       all the stages together; this recovers the parallelism but
       sucks because it rebuilds the expensive intermediates over and
       over again.
   (3) Materialize the intermediate files in the make rules and write
       recipes for one step at a time, which avoids doing extra work
       but sucks because if you delete any of the intermediate files
       make thenceforth assumes it needs to rebuild them and anything
       that came from them.

There should be a way to mark targets such that if they don't exist
they are only out of date if something that depends on them is also
out of date. As far as I know no such tag exists.

Note that this is closely related to (but not quite the same as) the
problem of depending on .h files that go away. I have argued in the
past that that problem needs a principled solution, and this problem
is another example of why.

I am not sure what that principled solution is.

Note that given disk sizes this scenario is much more relevant when
using make to drive benchmarking or testing than for software builds,
but it is also potentially applicable to things like .c files
generated from .y files.

Historically some make dialects have implicitly done something like
this when using chains of suffix rules, but (a) AFAIK not reliably and
(b) suffix rules per se are irrelevant to the issue.

>How-To-Repeat:
n/a

>Fix:
needs discussion.