Re: MAKE_JOBS_SAFE=[number]

[Marc Espie <espie%nerim.net@localhost>]

On Sun, Sep 23, 2012 at 06:52:30AM +0000, David Holland wrote:
> On Mon, Sep 17, 2012 at 04:01:29PM -0400, Julio Merino wrote:
>  > >> Is it possible to allow MAKE_JOBS_SAFE=[number] in addition to YES/NO?
>  > >
>  > > I don't think this makes sense. If -j16 fails, it is just a question of
>  > > scheduling whether -j8 fails as well.
>  > 
>  > Agreed.  If the software fails to build with any degree of
>  > parallelization greater than 1, the Makefiles are bogus.  As with
>  > threading bugs, it's just a matter of time before the problem appears.
> 
> It's not that simple. It's not like thread scheduling in C code, where
> the timeslices and variances caused by the scheduler are much larger
> than individual actions or blocks of code.
> 
> In a build, the amount of time each target takes may scale with your
> processor speed (or not) but is mostly going to be roughly the same
> each time and (more importantly) roughly the same each time with
> respect to how long other targets take. Also, make dispatches the
> targets in the same or nearly the same order each time; it is not
> random.

That's not quite true. It massively depends on your hardware and setup,
processor speed, number of cores, IO speed may vary widely.  In my
experience, nearly clean Makefiles may suddenly start to break when a new
setup appears.

Also, in OpenBSD, we introduced a debug option in our make to help parallel
makefiles break, by randomizing the order of the queue of buildable
targets...

> Suppose you have a target X, which is the first thing in the build,
> and another target Y that requires X to have been made but doesn't
> enforce this in make, and 10 other targets T0..T9 in between them. If
> you build with -j12, then X, T*, and Y will all be started at once,
> and the build will fail. If you build with -j11, then X and T* will be
> started at once, and the build will fail if and only if one of the T*
> targets finishes before X. If you build with -j10, two of them have
> to, and so on. If all the targets are roughly the same length
> (e.g. they're all ordinary compiles) or X is shorter than average, as
> is common with this kind of target (yacc runs, header generation
> scripts, etc.) then building with -j8 will pretty much always succeed.

Actually, yacc runs are often a problem. A lot of people out there do not
understand make, and they think that:

file.c file.h:  file.y
        some yacc rule

is going to work.

Not so, since that expands to two separate targets, and yacc may very well
be generating file.c while file.h is being used elsewhere. Oops.

> A better solution is to fix the offending makefiles, of course. But
> this isn't always easy, especially for the more regrettable kind of
> automake spew that depends heavily on gmake.

Well, more than that, small autohell packages will spend over half their time
in configure, a lot of the time.  Of course, since autoconf is so poorly
designed, the idiots have no way to figure out which test depends on what,
otherwise they could possibly parallelize the tests themselves...