Re: Issues with larger numbers with -j

[Marc Espie <espie%nerim.net@localhost>]

On Wed, Jun 14, 2017 at 08:51:45AM -0500, Jason Bacon wrote:
> On 06/14/17 04:20, Marc Espie wrote:
> >make isn't really an appropriate build tool for larger machines with
> >lots of cpu.
> >
> >best-of-the-breed has been ninja for years. I wish it were in more common
> >use, as some big projects that support it (chrome) tend to not always use
> >it, so they are missing dependencies as well.
> >
> >Anyway, ninja, is really good at representing complex dependency graphs
> >over a whole project, totally avoiding any kind of "recursive make"
> >behavior while eschewing extra complexity from some of its competitors.
> >
> >Ninja *is* a next gen build system. Started from scratch, avoided all the
> >mistakes that are going to plague make until the end of time.
> >
> >I don't know where pkgsrc stands these days, but changing generators to
> >ninja where possible (cmake -g Ninja for instance) yields huge benefits
> >in terms of parallelism for large software.
> >
> >Sometimes you will run into dependency chains that need fixing and
> >upstreaming... the more projects looking into this the better.
> 
> 30 years of software development teaching have made it clear to me that the
> tools are not the problem, the developers are.
> 
> A disciplined software engineer will efficiently produce great products with
> any reasonably modern language, OS, and build tools.  A sloppy one will
> produce garbage no matter how awesome the tools they use.
> 
> Learn how to use make properly and it will serve you well for almost any
> project.  In my view, if a project is too complex for make, it should be
> broken up into simpler modules, and dependencies should left to a package
> manager like pkgsrc.

Wrong. There are modern problems which are mostly unsolvable for various
reasons.

One being that all modern makes have different sets of bugs which trigger
under various circumstances.  It's at best very difficult to have lots
of files you will compile with different flags.  Also difficult to produce
object files in a consistent manner in different directories.
Stuff that produces dependencies in a dynamic way is another problem.
Rules that actually produce several different files, or dynamic names
are also an issue.

The other problem is scale. Modern machines have from 16 to 128 cpus and
more.  Which means that it becomes highly desireable to scale up to 128
simultaneous build processes.   Recursive make is out (well, it was a bad
idea to start with).  So are a lot of other tools.   Ninja can still 
cope as long as you give it a full set of dependencies.

Just because it's possible to do tricky things with make doesn't mean you
should do so...