Re: make: dynamic scaling of maxJobs ?

[David Holland <dholland-tech%netbsd.org@localhost>]

On Wed, Aug 14, 2013 at 02:45:36PM -0700, Simon J. Gerraty wrote:
 > > > I renamed the operative variables so that they can be set in the
 > > > environment.
 > > > This makes it easy to impose this strategy on old branches that know
 > > > nothing of it, this is likely the most useful approach.
 > > > Only level 0 make will pay attention - since it would be a bad idea
 > > for
 > > > more than one instance of make to attempt to adjust the token pool.
 > >
 > >It still seems to me that this is a hack and neither the right
 > >interface for the long term nor something we want to support in the
 > >long term.
 > 
 > Using tools to compose solutions is the unix way.
 > A tool that does one thing - report how many concurrent jobs would be
 > good right now fits very well.
 >
 > Allowing another tool - make - to leverage said tool fits well too.

Sure. Except that it's an entirely specious argument. You could make a
similar argument about e.g. DNS and ssh: instead of "ssh hostname",
you'd run "ssh -N nslookup hostname": you have a tool that does one
thing (look up hostnames) and you're allowing another tool (ssh) to
leverage it. This argument is bogus, and so is yours.

Meanwhile, do you really intend that every time you run make you need
to type in "make -j /home/me/bin/makejobs.sh"? Is that a good user
interface, composition or not?

It seems to me that "how many parallel jobs should I be starting?" is
a system-level question whose answers should be provided by a system
service. I pointed this out once already. You said it wouldn't be
portable that way, but that's nonsense. On platforms that don't have a
native facility you can provide the same logic you propose to put in
this external script -- you can even put it in an external script if
you like, although I think that's kind of silly.

Also, I think making it work only for the special case of toplevel
whole-project makes is not worthwhile. The job tokens infrastructure
should be improved so it works for real builds too; builds of src, for
example, or of some of the larger pkgsrc packages. It's great if it
works for your special case, but it's not much use to anyone else that
way.

 > >And (again? I forget, I've been kind of distracted lately) what about
 > >other programs that also support the -j idiom?
 > 
 > I'm spacing; which other tools/apps that support -j are you thinking
 > of?  and what about them?  If they take a -j arg you can always run them
 > as -j `maxJobs.sh`.  If they run long enough to matter you might want to
 > enhance them....

There's pigz, although it apparently uses -p instead of -j, and
there's been talk of adding -j to sort. And there are other programs
(both in and out of base) that would benefit from being parallelized
if the infrastructure to make it transparent existed.

-- 
David A. Holland
dholland%netbsd.org@localhost