Subject: Re: Simultaneous builds
To: NetBSD Packages Technical Discussion List <tech-pkg@netbsd.org>
From: Alistair Crooks <agc@pkgsrc.org>
List: tech-pkg
Date: 01/16/2002 13:34:20
Sorry, Greg, I don't have time to discuss this side issue right
now.  As I said, I'd be happy to see a well-discussed PR from you
on the subject of OBJMACHINE and OBJHOSTNAME, but mere contradiction
of anything I say isn't really the way to progress things.

Oh, and you are mistaken - _HOSTNAME is added to MAKEFLAGS, so that
submakes don't re-evaluate it. You can verify this by looking at
the output of PKG_DEBUG_LEVEL=2 during a normal make.

My apologies to other people reading this list for wasting their
time, I shall take my proposed changes to a more suitable forum.

Alistair


On Tue, Jan 15, 2002 at 03:56:38PM -0500, Greg A. Woods wrote:
> [ On Tuesday, January 15, 2002 at 20:31:42 (+0100), Alistair Crooks wrote: ]
> > Subject: Re: Simultaneous builds
> >
> > Funny, someone thanked me the other day for adding OBJHOSTNAME,
> > and said it was on his list of things to do.
> 
> I submit that whomever said they wanted such a feature didn't think
> through the problem completely.  It is entirely and totally unnecessary,
> even without the change I've suggested (the change I suggested is only
> necessary to support simultaneous builds on multiple hosts sharing the
> same pkgsrc source tree).  OBJHOSTMACHINE simply duplicates the ability
> already inherent in WRKOBJDIR (by virtue of the fact WRKOBJDIR can point
> to a place that will be unique on every build host, a place that can
> even be a symlink itself thus further making it totally adjustable and
> customisable).
> 
> > It is hardly
> > "complication".
> 
> It is indeed a complication, by the very definition that you've added
> more complexity to the expresisions which implement it, albeit a minor
> one in pkgsrc (though not nearly so minor in /usr/src).
> 
> On an otherwise virgin machine OBJHOSTNAME necessarily also causes at
> least one invocation of 'uname -n' too....  (one for every time make
> reads bsd.pkg.mk if I'm not mistaken)
> 
> >  There is even a way to do this without the
> > OBJHOSTNAME definitions that I added, but I thought it best to
> > codify practice.
> 
> Codifying bad practice doesn't help.....
> 
> > Please remember that we don't all share your ease
> > of modification of bsd.pkg.mk, familiarity with make(1), or ability
> > to use cvs (I'm not being facetious here - just pointing out that,
> > often, the intended audience is people who have been handed a
> > machine and told to administer it).
> 
> I understand, but that's all the more reason to avoid adding yet another
> unnecessary knob (i.e. one clearly duplicating existing capabilities)
> that makes understanding what's really necessary in a given scenario
> even more difficult.  If inexperienced people are expected to be able to
> drive this process then there needs to be one and only one way of doing
> any particular thing (and of course that one knob for each feature needs
> to be carefully and clearly documented too).
> 
> (you've no idea how long it took even me to figure out that OBJMACHINE
> was both unnecessary and even evil -- I spent many hours staring at the
> mess of intricate and inter-related expansions in bsd.obj.mk, the
> confusing and all-too-often incorrect documentation in bsd.README, and
> that's after doing a whole bunch of experiments to try to see what
> effect carefully calculated changes would have!)
> 
> > Moreover, I could have done
> > it without changing bsd.pkg.mk at all, by advising everyone to
> > set WRKDIR_BASENAME for themselves, but I prefer a handy knob for
> > these kind of things.
> 
> The only knob that's necessary is the existing WRKOBJDIR setting.  It
> can even be "hard-coded" in a shared mk.conf file so long (and MUST be
> at least consistently defined on all machines sharing a pkgsrc tree).
> 
> > > ***************
> > > *** 2506,2514 ****
> > >                 fi;                                                     \
> > >         fi
> > >   .  ifdef WRKOBJDIR
> > > !       -${_PKG_SILENT}${_PKG_DEBUG}                                    \
> > > !       ${RMDIR} ${BUILD_DIR} 2>/dev/null;                              \
> > > !       ${RM} -f ${WRKDIR_BASENAME}
> > >   .  endif
> > >   .endif
> > >   
> > > --- 2597,2603 ----
> > >                 fi;                                                     \
> > >         fi
> > >   .  ifdef WRKOBJDIR
> > > !       -${_PKG_SILENT}${_PKG_DEBUG}${RMDIR} ${BUILD_DIR} 2>/dev/null;
> > >   .  endif
> > >   .endif
> >
> > If you'd like to wrap these changes together under a suitable definition,
> > and send-pr them, I'm sure we'd look at them.
> 
> I could, but until/unless you "buy" into my assertion that OBJHOSTNAME
> is a totally unnecessary knob (which in context of read-only source
> partitions is equally as evil as OBJMACHINE) and agree that WRKOBJDIR
> should be set to a common target on all systems sharing a pkgsrc tree,
> my change above is irrelevant and useless to anyone but me (who will
> never in a lifetime ever set OBJHOSTNAME or OBJMACHINE (again) :-).
> 
> Meanwhile if you do agree with me on this topic then the need for the
> above change is very self obvious and necessary in conjunction with
> allowing simultaneous builds on separate hosts.
> 
> Indeed my change may even be necessary with OBJHOSTNAME, but disliking
> OBJHOSTNAME as much as I do I'll leave the reason unsaid....  :-)
> (anyone using OBJHOSTNAME and doing enough simultaneous builds on enough
> machines at the same time will inevitably discover it  :-)
> 
> > Anyway, your mail seems to object to a small part of the changes
> > I'm proposing, which went in two days ago, which other people want,
> > and which is a convenient knob.
> 
> Indeed that's all I'm complaining about.  I ask anyone who disagrees
> with me to do so by showing a concrete example where a common shared
> setting of WRKOBJDIR is not more than sufficient to allow for shared use
> of pkgsrc (either with my change above, or without also allowing for any
> simultaneous builds in any way).
> 
> > You have not commented on the main
> > part of the changes at all - the shlock ones, and the splitting of
> > the do-extract target.
> 
> I'm personally not really concerned about allowing simultaneous builds,
> just so long as the attempt to do so doesn't adversely affect things
> (especially the overall performance of "make" in pkgsrc).  While I agree
> with David that there are many times when better use of resources can be
> attained by doing simultaneous builds in pkgsrc, I'm not likely to try
> doing so myself any time soon.  I don't even use 'make -j' -- 'cc -pipe'
> is as far as I go to building things "simultaneously".  :-)
> 
> -- 
> 								Greg A. Woods
> 
> +1 416 218-0098;  <gwoods@acm.org>;  <g.a.woods@ieee.org>;  <woods@robohack.ca>
> Planix, Inc. <woods@planix.com>; VE3TCP; Secrets of the Weird <woods@weird.com>