Subject: Reducing bsd.pkg.mk recursion
To: None <tech-pkg@netbsd.org>
From: Todd Vierling <tv@duh.org>
List: tech-pkg
Date: 11/11/2004 22:48:13
There's a lot of it.  A whopping caboodle of it.  The problem is, it's
rather expensive to call ${MAKE} recursively (in some cases, a couple levels
deep) within a package when there's really no need to do so.

I have some changes in work which already improve bsd.pkg.mk build time
appreciably, especially on platforms where fork() is not cheap.  It's not
just fork(), though; it takes nonzero time for make to go reparse and
recalculate everything it just finished calculating in the package Makefile
the last time through.

I'll post the diffs for review soonish when they are reliable.  I'd like to
point out the specific points I'm addressing before diving fully into this,
in case I'm missing something here.  Please comment freely.

* PKG_PHASE: This whole concept forces a layer of ${MAKE} recursion.  The
  only things that use this, AFAICT, are bits in the wrapper, bl3, and
  reduce-depends code to avoid calculating things when the user hasn't
  requested a build mode requiring them.

  I believe these should all be possible to handle through proper use of
  ".if make()" conditionals rather than recursing just to set PKG_PHASE.
  The "real-*" targets could then be collapsed right onto their respective
  cookie timestamp targets.

  To accomplish this, I plan to add bits to bsd.pkg.mk to detect the
  "maximum phase" of the package build requested by the user, so that
  anything previously needing PKG_PHASE can simply check if a certain phase
  *would* be executed in the current make instance.  This should provide
  equivalent functionality all within a single make invocation.

* Many following targets are recursed, but could be collapsed into a
  sequence of subtargets for a certain build phase.  In some cases these
  would involve slight shellcode modifications (and some of those are
  actually simplifications).  In other cases it may involve splitting a do-*
  target, which is currently recursing, into several separate targets
  instead.

  Some candidates for splitting or otherwise de-recursing, [their parent
  targets in square brackets]:

  [fetch] check-vulnerable
  [fetch] batch-check-distfiles
  [real-su-package] package-links, delete-package (shellcode block)
  [package-links] delete-package-links
  [delete-package] delete-package-links
  [install] pre-install-script
  [install] pre-install
  [install] do-install
  [install] post-install
  [install] ${PLIST}
  [install] post-install-script
  [install] do-shlib-handling
  [install] fake-pkg
  [install] check-shlibs
  [*] show-shlib-type
  [do-su-package] do-su-install (allow one su instance to do both)

-- 
-- Todd Vierling <tv@duh.org> <tv@pobox.com>