Extracting versions from pkgsrc tree taking hours - how to address?

[David Brownlee <abs%absd.org@localhost>]

There are various tools which need to extract PKGNAME or similar
information from a pkgsrc tree, typically to compare against installed
versions or a set of binary packages.

The canonical way to do this is to run "make show-var
VARNAME=PKGNAME", which for a sample package on my Thinkpad T500
(pkgtools/lintpkgsrc) takes around 0.65 seconds.

For a full pkgsrc+wip tree (~14000 packages) this process would
probably take around 2 hours 45 minutes.

Some tools try to mitigate this by embedding partial Makefile parsers
(lintpkgsrc is a particularly gregarious example) and produce a result
in around 4 1/2 minutes (somewhat over 30 times faster), though the
knotty perl code that does this is difficult to maintain and would
make the world a better place by its absence.

Aleksey has a pkg_micro_src_summary which can extract PKGNAME and
similar in a subset of cases.

Caching information about the pkgsrc tree could help, but troublesome
given a pkgdir can generate different PKGNAMEs based on just about
anything from installed packages, mk.conf, environment variables to
phase of the moon (*).

Some questions/thoughts:

summary information format:
- If anything does generate summary information for pkgsrc tree is
there any reason not to use the same pkg_summary format used for
binary packages
- If not, I'll at least update lintpkgsrc to ingest and
excr--...export in that format

generating summary information:
- Should we have a specific in tree tool for generating pkgsrc summary
information, which longerterm lintpkgsrc and other tools should be
depending upon (even if they only use the output directly and there is
no cache).

speeding up parsing:
- Would it make sense to adjusting pkgsrc to help quick mechanical
parsing, for example if we moved to a default of defining PKGNAME and
deriving DISTNAME from PKGNAME,
- then the majority of pkgsrc Makefiles could be "quick parsed" on the
rule "contains PKGNAME= with no $', the remaining being passed through
make as normal
- future options involve automatically handling python versions, but
again this is easier if there is one definitive tool in tree for
extracting summary information from pkgsrc Makefiles..


(*) For reference, using time/phoon
PYTHON_VERSION_DEFAULT!=phoon -l 1|awk '/New/{print 25}/First/{print
26}/Full/{print 27}/Last/{print 28}'