Re: lintpkgsrc important changes RFC (Was: lintpkgsrc 'Out of memory!' error)

[David Brownlee <abs%netbsd.org@localhost>]

On 7 September 2012 16:34, Aleksey Cheusov <cheusov%tut.by@localhost> wrote:
>
> JFYI. In order to speed up Makefile processing I also use "heuristics"
> that partially supports bmake's language. If it fails, bmake is used
> as a fallback.
> See pkgtools/pkg_summary-utils.
>
> pkg_micro_src_summary is able to fetch only "simple" variables
> like PKGPATH or PKGNAME defined
> in Makefile or nearby but not in mk/* or bl3.mk.
> It is used by 'nih status -s' by default.
>
>    0 lintpkgsrc>time -p bmake show-vars VARNAMES='PKGNAME PKGPATH'
>    lintpkgsrc-4.84
>    pkgtools/lintpkgsrc
>    real 0.60
>    user 0.29
>    sys 0.16
>    0 lintpkgsrc>time -p pkg_micro_src_summary -f PKGNAME,PKGPATH
> pkgtools/lintpkgsrc
>    PKGNAME=lintpkgsrc-4.84
>    PKGPATH=pkgtools/lintpkgsrc
>
>    real 0.22
>    user 0.11
>    sys 0.12
>    0 lintpkgsrc>
>
> In this example, ~3-times faster.

On my ThinkPad my bmake show-vars time for lintpkgsrc is within 10% of yours.

"time lintpkgsrc -E /tmp/dbdump" on pkgsrc+wip  (modulus various
bleating about various packages in wip) takes:

lintpkgsrc -E /tmp/a  187.22s user 51.76s system 90% cpu 4:23.68 total
for 14066 packages

Calling "bmake show-vars" on every package would probably  take
roughly 2 hours 45 min on the same system (ignoring the fact that
"make show-vars for packages with many dependencies or much other
complexity take much longer).

Thats lintpkgsrc giving a >30 times speedup over "bmake show-vars" ,
but I agree a better longterm approach is probably to have some
mechanism to extract the values from "simple packages" and to fall
back to bmake show-vars" for more complex ones.

I also think we should adjust the standard Makefile template to make
this easier... - will start a different thread on tech-pkg@ to
continue that...