Re: wip/pkgchkxx: a complete reimplementation of pkg_chk and pkg_rolling-replace

[Greg Troxel <gdt%lexort.com@localhost>]

PHO <pho%cielonegro.org@localhost> writes:

> I just created a new package wip/pkgchkxx. This is a complete
> reimplementation of pkgtools/pkg_chk and pkgtools/pkg_rolling_replace
> in C++17.

WOW!  As much as I like to hate on C++, this is very cool and I'm
looking forward to trying it.

I wonder -- why did you decide on C++17 vs 14 vs 11?  This is a
foundational tool for package management, and I'm not really clear where
C++17 isn't ok with the base system compiler.  Is it just ancient LTS
(RHEL/CentOS)?  Also NetBSD 8?

We currently have a pkglint problem, where the main pkglint needs go,
and there is deficient pkglint for non-go machines.  That's different
in a key way, which is that pkglint is only really needed where people
do pkgsrc development, and pkg_rr is needed on any machine where people
build packages.

> When you update your pkgsrc tree on a running system and you have like
> 200 outdated packages to update, the chance of pkg_rr -u completing
> without any failures is somewhere between 5% and 0%. You usually have
> to fix broken things here and there, re-run pkg_rr repeatedly, and
> waste your time staring at "Checking for xxx packages (xxx=YES)" for
> several seconds each time you restart pkg_rr. I wanted to change that.

True, but the biggest time sink is pkg_chk for -u.

> * pkgchkxx(8) takes options fully compatible with pkg_chk(8).

You didn't say, and probably this is future, but the biggest problem
right now in the pkg_rr/pkg_chk world is failing to deal with
multiversion packages like pyNN-foo, where it basically errors if NN is
not PYTHON_VERSION_DEFAULT.  That requires pkg_chk to output both
PKGPATH and some version kv objects, and carry that around as the thing
that needs doing, rather than just PKGPATH.

> * pkgrrxx(8) takes options fully compatible with pkg_rolling-replace(8).
> * Unlike pkg_rr, you can run pkgrrxx as a non-root user. It makes use
>   of ${ROOT_CMD} whenever it needs a root access.

Amusingly I have an extra "sudo" in one place in my installed copy.  I
should get that committed, as I think we're going to be keeping the
shell implementation for a while, especially depending on how burdensome
the language requirement is.

> * "pkgrrxx -u" runs roughly N times faster than "pkg_rolling-replace
>   -u" where N is the number of CPUs you have.

I assume you mean mean the wall clock time of pkg_rr itself, not builds
that it calls.

> * "pkgchkxx -aur -b" runs 11x faster than "pkg_chk -aur -b" when
>   pkg_summary file is available.

I don't follow this.  I didn't think pkg_-chk used pkg_summary, and
pkg_summary is generally collected metadata for a bunch of binary
packages.  But this is about comparing installed packages to the source
tree.

Or is there some "make me a pkg_summary that describes what the results
of building every package would be"?

> * "pkgchkxx -l" runs 185x faster than "pkg_chk -l" when pkg_summary
>   file is available, and runs 24.8x faster when it's unavailable (and
>   needs to scan archives).

"archives"?

> [be afraid and backup]

Fair enough, but is there anything that this does other than "make
replace package clean" and pkg_admin set?  It really should be pretty
low risk.

> Also do not question the value of faster pkg_rr when the most
> time-consuming part of its job is to actually rebuild packages!

I don't follow this.

If you mean "even if the most time is the actual rebuild, it's still a
big improvement for pkg_rr to be faster", I agree.  Indeed for a pkg_rr
run that is going to take 24 hours, it doesn't really matter.  But it
does matter for things that are going to take 5 minutes.

If you mean something else, explain.

The other thing that is wrong with pkg_rr (and always has been, but it
has been increasingly painful) is that many packages have lots of
build-only dependencies.  pkg_rr goes over the installed set and
extracts the dependencies (which are recorded) but doesn't extract the
build deps (which need a make invocatino) until building each one.
However it would probably be better to get the TOOL_DEPENDS and
BUILD_DEPENDS dependencies up front, and this can be parallized.

Also in the brave new world of cross, it would be nice to be able to
somehow DTRT about TOOL/BUILD when running pkg_rr for a cross build.  I
have so far not contemplated or tried.