Re: pkg_rolling-replace unexpected behaviour

[Chavdar Ivanov <ci4ic4%gmail.com@localhost>]

On Fri, 22 Nov 2019 at 19:19, Greg Troxel <gdt%lexort.com@localhost> wrote:
>
> Chavdar Ivanov <ci4ic4%gmail.com@localhost> writes:
>
> > Perhaps I am wrong; I haven't read the source code yet. My
> > understanding for pkg_rolling replace was that it finds and rebuild
> > first the deepest in the dependence tree packages, ending with the
> > leaves.
>
> Yes, except that it's more complicated than "deepest"
>
>   it only considers rebuilding things that are unsafe_depends or
>   mismatched (or rebuild=YES)
>
>   the initial calculation of the graph only looks at package
>   dependencies, not build dependencies
>
>   before building a package, pkg_rr finds build deps and adds them and
>   recalculates.  usually a few rounds of this happen before it actually
>   rebuilds anything, and then it stablizes.
>
>   you said "deepest", but in general there is a complicated graph that
>   is not a tree, in that there are multiple edges where a tree doesn't
>   allow them, and there can be disconnected parts.  pkg_rr does a tsort
>   and picks the first thing.  This is, absent bugs, guaranteed to be
>   something such that there is no package that should be rebuilt before,
>   but there are usually other packages that didn't tsort first that
>   would also be correct first choices.

Thanks for the good explanation. I can imagine the spidersweb of
interdependencies X the impossible-to-factor combinations of variables
in /etc/mk.conf X OS versions X don't know what else...

>
> > I just cvs updated my pkkgsrc tree and started 'pkg_rolling-replace
> > -uvk', a mere 10 days after my last go, so didn't expect that many
> > packages to get updated.
>
> It really depends.  a perl update needs a lot of rebuilds (but this is
> usually not exciting).
>
> > I noticed the second package to be rebuilt
> > was
> > ...
> > -rw-r--r-- 2 root wheel   32355261 Nov 22 16:36 gimp-2.10.14nb1.tgz
> > ...
> > Then it proceeded with perl-5.30.1 and some modules, and again
> > ...
> > -rw-r--r-- 2 root wheel   32354941 Nov 22 17:19 gimp-2.10.14nb1.tgz
> > ....
> >
> > I have noticed similar behaviour before, but this time I was able to
> > catch it on screen (previously the repeated build package files would
> > have been overwritten).
> >
> > Any idea why?
>
> No, and probably either you have found a bug or perhaps have a corrupt
> pkgdb of some kind.  I am happy to look into any data.  By now, it works
> ok the way I run it, but it surely has bugs that trip in other
> circumstances.
>
> This sort of smells like failing to get all the dependency info and thus
> replacing things too early, then having them marked unsafe_depends and
> then doing it again.
>
> I would suggest:
>

I always pkgclean before rolling-replace,

>   run "pkg_admin rebuild-tree" (now, since I am suspicious, and this is
>   a good thing periodically).  Let me know if it reports unresolved
>   dependencies.

but haven't done rebuild-tree recently, so perhaps there is something
there. It is still building, I'll try it  after it finishes (which in
my case may carry on for 3-4 days, as the laptop overheats and I have
MAKE_JOBS=1 ...)


>
>   (always) clean up all work directories, every time you do anything
>   with pkgsrc, and certainly before a pkg_rr
>
>   run "pkg_rolling-replace -uvk < /dev/null > RR.000 2>&1 &" so that you
>   can look later
>
>
> I also suggest using ccache so that extra rebuilds don't hurt that much.

Well, you did suggest it to me this January and I started using it,
but then I hit a problem - see
https://gnats.netbsd.org/cgi-bin/query-pr-single.pl?number=54134 . I
actually built then a new VM with pkgsrc running -current of the
moment just to repeat the problem building openjade with ccache and
without it and the problem was confirmed, so I stopped using ccache
after that. I might try it again to see if the problem remains.

--
----