Re: Script pkg_tsort

[tlaronde%polynum.com@localhost]

On Mon, Apr 15, 2019 at 07:46:03PM -0400, Greg Troxel wrote:
> 
> >> pkg_rolling-replace does a bit of this.  If you end up packaging it, it
> >> might make sense to depend on it instead.
> 
> What I meant is that if your script, which figures out all the
> dependencies, could perhaps be called by pkg_rr instead of what pkg_rr
> is doing.  Basically decomposing it into simpler tools as you suggest.
> 

OK, so for this we are on same wavelength.

> > But pkg_rolling-replace ignores +PRESERVE.
> 
> It does not pay attention to +PRESERVE, but it doesn't call pkg_delete.
> It simply calls make replace, which is not documented to fail on
> +PRESERVE, and I don't think should fail in that case.
> 
> Are you intending to avoid doing a replace of things like pkg_install?
> 

No. The tsort does take into account +PRESERVE the goal being that the
listing finally produced can be taken as is for further action and there
will be no surprise with a package being replaced that was not intended
to be. For example, with a database manager (PostgreSQL to name one),
updating to an incompatible major version can render your data
inaccessible. You need to plan the dumping of the bases before switching
to a new version. It should not be updated, involontarily, as a side
effect.

> > I don't know either if pkg_rolling-replace takes pkg-vulnerabilities
> > into account (when not doing things but telling what it would have
> > done).
> 
> It does not.  Are you thinking about the situation where an installed
> package is not vulnerable but the new one is?  Or both the old and new?
> Or just old?  I have more or less viewed updating to a newer version as
> always neutral or positive, although I realize there could be a
> pathological case.
> 
> What is your proposed course of action (for the human who interprets the
> pkg_tsort output) relative to a list of things to replace and
> pkg-vulnerabilities?
> 

In the output as for now, there are the two fields: status of installed
(OK or VULNERABLE), status of the pkgsrc version (OK, VULNERABLE or
UNKNOWN when there is no such package in the pkgsrc version---then the
new "version" is UNAVAILABLE; I had the surprise to discover that this
is the case for some of my installed packages...).

> >
> > You have probably to call make since there is no simple way to parse the
> > Makefile, with all the inclusions and the variable expansions---and 
> > this is what takes a huge amount of time---I don't know for the pkgsrc
> > version of the package if I could simply take the distinfo file without
> > bothering with make(1) and the Makefile.
> >
> > But for the dependencies, one can perhaps avoid calling make by sed'ing 
> > the Makefile for the buildlink inclusions and then looking for the 
> > inclusions if this has not already be done for the packages (example:
> > one starts with a list of packages; the package itself is PROVIDE(d);
> > the dependencies are REQUIRE(d); after the first pass, sort(1) and
> > uniq(1) the REQUIRE(d); cat with PROVIDE(d), sort on name field and
> > if the pkg has no PROVIDE line, iterate with the list of not provided
> > ---should be faster than using awk(1) to compare names even in an array).
> 
> I prefer to call make, rather than try to reimplement part of it, for
> reliability.  The time to run is slightly annoying, but is not
> problematic for me.
> 
> If the tsort output is used to guide an order for make replace, then I
> think you need to know the dependencies of the package that will be
> built even if they are different than the dependencies of the package
> that is installed.
> 

Yes. The main goal is updating that is a make replace. So I will have to
add the dependencies of the new package in case they are not simple
perhaps updated versions of the installed ones.

But the program could be useful too for a not already installed package
to see if adding it will not end in trying to upgrade various already
installed packages breaking a bunch of installed packages (this happens
too much with huge multimedia packages).

> If your goals are other than a make replace plan, then I'd be interested
> in hearing about them.
> 
> > As I said, the script is relatively simple now and should be expanded
> > to list, when one gives a specific package or a list of specific
> > packages, what packages have to be updated and what packages not
> > listed will have to be updated too.
> 
> There's an interesting question of updating only if mismatched, and
> updating based on unsafe_depends.  With out-of-order replacing, I think
> one needs unsafe_depends to be sure to end up in a sound state.  I
> realize what you posted about only tosrts, but I am curious about your
> approach with the output.
> 

As said before, my approach is try to put all the informations in the
records produced. If, for the particular cases of +PRESERVE or
vulnerabilities, there is no simple way to "defer" the decision (user
can not, even with the informations, make a definite choice because
there are too many cross-dependencies that the flat listing can not
render), switches in the program will only produce a listing of what can
be done---and perhaps no listing if there is no way to upgrade or
install by sticking to old versions or not installing vulnerable
packages.

> > But the principle of splitting between providing a list of records that
> > can be script processed, and treating the packages is, IMHO, the sound
> > one.
> 
> Sure, no argument in principle about composing simpler tools.
> 
> What is your approach to the situation of starting with one package out
> of date, and no packages marked unsafe_depends?  Once that is replaced,
> things depending on it probably need to be too.

Yes, this is one of the goal: to know in advance, as far as possible,
the implications of trying to upgrade a package (or add a new one). I'm
sure I'm not the only one who tried to update "one" package because
there were bugs or crashes and seeing half of the packages being
updated with a significant number of packages broken, whether because of
forcing a replacement that broke remaining packages, or stopping
upgrading with the package replaced being definitively broken.

The question arising for me know being if it would not be a better place
to insert all this in the mk infrastructure: one start with a copy
read-only of the pkg database and the pkgsrc infrastructure adding fake
entries to the database for packages that checks for vulnerabilities and
dependencies (this can be done with the unionfs on a NetBSD for example)
without downloading, configuring or compiling---it is not 100% safe
since there can still be problems with these three; but I will not be able
to make better than that.

Other solution: using also unionfs for a fake copy of the system and
playing with PREFIX in order to compile in the subdirectory populated
and trying to upgrade the list of packages as given by pkg_tsort.

I'm planning to play a little with the latter this week to see if it
can work (avoiding the need to install a complete image of the system to
chroot).
-- 
        Thierry Laronde <tlaronde +AT+ polynum +dot+ com>
                     http://www.kergis.com/
                       http://www.sbfa.fr/
Key fingerprint = 0FF7 E906 FBAF FE95 FD89  250D 52B1 AE95 6006 F40C