Re: Why are depends rebuilt on "bmake update" ?

[Aleksej Saushev <asau%inbox.ru@localhost>]

Greg Troxel <gdt%ir.bbn.com@localhost> writes:

> Aleksej Saushev <asau%inbox.ru@localhost> writes:
>
>> I tried to recover from inconsistent "make replace" with pkg_rolling-replace
>> several times and always ended in a state with even more inconsistencies.
>> This led to rebuilding of all packages at least two times, and at least one
>> time I could get out with pkg_chk. In other cases I performed manual updates
>> to recover. This makes me believe that pkg_rolling-replace is not safe
>> mechanism.

"BUGS" section doesn't sound encouraging either.

> That sounds unusual.  (If dependencies are not expressed correctly, it
> can't work, but that doesn't sound like what happened to you.)
>
> Another problem I should have pointed out is that
>
>   building packages from one pkgsrc version
>
>   updating to newer pkgsrc (but not rebuilding anything)
>
>   make replace on some random package
>
>   pkg_rolling-replace -v
>
> will not work, because many packages will demand specific new versions
> of dependencies.

Alright, I see. It should build new dependency and stop at conflict.

> This is not actually a pkg_rr bug, but an underlying
> reality of make replace.  To avoid this, I do
>
>   pkg_rolling-replace -uv
>
> which also replaces mismatched packages.  Perhaps this is what you ran
> into.
>
> If you have more data abou the problem you had (logs, etc.) I'd be
> interested in seeing it.

I don't have more data, it's been long since I used it last time.
Problems that require hard work to recover don't encourage experimentation.

>> I still don't understand why it is recommended to recover from
>> inconsistencies.  The way it functions (utilizing unsafe operations)
>> doesn't make impression of doing it correctly either.
>
> The theory is actually pretty straightforward:
>
>   pkgsrc normally has a "all dependencies of a program exist, and
>   are the right versions" rule.
>
>   when you do 'make replace' on package A, then that rule, assuming it
>   was true before, remains true, except that for various packages b that
>   depend on A, the dependency might not be ok (wrong shlib version,
>   etc.)
>
>   pkgsrc sets flags unsafe_depends on all such packages b when it does a
>   make replace on A.  It clears the flag on A, because A was rebuilt
>   against what was installed, and thus by definition is ok at that
>   moment.
>
>   Thus, the new rule which is always true is "all dependencies are a
>   right/sound version, except that packages flagged unsafe_depends may
>   have bad versions of dependencies."
>
> So, if you do a bunch of make replace operations, then if you arrive
> with unsafe_depends not set on anything, you should have a consistent
> situation.
>
> pkg_rr topologically sorts all the packages and does the make replace in
> the right order, so that you only replace each package once.  So it is
> systematically cleaning the unsafe_depends flags.  That's really all
> there is to it.
>
> Taken together, I believe that this approach is entirely sound.  (It may
> fail because of package structural changes, but absent dependency bugs
> in pkgsrc, one should not be able to arrive at a state with no
> unsafe_depends flags and any inconsistency.  That's why I say that it is
> sound, even though sometimes it is unsuccessful.)

This sounds akin to FOL programming, thus there must be some catch.
Real life is a bit more complex, and non-monotonic changes occur...

Now I see what might cause problems when using it for updates.

Consider the following scenario.
There're two packages installed, A and B, A depends on B.

   A -> B

Now, B gets updated but A lags behind, instead B1 is introduced.

When running pkg_rr something triggers update of B.
It gets rebuilt, replaced, and triggers update of A.
Now the dependecies look like this:

   A -> [B1, yet to build]
        B

If B and B1 conflict, user is in trouble.
If B is high enough in forest, going back is even harder.

I seem to have some vague memory that I have seen something like that,
I'm not going to defend it though. Yet this update pattern happens from
time to time in pkgsrc.

> Separately from the theory, I have been running pkg_rr to update many
> systems for over 5 years, and it has basically worked (modulo the issues
> in my previous note, which are failures to replace, not integrity
> failures once replaced).

Sorry, but the above convinces me that pkg_rr is still way too dangerous
to recommend to those who may not understand the mechanics.

First, it is too easy to omit "u" flag, thus a user may stick with
inconsistent tree with no clear recovery plan.

Second, its operation isn't trivial _enough_ so that a user could have
enough clue on what's going on. You cannot expect that everyone knows
what topological sort is and understands consequences. Documentation
doesn't stress that either. Perhaps, man page should elaborate more
how it actually works, so that a qualified person could have better
grasp of underlying mechanics and could work out recovery plan.

pkg_chk is more obvious and straightforward to work with, since it walks
the dependency tree from given set of roots, and operator can just
remove what gets in a way and continue. In case of pkg_rr recovery
requires selectively removing a package that has active dependents,
and you have to do it carefully so that you don't kill the whole tree
of packages recursively. I admit that I might be biased towards pkg_chk,
but that's what my experience is.


-- 
HE CE3OH...