Re: python woes in netbsd-10 bulk builds

[Taylor R Campbell <riastradh%NetBSD.org@localhost>]

> Date: Sun, 14 Jul 2024 17:45:58 -0400
> From: Greg Troxel <gdt%lexort.com@localhost>
> 
> Taylor R Campbell <riastradh%NetBSD.org@localhost> writes:
> 
> > If we had a process with a well-defined criterion for when we switch
> > the symlink, a well-defined target that people could collaboratively
> > focus on and work on fixing, maybe we would have noticed this before
> > switching the symlink and putting egg all over our faces like we seem
> > to do every quarter.
> 
> It is well defined; you just don't like the definition.  Please stop the
> misinformation.

Every time I have asked you how to evaluate the definition, you have
avoided the question.

For example, I wrote, on April 10:

   Specific complaint:

      There are no concrete QA criteria that anyone can apply to evaluate
      a bulk build.

      Users don't know what to expect; developers lack guidance to
      prioritize collaboration on fixes; builders are making individual
      judgments on risk of personally screwing up and upsetting users and
      developers.

   (If you disagree with this complaint, please quote the concrete
   criteria, and show how to apply them to a bulk report, such as
   <https://mail-index.netbsd.org/pkgsrc-bulk/2024/04/03/msg025111.html>,
   without having to consult pkgsrc-pmc for a ruling on which packages
   are critical or not.)

Your private reply didn't address this, so in a followup on April 11,
I asked:

   I'd really appreciate if you could work through the steps to evaluate
   <https://mail-index.netbsd.org/pkgsrc-bulk/2024/04/03/msg025111.html>
   in a way that doesn't require pkgsrc-pmc to rule on what counts as
   critical.  I'm open to being convinced that this is feasible for me
   and builders to evaluate -- it's just not obvious to me how.

Your private reply still didn't address this, other than to suggest
that (a) the tools need to be better and (b) the procedure I followed
was generally right.

But the procedure I followed was to spend days randomly guessing what
bulk report might correspond to the packages we had uploaded but hadn't
yet published, and guessing whether anything was wrong with it, and
then digging into math/coinmp based on my guesses.

It is bonkers that anyone who operates builds should have to follow
the same guessing procedure I did every time they run a build _just to
check whether there is a problem_.  (Also my process _started_ by
consulting bulk-test-essential, which is apparently wrong.)

If you believe there is a well-defined QA criterion that can already
be evaluated without consulting pkgsrc-pmc and without invariably
turning into an abstract policy debate, please provide it and show how
to apply it.  And then maybe we can implement it!

> We have not switched the symlink,

The symlink was already switched before I wrote my message.  I don't
know when it was switched -- from memory, I think it was Jul 8, but it
could have been Jun 28.  This suggests we should also have a log for
when these things happen.

>                                   and your complaints about process are
> totally orthogonal to the current situation.

My complaint about the process is that it led to the current
situation, and my specific proposed remedy -- check
bulk-test-essential before publishing, and raise an alarm early --
would have avoided the current situation.

> > Apparently the pkg_summary.gz for the amd64 10.0_2024Q2 directory has
> > also failed to update for weeks:
> >
> > -rw-rw-r--  1 pkgmastr  netbsd     3315698 Jun 28 17:24 pkg_summary.gz
> 
> huh, interesting and I suspect once pointed out that will be resolved.

It is resolved now but we were for weeks publishing a completely
broken binary package repository.  This -- and the constant refrain of
broken binary packages every quarter -- is evidence that our process
is broken and needs some automated QA steps.

A QA process of `riastradh spot checks things that leap to mind and
asks why they are broken' is a joke.

> > So, I have suggestions for how to proceed.
> >
> > 1. Endorse bulk-test-essential as a target.
> 
> We had this discussion before.  The adopted policy values freshness for
> security purposes over availability.  That's not going to change on my watch.

Why do you think I am suggesting changing that priority?

I am only asking that the current state of affairs be baked into a
well-defined target that anyone can evaluate.

If some package _currently_ listed in bulk-test-essential is holding
things up, then when the alarm is raised -- like it should have been
back at the end of June when setuptools failed to build -- we can
review whether that package should still be considered essential.

> However, if you read the other message, it says "firefox doesn't build
> on 10 i386; does it build for anybody, and does anybody even try to use
> it on that platform?".  Which is exactly what you are suggesting.

So is the pmc-endorsed criterion whether firefox builds?  (That is,
after all, the only concretely evaluable criterion that is actually
listed at <https://pkgsrc.org/quarterly>: `In case A, it is better to
wait for the resolvable problems to be resolved, and then move the
symlink, so that users don't upgrade and e.g. end up with firefox
missing.')


I would also appreciate input on the other specific changes I
proposed:

2. put each build in a separate single-use directory and use 302
   redirects instead of symlinks into a repeated-use directory
3. include bulklogs or link to bulklogs in package upload
4. have a program to evaluate QA criteria and update the redirects
5. disable the cron job that regenerates pkg_summary and rely on
   builders to do it