Re: python 3.7 removal proposal (before upstream EOL!!)

[Joerg Sonnenberger <joerg%bec.de@localhost>]

On Tue, Nov 08, 2022 at 07:46:30PM -0500, Greg Troxel wrote:
> This all started because bulk builds where showing so many failures of
> py37- packages that the other failures were harder to see because of
> noise.  And, it went on for a while with nobody fixing it which is
> pretty good evidence of a blend between nobody cares enough and nobody
> knows how.   I wasn't concerned with exactly why, or how hard it might
> be to fix -- as the responsible person for the branch I care that it is
> or is not broken.

<sarcasm> Are we going to drop NetBSD 8 next because it is clearly in a
much worse shape? </sarscam>

Shockingly, I'm been very busy with work for the last year and that
seriously limited the time I could spend on other things. The whole py27
situation is already a bad enough time sink to not fight non-mission
critical issues. The whole numpy and ipython dependency stack doesn't
qualify for that, even if ipython is a bit annoying to not have.
Removing all of py37 for that is like trying to kill a hord of emus with
gatling guns.

> The breakage was first noted in mef@'s builds on 18 October.  wiz@
> posted about py37 fallout on 25 October, asking if anyone was willing to
> fix.  You didn't speak up, or fix it then.  adam@ was in favor of
> outright removal.  nia@ had some thoughtful comments about struggling
> with upstream instability and the reasons why we have lots of versions,
> but wasn't clearly outright against 3.7 removal.  And you did not say
> anything; in particular you did not say that 3.7 was important to you
> and therefore you would fix all the issues.

See above.

> After the dicussion petered out, wiz asked me what I thought, and we
> agreed that massive bulk build failures were not ok.  So I raised the
> question.  I specifically asked if anyone would volunteer to do the work
> for one kind of 3.7 accomodation and *nobody* said they were willing.
> You have *not yet* said you are willing to generally fix things, only
> things like "So far I haven't seen that many cases of difficult to fix
> breakage..."  which I consider an impolite assertion that other people
> should fix this for your benefit.

I'm not going to fix all problems. But I also find that it is an
entirely unreasonable and one sided barrier, especially since random
broken packages are the default state and not something most developers
care in general anyway. I don't mind spending the time to fix the
infrastructure or even to actually investigate root issues like the
whole setuptools_scm bug that seems to be responsible for half of the
direct failures at least.

> You also haven't explained why you need 37 in pkgsrc HEAD, vs using
> 2022Q3 for your porting (porting from 2.7 should have started 4 years
> ago, and has been seriously overdue for a while), vs just creating a
> python37 venv and pip installing what you need there.

Porting a large application is not done in a week. Even four years ago,
the whole library situation was a mess and a showstopper by itself. It's
also not what is paying the bills. Given how many (C) libraries have
actual exploitable bugs every month, it's not an option. If you force me
to use a venv and pip, I can just as well drop pkgsrc completely.

> We do not have a rule that packages have to be kept until formal EOL.
> The general idea is benefit vs maintenance pain and when we have lots of
> versions (python, pgsql), we do tend to prune at EOL, but almost always
> the pain is low and by EOL we genuinely don't think anyone is using
> them.  In this case, there was actual trouble, and one person who
> claimed that someday they would probably use it.  And, I read rough
> consensus as agreeing that the community of python module upstreams as
> decided that 3.7 is functionally EOL.

Funny, I've been told often enough that pkgsrc is for the benefit of the
users. Guess that no longer matters.

> To try to make things easier for you, I left the python37 package there
> (against the wishes of at least Adam).  The changes I did make are small
> and you can locally revert them and build what you want, and fix 37
> things.  That effort seems tiny compared to 2.7 porting that is so hard
> that it isn't done 2+ years after 2.7 EOL.

Given that Adam actively started to remove completely harmless py37 glue
in the tree in places that actually do matter for me, that doesn't help
at all. The whole Python 2.7 EOL is a clusterfuck and I'm not alone in
that assessment.

> > (1) Immediate stop any further removal of Python 3.7 support. I find it
> > frankly a major violation of pksgrc protocol to do so for a change that
> > has been contested before.
> >
> > (2) Revert the change proposed here.
> 
> We don't have a rule that one person can veto anything.

No, but we have a rule that unilateral decisions shouldn't be pushed
through while there is still a discussion going on.

> As for things being fixed, I have updated and done pkg_rr a few times,
> deleted all my py37 packages, and python37 from both the installed
> system and binary package dir, and then did
> 
>   /usr/pkgsrc/devel/py-iniconfig $ PYTHON_VERSIONS_ACCEPTED=37 make PYTHON_VERSION_REQUIRED=37 package
> 
> and this still resulted in "ERROR: Circular dependency detected".  I'll
> update and try again, but I am curious if it works for anyone else.

*shrug* Worked for me after fixing setuptools_scm.

> 
> > (3) (Re)introduce the last 3.7-supported versions of ipython and numpy
> > with the versioned Python dependency framework. For numpy, most packages
> > don't even need to change beyond dropping the (then) bogus py37
> > restriction.
> 
> That sounds like demanding other people maintain things so you can run
> funtionally obsolete python.  I agree that the python world is too quick
> to deprecate things, but I don't think pkgsrc should have a policy of
> not updating to versions of packages that fail with a
> technically-not-EOL python version.  We would of course be squeezed by
> that and needing new enough for other packages.  nia@'s suggestion about
> expanding versioned_dependencies makes sense, but so does the idea that
> it is a lot of work with little benefit.  So far, what we're mostly
> seeing is pretty old py3N being desupported; almost everyhing (maybe
> everything?) is working with 3.10 and even 3.8 is feeling really crufty
> to me.

No, I'm saying stop hurting other people because you don't care. This is
not a case of the "Python world" depcrating things too fast, but a few
individual packages pushing for various reasons things and a bunch of
packaging bugs in pkgsrc.

Joerg