Re: Build failure tooling improvements

[Greg Troxel <gdt%lexort.com@localhost>]

A few thoughts, not really connected

  - I don't think we should build for or rely on github.  That is
    entirely separate from CI in general, which I think is a good thing.
    I realize it may be convenient to run on github now, but I would
    steer to having as much as possible not depend on github-specific
    mechanisms.

    This comment is not anti-git at all; it's about not depending on a
    proprietary service that's beyond our control, and that violates
    open source licenses by using code for training large models.  Plus,
    it's unreliable -- recently an open source project has had its
    repositories set to "archived" by github.  Perhaps this is about
    sanctions, but it's hard to follow.

  - It would be great to enable people who want, especially on unusual
    platforms, to have build runners that get hooked in, perhaps with
    them getting to have some control.  I realize that's a vast and
    fuzzy scope.  I realize that older/slower platforms have trouble
    building in a timeframe that works for pre-commit checking.  Overall
    I see this as related to do "do not encoded dependencies on github".

  - I continue to perceive that setting up pbulk is non-trivial, and
    more that lots of others perceive this.  I should go through the
    process and see; I feel like there are lots of things published, and
    I wonder about integrating some into pbulk docs proper, and trying
    to make it so that common cases need also no thought.  This is a
    half-baked comment that is probably at least somewhat off the mark.

  - BulkTracker is getting better.  With the NetBSD releng
    autobuild/report, you can see graphs of test failure counts over
    time which I find helpful.  (I realize that's test and we're talking
    about build.)  There is so much in pkgsrc that it's very essy to get
    warning fatigue.  What I'd like to see is some kind of alerting on a
    big increase in failed packges, as likely indicative of a
    regression, and a big decrease.  (I say failed vs succesful, because
    removing a python version drops the successful package count a lot
    but that's not a bug.)

  - It would be great if when a package goes from building to failing
    the MAINTAINER and all those who touched it since the last
    successful build get a note, but I don't think this should go to
    any for-humans-lists or pkgsrc-bulk.   I think Taylor is starting to
    do something like this for the shadow builds.  It would be cool if
    this were able to distinguish from starting to fail on some but not
    all platforms vs failing on all.