Re: Build failure tooling improvements

[David Holland <dholland-pkgtech%netbsd.org@localhost>]

On Thu, Mar 20, 2025 at 05:29:49PM +0000, Jonathan Perkin wrote:
 > In the past I've done a lot of work, whether that's daily bulk builds
 > across multiple operating systems, the CI system, scan failure mails,
 > custom bulk builds, lots of additional checks, etc, but they haven't always
 > been well received or used as I'd hoped, which has obviously led to some
 > frustration on both sides.
 > 
 > So, how about a different approach.  What would you like me to work on?
 > What features would you like to see, and would actively use?  What is your
 > ideal workflow?  What would it take for your workflow to include
 > cross-platform checks prior to commit?  What resources do you need to help
 > ensure each individual commit is as high-quality as possible?

Here's one thing:

The original bulk tracker vision (which isn't what got built) included
at least two things that would make for a much more effective tool:

   - tracking commits as well as build results (which includes
     handling branches);
   - matching "the same" failures of one package across builds and
     targets;
   - matching "the same" failures across packages.

Some things these enable:

   - being easily able to tell target-specific failures from general
     ones;

   - being easily able to tell when an old problem reappears and what
     the change was that fixed it;

   - identifying when something broke and what commit broke it (e.g.
     its own update, update of a dep, infrastructure change, etc.),
     which often tells you a lot about why;

   - being able to see quickly if someone's committed a change since
     the failure you're looking at, so you don't waste time redoing
     someone else's fixes;

   - easily fishing out related fixes already applied to other
     packages;

   - being able to see what happened when an already broken package
     breaks further and when/if it later reverts to its prior state
     (this does happen, it can be caused by interactions among deps);

   - still being able to find and see the last actual result even in
     cases where currently there's a broken dep;

   - also having all this summary information in a database makes it
     possible to extract overview reports that currently can't be
     had, as well as statistics and other info.

Another goal was to be able to provide a list of packages that haven't
been built since they were last committed, or haven't been built since
something they depend on was last committed. These lists can then be
used for targeted limited builds. And then ideally it would be
possible to ingest the results of those limited builds without
confusing the database. (IIRC at the time there were complications
associated with being able to know what had succeeded in a limited
build report.)

All of this was formulated back when I was regularly working the bulk
reports, and the goals and reports are based on the things I was doing
at the time by hand. Things have changed some in the meantime, but not
I think by so much that these ideas are obsolete.


(also, another thing the current tool doesn't have that I imagine
would get a fair amount of use is a personal feed for sets of packages
you're interested in)


FWIW, it seems the original bulktracker project proposal is still on
the projects page in the wiki.

-- 
David A. Holland
dholland%netbsd.org@localhost