Re: Running bulk builds for finding bugs in packages

[Greg Troxel <gdt%lexort.com@localhost>]

Roland Illig <roland.illig%gmx.de@localhost> writes:

> On 30.05.2020 11:45, nia wrote:
>> On Sat, May 30, 2020 at 10:34:47AM +0200, Roland Illig wrote:
>>> A few days ago, I added a section to the pkgsrc guide containing ideas
>>> for bulk builds with non-default configurations, to find packages that
>>> don't stick to the pkgsrc conventions or to best practices.
>>>
>>> https://www.netbsd.org/docs/pkgsrc/bulk.html#bulk.var
>>>
>>> Did I miss any important build variants? Or are there any details that I
>>> got wrong or that should have been explained better?
>>
>> Do we really want to turn package Makefiles into a document of every
>> upstream code problem?
>
> Hmmm, probably not.
>
> I chose to do this for -Werror=char-subscripts because several C
> programmers do not know how to use the <ctype.h> functions correctly.
> Therefore I wanted to find out these packages and mark them, just to
> know that _if_ they crash mysteriously, there's a hint somewhere.

If this is really a database of detected warnings, stored in a
disrtibuted manner in a Makefile, it would seem to make more sense to
store it in some database.

So perhaps instead of storing it, perhaps a program (a new pkgtools
package?) that can run these sorts of test on a package, perhaps simply
by going to a package directory and running it, which might do even 50
builds with varying output.   That could be a thing to do when a package
behaves badly in hard-to-grasp ways.

> My main goal of running that strict bulk build was to report these bugs
> upstream, to have them fixed with the next update of the package.

I think that is the right goal.  pkgsrc really should just be wrapping
upstream, and unless it's necessary, I think upstream packages should be
fixed upstream.  That's a lot of why we have norm that patches (that fix
upstream problems) have to be filed upstream and have the upstream
ticket URL.   pkgsrc carrying patches for things that don't really need
fixing leads to more work for maintainers at update time.

> A similar compiler flag is -Wimplicit-function-declaration, which would
> catch a lot of old, rusty packages that declare their own string
> functions or that don't include <unistd.h>.  Finding and fixing these is
> a lot of work though, therefore I won't start that one.

I think if anyone feels like finding these and filing upstream bugs,
that's fine.  But I don't see it as a net benefit to patch or annotate
in pkgsrc.

> For a general build, I agree completely.  This whole section is about
> experimental bulk builds though, to find bugs.  And in these, I find it
> acceptable to rely on a particular compiler.

I think as long as the result of finding bugs is filing tickets and/or
fixes upstream, everybody can be ok with this.   Perhaps you can back
off the notion of adding things to Makefiles and make it more clear that
the right approach is to file/fix upstream.