Re: Test failures

[Andreas Gustafsson <gson%gson.org@localhost>]

Thor Lancelot Simon wrote:
> On Tue, Mar 06, 2012 at 10:36:04PM +0200, Jukka Ruohonen wrote:
> > 
> > Fair enough. But most of the failures would be catched if developers would
> > actually run the tests before committing.
> 
> Only if every developer ran every test before every commit, on, at least,
> every Tier I platform.

That is just not true.  The vast majority of failures would be caught
by running the tests on one platform - any platform.

> And, unfortunately, I suspect that is a burden which would lead many
> developers to simply not commit.  We're already asking developers to
> do full system builds, which take the best part of an hour; now
> you'd be asking them to maintain either a fully up-to-date VM for
> each Tier I platform, or a private copy of an automated framework to
> compile-install-test on such VMs.

No one has asked for changes to be tested on every Tier I platform.
The problem is not changes that work on one platform but fail on
another, it's changes that have not been tested _at all_.  Including
ones that don't even compile.

> Usually most developers (even I) don't screw up.  This is NetBSD-current,
> not a release branch, and so I believe some amount of "screw up and fix
> it later" is OK if it greatly speeds development.

It may speed development for the person doing it, but it slows
development for everyone else.  That's unfair, and I think the net
impact on the project is negative.
-- 
Andreas Gustafsson, gson%gson.org@localhost