Re: Test failures

[Jukka Ruohonen <jruohonen%iki.fi@localhost>]

On Tue, Mar 06, 2012 at 11:49:11PM -0500, Thor Lancelot Simon wrote:
> every Tier I platform.  And, unfortunately, I suspect that is a burden
> which would lead many developers to simply not commit.  We're already
> asking developers to do full system builds, which take the best part of
> an hour.

I agree. Personally I do not have the horsepower to even do full builds
before every commit. But I would probably run the test before a branch
merge or other big MI commit.

> Usually most developers (even I) don't screw up.  This is NetBSD-current,
> not a release branch, and so I believe some amount of "screw up and fix
> it later" is OK if it greatly speeds development.

Just because there is a HEAD branch, and "that's the way we have always
developed", does not mean better alternatives should not be considered.
I am not sure at all whether the current "post hoc QA" process speeds up the
development. Actually, one might argue that the overall effect is negative:
a lot of man-hours are still wasted on tracking regressions.

> to run all the anita tests before every commit.  I think periodic runs
> of the tests represent a good compromise between speed of development
> and speed of debugging.

Andreas would probably agree with me in that the most dangerous situation
in the current setting occurs when somone commits something that panics the
system. It gets eventually fixed, but in the meanwhile even hundreds of
commits have been made. This is where the current test setup fails.
And this is how blocker bugs such as #45677 and #46096 got introduced,
despite of having being caught by the test infrastructure.

- Jukka.