Re: Test failures

[Andreas Gustafsson <gson%gson.org@localhost>]

Thor Lancelot Simon wrote:
> On Wed, Mar 07, 2012 at 08:02:21PM +0200, Andreas Gustafsson wrote:
> > 
> > No one has asked for changes to be tested on every Tier I platform.
> > The problem is not changes that work on one platform but fail on
> > another, it's changes that have not been tested _at all_.  Including
> > ones that don't even compile.
> 
> I don't agree.  I think it is very infrequent that changes are checked
> in that "have not been tested _at all_".  What does clearly happen is
> some combination of:
> 
>       * Changes have not been tested with the full set of ANITA
>         tests, likely because the developer does not have (or
>         does not always have) a readily available environment for
>         doing so.
> 
>       * Changes have not been tested in a full system build, likely
>         because they seemed small enough to not need full-system
>         testing.
> 
>       * The developer thinks he or she did a full system build before
>         testing, but managed to screw up the dependencies somehow or
>         missed a local change that made something not rebuild.
> 
>         Unfortunately the build framework for RUMP is a major cause
>         of this.
> 
>       * The developer forgot to add a file with "cvs add", producing
>         a complete local build-and-test success but a failure for
>         others who check out the committed change.
> 
> I don't actually think requiring ANITA test runs on all changes before
> commit will catch some of these problems -- particularly not the last two
> problems -- and I do not think it is reasonable to eliminate the developer
> discretion that allows the first two problems to occur, because, let's
> face it, some changes are low-risk, and requiring everyone to have a full
> automated test framework on hand and run everything through it when they
> add a comment to /etc/services is a pretty high burden to bear when
> problems that arise in these ways can be very quickly fixed.

I'm not saying you need to run a full set of system tests when you add
a comment to /etc/services.  But to do a commit like the January 27
kmem changes without running the tests on at least one platform first
should not be considered acceptable.
-- 
Andreas Gustafsson, gson%gson.org@localhost