Re: Test failures

[Thor Lancelot Simon <tls%panix.com@localhost>]

On Wed, Mar 07, 2012 at 08:02:21PM +0200, Andreas Gustafsson wrote:
> 
> No one has asked for changes to be tested on every Tier I platform.
> The problem is not changes that work on one platform but fail on
> another, it's changes that have not been tested _at all_.  Including
> ones that don't even compile.

I don't agree.  I think it is very infrequent that changes are checked
in that "have not been tested _at all_".  What does clearly happen is
some combination of:

        * Changes have not been tested with the full set of ANITA
          tests, likely because the developer does not have (or
          does not always have) a readily available environment for
          doing so.

        * Changes have not been tested in a full system build, likely
          because they seemed small enough to not need full-system
          testing.

        * The developer thinks he or she did a full system build before
          testing, but managed to screw up the dependencies somehow or
          missed a local change that made something not rebuild.

          Unfortunately the build framework for RUMP is a major cause
          of this.

        * The developer forgot to add a file with "cvs add", producing
          a complete local build-and-test success but a failure for
          others who check out the committed change.

I don't actually think requiring ANITA test runs on all changes before
commit will catch some of these problems -- particularly not the last two
problems -- and I do not think it is reasonable to eliminate the developer
discretion that allows the first two problems to occur, because, let's
face it, some changes are low-risk, and requiring everyone to have a full
automated test framework on hand and run everything through it when they
add a comment to /etc/services is a pretty high burden to bear when
problems that arise in these ways can be very quickly fixed.

I see automated testing as an extremely useful tool, but I do not think
requiring it in the way you seem to be advocating is really a net win
in terms of developer productivity.

Thor