[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]
Re: Test failures
On Wed, Mar 07, 2012 at 08:02:21PM +0200, Andreas Gustafsson wrote:
> No one has asked for changes to be tested on every Tier I platform.
> The problem is not changes that work on one platform but fail on
> another, it's changes that have not been tested _at all_. Including
> ones that don't even compile.
I don't agree. I think it is very infrequent that changes are checked
in that "have not been tested _at all_". What does clearly happen is
some combination of:
* Changes have not been tested with the full set of ANITA
tests, likely because the developer does not have (or
does not always have) a readily available environment for
* Changes have not been tested in a full system build, likely
because they seemed small enough to not need full-system
* The developer thinks he or she did a full system build before
testing, but managed to screw up the dependencies somehow or
missed a local change that made something not rebuild.
Unfortunately the build framework for RUMP is a major cause
* The developer forgot to add a file with "cvs add", producing
a complete local build-and-test success but a failure for
others who check out the committed change.
I don't actually think requiring ANITA test runs on all changes before
commit will catch some of these problems -- particularly not the last two
problems -- and I do not think it is reasonable to eliminate the developer
discretion that allows the first two problems to occur, because, let's
face it, some changes are low-risk, and requiring everyone to have a full
automated test framework on hand and run everything through it when they
add a comment to /etc/services is a pretty high burden to bear when
problems that arise in these ways can be very quickly fixed.
I see automated testing as an extremely useful tool, but I do not think
requiring it in the way you seem to be advocating is really a net win
in terms of developer productivity.
Main Index |
Thread Index |