[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]
Re: Test failures
Thor Lancelot Simon wrote:
> On Wed, Mar 07, 2012 at 08:02:21PM +0200, Andreas Gustafsson wrote:
> > No one has asked for changes to be tested on every Tier I platform.
> > The problem is not changes that work on one platform but fail on
> > another, it's changes that have not been tested _at all_. Including
> > ones that don't even compile.
> I don't agree. I think it is very infrequent that changes are checked
> in that "have not been tested _at all_". What does clearly happen is
> some combination of:
> * Changes have not been tested with the full set of ANITA
> tests, likely because the developer does not have (or
> does not always have) a readily available environment for
> doing so.
> * Changes have not been tested in a full system build, likely
> because they seemed small enough to not need full-system
> * The developer thinks he or she did a full system build before
> testing, but managed to screw up the dependencies somehow or
> missed a local change that made something not rebuild.
> Unfortunately the build framework for RUMP is a major cause
> of this.
> * The developer forgot to add a file with "cvs add", producing
> a complete local build-and-test success but a failure for
> others who check out the committed change.
> I don't actually think requiring ANITA test runs on all changes before
> commit will catch some of these problems -- particularly not the last two
> problems -- and I do not think it is reasonable to eliminate the developer
> discretion that allows the first two problems to occur, because, let's
> face it, some changes are low-risk, and requiring everyone to have a full
> automated test framework on hand and run everything through it when they
> add a comment to /etc/services is a pretty high burden to bear when
> problems that arise in these ways can be very quickly fixed.
I'm not saying you need to run a full set of system tests when you add
a comment to /etc/services. But to do a commit like the January 27
kmem changes without running the tests on at least one platform first
should not be considered acceptable.
Andreas Gustafsson, gson%gson.org@localhost
Main Index |
Thread Index |