Re: Test failures

[Thor Lancelot Simon <tls%panix.com@localhost>]

On Wed, Mar 07, 2012 at 09:31:40AM +0200, Jukka Ruohonen wrote:
> 
> A related issue is that we have a clear policy of not allowing test
> failures. But for reason or another, this is not enforced. (And even if it
> would be, reverting changes should be done immediately due to the reasons
> mentioned; again something where humans/committees are not good at.)

A) I don't agree that all test failures involving rump are actually
   legitimate problems with the code under test, though most are.

B) I don't agree that automatic reversion -- or reversion at all -- of
   changes that cause test failures is the right policy.  In some
   cases, backing out an entire commit is the right thing to do, but in
   others, it needlessly makes a mess of the revision history and
   complictes life for everyone, but particularly the developer trying
   to fix the problem.  Often a small fix is a much better and cleaner
   solution than backing out an entire commit.

The workflow you seem to want is wildly at odds with the one the hundreds
of active NetBSD developers have used for decades.  By and large, what we
have done in the past has worked well, and I would like to see it made
incrementally better by any changes to take advantage of automatic testing,
not completely thrown away and replaced with something new.

In particular, having had fairly extensive experience with it in the past
I think that automatic reversion of commits is a terrible idea and even
more so when working with a version control system that does not have
changelists.  And a policy which calls for this -- even if humans
"automatically" do it, rather than software -- is a bad policy as far as
I am concerned.

A tree with less test failures today than yesterday is good, but it is
not the only good!  Sometimes progress is best served by temporarily
breaking things -- sometimes even if "temporarily" lasts for a long
time.

-- 
Thor Lancelot Simon                                          
tls%panix.com@localhost
  "The liberties...lose much of their value whenever those who have greater
   private means are permitted to use their advantages to control the course
   of public debate."                                   -John Rawls