Re: Test failures

[Thor Lancelot Simon <tls%panix.com@localhost>]

On Wed, Mar 07, 2012 at 10:34:31AM +0200, Jukka Ruohonen wrote:
> 
> Just to make sure: I think you kind of missed the whole point. Reverting
> changes is by definition a post hoc solution. No one is advocating something
> like that. (Albeit commits that break the system for weeks should be reverted
> IMO.) By contrast, proactive solutions run builds and test suites before a
> commit even makes into the main repository. I have worked with such setups
> and it works really well. But as has been noted in previous discussions,
> in NetBSD the biggest problem is likely the size of the code base.

I've worked with this kind of setup too.  I don't agree that it's
appropriate for commits to HEAD in the NetBSD repository, particularly
not with the version control system we are currently using.

We have one centralized repository.  Imposing a large burden before
allowing code to be committed to it all basically discourages developers
from committing their work.  The result is major pain merging changes at
commit time and potentially even lost work.

And we do not have changelists nor a VCS with really good merge tools
(are there even any really good open-source merge tools?) which increases
the pain of maintaining private, uncommitted code for long periods of
time -- or even working on private branches.

I think it's perfectly appropriate to run automated tests and have
strict requirements about failures before code is moved from development
to production (release, release-candidate, release-branch, whatever)
status.  But NetBSD-current is decidedly not that, and so what you seem
to be calling for is that no code that is in development status should
be checked in.

But that is quite expressly *not* how NetBSD-current is managed, and I
do not think it should be.  I think it is excellent that NetBSD-current
is expressly a development version of the code and that we very clearly
warn people to use it at their own risk.  I think it is excellent that
developers are free to experiment and be creative so long as they do
not leave the tree in a state in which other developers cannot work for
long periods of time.  If you want a stable branch -- we have several,
use them!  From my point of view, if you want the appropriate point at
which to impose draconian requirements about what code can and cannot
be checked in based on failure counts from automated tests, that point
is when the code is designated as stable -- which, the way the NetBSD
repository and processes are organized, would be when it's pulled up
to a stable branch or when we declare HEAD frozen before a new branch
is created.

Thor