Subject: Re: known-to-be-good tags on -HEAD? (Re: recent instability)
To: Chapman Flack <firstname.lastname@example.org>
From: Garrett D'Amore <email@example.com>
Date: 04/20/2006 20:25:18
Actually, what this really sounds to me like, is we need some kind of
automatic regression test suite that picks up nightlies and runs them.
If a test suite passes for a build, it could auto-tag them.
A website listing regression failures would be helpful, too.
You wouldn't be able to cover everything this way (e.g. kind of hard to
test all the different hardware combos), but it would be a good start,
Chapman Flack wrote:
> Jeremy C. Reed wrote:
>> It apparently works for the developer who made the commits (that
>> break for someone else).
> This reminds me of a question that has been nagging me for a bit: how
> many distinct platforms does a typical developer test on before
> or have access to for testing at all? I, for example, have only i386 hw
> Perhaps NetBSD should have KNOWN_GOOD_<port> tags added independently
> as versions are determined to work on different ports.
> The trick would be a good definition of the tag semantics. KNOWN_GOOD
> is really a misleading name--it's more like not-yet-known-bad, and
> has at least two important parameters: not yet known bad after testing
> by how many users and for how long. Perhaps the tag should be called
> something like CONFIDENCE_1W100_macppc, added as soon as a version has
> seen 1 week's use under normal workloads by 100 macppc users without
> any regression. It sounds like a good candidate for automation, with a
> command like send-sr (for sending "success reports" rather than problem
> reports :), and a receiving process that, upon receiving 100 1-week
> success reports for <port>, would move the CONFIDENCE_1W100_<port> tag,
> leaving a CONFIDENCE_1W100_<port>_PREVIOUS tag in its original place.
> Should reports of a significant regression cause the tag to move back?
> One would think the answer should be yes.
> The idea obviously generalizes to a small family of confidence tags that
> differ by number of users or duration of testing, and people could match
> their level of risk aversion to the tag they choose to sync to.
> The scheme would require send-sr to unambiguously know the date/time
> (for -D in cvs) of the checkout from which the running system was
> built, which could be kept in a file somewhere in the tree that is
> updated by a commit-script whenever anything is committed anywhere.
> In fact, that seems likely to be functionality we might already have
> though I've not stumbled on it yet--do we have such a thing? send-sr
> should probably also verify that the checked-out sources all correspond
> to the recorded revision date.
> I don't know enough about svn to guess if such a scheme would be easier
> or harder in svn. It doesn't seem infeasible in cvs.
> Thinking about it, I would actually call such a facility useful. Does
> anybody else think so? Any thoughts on what values for duration and
> number-of-users would make reasonably meaningful tags?
Garrett D'Amore, Principal Software Engineer
Tadpole Computer / Computing Technologies Division,
General Dynamics C4 Systems
Phone: 951 325-2134 Fax: 951 325-2191