[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]
Re: failures with NetBSD
"Julio M. Merino Vidal" <jmmv84%gmail.com@localhost> writes:
> On Mar 24, 2008, at 19:35 , Martin Husemann wrote:
>> On Mon, Mar 24, 2008 at 06:43:51PM +0100, Julio M. Merino Vidal wrote:
>>> before the conversion to ATF). I guess I really have to add an
>>> "Expected failure" output for test cases (it's in the to-do list for
>>> 0.6), so that we can clearly mark those that we know are broken and
>>> thus prevent confusion to end users.
>> I think "expected failure" is confusing - maybe add a "known to fail"
>> result and allow outputing some helpfull comment (like "PR bin/
> I like the idea of the helpful comment. However, I think "expected
> failure" is a widespread terminology and we should keep it.
Actually, in most test suites, an "Expected Failure" is a test where
you WANTED something to fail -- as in you fed invalid input in and the
error checking routines properly rejected it. You're talking about
something that is, in fact, a known bug, not an expected error.
> I have seen it already in several test suites. The idea of this
> feature is that, in the test case, you do some check that should
> fail and, if it does, you abort the test with "expected failure".
> I.e. you must still run the test and not omit it. "Known to fail"
> seems to indicate that the test is not run, because we already know
> it will fail.
> Which leads us to the need to also have unexpected passes, which are
> the contrary of the above and are used to detect test cases that have
> been unexpectedly fixed or their problems have been hidden.
Perry E. Metzger perry%piermont.com@localhost
Main Index |
Thread Index |