ATF-devel archive

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]

Re: failures with NetBSD



"Julio M. Merino Vidal" <jmmv84%gmail.com@localhost> writes:
> On Mar 24, 2008, at 19:35 , Martin Husemann wrote:
>
>> On Mon, Mar 24, 2008 at 06:43:51PM +0100, Julio M. Merino Vidal wrote:
>>> before the conversion to ATF).  I guess I really have to add an
>>> "Expected failure" output for test cases (it's in the to-do list for
>>> 0.6), so that we can clearly mark those that we know are broken and
>>> thus prevent confusion to end users.
>>
>> I think "expected failure" is confusing - maybe add a "known to fail"
>> result and allow outputing some helpfull comment (like "PR bin/
>> xxxxx")?
>
> I like the idea of the helpful comment.  However, I think "expected
> failure" is a widespread terminology and we should keep it.

Actually, in most test suites, an "Expected Failure" is a test where
you WANTED something to fail -- as in you fed invalid input in and the
error checking routines properly rejected it. You're talking about
something that is, in fact, a known bug, not an expected error.

Perry

> I have seen it already in several test suites.  The idea of this
> feature is that, in the test case, you do some check that should
> fail and, if it does, you abort the test with "expected failure".
> I.e. you must still run the test and not omit it.  "Known to fail"
> seems to indicate that the test is not run, because we already know
> it will fail.
>
> Which leads us to the need to also have unexpected passes, which are
> the contrary of the above and are used to detect test cases that have
> been unexpectedly fixed or their problems have been hidden.
>

-- 
Perry E. Metzger                perry%piermont.com@localhost


Home | Main Index | Thread Index | Old Index