ATF-devel archive

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]

Re: failures with NetBSD

On Mar 24, 2008, at 19:35 , Martin Husemann wrote:

On Mon, Mar 24, 2008 at 06:43:51PM +0100, Julio M. Merino Vidal wrote:
before the conversion to ATF).  I guess I really have to add an
"Expected failure" output for test cases (it's in the to-do list for
0.6), so that we can clearly mark those that we know are broken and
thus prevent confusion to end users.

I think "expected failure" is confusing - maybe add a "known to fail"
result and allow outputing some helpfull comment (like "PR bin/ xxxxx")?

I like the idea of the helpful comment. However, I think "expected failure" is a widespread terminology and we should keep it. I have seen it already in several test suites. The idea of this feature is that, in the test case, you do some check that should fail and, if it does, you abort the test with "expected failure". I.e. you must still run the test and not omit it. "Known to fail" seems to indicate that the test is not run, because we already know it will fail.

Which leads us to the need to also have unexpected passes, which are the contrary of the above and are used to detect test cases that have been unexpectedly fixed or their problems have been hidden.

Home | Main Index | Thread Index | Old Index