Re: xfail: expected failures

[Garrett Cooper <yanegomi%gmail.com@localhost>]

On Wed, Jun 16, 2010 at 1:49 PM, Antti Kantee <pooka%cs.hut.fi@localhost> wrote:
> [Julio requested I send a small note to this list since there have been
> others interested in the feature]
>
> Today I added "xfail" to the NetBSD atf tree.  What it essentially
> does is negates the logical success/fail for a test.  This is to help
> differentiate between currently known but not yet fixed problems and
> unexpected problems.  In the test report, xfailed tests are counted
> under "expected failures".  If an xfail test unexpectedly succeeds,
> it is counted as a failure.
>
> Currently, an xfail test is indicated by setting the xfail var and giving
> preferably some tracking system identifier as the reason, e.g.:
>
>        atf_tc_set_md_var(tc, "xfail", "PR kern/43456");
>
> Xfail will be present in atf 0.10 (according to Julio ;).  The current
> implementation is a big binary hammer, meaning it is not possible to
> specify where the test is expected to fail.  This may change for the
> 0.10 release.  If anyone has any other ideas about what they would like to
> see in the this department, I guess now would be a good time to speak up.

    Apart from the fact that `expected failure' is a bit ambiguous
(compare positive testcase failures which are known to fail under
certain conditions vs negative testcase failures, which are genuinely
expected), this seems to make sense given what other folks have posted
elsewhere on the subject. I don't think that POSIX actually says
anything genuinely helpful on the topic...
Thanks,
-Garrett