ATF-devel archive

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]

xfail: expected failures



[Julio requested I send a small note to this list since there have been
others interested in the feature]

Today I added "xfail" to the NetBSD atf tree.  What it essentially
does is negates the logical success/fail for a test.  This is to help
differentiate between currently known but not yet fixed problems and
unexpected problems.  In the test report, xfailed tests are counted
under "expected failures".  If an xfail test unexpectedly succeeds,
it is counted as a failure.

Currently, an xfail test is indicated by setting the xfail var and giving
preferably some tracking system identifier as the reason, e.g.:

        atf_tc_set_md_var(tc, "xfail", "PR kern/43456");

Xfail will be present in atf 0.10 (according to Julio ;).  The current
implementation is a big binary hammer, meaning it is not possible to
specify where the test is expected to fail.  This may change for the
0.10 release.  If anyone has any other ideas about what they would like to
see in the this department, I guess now would be a good time to speak up.

cheers,
  antti

p.s. I did not yet add the regression tests for this feature.  *shame*


Home | Main Index | Thread Index | Old Index