Re: xfail: expected failures

[Julio Merino <jmmv%NetBSD.org@localhost>]

On Mon, Jun 28, 2010 at 2:09 AM, Garrett Cooper <yanegomi%gmail.com@localhost> 
wrote:
>
> BTW, this comment from the wiki looks wrong:
>
>   atf_tc_expect_pass();  /* If reached, immediately exits with failure. */

Well, the comment is actually right.  It is a matter of
simplification: if the test case was expecting any kind of fatal
failure, attempting to change the expectations is an immediate error
because that fatal failure did not happen.  It'd be possible to do
some other thing, but it'd be too complicated and require active
communication between the test case and atf-run.

As the doc says:

-----
* signal, exit and death: The test program pre-creates the result file
with a message stating the expected termination so that, when the
program actually terminates, atf-run can find the result file and know
that the premature exit was expected. Setting expect to any other
value after it has already been set to any of these three triggers an
immediate failure of the test case (because the program did not
terminate as expected). If an expected termination is detected, the
test case reports an expected_signal / expected_exit result.
-----

The only case in which it is legitimate to revert to "pass" is when
you were expecting a failure and the test case raised a non-fatal
failure (such as in the other example in the document).

-- 
Julio Merino