Re: xfail: expected failures

[Julio Merino <jmmv%NetBSD.org@localhost>]

On Mon, Jun 28, 2010 at 12:16 PM, Antti Kantee <pooka%cs.hut.fi@localhost> 
wrote:
> On Mon Jun 28 2010 at 07:47:56 +0100, Julio Merino wrote:
>> On Mon, Jun 28, 2010 at 2:09 AM, Garrett Cooper 
>> <yanegomi%gmail.com@localhost> wrote:
>> >
>> > BTW, this comment from the wiki looks wrong:
>> >
>> >   atf_tc_expect_pass();  /* If reached, immediately exits with failure. */
>>
>> Well, the comment is actually right.  It is a matter of
>> simplification: if the test case was expecting any kind of fatal
>> failure, attempting to change the expectations is an immediate error
>> because that fatal failure did not happen.  It'd be possible to do
>> some other thing, but it'd be too complicated and require active
>> communication between the test case and atf-run.
>
> The comment will become wrong once the feature under test is fixed and
> the failure expectency is removed.  However, your test will do nothing
> to draw attention to this.

That's not the case.  We have the following:

atf_tc_expect_death("The following code will crash");
abort() / exit()
atf_tc_expect_pass();

The code above, as is, will fail the test case with a "expected_death"
reason, which will count towards the expected failures count.

If we remove the abort() call, the test will reach the
atf_tc_expect_pass() call, at which point it will be reported as
"failed" because it did not crash as it was intended.  And thus if the
condition under test gets fixed, the attention of the developer will
be drawn.

(Replace abort() by your favorite buggy function that crashes or exits.)

> Also, another thing that came to mind is that the semantics don't allow
> to write tests which can fail non-deterministically in two places:
>
>  stage1();      /* might fail here */
>  prep_stage2(); /* should not fail here and must be done after stage 1 */
>  stage2();      /* should latest fail here */
>
> I don't know how common such a scenario would be, but simple gating with
> xfail_enter() / xfail_exit() works there too.

At the moment, every xfail enter/exit block must have at least one
failure in between.  It's more of an "all block must fail" than just a
"at least one block should fail".  Not sure which of the two
approaches makes more sense.  If we stick to the principle of "each
test case must test one scenario", then expecting a failure on just
one block makes sense and expecting a failure in every block probably
does not.  I think.

If you have two conditions that might fail, just put them in the same
block as non-fatal checks.  How realistic is that you need to do other
stuff in between for which you must do regular checks?

-- 
Julio Merino