Re: xfail: expected failures

[Antti Kantee <pooka%cs.hut.fi@localhost>]

On Mon Jun 28 2010 at 07:47:56 +0100, Julio Merino wrote:
> On Mon, Jun 28, 2010 at 2:09 AM, Garrett Cooper 
> <yanegomi%gmail.com@localhost> wrote:
> >
> > BTW, this comment from the wiki looks wrong:
> >
> >   atf_tc_expect_pass();  /* If reached, immediately exits with failure. */
> 
> Well, the comment is actually right.  It is a matter of
> simplification: if the test case was expecting any kind of fatal
> failure, attempting to change the expectations is an immediate error
> because that fatal failure did not happen.  It'd be possible to do
> some other thing, but it'd be too complicated and require active
> communication between the test case and atf-run.

The comment will become wrong once the feature under test is fixed and
the failure expectency is removed.  However, your test will do nothing
to draw attention to this.

Also, another thing that came to mind is that the semantics don't allow
to write tests which can fail non-deterministically in two places:

  stage1();      /* might fail here */
  prep_stage2(); /* should not fail here and must be done after stage 1 */
  stage2();      /* should latest fail here */

I don't know how common such a scenario would be, but simple gating with
xfail_enter() / xfail_exit() works there too.