[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]
Re: xfail: expected failures
On Mon Jun 28 2010 at 12:31:33 +0100, Julio Merino wrote:
> On Mon, Jun 28, 2010 at 12:16 PM, Antti Kantee <pooka%cs.hut.fi@localhost>
> > On Mon Jun 28 2010 at 07:47:56 +0100, Julio Merino wrote:
> >> On Mon, Jun 28, 2010 at 2:09 AM, Garrett Cooper
> >> <yanegomi%gmail.com@localhost> wrote:
> >> >
> >> > BTW, this comment from the wiki looks wrong:
> >> >
> >> > atf_tc_expect_pass(); /* If reached, immediately exits with failure.
> >> > */
> >> Well, the comment is actually right. It is a matter of
> >> simplification: if the test case was expecting any kind of fatal
> >> failure, attempting to change the expectations is an immediate error
> >> because that fatal failure did not happen. It'd be possible to do
> >> some other thing, but it'd be too complicated and require active
> >> communication between the test case and atf-run.
> > The comment will become wrong once the feature under test is fixed and
> > the failure expectency is removed. However, your test will do nothing
> > to draw attention to this.
> That's not the case. We have the following:
> atf_tc_expect_death("The following code will crash");
> abort() / exit()
I did write "comment", so it's a little unfair to analyze a situation
in which you remove the comment... ;)
> > Also, another thing that came to mind is that the semantics don't allow
> > to write tests which can fail non-deterministically in two places:
> > stage1(); /* might fail here */
> > prep_stage2(); /* should not fail here and must be done after stage 1 */
> > stage2(); /* should latest fail here */
> > I don't know how common such a scenario would be, but simple gating with
> > xfail_enter() / xfail_exit() works there too.
> At the moment, every xfail enter/exit block must have at least one
> failure in between. It's more of an "all block must fail" than just a
> "at least one block should fail". Not sure which of the two
> approaches makes more sense. If we stick to the principle of "each
> test case must test one scenario", then expecting a failure on just
> one block makes sense and expecting a failure in every block probably
> does not. I think.
How many times can a test exit or timeout or get a fatal signal?
> If you have two conditions that might fail, just put them in the same
> block as non-fatal checks.
You can't if the condition is a crash.
> How realistic is that you need to do other
> stuff in between for which you must do regular checks?
It's more a question of being a prep stage which you do not except
Anyway, my main gripe is still with the totally misleading name.
But whatever, I'll live ;)
Main Index |
Thread Index |