Re: lib/45143 (add -f to atf-run(1))

[Jukka Ruohonen <jruohonen%iki.fi@localhost>]

The following reply was made to PR lib/45143; it has been noted by GNATS.

From: Jukka Ruohonen <jruohonen%iki.fi@localhost>
To: gnats-bugs%NetBSD.org@localhost
Cc: jmmv%NetBSD.org@localhost, gnats-admin%NetBSD.org@localhost, 
netbsd-bugs%NetBSD.org@localhost,
        jruohonen%iki.fi@localhost
Subject: Re: lib/45143 (add -f to atf-run(1))
Date: Thu, 18 Aug 2011 11:32:43 +0300

 On Mon, Aug 15, 2011 at 12:50:03PM +0000, Julio Merino wrote:
 >  >   If you have tests that are skipped because they don't work, the
 >  >   necessary first step toward making them start working is to run them
 >  >   anyway and see what happens.
 >  
 >  Uh... of course.  But if you are already attempting to debug a failing 
 >  test case, it's equally easy for you to edit the code and remove what 
 >  you think is faulty.  Using a "force flag" in this scenario provides 
 >  zero value.
 >  
 >  And, "are skipped because they don't work" ?  Can you provide an example 
 >  of that?  What kind of "don't work" are you referring to?
 
 This has nothing to do with "broken tests".
 
 But consider an example: tests can be skipped on purpose because these cause
 a panic.  Perhaps this occurs on some emulator but not on real hardware, or
 perhaps it occurs on arch-X but not on arch-Y.  You don't always have the
 sources ready (cf.  dedicated test infrastructure systems or end-user
 systems, as advertised in tests(7)). But you still want to see whether you
 are affected.  Comparable case: if it is known that a test causes a panic on
 some, largely unknown, set of machines, you don't even want to go and try to
 "fix" the test in the source tree.
 
 - Jukka.