Re: Test interdependences, and globals

[Julio Merino <jmmv%NetBSD.org@localhost>]

On Tue, Jul 6, 2010 at 10:55 PM, Cliff Wright <cliff%snipe444.org@localhost> 
wrote:
> On Tue, 6 Jul 2010 21:40:27 +0100
> Julio Merino <jmmv%NetBSD.org@localhost> wrote:
>
>
>> Can you provide some specific examples of why are you trying to do
>> this?  Why can't you recreate that unique value as a setup step for
>> every test case?
>
> The testing we are doing is far to complicated to do as a single test 
> program. The concept of levels (e.g test suite, test program) works very well 
> for us. In the early setup, multiple test programs are run in a specific 
> sequence to get a gui in the right state to display a page of buttons, one of 
> which generates the unique value that will never be generated again. To 
> combine all these steps into a single test program, would complicate our 
> ability to test each little step(does the window exist?, did the menu 
> pop-up?, is the button green?). So even if I could regenerate the unique 
> value(and I can't), as soon as I run the next test (which might be: find and 
> press the yellow button that has this unique label) the unique value from the 
> previous test is lost. This breakup of an immense test into smaller tests 
> (and then into even smaller tests) is very important to us.

The scenario you are describing corresponds to one test case "does the
window have all X properties?".  The test case, as its first step,
sets up the GUI window.  Then, it performs all the *non-fatal* checks
(does the window exist? did the menu pop-up? is the button green?) and
then exits.  Doing all these checks as non-fatal means that all of
them will run regardless of their results.  Why can't you do this?

Passing state through test cases or test programs is NOT supported and
won't be.  Future versions of atf will run tests in parallel (which
means you most likely won't have any control on their order); and,
even further in the future, it'd be nice to allow running such tests
in separate machines (for better isolation for example).  Doing what
you suggest breaks all these possibilities or just makes them
unfeasible.

>> Adding test dependencies would be possible, albeit tricky, but it
>> won't happen without a real and convincing use case for it.  Can you
>> elaborate?  If the first test fails, what's the problem of running the
>> other?  They'll just report failure, which is OK because they *are*
>> actually failing.
>
> The problem particularly with a gui is the following steps might effect the 
> gui(e.g the wrong window is now open) so that the following suite(group) of 
> tests might now fail when they otherwise would have succeeded. We will be 
> running full tests with thousands of steps, so having usable results for 
> individual groups is very important.

As mentioned above: you just have one test case.  After setting up the
window, you perform a *fatal* check to know if the correct window
popped up or not.  Once you know it's the good one, then you perform
non-fatal checks from that point onwards.  But all within the same
test case.  If the fatal check fails, you prematurely exit the test
case and don't do anything else.  If the fatal check succeeds, then
you get a detailed output of the results of the non-fatal checks.

>  If you add dependencies, you are hiding real tests
>> on the assumption that their failures won't be helpful (and in most
>> cases such results are helpful because they provide additional data
>> points of why things failed).
> I can't get specific, so here is a for instance. Say I want to run a suite of 
> tests on xcalc, 1 suite with it in rpn mode, and one suite with it not in rpn 
> mode. Say I find a major error in rpn mode. I would now like to skip all 
> other rpn tests, and now run the non-rpn tests. Any rpn tests that now run 
> would have to be ignored even if they passed(could be bogus results). When 
> running very large tests that could have over thousands of  steps, skipping 
> bad groups of tests will effect both time, and being able to interpret the 
> results.

Test cases that can provide false positives are broken.  You should
not have any of those.

-- 
Julio Merino