Re: Test interdependences, and globals

[Cliff Wright <cliff%snipe444.org@localhost>]

On Tue, 6 Jul 2010 21:40:27 +0100
Julio Merino <jmmv%NetBSD.org@localhost> wrote:

 
> Can you provide some specific examples of why are you trying to do
> this?  Why can't you recreate that unique value as a setup step for
> every test case?

The testing we are doing is far to complicated to do as a single test program. 
The concept of levels (e.g test suite, test program) works very well for us. In 
the early setup, multiple test programs are run in a specific sequence to get a 
gui in the right state to display a page of buttons, one of which generates the 
unique value that will never be generated again. To combine all these steps 
into a single test program, would complicate our ability to test each little 
step(does the window exist?, did the menu pop-up?, is the button green?). So 
even if I could regenerate the unique value(and I can't), as soon as I run the 
next test (which might be: find and press the yellow button that has this 
unique label) the unique value from the previous test is lost. This breakup of 
an immense test into smaller tests (and then into even smaller tests) is very 
important to us.

 
 
> Adding test dependencies would be possible, albeit tricky, but it
> won't happen without a real and convincing use case for it.  Can you
> elaborate?  If the first test fails, what's the problem of running the
> other?  They'll just report failure, which is OK because they *are*
> actually failing.
 
The problem particularly with a gui is the following steps might effect the 
gui(e.g the wrong window is now open) so that the following suite(group) of 
tests might now fail when they otherwise would have succeeded. We will be 
running full tests with thousands of steps, so having usable results for 
individual groups is very important.

 If you add dependencies, you are hiding real tests
> on the assumption that their failures won't be helpful (and in most
> cases such results are helpful because they provide additional data
> points of why things failed).
I can't get specific, so here is a for instance. Say I want to run a suite of 
tests on xcalc, 1 suite with it in rpn mode, and one suite with it not in rpn 
mode. Say I find a major error in rpn mode. I would now like to skip all other 
rpn tests, and now run the non-rpn tests. Any rpn tests that now run would have 
to be ignored even if they passed(could be bogus results). When running very 
large tests that could have over thousands of  steps, skipping bad groups of 
tests will effect both time, and being able to interpret the results.  
> 
> -- 
> Julio Merino


-- 
Cliff Wright <cliff%snipe444.org@localhost>