[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]
Re: Test interdependences, and globals
On Tue, 6 Jul 2010 21:40:27 +0100
Julio Merino <jmmv%NetBSD.org@localhost> wrote:
> Can you provide some specific examples of why are you trying to do
> this? Why can't you recreate that unique value as a setup step for
> every test case?
The testing we are doing is far to complicated to do as a single test program.
The concept of levels (e.g test suite, test program) works very well for us. In
the early setup, multiple test programs are run in a specific sequence to get a
gui in the right state to display a page of buttons, one of which generates the
unique value that will never be generated again. To combine all these steps
into a single test program, would complicate our ability to test each little
step(does the window exist?, did the menu pop-up?, is the button green?). So
even if I could regenerate the unique value(and I can't), as soon as I run the
next test (which might be: find and press the yellow button that has this
unique label) the unique value from the previous test is lost. This breakup of
an immense test into smaller tests (and then into even smaller tests) is very
important to us.
> Adding test dependencies would be possible, albeit tricky, but it
> won't happen without a real and convincing use case for it. Can you
> elaborate? If the first test fails, what's the problem of running the
> other? They'll just report failure, which is OK because they *are*
> actually failing.
The problem particularly with a gui is the following steps might effect the
gui(e.g the wrong window is now open) so that the following suite(group) of
tests might now fail when they otherwise would have succeeded. We will be
running full tests with thousands of steps, so having usable results for
individual groups is very important.
If you add dependencies, you are hiding real tests
> on the assumption that their failures won't be helpful (and in most
> cases such results are helpful because they provide additional data
> points of why things failed).
I can't get specific, so here is a for instance. Say I want to run a suite of
tests on xcalc, 1 suite with it in rpn mode, and one suite with it not in rpn
mode. Say I find a major error in rpn mode. I would now like to skip all other
rpn tests, and now run the non-rpn tests. Any rpn tests that now run would have
to be ignored even if they passed(could be bogus results). When running very
large tests that could have over thousands of steps, skipping bad groups of
tests will effect both time, and being able to interpret the results.
> Julio Merino
Cliff Wright <cliff%snipe444.org@localhost>
Main Index |
Thread Index |