Re: Test interdependences, and globals

[Julio Merino <jmmv%NetBSD.org@localhost>]

On Wed, Jul 7, 2010 at 5:43 PM, Cliff Wright <cliff%snipe444.org@localhost> 
wrote:
> On Wed, 7 Jul 2010 11:07:37 +0100
> Julio Merino <jmmv%NetBSD.org@localhost> wrote:
>
> While a math library where each test is isolated would work fine, a gui would 
> not as they have state. Example a test that should be skipped instead pops up 
> a new window that blocks out an underlying window. The tests that would have 
> worked for the next group of steps on the underlying window now fails. Now I 
> have a false negative. Which for very small tests might be fine, but when 
> running with thousands of steps possibly for days this will not work. The 
> only way around this that I could think of would be to restart the gui for 
> every test, and put all the steps into a single test, but with thousands of 
> steps this will take longer than the current manual method.

Yes; your tests should be restarting the GUI for every single case to
ensure that the test starts with a known well-defined state.  If
that's too costly or impossible to do, then it's a symptom that the
code was not written with testability in mind.

(That said, I have never tested GUIs and, therefore, there hasn't been
any thought about this, at all, in ATF.)

>  ATF seems to be closer to what I want than others, so looks like I can just 
> put in my hacks to get what I need. I find the levels, and grouping that 
> currently exist very useful, so hope this feature remains.

Could you elaborate on why you think ATF suits you better than other
systems?  I'm just curious.

-- 
Julio Merino