[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]
Re: About non-deterministic tests
On 9/18/11 9:11 AM, Jukka Ruohonen wrote:
> This is not really related to ATF, but more of a question related to testing
> generally and specifically testing done in NetBSD.
> In NetBSD, and I guess in anything that does system testing, many tests for
> known defects are not failing always but just occasionally. The examples
> range from race condition tests to different timing tests. However, it
> would be much easier to keep track of the results if the baseline would be
> "zero failures" (at least on the architectures that are part of the daily
> Now perhaps it would be possible to modify these tests so that they always
> fail. But as we are reaching 3000 test cases, it does not seem reasonable
> that few tests are trying to catch failures by running ten minutes. In
> addition, it is known that Qemu contains severe timer issues (cf. Anita).
Yeah, this is a feature requested by pooka@ a long time ago and it would
definitely make some tests easier to write.
I think we should have a way to:
- Declare this kind of test cases more easily, without having to
resort to weird expect/fail combinations.
- The user should be able to tune for how long the race condition
tests run. The defaults should probably be to run for a reasonable
amount of time (10 minutes is not reasonable), but the user may
want to run things in a more exhaustive manner. Plus, with
parallel execution, running long tests should not be that of a big
I have filed http://code.google.com/p/kyua/issues/detail?id=27 with the
details above. Do you think this covers what you have in mind?
Julio Merino / @jmmv
Main Index |
Thread Index |