On Fri, 22 Apr 2011, Paul Goyette wrote:
On Thu, 21 Apr 2011, Martin Husemann wrote:This brought it down to "normal" levels for me: Summary for 400 test programs: 2208 passed test cases. 9 failed test cases. 56 expected failed test cases. 52 skipped test cases.My test-bed has decided to revert to an earlier failure mode after several months of behaving nicely. I have had one successful build, however the atf-run's xml output file was truncated in mid-flight.The "ticker" output file indicates that things are closer to normal now: Summary for 402 test programs: 2194 passed test cases. 18 failed test cases. 56 expected failed test cases. 62 skipped test cases.Prior to this episode, there was a long period of only a handful of test failures, so the current 18 is still quite a bit higher than expected. Still ,it is much better than the previous 38 failures.I'm hoping the test-bed system goes back into remission soon, so we can get some reliable results.
The test-bed has once again settled down, and is producing results that are at least repeatable. :)
The original problems that prompted this thread have NOT been totally resolved. There are still about a dozen test cases that had been working are now consistently failing.
There seems to be a real problem in the filesystem code that may be causing corruption. It is probably triggered by the tests that fill up a filesystem, but once this occurs it seems that other files (even on other filesystems!) are affected. The earlier comment about the "xml output file was truncated in mid-flight" reflects this corruption.
It does not happen every time, but in the past 26 test-passes it has occurred 17 times.
There may be other conditions required to trigger this current bug. One possibility is that the test-bed runs qemu in a fairly low-memory environment - 128MB! This may (or may not) also explain why gson's i386 test bed is not seeing the problem, since i386 consumes somewhat less memory.
Since it's not crashing any more, and the file corruption is not apparent or obvious until the end of the test run, I am unsure how to proceed debugging this problem. Suggestions would be welcomed.
------------------------------------------------------------------------- | Paul Goyette | PGP Key fingerprint: | E-mail addresses: | | Customer Service | FA29 0E3B 35AF E8AE 6651 | paul at whooppee.com | | Network Engineer | 0786 F758 55DE 53BA 7731 | pgoyette at juniper.net | | Kernel Developer | | pgoyette at netbsd.org | -------------------------------------------------------------------------