Re: Report: LLDB from trunk is running on NetBSD once again!

[christos%zoulas.com@localhost (Christos Zoulas)]

On Mar 3, 11:19am, mgorny%gentoo.org@localhost (=?UTF-8?Q?Micha=C5=82_G=C3=B3rny?=) wrote:
-- Subject: Re: Report: LLDB from trunk is running on NetBSD once again!

| As Kamil mentioned, that's non-trivial to establish, especially for two
| reasons:
| 
| 1. LLDB's test suite is built on top of multiple combined test runners
| with a lot of historical baggage.  As a result, multiple 'subtests' are
| mapped into one lit test case, and sometimes 1 test in the output may
| equal over 10 different real test cases.  This also loses the original
| XFAIL semantics, as the test wrapper merges it into PASS.  So the final
| report can only tell how many new-style (lit) tests fail because old-
| style tests are all merged into passes.
| 
| 2. Some tests are marked XFAIL on practically all platforms (sometimes
| with notes indicating the test case is broken), so it is hard to
| establish how much of the work is NetBSD-specific, and how much is
| cross-platform upstream work.
| 
| 3. Finally, some of the test passes (especially tests for 'something bad
| not happening') may be false negatives right now, and may start failing
| once we fix the underlying feature.
| 
| Nevertheless, I'll try to summarize it in some numbers, using poor man's
| grep.
| 
| The Python test suite has 1403 cases (grep 'def test.*self').
| At the moment I have 165 marked expectedFailureNetBSD, and 16 instances
| of skipIfNetBSD (note that this includes a few Darwin-specific tests
| which are irrelevant to us).
| 
| The pure lit test suite has 209 cases (looking at file counts), of which
| I have 16 marked 'XFAIL: system-netbsd'.
| 
| Unittests have 678 cases.  Only 1 is skipped and that only temporarily
| until the pty kernel fix is in netbsd-8 for a while.
| 
| Please note that those numbers are rough.

Thanks, that gives us a much better idea of what's going on (even if the
numbers can't be accurate).

Best,

christos