On Sat, 2019-03-02 at 23:56 +0000, Christos Zoulas wrote: > In article <f6d4ca20179f1a2b4278e411a86ccc21ad203e59.camel%NetBSD.org@localhost>, > Micha Górny <mgorny%NetBSD.org@localhost> wrote: > > -=-=-=-=-=- > > > > Hello, > > > > I've just published the report on my first month of work on LLDB: > > http://blog.netbsd.org/tnf/entry/lldb_from_trunk_is_running > > > > Long story short, I'm working on LLDB as contracted by the NetBSD > > Foundation. I've been able to fix the regression preventing LLDB from > > starting processes, and I've been working on getting the test suite to > > work reliably (passing or xfailing as appropriate), in order to enable > > continuous integration and regression catching. > > > > In the report I've detailed on the more interesting issues I've found, > > and summarized what I'm about to do next. > > Thanks a lot! This is good progress. Can you also say how many tests > are still failing and what's your estimate of the effort fixing them? > I.e. do you think that we are missing functionality, are there bugs > we need to fix etc? > As Kamil mentioned, that's non-trivial to establish, especially for two reasons: 1. LLDB's test suite is built on top of multiple combined test runners with a lot of historical baggage. As a result, multiple 'subtests' are mapped into one lit test case, and sometimes 1 test in the output may equal over 10 different real test cases. This also loses the original XFAIL semantics, as the test wrapper merges it into PASS. So the final report can only tell how many new-style (lit) tests fail because old- style tests are all merged into passes. 2. Some tests are marked XFAIL on practically all platforms (sometimes with notes indicating the test case is broken), so it is hard to establish how much of the work is NetBSD-specific, and how much is cross-platform upstream work. 3. Finally, some of the test passes (especially tests for 'something bad not happening') may be false negatives right now, and may start failing once we fix the underlying feature. Nevertheless, I'll try to summarize it in some numbers, using poor man's grep. The Python test suite has 1403 cases (grep 'def test.*self'). At the moment I have 165 marked expectedFailureNetBSD, and 16 instances of skipIfNetBSD (note that this includes a few Darwin-specific tests which are irrelevant to us). The pure lit test suite has 209 cases (looking at file counts), of which I have 16 marked 'XFAIL: system-netbsd'. Unittests have 678 cases. Only 1 is skipped and that only temporarily until the pty kernel fix is in netbsd-8 for a while. Please note that those numbers are rough. -- Best regards, Michał Górny
Attachment:
signature.asc
Description: This is a digitally signed message part