tech-toolchain archive

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]

Re: Report: LLDB from trunk is running on NetBSD once again!

On Sat, 2019-03-02 at 23:56 +0000, Christos Zoulas wrote:
> In article <>,
> Micha  Górny  <> wrote:
> > -=-=-=-=-=-
> > 
> > Hello,
> > 
> > I've just published the report on my first month of work on LLDB: 
> >
> > 
> > Long story short, I'm working on LLDB as contracted by the NetBSD
> > Foundation.  I've been able to fix the regression preventing LLDB from
> > starting processes, and I've been working on getting the test suite to
> > work reliably (passing or xfailing as appropriate), in order to enable
> > continuous integration and regression catching.
> > 
> > In the report I've detailed on the more interesting issues I've found,
> > and summarized what I'm about to do next.
> Thanks a lot! This is good progress. Can you also say how many tests
> are still failing and what's your estimate of the effort fixing them?
> I.e. do you think that we are missing functionality, are there bugs
> we need to fix etc?

As Kamil mentioned, that's non-trivial to establish, especially for two

1. LLDB's test suite is built on top of multiple combined test runners
with a lot of historical baggage.  As a result, multiple 'subtests' are
mapped into one lit test case, and sometimes 1 test in the output may
equal over 10 different real test cases.  This also loses the original
XFAIL semantics, as the test wrapper merges it into PASS.  So the final
report can only tell how many new-style (lit) tests fail because old-
style tests are all merged into passes.

2. Some tests are marked XFAIL on practically all platforms (sometimes
with notes indicating the test case is broken), so it is hard to
establish how much of the work is NetBSD-specific, and how much is
cross-platform upstream work.

3. Finally, some of the test passes (especially tests for 'something bad
not happening') may be false negatives right now, and may start failing
once we fix the underlying feature.

Nevertheless, I'll try to summarize it in some numbers, using poor man's

The Python test suite has 1403 cases (grep 'def test.*self').
At the moment I have 165 marked expectedFailureNetBSD, and 16 instances
of skipIfNetBSD (note that this includes a few Darwin-specific tests
which are irrelevant to us).

The pure lit test suite has 209 cases (looking at file counts), of which
I have 16 marked 'XFAIL: system-netbsd'.

Unittests have 678 cases.  Only 1 is skipped and that only temporarily
until the pty kernel fix is in netbsd-8 for a while.

Please note that those numbers are rough.

Best regards,
Michał Górny

Attachment: signature.asc
Description: This is a digitally signed message part

Home | Main Index | Thread Index | Old Index