Re: failing regression tests (please fix them!)

[Antti Kantee <pooka%cs.hut.fi@localhost>]

On Wed Jun 02 2010 at 15:05:43 +0300, Antti Kantee wrote:
> Hi,
> 
> We still have 24 failing regression tests in the standard "anita test" run:
> 
> Failed test cases:
>     ipf/t_ipf:bpf1, ipf/t_ipf:bpf_f1, ipf/t_ipf:n1, ipf/t_ipf:n11,
>     ipf/t_ipf:n2, ipf/t_ipf:n4, ipf/t_ipf:n5, ipf/t_ipf:n6, ipf/t_ipf:ni10,
>     ipf/t_ipf:ni11, ipf/t_ipf:ni12, ipf/t_ipf:ni19, ipf/t_ipf:ni20,
>     ipf/t_ipf:ni5, kernel/t_umount:umount, lib/libevent/t_event:kqueue,
>     lib/libevent/t_event:poll, lib/libevent/t_event:select,
>     net/sys/t_connect:low_port, util/df/t_df:normal,
>     util/grep/t_grep:basic, util/grep/t_grep:file_exp,
>     util/sh/t_expand:strip, util/sh/t_set_e:all
> 
> Most of them look like broken / unupdated tests.  (There's also a bunch
> of tests which are skipped for incorrect reasons.)
> 
> As should be well-known at least for developers, the details are available
> on Andreas' most excellent automatic build/install/test page at:
> http://www.gson.org/netbsd/bugs/build/

Ok, now we have 3 failed tests, and a good number of the tests were
fixed instead of disabled:

Failed test cases:
    fs/nullfs/t_basic:twistymount, lib/semaphore/pthread/t_sem_pth:unlink,
    util/sh/t_expand:strip

This first two have PRs for them (in the test descr) and the last one
seems like a bug, so I didn't disable it.

We have talked a little with Julio about a feature to get tests with an
associated PR classified separately -- it's useful to have regression
tests from bug reports committed into tests as soon as possible, but
they shouldn't cause the whole run to fail, as some PRs can be extremely
tricky to fix and it can take a while before they are taken care of.

stay tuned ...