Re: Test runs timing out on b5

[Andreas Gustafsson <gson%gson.org@localhost>]

Robert Elz wrote:
>   |   lib/libc/gen/t_sleep:kevent
>   |   lib/libevent/t_event:kqueue
>   |   lib/libevent/t_event:poll
>   |   lib/libevent/t_event:select
> 
> As I recall those have been timing out for ages (I assumed probably
> related to the qemu timimg issue, as they're all timer sensitive.)

Right, those should not have been included in the list.  That was
sloppy editing on my part.

> So that leaves just ...
> 
>   |   lib/libc/ssp/t_ssp:fgets
>   |   lib/libc/ssp/t_ssp:getcwd
>   |   lib/libc/ssp/t_ssp:gets
>   |   lib/libc/ssp/t_ssp:memcpy
>   |   lib/libc/ssp/t_ssp:memmove
>   |   lib/libc/ssp/t_ssp:memset
>   |   lib/libc/ssp/t_ssp:raw
>   |   lib/libc/ssp/t_ssp:read
>   |   lib/libc/ssp/t_ssp:readlink
>   |   lib/libc/ssp/t_ssp:snprintf
>   |   lib/libc/ssp/t_ssp:sprintf
>   |   lib/libc/ssp/t_ssp:stpcpy
>   |   lib/libc/ssp/t_ssp:stpncpy
>   |   lib/libc/ssp/t_ssp:strcat
>   |   lib/libc/ssp/t_ssp:strcpy
>   |   lib/libc/ssp/t_ssp:strncat
>   |   lib/libc/ssp/t_ssp:strncpy
>   |   lib/libc/ssp/t_ssp:vsnprintf
>   |   lib/libc/ssp/t_ssp:vsprintf
> 
> as new failures.   I cannot see any reason changes to syslog could be
> in any way related (none of the ssp tests use syslog),

security(7) says "Upon detection of a buffer overrun, SSP will
immediately abort execution of the program and send a log message
to syslog(3)."

> however at about the same time ...
> 
> commit 2017.01.12.00.43.55 christos src/lib/libc/include/extern.h 1.25
> commit 2017.01.12.00.43.55 christos src/lib/libc/string/strerror_ss.c 1.2
> 
> and these - according to the i386 build log where they did start timing
> out before the build broke, yet again, these were the last changes before
> the timeouts started ...

The tests of 2017.01.12.00.38.01 sources yielded "new failures: 19
test cases" rather than "did not complete" because I increased the
timeout before that test was run.  This result should be considered
a failure just like the "did not complete" ones, so the syslog changes
of 2017.01.12.00.38.01 are still the likely cause.

I have now also verified this by testing on real hardware:

  http://www.gson.org/netbsd/bugs/build/amd64-baremetal/commits-2017.01.html#2017.01.12.00.38.01

> the build after the syslog changes has, just
> recently, completes its test run - now b5 appears to be checking to see
> if
> 
> commit 2017.01.12.00.38.25 riastradh src/lib/libc/README 1.6
> 
> might have caused the issue!   We really should make things a little smarter
> and know that changes to readmes or CHANGES files, etc, cannot have any
> effect on anything, and not do a build when that is the only change, and
> not bother doing, as now, and bisecting the changes treating this as if it
> might have caused a problem.

CPU time is cheap, and I think there is value in being able to say
that you have actually tested the sources immediately before and after
a suspect commit, rather than relying on the assumption that certain
types of commits can't possibly cause problems.
-- 
Andreas Gustafsson, gson%gson.org@localhost