Re: port-arm/55598: ChaCha self-test sometimes fails on evbarm-earmv7hf testbed

[Andreas Gustafsson <gson%gson.org@localhost>]

The following reply was made to PR port-arm/55598; it has been noted by GNATS.

From: Andreas Gustafsson <gson%gson.org@localhost>
To: riastradh%netbsd.org@localhost,
Cc: gnats-bugs%netbsd.org@localhost
Subject: Re: port-arm/55598: ChaCha self-test sometimes fails on evbarm-earmv7hf testbed
Date: Sat, 22 Aug 2020 22:50:41 +0300

 Taylor,
 
 You wrote:
 >  Are these the only instances where it failed?
 
 Almost - there have been four instances so far (counting a case where
 the system rebooted and then failed again as one, not two).
 
 > Are there other instances where it succeeded, when running exactly
 > the same software?
 
 That question is only meaningful if the exact same software has been
 tested more than once, but the testbed doesn't do that.
 
 It's probably more useful to consider the full set of outcomes from
 the last 25 test runs (these include all the failures).  A zero
 in the rightmost column means success, nonzero means failure:
 
   lyta /bracket/evbarm-earmv7hf/results $ zgrep -c 'chacha: self-test failed' 2020/*/test.log.gz | tail -25
   2020/2020.08.02.01.36.46/test.log.gz:0
   2020/2020.08.02.10.49.53/test.log.gz:0
   2020/2020.08.03.07.25.59/test.log.gz:0
   2020/2020.08.03.20.43.41/test.log.gz:0
   2020/2020.08.04.04.22.02/test.log.gz:1
   2020/2020.08.04.14.35.24/test.log.gz:0
   2020/2020.08.05.01.58.42/test.log.gz:0
   2020/2020.08.05.10.33.01/test.log.gz:0
   2020/2020.08.05.16.36.33/test.log.gz:0
   2020/2020.08.06.07.38.54/test.log.gz:0
   2020/2020.08.06.16.23.52/test.log.gz:0
   2020/2020.08.07.00.43.52/test.log.gz:0
   2020/2020.08.07.07.19.45/test.log.gz:0
   2020/2020.08.07.23.31.07/test.log.gz:0
   2020/2020.08.08.19.39.28/test.log.gz:1
   2020/2020.08.17.09.03.03/test.log.gz:0
   2020/2020.08.17.19.41.32/test.log.gz:0
   2020/2020.08.18.03.02.50/test.log.gz:0
   2020/2020.08.18.14.32.34/test.log.gz:0
   2020/2020.08.18.23.31.50/test.log.gz:0
   2020/2020.08.19.09.22.05/test.log.gz:0
   2020/2020.08.19.22.47.09/test.log.gz:2
   2020/2020.08.20.11.09.56/test.log.gz:1
   2020/2020.08.21.17.51.31/test.log.gz:0
   2020/2020.08.22.01.43.07/test.log.gz:0
 
 >  (There have been some changes to sys/crypto/chacha over the time
 >  period covered by those three dates.)  Does tests/sys/crypto/chacha
 >  fail randomly if repeated?
 
 That would have to be tested separately, but I'm not sure I see the
 point since the reported issue is not the failure of those tests but
 of the kernel's built in self test, and we already know the tests in
 sys/crypt/chacha passed in all 25 of the above runs.  If you think
 testing this will actually yield some useful information, which
 version would you like me to repeatedly test?
 -- 
 Andreas Gustafsson, gson%gson.org@localhost