kern/56580: evbarm-earmv7hf testbed sometimes loses serial console keystrokes

[gson%gson.org@localhost (Andreas Gustafsson)]

>Number:         56580
>Category:       kern
>Synopsis:       evbarm-earmv7hf testbed sometimes loses serial console keystrokes
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sat Dec 25 16:00:01 +0000 2021
>Originator:     Andreas Gustafsson
>Release:        NetBSD-current
>Organization:
  
>Environment:
System: NetBSD
Architecture: arm
Machine: evbarm
>Description:

Test runs on the TNF earm-earmv7hf testbed sometimes fail because
shell commands sent to the serial console of the system under test do
not get executed as expected.

It looks like the serial console is losing some of the keystrokes and
echoing the remaining ones twice.  Grepping the logs for timeouts
occurring at shell prompts shows four cases where this has happened
so far:

  lyta /bracket/evbarm-earmv7hf/results $ find . -name test.log.gz -print | xargs zgrep '#.*Traceback'
  ./2019/2019.09.27.08.57.12/test.log.gz:anita-root-shell-prompt-1569607974.5# ecoei_tts$=ecoei_tts$=Traceback (most recent call last):
  ./2021/2021.11.16.17.41.23/test.log.gz:armv7# exc/i/hexc/i/hTraceback (most recent call last):
  ./2021/2021.12.19.22.03.41/test.log.gz:armv7# PS=aiaro-hl-rm't13946.400#'PS=aiaro-hl-rm't13946.400#'Traceback (most recent call last):
  ./2021/2021.12.24.18.16.11/test.log.gz:armv7# exc/i/hexc/i/hTraceback (most recent call last):

These involve multiple different shell commands; let's pick the last
one for closer examination.

In a successful test run, the console log will contain the following
shell prompt and echoed shell command:

  armv7# exec /bin/sh

but in this failed run, it instead contains:

  armv7# exc/i/hexc/i/h

which could be explained by the characters marked with ^ below
missing:

  armv7# exec /bin/sh
           ^ ^ ^ ^ ^

and the remaining partial command being echoed twice.  The return
keypress terminating the command is apparently also lost, because the
log does not include the error message that would result if the shell
actually tried to execute the partial command.

The full console log from this run is at:

  http://releng.netbsd.org/b5reports/evbarm-earmv7hf/2021/2021.12.24.18.16.11/test.log

>How-To-Repeat:

>Fix: