kern/41058: General software failures with kernels built for arm/NSLU2 kernels

[don%donhayford.com@localhost]

>Number:         41058
>Category:       kern
>Synopsis:       General software failures with kernels built for arm/NSLU2 
>kernels
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sun Mar 22 13:35:00 +0000 2009
>Originator:     Donald T Hayford
>Release:        -current from 4/26/2008 through 3/14/2009
>Organization:
self
>Environment:
NetBSD netbsd. 5.99.01 NetBSD 5.99.01 (NSLU2_ALL) #0: Sun Mar  8 22:42:32 EDT 
2009  
hayford@free-2k:/usr/home/hayford/netbsd-20081107/obj-armeb/sys/arch/evbarm/compile/NSLU2_ALL
>Description:
There have been recent reports of certain problems with NetBSD on the NSLU2, 
both -current and older.  For example, see:
http://mail-index.netbsd.org/port-arm/2009/03/08/msg000693.html
and the associated thread for more information.  
The symptoms are a general malaise with various bits of software failing (cc 
reporting internal errors, md5 returning inconsistent results) at random times. 
 The test that seems to work best, though it is not a great test, is to md5 a 
large file.  Older versions of the kernel will return a different (and 
incorrect) result occasionally (1 or 2 times out of 30).  Newer kernels fail 
significantly more often.  For example here are the results for a kernel built 
using a cvs date of 11/07/2008:

MD5 (pkgsrc-2008Q3.tar.gz) = 0710cd9aa73651cd00b274fed0399a3d (Note: this is 
the correct answer)
MD5 (pkgsrc-2008Q3.tar.gz) = 52925bf5588c9643ee58f4f24ab024b0
MD5 (pkgsrc-2008Q3.tar.gz) = b5af39add036cd54f78ff38384a60370
MD5 (pkgsrc-2008Q3.tar.gz) = f0d5f16bdc59d09f906557149bc9a326
MD5 (pkgsrc-2008Q3.tar.gz) = 0710cd9aa73651cd00b274fed0399a3d
MD5 (pkgsrc-2008Q3.tar.gz) = 0f6c17c863a497165e78729c6c19f042
MD5 (pkgsrc-2008Q3.tar.gz) = 767c33a02ce12ef37b0cfed747586451
MD5 (pkgsrc-2008Q3.tar.gz) = 39ae3194a7306e5cafdd1767bd26d141
MD5 (pkgsrc-2008Q3.tar.gz) = 550be4e993b76ccd8c218c6b59e69f03
MD5 (pkgsrc-2008Q3.tar.gz) = 60beb07311a37b4076d39aa85eadf748
MD5 (pkgsrc-2008Q3.tar.gz) = 0710cd9aa73651cd00b274fed0399a3d
MD5 (pkgsrc-2008Q3.tar.gz) = 0710cd9aa73651cd00b274fed0399a3d
MD5 (pkgsrc-2008Q3.tar.gz) = 680e4b443e0984efdc3f8f3ac6d67b62
MD5 (pkgsrc-2008Q3.tar.gz) = 273d8d3bf711ab59fb55bede9cb077d2
MD5 (pkgsrc-2008Q3.tar.gz) = 25edba177456c0059964ddff9371095d
MD5 (pkgsrc-2008Q3.tar.gz) = b0cc81b7fc8d66e8e65d174b432762d6
MD5 (pkgsrc-2008Q3.tar.gz) = 0710cd9aa73651cd00b274fed0399a3d
MD5 (pkgsrc-2008Q3.tar.gz) = 0710cd9aa73651cd00b274fed0399a3d
MD5 (pkgsrc-2008Q3.tar.gz) = 0951fb446714bfc3842dd57891e7288a
MD5 (pkgsrc-2008Q3.tar.gz) = 7c7b8bb852e567dec180bee4fbb70888
MD5 (pkgsrc-2008Q3.tar.gz) = a426775dee944c97159098e0119efa26
MD5 (pkgsrc-2008Q3.tar.gz) = 0710cd9aa73651cd00b274fed0399a3d
MD5 (pkgsrc-2008Q3.tar.gz) = bc60fe91f8409fda6fc8cbb834b45102
MD5 (pkgsrc-2008Q3.tar.gz) = a2725280cd57be4e388903e933708852
MD5 (pkgsrc-2008Q3.tar.gz) = 9902781605916302983831b48947592b
MD5 (pkgsrc-2008Q3.tar.gz) = 3adf6c0a2616b7f9720b9874e013b8b2
MD5 (pkgsrc-2008Q3.tar.gz) = 86c0e9d9ba56bde4587d01d5606e9a90
MD5 (pkgsrc-2008Q3.tar.gz) = 0710cd9aa73651cd00b274fed0399a3d
MD5 (pkgsrc-2008Q3.tar.gz) = 0710cd9aa73651cd00b274fed0399a3d
MD5 (pkgsrc-2008Q3.tar.gz) = c442a730b262580d7f3ba6ef44622057
MD5 (pkgsrc-2008Q3.tar.gz) = 72d8b997c41de9da6d7fbe0202115ca6
MD5 (pkgsrc-2008Q3.tar.gz) = fc963410678d6b514580762e0feb2126
MD5 (pkgsrc-2008Q3.tar.gz) = 16209fe1355c850392dd46775cd8547e
MD5 (pkgsrc-2008Q3.tar.gz) = 0710cd9aa73651cd00b274fed0399a3d
MD5 (pkgsrc-2008Q3.tar.gz) = 0710cd9aa73651cd00b274fed0399a3d
MD5 (pkgsrc-2008Q3.tar.gz) = 7c11cbe472c9e16fff77b20d9bede6b7
# uname -a
NetBSD netbsd. 5.99.01 NetBSD 5.99.01 (NSLU2_ALL) #0: Sun Mar  8 22:42:32 EDT 
2009  
hayford@free-2k:/usr/home/hayford/netbsd-20081107/obj-armeb/sys/arch/evbarm/compile/NSLU2_ALL

I have tested kernels back to 5/5/2008 and still see the problem, though the 
failure rates are pretty low (1 in 30 to 1 in 50).  I tried to build a kernel 
from 4/26/2008, but was unable to, for some reason.  However, I was able to 
substitute the following two directories from 4/26/2008 into later builds 
(5/12/2008, 5/16/2008 and 6/1/2008) and get the kernel and world to build 
correctly:
    src/sys/arch/arm
    src/sys/arch/evbarm
These bastardized kernels seem to run correctly - i.e., no failures in more 
than 100 trials.  I tried to be more selective with the files that were 
substituted (i.e., just the arm/xscale directory or just the evbarm/nslu2 
directory), but unfortunately there are enough dependencies scattered about 
that the build fails.  Likewise, I can't substitute these directories into 
anything approaching -current and build correctly.
>How-To-Repeat:
Build any kernel from 5/5/2008 to present for the NSLU2.
Run md5 on any large file (I use pkgsrc-2008Q3.tar.gz).  Do it several times 
and you will get different (and incorrect) results.
>Fix:
Using src/sys/arch/arm and src/sys/arch/evbarm from 4/26/2008 with src from 
5/12/2008 - 6/1/2008 will build and apparently run correctly.