On Sun, 3 Sep 2006, Valeriy E. Ushakov wrote:

> On Sun, Sep 03, 2006 at 16:45:27 +0100, Stephen Borrill wrote:
>
>> I'm running an NFS 3.1_RC1 userland with a GENERIC 4.99.1 kernel on my
>> Jornada 690. I've 128MB of NFS swap configured and I've been compiling up
>> (some of) pkgsrc. My network card is a Linksys WPC11 v3 (wi0). At
>> seemingly random times (sometimes after 24 hours of uptime, but sometimes
>> only an hour), I've seen the machine apparently lock up (logged in over
>> ssh), but the screen's been blank when I've got to the machine itself and
>> I've hit the reset button. Today however, I caught it at a panic:
>>
>> fatal address error (load) in kernel mode
>>  spc 0 ssr 400001f0
>> panic: general_exception
>> kloader: kernel file name: /netbsd
>> panic: tlb_exception: no copyin/out fault handler (page not found)
>> expevt=c0 va=c0077000 ssr=400001f1 spc=8c0766d6 lwp=0x8cf2ba4c onfault=0x0
>>
>> Would I have more joy with a kernel that wasn't -current? Am I just
>> hitting a lack of memory?
>
> Hmm, EXPEVT=C0 is protection violation on store.  Can you do in DDB:
>
> db> bt                  # backtrace

Hmm, a couple of different ones. Here's the first that I wrote by hand 
before I attached a serial console:

cpu_Debugger() at netbsd:panic+0xaa
panic() at netbsd: tlb_exception+0x45e
tlb_exception() at 0x8c0004b8
<EXPEVT 040; SSR=40000131> at netbsd:nfs_sigintr+0x16
nfs_sigintr() at netbsd:nfstimer+0x44
nfs_timer() at netbsd:softclock+0x1d2
softclock() at netbsd:saoftintr_dispatch+0x62
tmul_intr() at netbsd:intc_intr+0xa0
intc_intr() at 0x8c000680
<EXPEVT 000; SSR=40000100> at netbsd:hd6446x_int_resume+0x1e
hd6446x_intr_resume() at 0
() at 0

And here's the one I've just had:

cpu_Debugger() at netbsd:panic+0xaa
panic() at netbsd:tlb_exception+0x45e
tlb_exception() at 0x8c0004b8
<EXPEVT 040; SSR=400001f1> at netbsd:nfs_rcvlock+0x1c
nfs_rcvlock() at netbsd:nfs_reply+0x30
nfs_reply() at netbsd:nfs_request+0x4f6
nfs_request() at netbsd:nfs_readrpc+0x374
nfs_readrpc() at netbsd:nfs_doio+0x154
nfs_doio() at netbsd:nfs_strategy+0x34
nfs_strategy() at netbsd:VOP_STRATEGY+0x1e
VOP_STRATEGY() at netbsd:sw_reg_start+0x5e
sw_reg_start() at netbsd:swstrategy+0x2d6
swstrategy() at netbsd:spec_strategy+0x118
spec_strategy() at netbsd:VOP_STRATEGY+0x1e
VOP_STRATEGY() at netbsd:uvm_swap_io+0xf2
uvm_swap_io() at netbsd:uvm_swap_get+0x3c
uvm_swap_get() at netbsd:uvmfault_anonget+0x158
uvmfault_anonget() at netbsd:uvm_fault_internal+0x7c6
uvm_fault_internal() at netbsd:tlb_exception+0x236
tlb_exception() at 0x8c0004b8
<EXPEVT 040; SSR=00000000> at 0x6444ea


> db> x/i 8c0766d6        # spc

From the first:
netbsd:nfs_sigintr+0x16:	mov.l	@(4,r4), r0

The second:
netbsd:nfs_rcvlock+0x1c:        mov.l   @r8, r0

> db> show reg

From the first:
r0	0x1
r1	0x8c1002a8	cpu_Debugger
r2	0xbd3
r3	0x8c1e9000	_prop_string_pool+0x3d490
r4	0x400001f0
r5	0
r6	0
r7	0x8c14b939	_link_session_pool+0x7a9
r8	0xc2321de4
r9	0x8c0bf770	printf
r10	0x8c154664	db_eregs+0x174
r11	0x100
r12	0x4
r13	0x8c191470	nfsstats+0x7c
r14	0xc2321e1c
r15	0xc2321e1c
pr	0x8c0bfd12	panic+0xaa
spc	0x8c1002ae	cpu_Debugger+0x6
ssr	0x400001f1
mach	0
macl	0x47e00
netbsd:cup_Debugger+0x6:	mov	r14,r15

And the second:
r0          0x1
r1          0x8c1002a8  cpu_Debugger
r2          0xd22
r3          0x8c1e9000  _prop_string_pool+0x3d490
r4          0x400001f0
r5          0xd
r6          0
r7          0x8c14b939  _link_session_pool+0x7a9
r8          0xc2436b24
r9          0x8c0bf770  printf
r10         0x8c154664  db_eregs+0x174
r11         0x100
r12         0x2537782c
r13         0x2537763c
r14         0xc2436b5c
r15         0xc2436b5c
pr          0x8c0bfd12  panic+0xaa
spc         0x8c1002ae  cpu_Debugger+0x6
ssr         0x400001f1
mach        0x1
macl        0x839b680
netbsd:cpu_Debugger+0x6:        mov     r14, r15

> db> mach tlb

Didn't get one for the first as it was too long.

From the second:
VPN only-mode, multiple virtual storage mode
ASID=21 (cc1plus)---TLB DUMP---
    VPN      ASID    PFN  AREA VDCGWtPR  SZ
           U/K                       U/K
  [way 0]
0xc0640000 K   0 0x0cb6d000 3 |.||x _r  4K
0xc0641000 K   0 0x0e682000 3 |.||x _r  4K
0x00402000 U   9 0x0cbd9000 3 |.|.x rr  4K
0xc0683000 K   0 0x0c600000 3 |.||x _r  4K
0xc0684000 K   0 0x0ed21000 3 |.||x _r  4K
0xc0685000 K   0 0x0e94a000 3 |.||x _r  4K
0xc0686000 K   0 0x0c32a000 3 |.||x _r  4K
0xc0687000 K   0 0x0ca95000 3 |.||x _r  4K
0xc0688000 K   0 0x0cd9d000 3 |.||x _r  4K
0x25049000 U  21 0x0c973000 3 |.|.x rr  4K
0x00baa000 U  21 0x0ccf4000 3 |||.x ww  4K
0xc006b000 K   0 0x0ef2a000 3 ||||x _w  4K
0xc062c000 K   0 0x0e672000 3 |.||x _r  4K
0xc006d000 K   0 0x0ef28000 3 ||||x _w  4K
0xc002e000 K   0 0x0ef6e000 3 ||||x _w  4K
0xc066f000 K   0 0x0c6fb000 3 |.||x _r  4K
0xc0670000 K   0 0x0e849000 3 |.||x _r  4K
0xc0671000 K   0 0x0ea65000 3 |.||x _r  4K
0xc0672000 K   0 0x0ed01000 3 |.||x _r  4K
0xc0673000 K   0 0x0ee49000 3 |.||x _r  4K
0xc0674000 K   0 0x0cd81000 3 |.||x _r  4K
0xc0675000 K   0 0x0c8ee000 3 |.||x _r  4K
0xc0676000 K   0 0x0ee16000 3 |.||x _r  4K
0xc0677000 K   0 0x0e063000 3 |.||x _r  4K
0xc0678000 K   0 0x0eea8000 3 |.||x _r  4K
0xc0679000 K   0 0x0ea64000 3 |.||x _r  4K
0xc067a000 K   0 0x0edec000 3 |.||x _r  4K
0xc067b000 K   0 0x0ed0f000 3 |.||x _r  4K
0xc067c000 K   0 0x0e3f6000 3 |.||x _r  4K
0xc067d000 K   0 0x0e2cb000 3 |.||x _r  4K
0xc007e000 K   0 0x0e1fb000 3 |.||x _r  4K
0xc065f000 K   0 0x0e171000 3 |.||x _r  4K
  [way 1]
0xc0660000 K   0 0x0e00c000 3 |.||x _r  4K
0xc0661000 K   0 0x0e9f7000 3 |.||x _r  4K
0xc0642000 K   0 0x0e20e000 3 |.||x _r  4K
0xc2423000 K   0 0x0c51b000 3 ||||x _w  4K
0xc0004000 K   0 0x0eff4000 3 ||||x _w  4K
0xc0645000 K   0 0x0cc66000 3 |.||x _r  4K
0xc0006000 K   0 0x0efe2000 3 ||||x _w  4K
0xc0647000 K   0 0x0c447000 3 |.||x _r  4K
0xc0648000 K   0 0x0eea0000 3 |.||x _r  4K
0xc0649000 K   0 0x0c855000 3 |.||x _r  4K
0xc064a000 K   0 0x0e0c0000 3 |.||x _r  4K
0xc064b000 K   0 0x0cd64000 3 |.||x _r  4K
0xc064c000 K   0 0x0ed25000 3 |.||x _r  4K
0xc064d000 K   0 0x0e076000 3 |.||x _r  4K
0xc006e000 K   0 0x0ef27000 3 ||||x _w  4K
0x2502f000 U  21 0x0e9f6000 3 |||.x ww  4K
0xc0070000 K   0 0x0ef25000 3 ||||x _w  4K
0x25051000 U  21 0x0c890000 3 |.|.x ww  4K
0x25072000 U  21 0x0ccfc000 3 |||.x ww  4K
0xc0053000 K   0 0x0ef47000 3 ||||x _w  4K
0x25034000 U  21 0x0edcc000 3 |.|.x rr  4K
0x25075000 U  21 0x0cdfe000 3 |.|.x rr  4K
0xc2436000 K   0 0x0c7bb000 3 ||||x _w  4K
0x241f7000 U  21 0x0e366000 3 |||.x ww  4K
0x00638000 U  21 0x0e5d0000 3 |.|.x rr  4K
0xc0079000 K   0 0x0c266000 3 ||||x _w  4K
0xc009a000 K   0 0x0c665000 3 ||||x _w  4K
0xc063b000 K   0 0x0ec63000 3 |.||x _r  4K
0xc063c000 K   0 0x0e353000 3 |.||x _r  4K
0xc063d000 K   0 0x0eb0d000 3 |.||x _r  4K
0xc009e000 K   0 0x0c620000 3 ||||x _w  4K
0xc067f000 K   0 0x0c943000 3 |.||x _r  4K
  [way 2]
0xc0680000 K   0 0x0c73d000 3 |.||x _r  4K
0xc0681000 K   0 0x0c391000 3 |.||x _r  4K
0xc0662000 K   0 0x0e032000 3 |.||x _r  4K
0xc0663000 K   0 0x0e595000 3 |.||x _r  4K
0xc0664000 K   0 0x0cea5000 3 |.||x _r  4K
0xc0665000 K   0 0x0e04d000 3 |.||x _r  4K
0xc0666000 K   0 0x0e477000 3 |.||x _r  4K
0xc0667000 K   0 0x0c84f000 3 |.||x _r  4K
0xc0668000 K   0 0x0eaea000 3 |.||x _r  4K
0xc0669000 K   0 0x0e088000 3 |.||x _r  4K
0xc066a000 K   0 0x0eb75000 3 |.||x _r  4K
0xc066b000 K   0 0x0ce84000 3 |.||x _r  4K
0xc006c000 K   0 0x0ef29000 3 ||||x _w  4K
0xc066d000 K   0 0x0c50a000 3 |.||x _r  4K
0xc064e000 K   0 0x0e8b5000 3 |.||x _r  4K
0xc064f000 K   0 0x0eb8d000 3 |.||x _r  4K
0xc0650000 K   0 0x0e7e5000 3 |.||x _r  4K
0xc0651000 K   0 0x0e806000 3 |.||x _r  4K
0xc0652000 K   0 0x0eb8a000 3 |.||x _r  4K
0xc0653000 K   0 0x0c85a000 3 |.||x _r  4K
0xc0654000 K   0 0x0c7d5000 3 |.||x _r  4K
0xc0655000 K   0 0x0c97e000 3 |.||x _r  4K
0xc0656000 K   0 0x0c427000 3 |.||x _r  4K
0xc0657000 K   0 0x0e7fe000 3 |.||x _r  4K
0xc0658000 K   0 0x0e283000 3 |.||x _r  4K
0xc0659000 K   0 0x0ede2000 3 |.||x _r  4K
0xc065a000 K   0 0x0cb30000 3 |.||x _r  4K
0xc065b000 K   0 0x0ccb8000 3 |.||x _r  4K
0xc065c000 K   0 0x0e9b2000 3 |.||x _r  4K
0xc065d000 K   0 0x0c621000 3 |.||x _r  4K
0xc063e000 K   0 0x0eb93000 3 |.||x _r  4K
0xc009f000 K   0 0x0c641000 3 ||||x _w  4K
  [way 3]
0x204e0000 U   9 0x0c2f9000 3 |.|.x rr  4K
0xc2321000 K   0 0x0c27a000 3 ||||x _w  4K
0xc0042000 K   0 0x0ef58000 3 ||||x _w  4K
0x207e3000 U  21 0x0ce4f000 3 |.|.x rr  4K
0x00644000 U  21 0x0ea63000 3 |.|.x rr  4K
0xc0025000 K   0 0x0ef77000 3 ||||x _w  4K
0xc0626000 K   0 0x0e341000 3 ||||x _w  4K
0xc0627000 K   0 0x0e7aa000 3 |.||x _r  4K
0xc0628000 K   0 0x0e2ef000 3 |.||x _r  4K
0xc0629000 K   0 0x0cae3000 3 |.||x _r  4K
0xc006a000 K   0 0x0ef2b000 3 ||||x _w  4K
0xc062b000 K   0 0x0e6f4000 3 |.||x _r  4K
0x241cc000 U  21 0x0cc4c000 3 |.|.x rr  4K
0xc000d000 K   0 0x0efda000 3 ||||x _w  4K
0xc066e000 K   0 0x0c813000 3 |.||x _r  4K
0xc006f000 K   0 0x0ef26000 3 ||||x _w  4K
0x25030000 U  21 0x0c72a000 3 |.|.x rr  4K
0x241f1000 U  21 0x0cd52000 3 |.|.x rr  4K
0xc0052000 K   0 0x0ef48000 3 ||||x _w  4K
0x25053000 U  21 0x0c8f2000 3 |||.x ww  4K
0x24ff4000 U  21 0x0ecc1000 3 |.|.x rr  4K
0x25035000 U  21 0x0e30a000 3 |||.x ww  4K
0x00ad6000 U  21 0x0cbb2000 3 |||.x ww  4K
0xc0077000 K   0 0x0c25d000 3 ||||x _w  4K
0x25058000 U  21 0x0c3b0000 3 |.|.x rr  4K
0x24fb9000 U  21 0x0ea2b000 3 |.|.x rr  4K
0x2503a000 U  21 0x0ebff000 3 |||.x ww  4K
0x2053b000 U   9 0x0c39b000 3 |.|.x rr  4K
0x24ffc000 U  21 0x0c44d000 3 |||.x ww  4K
0xc009d000 K   0 0x0c5d6000 3 ||||x _w  4K
0xc065e000 K   0 0x0e91c000 3 |.||x _r  4K
0xc063f000 K   0 0x0ec5f000 3 |.||x _r  4K

FYI, at the time of the panic top was showing:
Memory: 16M Ac, 8104K Inact, 388K Wired, 2968K Exec, 1684K File, 856K Free
Swap: 127M Total, 89M Used, 39M Free

(Yes, it was swapping hard!).

HTH,

-- 
Stephen