port-sparc64/38297: sparc64 smp panic: pmap_clear_reference(): 0xN still referenced!

[he%NetBSD.org@localhost]

>Number:         38297
>Category:       port-sparc64
>Synopsis:       sparc64 smp panic: pmap_clear_reference(): 0xN still 
>referenced!
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    port-sparc64-maintainer
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Tue Mar 25 19:50:00 +0000 2008
>Originator:     Havard Eidnes
>Release:        NetBSD 4.99.58
>Organization:
        I try...
>Environment:
System: NetBSD 4.99.58 (GENERIC.MP) #3: Tue Mar 25 04:02:38 CET 2008
        
he%quattro.urc.uninett.no@localhost:/u/build/HEAD/obj/sparc64/sys/arch/sparc64/compile/GENERIC.MP
Architecture: sparc64
Machine: sparc64
>Description:
        While testing out the new sparc64 SMP code, the kernel
        panic'ed after, well, a half-hour of operation?

        The machine is a Sun Ultra Enterprise 450, and it runs
        an LFS file system on it's non-system disk, where I had
        started rtorrent for the 4.0 ISO images.

        User-land is 4.0.

        The kernel includes the DEBUG + DIAGNOSTIC + LOCKDEBUG
        options.

        The console log follows, partly driven by some advice from
        other developers, I've replaced some of the "greater than"
        chars in an attempt to play nice with GNATS:

Sun Enterprise 450 (4 X UltraSPARC-II 400MHz), No Keyboard
OpenBoot 3.14, 4096 MB memory installed, Serial #11618221.
Ethernet address 8:0:20:b1:47:ad, Host ID: 80b147ad.



Rebooting with command: boot disk netbsd.mp -s                        
Boot device: /pci@1f,4000/scsi@3/disk@0,0  File and args: netbsd.mp -s
NetBSD IEEE 1275 Bootblock
:: NetBSD/sparc64 OpenFirmware Boot, Revision 1.9
:: (he%quattro.urc.uninett.no@localhost, Thu Jun 29 16:29:07 CEST 2006)
=0x858188
Loading netbsd.mp: 7174864+360776+643016 [516288+330109]=0x9c4250
Loaded initial symtab at 0x18f5110, strtab at 0x19738d0, # entries 20370
xtlb[0]: Tag: 1000000 Data: e0000000ff800076
xtlb[1]: Tag: 1400000 Data: e0000000ff400076
xtlb[2]: Tag: 1800000 Data: e0000000ff000076
prom_get_msgbuf: Cannot recover msgbuf on E250
prom_get_msgbuf: allocated new buf at 00000000
prom_get_msgbuf: claiming new buf at 00000000
consinit()
stdin node = f00701f0
stdout package = f00701f0
buffer @ 0x1c05ca0
console is /pci@1f,4000/ebus@1/se@14,400000:a
Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
    2006, 2007, 2008
    The NetBSD Foundation, Inc.  All rights reserved.
Copyright (c) 1982, 1986, 1989, 1991, 1993
    The Regents of the University of California.  All rights reserved.

NetBSD 4.99.58 (GENERIC.MP) #3: Tue Mar 25 04:02:38 CET 2008
        
he%quattro.urc.uninett.no@localhost:/u/build/HEAD/obj/sparc64/sys/arch/sparc64/compile/GENERIC.MP
total memory = 4096 MB
avail memory = 4006 MB
mainbus0 (root): SUNW,Ultra-4 (Sun Enterprise 450): hostid 80b147ad
cpu0 at mainbus0: SUNW,UltraSPARC-II @ 400 MHz, UPA id 0
cpu0: 32K instruction (32 b/l), 16K data (32 b/l), 4096K external (64 b/l)
cpu1 at mainbus0: SUNW,UltraSPARC-II @ 400 MHz, UPA id 1
cpu1: 32K instruction (32 b/l), 16K data (32 b/l), 4096K external (64 b/l)
cpu2 at mainbus0: SUNW,UltraSPARC-II @ 400 MHz, UPA id 2
cpu2: 32K instruction (32 b/l), 16K data (32 b/l), 4096K external (64 b/l)
cpu3 at mainbus0: SUNW,UltraSPARC-II @ 400 MHz, UPA id 3
cpu3: 32K instruction (32 b/l), 16K data (32 b/l), 4096K external (64 b/l)
psycho0 at mainbus0 addr 0xfffb4000
psycho0: SUNW,psycho: impl 0, version 4: ign 7c0 bus range 0 to 0; PCI bus 0
extent `psycho mem' (0x0 - 0x7fffffff), flags = 0x0
     0x0 - 0x16fff
extent `psycho io' (0x0 - 0xffff), flags = 0x0
     0x0 - 0x8ff
decreasing interrupt retry timeout from fffff to 0xff
DVMA map: fe000000 to ffffe000
IOTSB: 4344000 to 434c000
memory range: 000001ff 80000000
pci0 at psycho0
ebus0 at pci0 dev 1 function 0
ebus0: Sun Microsystems PCIO Ebus2, revision 0x01
auxio0 at ebus0 addr 726000-726003, 728000-728003, 72a000-72a003, 
72c000-72c003, 72f000-72f003
power at ebus0 addr 724000-724003 ipl 2021 ipl 2034 not configured
SUNW,pll at ebus0 addr 504000-504002 not configured
sc at ebus0 addr 500000-500007 not configured
sab0 at ebus0 addr 400000-40007f ipl 43: rev 3.2
sabtty0 at sab0 port 0: console i/o
sabtty1 at sab0 port 1
com0 at ebus0 addr 3083f8-3083ff ipl 41: ns16550a, working fifo
kbd0 at com0
com1 at ebus0 addr 3062f8-3062ff ipl 42: ns16550a, working fifo
ms0 at com1
wsmouse0 at ms0 mux 0
lpt0 at ebus0 addr 3043bc-3043cb, 300398-300399, 700000-70000f ipl 2018
fdthree at ebus0 addr 3023f0-3023f7, 706000-70600f, 720000-720003 ipl 2023 not 
configured
clock0 at ebus0 addr 0-1fff: mk48t59
flashprom at ebus0 addr 0-fffff, 0-fffff not configured
envctrl0 at ebus0 addr 600000-600003 ipl 2024 ipl 2021
envctrl0: Ultra Enterprise 450 environmental monitoring
envctrl0: PS 0: 550W, online
envctrl0: PS 1: 550W, online
envctrl0: PS 2: absent
hme0 at pci0 dev 1 function 1: Sun Happy Meal Ethernet, rev. 1
hme0: interrupting at ivec 37e1
hme0: Ethernet address 08:00:20:b1:47:ad
nsphy0 at hme0 phy 1: DP83840 10/100 media interface, rev. 1
nsphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
esiop0 at pci0 dev 3 function 0: Symbios Logic 53c875 (ultra-wide scsi)
esiop0: using on-board RAM
esiop0: interrupting at ivec 1fe0
esiop0: alloc new tag DSA table at PHY addr 0xfe066000
scsibus0 at esiop0: 16 targets, 8 luns per target
esiop1 at pci0 dev 2 function 0: Symbios Logic 53c875 (ultra-wide scsi)
esiop1: using on-board RAM
esiop1: interrupting at ivec 1fe6
esiop1: alloc new tag DSA table at PHY addr 0xfe068000
scsibus1 at esiop1: 16 targets, 8 luns per target
psycho1 at mainbus0 addr 0xfffc6000
psycho1: SUNW,psycho: impl 0, version 4: ign 7c0 bus range 128 to 128; PCI bus 
128
extent `psycho mem' (0x0 - 0x7fffffff), flags = 0x0
     0x0 - 0xfff
extent `psycho io' (0x0 - 0xffff), flags = 0x0
     0x0 - 0x3ff
memory range: 000001ff 00000000
pci1 at psycho1
timer0 at mainbus0 addr 0xfff9fc00 irq vectors 7ec
mc at mainbus0 not configured
psycho2 at mainbus0 addr 0xfff5e000
psycho2: SUNW,psycho: impl 0, version 4: ign 100 bus range 0 to 0; PCI bus 0
extent `psycho mem' (0x0 - 0x7fffffff), flags = 0x0
     0x0 - 0xfff
extent `psycho io' (0x0 - 0xffff), flags = 0x0
     0x0 - 0x3ff
decreasing interrupt retry timeout from fffff to 0xff
DVMA map: fe000000 to ffffe000
IOTSB: 43b6000 to 43be000
memory range: 000001c9 80000000
pci2 at psycho2
psycho3 at mainbus0 addr 0xfff70000
psycho3: SUNW,psycho: impl 0, version 4: ign 100 bus range 128 to 128; PCI bus 
128
extent `psycho mem' (0x0 - 0x7fffffff), flags = 0x0
     0x0 - 0xfff
extent `psycho io' (0x0 - 0xffff), flags = 0x0
     0x0 - 0x3ff
memory range: 000001c9 00000000
pci3 at psycho3
psycho4 at mainbus0 addr 0xfff36000
psycho4: SUNW,psycho: impl 0, version 4: ign 180 bus range 0 to 0; PCI bus 0
extent `psycho mem' (0x0 - 0x7fffffff), flags = 0x0
     0x0 - 0x180ffff
extent `psycho io' (0x0 - 0xffff), flags = 0x0
     0x0 - 0x3ff
decreasing interrupt retry timeout from fffff to 0xff
DVMA map: fe000000 to ffffe000
IOTSB: 43be000 to 43c6000
memory range: 000001cd 80000000
pci4 at psycho4
genfb0 at pci4 dev 4 function 0: 3D Labs GLINT Permedia 2V
genfb0: bogus parameters, unable to continue
psycho5 at mainbus0 addr 0xfff48000
psycho5: SUNW,psycho: impl 0, version 4: ign 180 bus range 128 to 128; PCI bus 
128
extent `psycho mem' (0x0 - 0x7fffffff), flags = 0x0
     0x0 - 0xfff
extent `psycho io' (0x0 - 0xffff), flags = 0x0
     0x0 - 0x3ff
memory range: 000001cd 00000000
pci5 at psycho5
pcons at mainbus0 not configured
Using %tick -- intr in 4000000 cycles
Using %tick -- intr in 4000000 cycles
Using %tick -- intr in 4000000 cycles
wskbd0 at kbd0scsibus0: waiting 2 seconds for devices to settle...
 mux 1Using %tick -- intr in 4000000 cycles
scsibus1: waiting 2 seconds for devices to settle...

esiop0: alloc newcdb at PHY addr 0xfe06a000
sd0 at scsibus0 target 0 lun 0: <FUJITSU, MAJ3182M SUN18G, 0804> disk fixed
sd0: 17274 MB, 7508 cyl, 19 head, 248 sec, 512 bytes/sect x 35378533 sectors
sd0: sync (50.00ns offset 16), 16-bit (40.000MB/s) transfers, tagged queueing
sd1 at scsibus0 target 1 lun 0: <FUJITSU, MAJ3182M SUN18G, 0804> disk fixed
sd1: 17274 MB, 7508 cyl, 19 head, 248 sec, 512 bytes/sect x 35378533 sectors
sd1: sync (50.00ns offset 16), 16-bit (40.000MB/s) transfers, tagged queueing
kbd0: reset failed
raidattach: Asked for 8 units
Kernelized RAIDframe activated
esiop1: alloc newcdb at PHY addr 0xfe06c000
cd0 at scsibus1 target 6 lun 0: <TOSHIBA, XM6201TASUN32XCD, 1103> cdrom 
removable
cd0: sync (100.00ns offset 16), 8-bit (10.000MB/s) transfers
Searching for RAID components...
root on sd0a dumps on sd0b
mountroot: trying lfs...
mountroot: trying ffs...
root file system type: ffs
init: copying out flags `-s' 3
init: copying out path `/sbin/init' 11
Enter pathname of shell or RETURN for /bin/sh: 
# 
# 
# ps ax
PID TTY   STAT    TIME COMMAND
  0 ?     DKl  0:00.16 [system]
  1 ?     Is   0:00.13 init -s 
  2 ttyh0 Ss   0:00.08 -sh 
  3 ttyh0 R+   0:00.02 ps -ax 
# exit
Tue Mar 25 18:30:04 CET 2008
swapctl: adding /dev/sd0b as swap device at priority 0
Checking for botched superblock upgrades: done.
Starting file system checks:
/dev/rsd0a: file system is clean; not checking
/dev/rsd1a: file system is clean; not checking
Setting tty flags.
Setting sysctl variables:
Starting network.
Hostname: torbidinsu.urc.uninett.no
IPv6 mode: host
Configuring network interfaces: hme0.
add net default: gateway 158.38.152.1
Adding interface aliases:
Building databases...
Starting syslogd.
Mar 25 18:30:11 torbidinsu /netbsd: kbd0: reset failed
Checking for core dump...
savecore: /dev/sd0b: Device busy
Mar 25 18:30:11 torbidinsu savecore: /dev/sd0b: Device busy
Setting date via ntp.
Mounting all filesystems...
Clearing /tmp.
Checking quotas: done.
Setting securelevel: kern.securelevel: 0 -> 1
Starting virecover.
Starting local daemons:.
Updating motd.
Starting ntpd.
Starting sshd.
postfix/postfix-script: starting the Postfix mail system
Starting inetd.
Starting cron.
Tue Mar 25 18:30:27 CET 2008

NetBSD/sparc64 (torbidinsu.urc.uninett.no) (console)

login: pmap_clear_reference(): 0x51f5350 still referenced!
Stopped in pid 0.33 (system) at netbsd:cpu_Debugger+0x4:        nop
db{2}: tra
uvmpdpol_pagedeactivate(51f5350, 12897e0, 151c000, f9da000, 18d24a8, f9da000) 
at netbsd:uvmpdpol_pagedeactivate+0x90
uvmpdpol_balancequeue(0, 1, 18d24a8, 0, 0, 0) at 
netbsd:uvmpdpol_balancequeue+0xa8
uvm_pageout(52ee4, 7dd66, 2a, 1e, 18d2400, 0) at netbsd:uvm_pageout+0x414
lwp_trampoline(f0075a4c, 0, 10fc00, fff7fd00, 10e9d8, fff7fe00) at 
netbsd:lwp_trampoline+0x8
db{2}: machine cpu 0
db{0}: tra
pmap_extract(110b7550, 41a66000, 0, 41a6c000, 10cb42d8, badcafe) at 
netbsd:pmap_extract+0x18
uvm_fault_internal(0, 41a66000, 1, 1, 10af9bf8, ffffffffffffffff) at 
netbsd:uvm_fault_internal+0x3a8
data_access_fault(10af9ed0, 30, 41a6c000, 41a6c0d1, 1824c00, 800801) at 
netbsd:data_access_fault+0x2cc
?(0, 0, ac06027dc095fd90, cc40c69f, 7fe698, 100) at 0x1008bb4
db{0}: machine cpu 1
db{1}: tra
mutex_vector_exit(f9bd050, f9bd050, f9bd050, f9c85c0, 1, f9c8610) at 
netbsd:mutex_vector_exit+0x68
mi_switch(f9c85c0, f9bd050, 0, f9c85c0, e0018000, 0) at netbsd:mi_switch+0x14c
idle_loop(1685800, 1684800, 1684c00, 1684c00, 1684c00, 1684c00) at 
netbsd:idle_loop+0xd0
cpu_mp_startup(ffffffffffffffff, 100000000, 0, 0, 0, 0) at 
netbsd:cpu_mp_startup+0x110
db{1}: machine cpu 2
db{2}: tra
uvmpdpol_pagedeactivate(51f5350, 12897e0, 151c000, f9da000, 18d24a8, f9da000) 
at netbsd:uvmpdpol_pagedeactivate+0x90
uvmpdpol_balancequeue(0, 1, 18d24a8, 0, 0, 0) at 
netbsd:uvmpdpol_balancequeue+0xa8
uvm_pageout(52ee4, 7dd66, 2a, 1e, 18d2400, 0) at netbsd:uvm_pageout+0x414
lwp_trampoline(f0075a4c, 0, 10fc00, fff7fd00, 10e9d8, fff7fe00) at 
netbsd:lwp_trampoline+0x8
db{2}: machine cpu 3
db{3}: tra
mutex_vector_enter(18d87c0, f9bd0f0, 151c000, f9d6300, 3, fffffff) at 
netbsd:mutex_vector_enter+0x7c
mi_switch(f9d6300, f9bd0f0, 0, f9d6300, e0018000, 0) at netbsd:mi_switch+0x2f0
idle_loop(1685800, 1684800, 1684c00, 1684c00, 1684c00, 1684c00) at 
netbsd:idle_loop+0xd0
cpu_mp_startup(ffffffffffffffff, 100000000, 0, 0, 0, 0) at 
netbsd:cpu_mp_startup+0x110
db{3}: machine cpu 2
db{2}: show reg
tstate      0x448000606
pc          0x144e9e4   cpu_Debugger+0x4
npc         0x144e9e8   cpu_Debugger+0x8
ipl         0
y           0
g0          0
g1          0x100000000
g2          0x100f8000
g3          0
g4          0xe0018000
g5          0x204
g6          0x21
g7          0
o0          0x16c70d8   copyright+0x81460
o1          0x51f5350
o2          0x1f2614
o3          0x1ffc92
o4          0x4020
o5          0
o6          0x109f12a1
o7          0x1443df4   pmap_clear_reference+0x2b4
l0          0
l1          0
l2          0
l3          0
l4          0
l5          0
l6          0
l7          0
i0          0
i1          0
i2          0
i3          0
i4          0
i5          0
i6          0
i7          0
f0          0
f2          0
f4          0
f6          0
f8          0x499b8d135f000000
f10         0x1371a2
f12         0
f14         0
f16         0
f18         0
f20         0
f22         0
f24         0
f26         0
f28         0
f30         0
f32         0
f34         0xffffffffffffa680
f36         0x410be01c
f38         0x411273c3
f40         0
f42         0
f44         0x410bde00
f46         0
f48         0
f50         0
f52         0
f54         0
f56         0
f58         0
f60         0
f62         0
fsr         0x421
gsr         0
netbsd:cpu_Debugger+0x4:        nop
db{2}: show uvm
Current UVM status:
  pagesize=8192 (0x2000), pagemask=0x1fff, pageshift=13
  515430 VM pages: 340792 active, 166331 inactive, 666 wired, 168 free
  pages  5362 anon, 500655 file, 1773 exec
  freemin=32, free-target=42, wired-max=171810
  faults=971796, traps=473692, intrs=2808957, ctxswitch=2381105
  softint=715026, syscalls=1266042, swapins=147, swapouts=164
  fault counts:
    noram=0, noanon=0, pgwait=0, pgrele=0
    ok relocks(total)=262389(262391), anget(retrys)=114904(0), amapcopy=4963
    neighbor anon/obj pg=7630/492026, gets(lock/unlock)=414391/262391
    cases: anon=111086, anoncow=3806, obj=411096, prcopy=3293, przero=9143
  daemon and swap counts:
    woke=2146, revs=2146, scans=346113, obscans=339844, anscans=0
    busy=0, freed=339844, reactivate=552, deactivate=512490
    pageouts=0, pending=0, nswget=0
    nswapdev=1, swpgavail=640242
    swpages=640242, swpginuse=0, swpgonly=0, paging=0
db{2}: show page 0x51f5350
PAGE 0x51f5350:
  flags=0, pqflags=2<ANON>, wire_count=0, pa=0xf930a000
  uobject=0x0, uanon=0x111a3e20, offset=0x0 loan_count=0
  [page ownership tracking disabled]
db{2}: show lock
Sorry, no record of a lock with address 0x51f5350 found.
db{2}: show lock pmap_lock
lock address : 0x000000000189fad8 type     :     sleep/adaptive
shared holds :                  0 exclusive:                  1
shares wanted:                  0 exclusive:                  1
current cpu  :                  2 last held:                  2
current lwp  : 0x000000000f9da000 last held: 0x000000000f9da000
last locked  : 0x0000000001443b50 unlocked : 0x0000000001443d84
initialized  : 0x00000000014458a4
owner field  : 0x000000000f9da000 wait/spin:                0/0

Turnstile chain at 0x18d8320.
=: No active turnstile for this lock.
db{2}: 


>How-To-Repeat:
        Has not yet proven to be repeatable...

>Fix:
        Sorry, don't know.