port-amd64/49295: Crashes in NetBSD running on KVM

[ddeus.netbsd%mailnull.com@localhost]

>Number:         49295
>Category:       port-amd64
>Synopsis:       Crashes in NetBSD running on KVM
>Confidential:   no
>Severity:       critical
>Priority:       high
>Responsible:    port-amd64-maintainer
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sun Oct 19 10:10:00 +0000 2014
>Originator:     Oleksandr Kozachuk
>Release:        CURRENT
>Organization:
>Environment:
>Description:
NetBSD crashes often if running in KVM.

I have started it on debian with following command:
exec kvm \
 -m 8192 -cpu host -smp 8 \
 -drive file=/dev/vgvm/vm,cache=writeback,if=virtio \
 -net nic,model=virtio,macaddr=00:50:56:00:36:6D \
 -net tap,vlan=0,ifname=tap0,script=/etc/service/vm/ifup \
 -vnc localhost:0

dmesg looks like follows:
NetBSD 7.99.1 (VM) #2: Sat Oct 18 13:51:26 CEST 2014
        root%bohrium.kozachuk.info@localhost:/usr/obj/amd64/sys/arch/amd64/compile/VM
total memory = 8191 MB
avail memory = 7946 MB
kern.module.path=/stand/amd64/7.99.1/modules
timecounter: Timecounters tick every 10.000 msec
timecounter: Timecounter "i8254" frequency 1193182 Hz quality 100
Bochs Bochs
mainbus0 (root)
ACPI: RSDP 0xfda10 000014 (v00 BOCHS )
ACPI: RSDT 0xdfffe050 000038 (v01 BOCHS  BXPCRSDT 00000001 BXPC 00000001)
ACPI: FACP 0xdffffd10 000074 (v01 BOCHS  BXPCFACP 00000001 BXPC 00000001)
ACPI: DSDT 0xdfffe090 001121 (v01   BXPC   BXDSDT 00000001 INTL 20100528)
ACPI: FACS 0xdffffcc0 000040
ACPI: SSDT 0xdffff970 000345 (v01 BOCHS  BXPCSSDT 00000001 BXPC 00000001)
ACPI: APIC 0xdffff850 0000B0 (v01 BOCHS  BXPCAPIC 00000001 BXPC 00000001)
ACPI: HPET 0xdffff810 000038 (v01 BOCHS  BXPCHPET 00000001 BXPC 00000001)
ACPI: SSDT 0xdffff1c0 000644 (v01   BXPC BXSSDTPC 00000001 INTL 20100528)
ACPI: All ACPI Tables successfully acquired
cpu0 at mainbus0 apid 0: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz, id 0x306a9
cpu1 at mainbus0 apid 1: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz, id 0x306a9
cpu2 at mainbus0 apid 2: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz, id 0x306a9
cpu3 at mainbus0 apid 3: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz, id 0x306a9
cpu4 at mainbus0 apid 4: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz, id 0x306a9
cpu5 at mainbus0 apid 5: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz, id 0x306a9
cpu6 at mainbus0 apid 6: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz, id 0x306a9
cpu7 at mainbus0 apid 7: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz, id 0x306a9
ioapic0 at mainbus0 apid 8: pa 0xfec00000, version 0x11, 24 pins
acpi0 at mainbus0: Intel ACPICA 20131218
acpi0: X/RSDT: OemId <BOCHS ,BXPCRSDT,00000001>, AslId <BXPC,00000001>
LNKS: ACPI: Found matching pin for 0.1.INTA at func 3: 9
LNKC: ACPI: Found matching pin for 0.3.INTA at func 0: 11
LNKD: ACPI: Found matching pin for 0.4.INTA at func 0: 11
acpi0: SCI interrupting at int 9
timecounter: Timecounter "ACPI-Safe" frequency 3579545 Hz quality 900
pckbc0 at acpi0 (KBD, PNP0303) (kbd port): io 0x60,0x64 irq 1
pckbc1 at acpi0 (MOU, PNP0F13) (aux port): irq 12
FDC0 (PNP0700) at acpi0 not configured
LPT (PNP0400) at acpi0 not configured
COM1 (PNP0501) at acpi0 not configured
HPET (PNP0103) at acpi0 not configured
ACPI: Enabled 16 GPEs in block 00 to 0F
ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S0_] (20131218/hwxface-646)
ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S1_] (20131218/hwxface-646)
ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S2_] (20131218/hwxface-646)
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard
pms0 at pckbc0 (aux slot)
pckbc0: using irq 12 for aux slot
wsmouse0 at pms0 mux 0
pci0 at mainbus0 bus 0: configuration mode 1
pci0: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
pchb0 at pci0 dev 0 function 0: vendor 8086 product 1237 (rev. 0x02)
pcib0 at pci0 dev 1 function 0: vendor 8086 product 7000 (rev. 0x00)
piixide0 at pci0 dev 1 function 1: Intel 82371SB IDE Interface (PIIX3) (rev. 0x00)
piixide0: bus-master DMA support present
piixide0: primary channel wired to compatibility mode
piixide0: primary channel interrupting at ioapic0 pin 14
atabus0 at piixide0 channel 0
piixide0: secondary channel wired to compatibility mode
piixide0: secondary channel interrupting at ioapic0 pin 15
atabus1 at piixide0 channel 1
piixpm0 at pci0 dev 1 function 3: vendor 8086 product 7113 (rev. 0x03)
timecounter: Timecounter "piixpm0" frequency 3579545 Hz quality 1000
piixpm0: 24-bit timer
piixpm0: LNKS: _SRS failed: AE_NOT_FOUND
interrupting at ioapic0 pin 9
iic0 at piixpm0: I2C bus
vga0 at pci0 dev 2 function 0: vendor 1013 product 00b8 (rev. 0x00)
wsdisplay0 at vga0 kbdmux 1: console (80x25, vt100 emulation), using wskbd0
wsmux1: connecting to wsdisplay0
drm at vga0 not configured
virtio0 at pci0 dev 3 function 0
virtio0: Virtio Network Device (rev. 0x00)
vioif0 at virtio0: Ethernet address 00:50:56:00:36:6d
virtio0: allocated 20480 byte for virtqueue 0 for rx, size 256
virtio0: using 8192 byte (512 entries) indirect descriptors
virtio0: allocated 81920 byte for virtqueue 1 for tx, size 256
virtio0: using 69632 byte (4352 entries) indirect descriptors
virtio0: allocated 8192 byte for virtqueue 2 for control, size 64
virtio0: interrupting at ioapic0 pin 11
virtio1 at pci0 dev 4 function 0
virtio1: Virtio Block Device (rev. 0x00)
ld0 at virtio1
virtio1: allocated 45056 byte for virtqueue 0 for I/O request, size 128
virtio1: using 36864 byte (2304 entries) indirect descriptors
ld0: 900 GB, 16383 cyl, 16 head, 63 sec, 512 bytes/sect x 1887436800 sectors
virtio1: interrupting at ioapic0 pin 11
isa0 at pcib0
lpt0 at isa0 port 0x378-0x37b irq 7
com0 at isa0 port 0x3f8-0x3ff irq 4: ns16550a, working fifo
attimer0 at isa0 port 0x40-0x43
pcppi0 at isa0 port 0x61
midi0 at pcppi0: PC speaker
sysbeep0 at pcppi0
attimer0: attached to pcppi0
acpicpu0 at cpu0: ACPI CPU
acpicpu0: C1: HLT, lat   0 us, pow     0 mW
vmt0 at cpu0: Unknown
vmware: open failed, eax=564d5868, ecx=0000001e, edx=00005658
vmt0: failed to open backdoor RPC channel (TCLO protocol)
acpicpu1 at cpu1: ACPI CPU
acpicpu2 at cpu2: ACPI CPU
acpicpu3 at cpu3: ACPI CPU
acpicpu4 at cpu4: ACPI CPU
acpicpu5 at cpu5: ACPI CPU
acpicpu6 at cpu6: ACPI CPU
acpicpu7 at cpu7: ACPI CPU
timecounter: Timecounter "clockinterrupt" frequency 100 Hz quality 0
IPsec: Initialized Security Association Processing.
atapibus0 at atabus1: 2 targets
cd0 at atapibus0 drive 0: <QEMU DVD-ROM, QM00003, 1.1.2> cdrom removable
cd0: 32-bit data port
cd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 5 (Ultra/100)
cd0(piixide0:1:0): using PIO mode 4, DMA mode 2 (using DMA)
Kernelized RAIDframe activated
boot device: ld0
root on ld0a dumps on ld0b
/: replaying log to memory
root file system type: ffs
/: replaying log to disk
tap3: Ethernet address f2:0b:a4:d2:c1:15

Backtraces are often different, by they look similar to this:

trap type 6 code 2 rip ffffffff8049015f cs 8 rflags 10246 cr2 ffff80008e8e8000 ilevel 6 rsp fffffe810e969ba0
curlwp 0xfffffe821f734840 pid 0.2 lowest kstack 0xfffffe810e9672c0
panic: trap
cpu0: Begin traceback...
vpanic() at netbsd:vpanic+0x13c
snprintf() at netbsd:snprintf
startlwp() at netbsd:startlwp
alltraps() at netbsd:alltraps+0x96
ld_virtio_vq_done1() at netbsd:ld_virtio_vq_done1+0x78
ld_virtio_vq_done() at netbsd:ld_virtio_vq_done+0x37
virtio_vq_intr() at netbsd:virtio_vq_intr+0x70
virtio_intr() at netbsd:virtio_intr+0x38
intr_biglock_wrapper() at netbsd:intr_biglock_wrapper+0x19
Xintr_ioapic_level5() at netbsd:Xintr_ioapic_level5+0xf2
--- interrupt ---
inl() at netbsd:inl+0x7
AcpiHwReadPort() at netbsd:AcpiHwReadPort+0xc1
AcpiHwRead() at netbsd:AcpiHwRead+0x61
acpitimer_read_fast() at netbsd:acpitimer_read_fast+0x11
acpicpu_cstate_idle_enter() at netbsd:acpicpu_cstate_idle_enter+0x30
acpicpu_cstate_idle() at netbsd:acpicpu_cstate_idle+0x6d
idle_loop() at netbsd:idle_loop+0xe8
cpu0: End traceback...

dumping to dev 19,1 (offset=807, size=2097051):
dump fatal page fault in supervisor mode
trap type 6 code 0 rip ffffffff80490154 cs 8 rflags 10206 cr2 ffff80008e8f2ff8 ilevel 8 rsp fffffe810e969770
curlwp 0xfffffe821f734840 pid 0.2 lowest kstack 0xfffffe810e9672c0
Skipping crash dump on recursive panic
panic: trap
cpu0: Begin traceback...
vpanic() at netbsd:vpanic+0x13c
snprintf() at netbsd:snprintf
startlwp() at netbsd:startlwp
alltraps() at netbsd:alltraps+0x96
ld_virtio_dump() at netbsd:ld_virtio_dump+0x1ab
lddump() at netbsd:lddump+0x149
dumpsys_seg() at netbsd:dumpsys_seg+0xc7
dump_seg_iter() at netbsd:dump_seg_iter+0xce
dodumpsys() at netbsd:dodumpsys+0x24c
dumpsys() at netbsd:dumpsys+0x1d
vpanic() at netbsd:vpanic+0x145
snprintf() at netbsd:snprintf
startlwp() at netbsd:startlwp
alltraps() at netbsd:alltraps+0x96
ld_virtio_vq_done1() at netbsd:ld_virtio_vq_done1+0x78
ld_virtio_vq_done() at netbsd:ld_virtio_vq_done+0x37
virtio_vq_intr() at netbsd:virtio_vq_intr+0x70
virtio_intr() at netbsd:virtio_intr+0x38
intr_biglock_wrapper() at netbsd:intr_biglock_wrapper+0x19
Xintr_ioapic_level5() at netbsd:Xintr_ioapic_level5+0xf2
--- interrupt ---
inl() at netbsd:inl+0x7
AcpiHwReadPort() at netbsd:AcpiHwReadPort+0xc1
AcpiHwRead() at netbsd:AcpiHwRead+0x61
acpitimer_read_fast() at netbsd:acpitimer_read_fast+0x11
acpicpu_cstate_idle_enter() at netbsd:acpicpu_cstate_idle_enter+0x30
acpicpu_cstate_idle() at netbsd:acpicpu_cstate_idle+0x6d
idle_loop() at netbsd:idle_loop+0xe8
cpu0: End traceback...
rebooting...





>How-To-Repeat:
Start NetBSD current in KVM and do some load on CPU and disks.
>Fix: