port-amd64/46094: cpu0 timer sometimes far off

[martin%NetBSD.org@localhost]

>Number:         46094
>Category:       port-amd64
>Synopsis:       cpu0 timer sometimes far off
>Confidential:   no
>Severity:       critical
>Priority:       high
>Responsible:    port-amd64-maintainer
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sat Feb 25 08:15:00 +0000 2012
>Originator:     Martin Husemann
>Release:        NetBSD 6.99.3
>Organization:
The NetBSD Foundation, Inc.
>Environment:
System: NetBSD martins.aprisoft.de 6.99.3 NetBSD 6.99.3 (GENERIC) #18: Fri Feb 
24 21:20:16 CET 2012 
martin%martins.aprisoft.de@localhost:/usr/src/sys/arch/amd64/compile/GENERIC 
amd64
Architecture: x86_64
Machine: amd64
>Description:

Sometimes the interrupt rate of the cpu0 timer is way lower than the expected
HZ on this machine (default, so 100). Sometimes it is pretty close. I have
seen 18Hz and 99Hz. With close to 100Hz the machine is fast and snappy, with
18Hz it is mostly unusable.

I have no idea, what makes the differences, it seems to vary from boot to
boot. The bios has c1e (sp?) enabled, but disabling it did not seem to change
things at a quick glance.

Right now, for example, vmstat -i shows:

interrupt                                     total     rate
TLB shootdown                                125050      118
cpu0 timer                                    74044       70
ioapic0 pin 19                               149734      141
ioapic0 pin 17                                    3        0
ioapic0 pin 18                                   39        0
ioapic0 pin 16                                 8930        8
Total                                        357800      338

The relevant timer sysctls are:
kern.timecounter.choice = TSC(q=3000, f=3010199210 Hz) clockinterrupt(q=0, 
f=100 Hz) hpet0(q=2000, f=14318180 Hz) ACPI-Fast(q=1000, f=3579545 Hz) 
lapic(q=-100, f=200733248 Hz) i8254(q=100, f=1193182 Hz) dummy(q=-1000000, 
f=1000000 Hz)
kern.timecounter.hardware = TSC
kern.timecounter.timestepwarnings = 0

Changing those, when it already was in a "degraded" mode, seemed to not help,
I haven't tried hardwiring another one during boot yet.

CPU identification:
cpu0: AMD Unknown AMD64 CPU (686-class), 3010.20 MHz, id 0x100fa0
cpu0: features  0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR>
cpu0: features  0x178bfbff<PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2>
cpu0: features  0x178bfbff<HTT>
cpu0: features2 0x802009<SSE3,MONITOR,CX16,POPCNT>
cpu0: features3 0xefd3fbff<SYSCALL/SYSRET,NOX,MXX,FFXSR,P1GB,RDTSCP,LONG>
cpu0: features3 0xefd3fbff<3DNOW2,3DNOW>
cpu0: features4 0x37ff<LAHF,CMPLEGACY,SVM,EAPIC,ALTMOVCR0,LZCNT,SSE4A>
cpu0: features4 0x37ff<MISALIGNSSE,3DNOWPREFETCH,OSVW,IBS,SKINIT,WDT>
cpu0: "AMD Phenom(tm) II X6 1075T Processor"
cpu0: I-cache 64KB 64B/line 2-way, D-cache 64KB 64B/line 2-way
cpu0: L2 cache 512KB 64B/line 16-way
cpu0: L3 cache 6MB 64B/line 48-way
cpu0: ITLB 32 4KB entries fully associative, 16 2MB entries fully associative
cpu0: DTLB 48 4KB entries fully associative, 48 2MB entries fully associative
cpu0: L2 ITLB 512 4KB entries 4-way
cpu0: L2 DTLB 512 4KB entries 4-way, 128 2MB entries 2-way
cpu0: L1 1GB page DTLB 48 1GB entries fully associative
cpu0: L2 1GB page DTLB 16 1GB entries 8-way
cpu0: Initial APIC ID 0
cpu0: AMD Power Management features: 0x3f9<TS,TTP,HTC,STC,100,HWP,TSC,CPB>
cpu0: SVM Rev. 1
cpu0: SVM NASID 64
cpu0: SVM features 0x40f<NP,LbrVirt,SVML,NRIPS,PauseFilter>
cpu0: family 0f model 0a extfamily 01 extmodel 00


Full dmesg:

Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
    2006, 2007, 2008, 2009, 2010, 2011, 2012
    The NetBSD Foundation, Inc.  All rights reserved.
Copyright (c) 1982, 1986, 1989, 1991, 1993
    The Regents of the University of California.  All rights reserved.

NetBSD 6.99.3 (GENERIC) #18: Fri Feb 24 21:20:16 CET 2012
        
martin%martins.aprisoft.de@localhost:/usr/src/sys/arch/amd64/compile/GENERIC
total memory = 16382 MB
avail memory = 15891 MB
timecounter: Timecounters tick every 10.000 msec
timecounter: Timecounter "i8254" frequency 1193182 Hz quality 100
System manufacturer System Product Name (System Version)
mainbus0 (root)
cpu0 at mainbus0 apid 0: AMD Phenom(tm) II X6 1075T Processor, id 0x100fa0
cpu1 at mainbus0 apid 1: AMD Phenom(tm) II X6 1075T Processor, id 0x100fa0
cpu2 at mainbus0 apid 2: AMD Phenom(tm) II X6 1075T Processor, id 0x100fa0
cpu3 at mainbus0 apid 3: AMD Phenom(tm) II X6 1075T Processor, id 0x100fa0
cpu4 at mainbus0 apid 4: AMD Phenom(tm) II X6 1075T Processor, id 0x100fa0
cpu5 at mainbus0 apid 5: AMD Phenom(tm) II X6 1075T Processor, id 0x100fa0
ioapic0 at mainbus0 apid 6: pa 0xfec00000, version 21, 24 pins
acpi0 at mainbus0: Intel ACPICA 20110623
acpi0: X/RSDT: OemId <030811,XSDT1042,20110308>, AslId <MSFT,00000097>
acpi0: SCI interrupting at int 9
timecounter: Timecounter "ACPI-Fast" frequency 3579545 Hz quality 1000
hpet0 at acpi0: high precision event timer (mem 0xfed00000-0xfed00400)
timecounter: Timecounter "hpet0" frequency 14318180 Hz quality 2000
acpiec0 at acpi0 (EC0, PNP0C09): io 0x62,0x66
attimer1 at acpi0 (TMR, PNP0100): io 0x40-0x43 irq 0
pcppi1 at acpi0 (SPKR, PNP0800): io 0x61
midi0 at pcppi1: PC speaker
sysbeep0 at pcppi1
UAR1 (PNP0501) at acpi0 not configured
aibs0 at acpi0 (ASOC, ATK0110-16843024): ASUSTeK AI Booster
OMSC (PNP0C02) at acpi0 not configured
RMSC (PNP0C02) at acpi0 not configured
SIOR (PNP0C02) at acpi0 not configured
PCIE (PNP0C02) at acpi0 not configured
RMEM (PNP0C01) at acpi0 not configured
acpibut0 at acpi0 (PWRB, PNP0C0C-170): ACPI Power Button
acpiwmi0 at acpi0 (AOD, PNP0C14-0): ACPI WMI Interface
acpiwmibus at acpiwmi0 not configured
attimer1: attached to pcppi1
pci0 at mainbus0 bus 0: configuration mode 1
pci0: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
pchb0 at pci0 dev 0 function 0: vendor 0x1002 product 0x5957 (rev. 0x00)
ppb0 at pci0 dev 2 function 0: vendor 0x1002 product 0x5978 (rev. 0x00)
ppb0: PCI Express 2.0 <Root Port of PCI-E Root Complex>
pci1 at ppb0 bus 5
pci1: i/o space, memory space enabled, rd/line, wr/inv ok
vga0 at pci1 dev 0 function 0: vendor 0x10de product 0x0e22 (rev. 0xa1)
wsdisplay0 at vga0 kbdmux 1: console (80x25, vt100 emulation)
wsmux1: connecting to wsdisplay0
drm at vga0 not configured
hdaudio0 at pci1 dev 0 function 1: HD Audio Controller
hdaudio0: interrupting at ioapic0 pin 19
hdafg0 at hdaudio0: NVIDIA product 0012
hdafg0: DP00 8ch: Digital Out [Jack]
hdafg0: 8ch/0ch 48000Hz PCM16*
hdafg1 at hdaudio0: NVIDIA product 0012
hdafg1: DP00 8ch: Digital Out [Jack]
hdafg1: 8ch/0ch 48000Hz PCM16*
hdafg2 at hdaudio0: NVIDIA product 0012
hdafg2: DP00 8ch: Digital Out [Jack]
hdafg2: 8ch/0ch 48000Hz PCM16*
hdafg3 at hdaudio0: NVIDIA product 0012
hdafg3: DP00 8ch: Digital Out [Jack]
hdafg3: 8ch/0ch 48000Hz PCM16*
ppb1 at pci0 dev 9 function 0: vendor 0x1002 product 0x597e (rev. 0x00)
ppb1: PCI Express 2.0 <Root Port of PCI-E Root Complex>
pci2 at ppb1 bus 4
pci2: i/o space, memory space enabled, rd/line, wr/inv ok
jmide0 at pci2 dev 0 function 0: vendor 0x197b product 0x2361 (rev. 0x10)
jmide0: 1 PATA port, 1 SATA port
jmide0: interrupting at ioapic0 pin 17
ahcisata0 at jmide0
ahcisata0: AHCI revision 1.10, 1 ports, 32 slots, CAP 
0xc722ff00<PSC,SSC,PMD,SPM,ISS=0x2=Gen2,SCLO,SAL,SALP,SNCQ,S64A>
atabus0 at ahcisata0 channel 0
jmide0: PCI IDE interface used
jmide0: bus-master DMA support present
jmide0: primary channel wired to native-PCI mode
jmide0: primary channel is unused
jmide0: secondary channel wired to native-PCI mode
jmide0: secondary channel is PATA
atabus1 at jmide0 channel 1
ppb2 at pci0 dev 10 function 0: vendor 0x1002 product 0x597f (rev. 0x00)
ppb2: PCI Express 2.0 <Root Port of PCI-E Root Complex>
pci3 at ppb2 bus 3
pci3: i/o space, memory space enabled, rd/line, wr/inv ok
vendor 0x1033 product 0x0194 (USB serial bus, interface 0x30, revision 0x03) at 
pci3 dev 0 function 0 not configured
ahcisata1 at pci0 dev 17 function 0: vendor 0x1002 product 0x4391 (rev. 0x40)
ahcisata1: interrupting at ioapic0 pin 19
ahcisata1: 64-bit DMA
ahcisata1: AHCI revision 1.20, 6 ports, 32 slots, CAP 
0xf732ff05<PSC,SSC,PMD,SPM,ISS=0x3=Gen3,SCLO,SAL,SALP,SMPS,SSNTF,SNCQ,S64A>
atabus2 at ahcisata1 channel 0
atabus3 at ahcisata1 channel 1
atabus4 at ahcisata1 channel 2
atabus5 at ahcisata1 channel 3
atabus6 at ahcisata1 channel 4
atabus7 at ahcisata1 channel 5
ohci0 at pci0 dev 18 function 0: vendor 0x1002 product 0x4397 (rev. 0x00)
ohci0: interrupting at ioapic0 pin 18
ohci0: OHCI version 1.0, legacy support
usb0 at ohci0: USB revision 1.0
ehci0 at pci0 dev 18 function 2: vendor 0x1002 product 0x4396 (rev. 0x00)
ehci0: interrupting at ioapic0 pin 17
ehci0: dropped intr workaround enabled
ehci0: EHCI version 1.0
ehci0: companion controller, 5 ports each: ohci0
usb1 at ehci0: USB revision 2.0
ohci1 at pci0 dev 19 function 0: vendor 0x1002 product 0x4397 (rev. 0x00)
ohci1: interrupting at ioapic0 pin 18
ohci1: OHCI version 1.0, legacy support
usb2 at ohci1: USB revision 1.0
ehci1 at pci0 dev 19 function 2: vendor 0x1002 product 0x4396 (rev. 0x00)
ehci1: interrupting at ioapic0 pin 17
ehci1: dropped intr workaround enabled
ehci1: EHCI version 1.0
ehci1: companion controller, 5 ports each: ohci1
usb3 at ehci1: USB revision 2.0
piixpm0 at pci0 dev 20 function 0: vendor 0x1002 product 0x4385 (rev. 0x42)
piixpm0: polling (SB800)
iic0 at piixpm0: I2C bus
sc: 0xffff80001d1df000
tag: 0xfffffe842f7bca48
self: 0xfffffe811d58b3c0
devname: iic0
hdaudio1 at pci0 dev 20 function 2: HD Audio Controller
hdaudio1: interrupting at ioapic0 pin 16
hdafg4 at hdaudio1: VIA VT1818S
hdafg4: DAC00 8ch: Speaker [Jack], HP Out [Jack]
hdafg4: ADC01 2ch: Line In [Jack], Mic In [Jack]
hdafg4: HDMI02 2ch: Digital Out [Jack]
hdafg4: DIG03 2ch: SPDIF Out [Jack]
hdafg4: 8ch/2ch 48000Hz PCM16*
audio0 at hdafg4: full duplex, playback, capture, independent
pcib0 at pci0 dev 20 function 3: vendor 0x1002 product 0x439d (rev. 0x40)
ppb3 at pci0 dev 20 function 4: vendor 0x1002 product 0x4384 (rev. 0x40)
pci4 at ppb3 bus 2
pci4: i/o space, memory space enabled
fwohci0 at pci4 dev 8 function 0: vendor 0x1106 product 0x3044 (rev. 0xc0)
fwohci0: interrupting at ioapic0 pin 20
fwohci0: OHCI version 1.10 (ROM=1)
fwohci0: No. of Isochronous channels is 4.
fwohci0: EUI64 00:1f:c6:00:00:0a:ca:1a
fwohci0: Phy 1394a available S400, 2 ports.
fwohci0: Link S400, max_rec 2048 bytes.
ieee1394if0 at fwohci0: IEEE1394 bus
fwip0 at ieee1394if0: IP over IEEE1394
fwohci0: Initiate bus reset
ohci2 at pci0 dev 20 function 5: vendor 0x1002 product 0x4399 (rev. 0x00)
ohci2: interrupting at ioapic0 pin 18
ohci2: OHCI version 1.0, legacy support
usb4 at ohci2: USB revision 1.0
ppb4 at pci0 dev 21 function 0: vendor 0x1002 product 0x43a0 (rev. 0x00)
ppb4: PCI Express 2.0 <Root Port of PCI-E Root Complex>
pci5 at ppb4 bus 1
pci5: i/o space, memory space enabled, rd/line, wr/inv ok
re0 at pci5 dev 0 function 0: RealTek 8168/8111 PCIe Gigabit Ethernet (rev. 
0x06)
re0: interrupting at ioapic0 pin 16
re0: Ethernet address bc:ae:c5:46:16:58
re0: using 256 tx descriptors
rgephy0 at re0 phy 7: RTL8169S/8110S/8211 1000BASE-T media interface, rev. 4
rgephy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 
1000baseT-FDX, auto
ohci3 at pci0 dev 22 function 0: vendor 0x1002 product 0x4397 (rev. 0x00)
ohci3: interrupting at ioapic0 pin 18
ohci3: OHCI version 1.0, legacy support
usb5 at ohci3: USB revision 1.0
ehci2 at pci0 dev 22 function 2: vendor 0x1002 product 0x4396 (rev. 0x00)
ehci2: interrupting at ioapic0 pin 17
ehci2: dropped intr workaround enabled
ehci2: EHCI version 1.0
ehci2: companion controller, 4 ports each: ohci3
usb6 at ehci2: USB revision 2.0
pchb1 at pci0 dev 24 function 0: vendor 0x1022 product 0x1200 (rev. 0x00)
pchb2 at pci0 dev 24 function 1: vendor 0x1022 product 0x1201 (rev. 0x00)
pchb3 at pci0 dev 24 function 2: vendor 0x1022 product 0x1202 (rev. 0x00)
pchb4 at pci0 dev 24 function 3: vendor 0x1022 product 0x1203 (rev. 0x00)
amdtemp0 at pchb4: AMD CPU Temperature Sensors (Family10h)
pchb5 at pci0 dev 24 function 4: vendor 0x1022 product 0x1204 (rev. 0x00)
isa0 at pcib0
com0 at isa0 port 0x3f8-0x3ff irq 4: ns16550a, working fifo
pckbc0 at isa0 port 0x60-0x64
acpicpu0 at cpu0: ACPI CPU
acpicpu0: C1: HLT, lat   0 us, pow     0 mW
acpicpu0: C2: I/O, lat  75 us, pow     1 mW, bus master check
acpicpu0: P0: FFH, lat   4 us, pow 19507 mW, 3000 MHz
acpicpu0: P1: FFH, lat   4 us, pow 14500 mW, 2300 MHz
acpicpu0: P2: FFH, lat   4 us, pow 10535 mW, 1600 MHz
acpicpu0: P3: FFH, lat   4 us, pow  6210 mW,  800 MHz
acpicpu0: T0: I/O, lat   1 us, pow     0 mW, 100 %
acpicpu0: T1: I/O, lat   1 us, pow     0 mW,  88 %
acpicpu0: T2: I/O, lat   1 us, pow     0 mW,  76 %
acpicpu0: T3: I/O, lat   1 us, pow     0 mW,  64 %
acpicpu0: T4: I/O, lat   1 us, pow     0 mW,  52 %
acpicpu0: T5: I/O, lat   1 us, pow     0 mW,  40 %
acpicpu0: T6: I/O, lat   1 us, pow     0 mW,  28 %
acpicpu0: T7: I/O, lat   1 us, pow     0 mW,  16 %
acpicpu1 at cpu1: ACPI CPU
acpicpu2 at cpu2: ACPI CPU
acpicpu3 at cpu3: ACPI CPU
acpicpu4 at cpu4: ACPI CPU
acpicpu5 at cpu5: ACPI CPU
fwohci0: BUS reset
fwohci0: node_id=0xc800ffc0, gen=1, CYCLEMASTER mode
ieee1394if0: 1 nodes, maxhop <= 0 cable IRM irm(0) (me)
ieee1394if0: bus manager 0
timecounter: Timecounter "clockinterrupt" frequency 100 Hz quality 0
timecounter: Timecounter "TSC" frequency 3010199210 Hz quality 3000
uhub0 at usb0: vendor 0x1002 OHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 5 ports with 5 removable, self powered
uhub1 at usb1: vendor 0x1002 EHCI root hub, class 9/0, rev 2.00/1.00, addr 1
uhub1: 5 ports with 5 removable, self powered
uhub2 at usb2: vendor 0x1002 OHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub2: 5 ports with 5 removable, self powered
uhub3 at usb3: vendor 0x1002 EHCI root hub, class 9/0, rev 2.00/1.00, addr 1
uhub3: 5 ports with 5 removable, self powered
uhub4 at usb4: vendor 0x1002 OHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub4: 2 ports with 2 removable, self powered
uhub5 at usb5: vendor 0x1002 OHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub5: 4 ports with 4 removable, self powered
uhub6 at usb6: vendor 0x1002 EHCI root hub, class 9/0, rev 2.00/1.00, addr 1
uhub6: 4 ports with 4 removable, self powered
ahcisata1 port 0: device present, speed: 6.0Gb/s
ahcisata1 port 1: device present, speed: 6.0Gb/s
ahcisata1 port 2: device present, speed: 1.5Gb/s
ehci1: handing over low speed device on port 1 to ohci1
wd0 at atabus2 drive 0
wd0: <ST2000DL003-9VT166>
wd0: drive supports 16-sector PIO transfers, LBA48 addressing
wd0: 1863 GB, 3876021 cyl, 16 head, 63 sec, 512 bytes/sect x 3907029168 sectors
wd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133)
wd0(ahcisata1:0:0): using PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133) 
(using DMA)
wd1 at atabus3 drive 0
wd1: <M4-CT256M4SSD2>
wd1: drive supports 16-sector PIO transfers, LBA48 addressing
wd1: 238 GB, 496149 cyl, 16 head, 63 sec, 512 bytes/sect x 500118192 sectors
ehci0: handing over full speed device on port 2 to ohci0
wd1: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 5 (Ultra/100)
wd1(ahcisata1:1:0): using PIO mode 4, DMA mode 2, Ultra-DMA mode 5 (Ultra/100) 
(using DMA)
atapibus0 at atabus4: 1 targets
cd0 at atapibus0 drive 0: <ATAPI   iHAS124   B, 3524223 3B812050134, AL0Q> 
cdrom removable
cd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 5 (Ultra/100)
cd0(ahcisata1:2:0): using PIO mode 4, DMA mode 2, Ultra-DMA mode 5 (Ultra/100) 
(using DMA)
ehci0: handing over low speed device on port 5 to ohci0
Kernelized RAIDframe activated
pad0: outputs: 44100Hz, 16-bit, stereo
audio1 at pad0: half duplex, playback, capture
boot device: wd0
root on wd0a dumps on wd0b
root file system type: ffs
ugen0 at uhub0 port 2
uhidev0 at uhub2 port 1 configuration 1 interface 0
ugen0: Syncrosoft eLicenser, rev 1.00/1.01, addr 2
uhidev0: Logitech USB Keyboard, rev 1.10/66.00, addr 2, iclass 3/1
ukbd0 at uhidev0: 8 modifier keys, 6 key codes
wskbd0 at ukbd0: console keyboard, using wsdisplay0
uhidev1 at uhub2 port 1 configuration 1 interface 1
uhidev1: Logitech USB Keyboard, rev 1.10/66.00, addr 2, iclass 3/0
uhidev1: 3 report ids
uhid0 at uhidev1 reportid 1: input=1, output=0, feature=0
uhid1 at uhidev1 reportid 2: input=1, output=0, feature=0
uhid2 at uhidev1 reportid 3: input=3, output=0, feature=0
aibs0: warning under limit on 'Chassis Fan Speed'
aibs0: warning under limit on 'Power Fan Speed'
wsdisplay0: screen 1 added (80x25, vt100 emulation)
wsdisplay0: screen 2 added (80x25, vt100 emulation)
wsdisplay0: screen 3 added (80x25, vt100 emulation)
wsdisplay0: screen 4 added (80x25, vt100 emulation)


>How-To-Repeat:
no idea - happens on this machine "most of the time"

>Fix:
can try patches, but have no idea myself.