Re: Kernel (9.99.44) responsiveness issues

[Andrew Doran <ad%netbsd.org@localhost>]

On Sun, Feb 02, 2020 at 01:02:58PM +0100, Kamil Rytarowski wrote:

> I keep observing responsiveness issues on NetBSD-current. This happened
> in last 2 months.
> 
> Whenever I start building something with -j${CORES}, I have significant
> delays of responsiveness in other applications.

What does your disk I/O look like during this time?  Are you using ffs
logging?  Do you have a tmpfs /tmp or a custom TMPDIR set?  You have enough
CPUs there to hammer the disk with a compile job given the right
circumstances.

At the end of the crash(8) manual page someone has added a nice recipe to
get backtraces from all of the kernel stacks.  If you added a "grep tstile"
in the pipeline you could see where these particular threads are blocking.

Andrew

> load averages:  2.69,  5.56,  6.22;               up 0+01:32:42 12:12:34
> 71 processes: 69 sleeping, 2 on CPU
> CPU states:  0.0% user,  0.0% nice,  0.0% system,  0.1% interrupt, 99.8%
> idle
> Memory: 19G Act, 9639M Inact, 416K Wired, 34M Exec, 19G File, 43M Free
> Swap: 64G Total, 64G Free
> 
>   PID USERNAME PRI NICE   SIZE   RES STATE      TIME   WCPU    CPU COMMAND
>     0 root       0    0     0K   87M CPU/7      0:40  0.00%  0.49% [system]
> 15823 root      85    0    16M 2508K poll/1     0:01  0.00%  0.00% nbmake
> 25446 kamil     43    0    28M 2452K CPU/0      0:00  0.00%  0.00% top
> 14117 root     114    0    27M 3356K tstile/0   0:00  0.00%  0.00% ld
> 29088 root     114    0    27M 3280K tstile/3   0:00  0.00%  0.00% ld
> 20839 root     114    0    27M 3208K tstile/1   0:00  0.00%  0.00% ld
> 19550 root     114    0    26M 3184K tstile/6   0:00  0.00%  0.00% ld
> 13716 root     114    0    26M 3104K tstile/2   0:00  0.00%  0.00% ld
>  8758 root     114    0    26M 3048K tstile/7   0:00  0.00%  0.00% ld
>   240 root     114    0    26M 2580K tstile/0   0:00  0.00%  0.00% ld
> 
> I can see in top(1) that processes are locked in turnstiles and load
> goes down.
> 
> $ uname -a
> NetBSD chieftec 9.99.44 NetBSD 9.99.44 (GENERIC) #0: Fri Jan 31 19:26:07
> CET 2020
> root@chieftec:/public/netbsd-root/sys/arch/amd64/compile/GENERIC amd64
> 
> 135 kamil@chieftec /home/kamil $ cpuctl list
> 
> Num  HwId Unbound LWPs Interrupts Last change              #Intr
> ---- ---- ------------ ---------- ------------------------ -----
> 0    0    online       intr       Sun Feb  2 10:40:26 2020 13
> 1    2    online       intr       Sun Feb  2 10:40:26 2020 0
> 2    4    online       intr       Sun Feb  2 10:40:26 2020 0
> 3    6    online       intr       Sun Feb  2 10:40:26 2020 0
> 4    1    online       intr       Sun Feb  2 10:40:26 2020 0
> 5    3    online       intr       Sun Feb  2 10:40:26 2020 0
> 6    5    online       intr       Sun Feb  2 10:40:26 2020 0
> 7    7    online       intr       Sun Feb  2 10:40:26 2020 0
> 136 kamil@chieftec /home/kamil $ cpuctl identify 0
> Cannot bind to target CPU.  Output may not accurately describe the target.
> Run as root to allow binding.
> 
> cpu0: highest basic info 0000000d
> cpu0: highest extended info 80000008
> cpu0: "Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz"
> cpu0: Intel Xeon E3-1200v2 and 3rd gen core, Ivy Bridge (686-class),
> 3392.48 MHz
> cpu0: family 0x6 model 0x3a stepping 0x9 (id 0x306a9)
> cpu0: features
> 0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE>
> cpu0: features
> 0xbfebfbff<MCA,CMOV,PAT,PSE36,CLFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2>
> cpu0: features 0xbfebfbff<SS,HTT,TM,SBF>
> cpu0: features1 0x7fbae3ff<SSE3,PCLMULQDQ,DTES64,MONITOR,DS-CPL,VMX,SMX,EST>
> cpu0: features1 0x7fbae3ff<TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE41,SSE42,X2APIC>
> cpu0: features1
> 0x7fbae3ff<POPCNT,DEADLINE,AES,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
> cpu0: features2 0x28100800<SYSCALL/SYSRET,XD,RDTSCP,EM64T>
> cpu0: features3 0x1<LAHF>
> cpu0: features5 0x281<FSGSBASE,SMEP,ERMS>
> cpu0: xsave features 0x7<x87,SSE,AVX>
> cpu0: xsave instructions 0x1<XSAVEOPT>
> cpu0: xsave area size: current 832, maximum 832, xgetbv enabled
> cpu0: enabled xsave 0x7<x87,SSE,AVX>
> cpu0: I-cache 32KB 64B/line 8-way, D-cache 32KB 64B/line 8-way
> cpu0: L2 cache 256KB 64B/line 8-way
> cpu0: L3 cache 8MB 64B/line 16-way
> cpu0: 64B prefetching
> cpu0: ITLB 64 4KB entries 4-way, 2M/4M: 8 entries
> cpu0: DTLB 64 4KB entries 4-way, 2M/4M: 32 entries (L0)
> cpu0: L2 STLB 512 4KB entries 4-way
> cpu0: Initial APIC ID 1
> cpu0: Cluster/Package ID 0
> cpu0: Core ID 0
> cpu0: SMT ID 1
> cpu0: MONITOR/MWAIT extensions 0x3<EMX,IBE>
> cpu0: monitor-line size 64
> cpu0: C1 substates 2
> cpu0: C2 substates 1
> cpu0: C3 substates 1
> cpu0: DSPM-eax 0x77<DTS,IDA,ARAT,PLN,ECMD,PTM>
> cpu0: DSPM-ecx 0x9<HWF,EPB>
> cpu0: SEF highest subleaf 00000000
> cpu0: Perfmon-eax 0x7300403<VERSION=0x3,GPCounter=0x4,GPBitwidth=0x30>
> cpu0: Perfmon-eax 0x7300403<Vectorlen=0x7>
> cpu0: Perfmon-edx 0x603<FixedFunc=0x3,FFBitwidth=0x30>
> cpu0: microcode version 0x15, platform ID 1
>