Re: kern/50313: processes get stuck doing exec

[Havard Eidnes <he%uninett.no@localhost>]

>   > Memory resource pool statistics
>   > Name        Size Requests Fail Releases Pgreq Pgrel Npage Hiwat Minpg Maxpg Idle
>   > ...
>   > execargs    262144  40498    0    40498     9     7     2     4     0    16    2
>   > ...
>
>  Out of curiosity: how much RAM is in the system?

Physically 4G, but since this is i386, my kernel sees

total memory = 3316 MB
avail memory = 3248 MB

>  Because exec args are big, unless there's a throttle (I don't
>  think we have one) it's fairly easy for transient demand to
>  take up a lot of space, which can fill up the kernel map or
>  plain run out of memory.

Do we have a meter for the kernel map, inspectable from user-land
in some way?

When this problem strikes, it's usually going downhill.
Presently my host has a load average of 10, mostly caused by:

  PID TTY    STAT      TIME COMMAND
...
 1311 ?      D      0:00.00 cron: running job 
 2828 ?      Rs     0:00.00 (local)
 3101 ?      RVs    0:00.00 (cron)
 5471 ?      D      0:00.00 cron: running job 
 7074 ?      D      0:00.00 cron: running job 
23096 ?      RVs    0:00.00 (cron)
23277 ?      RVs    0:00.00 (cron)
23347 ?      RVs    0:00.00 (cron)
25722 ?      RVs    0:00.00 (cron)
26808 ?      D      0:00.00 cron: running job 
29057 ?      R      0:00.00 (sh)

and some mail delivery appears to get stuck:

Oct 11 01:17:03 smistad postfix/qmgr[14168]: D81A13D0B8: skipped, still being delivered

Some of these are stuck doing uvm_km_alloc(), some of them stuck
in fork:

crash> t/t 0t5471
trace: pid 5471 lid 1 at 0xdd07ebe4
sleepq_block(0,0,c0be3a51,c0cda590,8,c4677c00,c4675280,c9002034,0,cf7fadfc) at sleepq_block+0xa3
cv_wait(cf7fadfc,c4614f00,ddbb3000,80000000,0,0,c0803314,c8c2ad40,dd07ec88,0) at cv_wait+0x126
fork1(c4c90560,3,14,0,0,c0803314,0,dd07ed28,0,dd07ed3c) at fork1+0x6a0
sys___vfork14(c4c90560,dd07ed00,dd07ed28,c0803389,c77357ac,11a,bbab9000,bbab9d30,c77357ac,cf7fade4) at sys___vfork14+0x50
syscall(dd07ed48,bbbb00b3,ab,bfbf001f,bbba001f,1c20,0,bfbfed98,0,bb91e153) at syscall+0xaa
crash> 

Regards,

- Håvard