Subject: ARP lockup (arp_drain: locked; punting)
To: None <tech-net@NetBSD.org>
From: Edgar =?unknown-8bit?B?RnXf?= <efnbl05@bn2.maus.net>
List: tech-net
Date: 03/24/2006 11:46:21
We're experiencing ARP lockups on one of our servers since upgrading
from 2.0.2 to 3.0.

The only uncommon thing about the machine in question is that it has two
network interfaces connected to the same ``wire''. We run two IP nets
(131.220.120/24 and 131.220.146/24) on the same ethernet (until I find the
time to partition the switches) and this was the way to make DHCP work.

The lockups occur at 04:48; I can't find anything special going on at
that time apart from our FAI clients starting a softupdate at 04:30;
that shouldn't put any heavy load on that particular server.

I get ``arp_drain: locked; punting'' in syslog and from that point on,
ARP seems to stop functioning. I can ping -n the machine's own IP,
but nothing else.

The interfaces in question are epic's, so this can't be the nfe memory
leak discussed in current-users.

The machine acts as a DHCP server, HTTP/FTP server and HTTP/FTP/RTSP proxy.
It also runs some internal list using Mailman.

vmstat -v -m -W (in the locked-up state) says:

Memory statistics by bucket size
    Size   In Use   Free   Requests  HighWater  Couldfree
      16     1658    390     100741    1280          0
      32      311    201      54880     640          0
      64      704   2048      49545     320        720
     128      242    110      16648     160          0
     256      358     26      10956      80          0
     512       80     40      33893      40          0
    1024      264    248      26781      20      11592
    2048       34     28       3364      10       3100
    4096       54     21        280       5        206
    8192       20      0         37       5          4
   16384       25      0         37       5          0
   32768       20      0         46       5          0
   65536        5      0          5       5          0

Memory usage type by bucket size
    Size  Type(s)
      16  USB, VM swap, UVM amap, ifaddr, namecache, soname, pipe, in_multi,
	  ip_moptions, routetbl, temp, pcb, devbuf, exec, kernfs mount,
	  RAIDframe, xform
      32  USB, AGP, VM swap, UVM amap, ifmedia, ether_multi, ifaddr, soname,
	  sysctldata, proc, fragtbl, routetbl, temp, pcb, devbuf, exec, kevent,
	  RAIDframe, xform
      64  USB, UVM amap, vnodes, prop, in_multi, routetbl, temp, pcb, DMA map,
	  devbuf, MFS node, NFS diroff, NFS req, NDP
     128  USB device, USB, AGP, VM swap, UVM amap, ifaddr, soname, ttys, prop,
	  sysctlnode, routetbl, temp, devbuf, kevent, inodedep, RAIDframe,
	  key mgmt, NDP
     256  USB, UVM amap, ifaddr, vnodes, subproc, proc, routetbl, temp, DMA map,
	  devbuf, exec, kevent, file desc, newblk, NFS diroff
     512  USB device, VM map, UVM amap, ifaddr, proc, UFS mount, temp, pcb,
	  devbuf, ioctlops, file desc, NFS mount, RAIDframe, crypto
    1024  VM page, UVM aobj, UVM amap, ttys, sysctlnode, temp, devbuf, exec,
	  kevent, NQNFS Lease, RAIDframe, key mgmt
    2048  UVM amap, ifaddr, sysctlnode, in_multi, UFS mount, temp, devbuf, exec,
	  RAIDframe
    4096  UVM amap, mount, namecache, sysctlnode, temp, devbuf, RAIDframe
    8192  UVM amap, sysctlnode, UFS mount, pagedep, RAIDframe
   16384  UVM amap, sysctlnode, RAIDframe
   32768  UVM amap, namecache, UFS mount, devbuf, inodedep, NFS node,
	  ISOFS mount, RAIDframe
   65536  UVM amap, temp, devbuf

Memory statistics by type                           Type  Kern
         Type  InUse MemUse HighUse  Limit Requests Limit Limit Size(s)
    USB device     4     2K      2K 58752K        4    0     0  128,512
           USB    16     2K      2K 58752K       16    0     0  16,32,64,128,256
           AGP     2     1K      1K 58752K        2    0     0  32,128
       VM swap     5     1K      1K 58752K        5    0     0  16,32,128
       VM page     1     1K      1K 58752K        1    0     0  1024
        VM map     4     2K      2K 58752K        4    0     0  512
      UVM aobj     1     1K      1K 58752K        1    0     0  1024
      UVM amap  2401   464K    562K 58752K   192935    0     0  16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536
       ifmedia    12     1K      1K 58752K       12    0     0  32
   ether_multi    10     1K      1K 58752K       10    0     0  32
        ifaddr    22     7K      7K 58752K       22    0     0  16,32,128,256,512,2048
         mount    10    40K     40K 58752K       11    0     0  4096
        vnodes    43     7K    126K 58752K     9481    0     0  64,256
     namecache     4    69K     69K 58752K        4    0     0  16,4096,32768
        soname     4     1K     12K 58752K     2976    0     0  16,32,128
          ttys    36    23K     23K 58752K       36    0     0  128,1024
          pipe     0     0K      1K 58752K       13    0     0  16
          prop     2     1K      1K 58752K        2    0     0  64,128
    sysctldata     0     0K      1K 58752K        1    0     0  32
    sysctlnode    52    88K     89K 58752K       76    0     0  128,1024,2048,4096,8192,16384
       subproc     4     1K      3K 58752K       62    0     0  256
          proc     8     1K      2K 58752K       16    0     0  32,256,512
      in_multi    34     4K      4K 58752K       34    0     0  16,64,2048
   ip_moptions     1     1K      1K 58752K        1    0     0  16
     UFS mount    21    75K     75K 58752K       21    0     0  512,2048,8192,32768
       fragtbl     0     0K      1K 58752K       77    0     0  32
      routetbl   101     7K     12K 58752K      633    0     0  16,32,64,128,256
          temp    12    58K     62K 58752K    53640    0     0  16,32,64,128,256,512,1024,2048,4096,65536
           pcb    22     4K     13K 58752K     3069    0     0  16,32,64,512
       DMA map   400    74K     74K 58752K      400    0     0  64,256
        devbuf   238   262K    262K 58752K      660    0     0  16,32,64,128,256,512,1024,2048,4096,32768,65536
          exec     2     1K      7K 58752K     6377    0     0  16,32,256,1024,2048
        kevent     7     3K      3K 58752K     4212    0     0  32,128,256,1024
      ioctlops     0     0K      1K 58752K       31    0     0  512
     file desc     1     1K      5K 58752K      300    0     0  256,512
      MFS node     1     1K      1K 58752K        1    0     0  64
        newblk     1     1K      1K 58752K        1    0     0  256
      inodedep     1    32K     33K 58752K        4    0     0  128,32768
       pagedep     1     8K      8K 58752K        1    0     0  8192
     NFS mount     3     2K      2K 58752K        3    0     0  512
    NFS diroff    31     3K      4K 58752K       40    0     0  64,256
       NFS req     0     0K      1K 58752K      362    0     0  64
   NQNFS Lease     1     1K      1K 58752K        1    0     0  1024
      NFS node     1    32K     32K 58752K        1    0     0  32768
  kernfs mount     1     1K      1K 58752K        1    0     0  16
   ISOFS mount     1    32K     32K 58752K        1    0     0  32768
     RAIDframe    65   690K    720K 58752K       76    0     0  16,32,128,512,1024,2048,4096,8192,16384,32768
      key mgmt   181   175K    427K 58752K    21542    0     0  128,1024
           NDP     6     1K      1K 58752K        8    0     0  64,128
        crypto     1     1K      1K 58752K        1    0     0  512
         xform     0     0K      1K 58752K       28    0     0  16,32

Memory totals:  In Use    Free    Requests
                 2164K    569K      297216

Memory resource pool statistics
Name              Size    Requests Fail    Releases InUse Avail  Pgreq  Pgrel Npage PageSz Hiwat Minpg Maxpg Idle Flags   Util
phpool-0            40        1286    0         525   761    47      8      0     8   4096     8     0   inf    0 0x040  92.9%
phpool-2            44           0    0           0     0     0      0      0     0   4096     0     0   inf    0 0x040   ---
phpool-4            44           0    0           0     0     0      0      0     0   4096     0     0   inf    0 0x040   ---
phpool-8            48           0    0           0     0     0      0      0     0   4096     0     0   inf    0 0x040   ---
phpool-16           56           0    0           0     0     0      0      0     0   4096     0     0   inf    0 0x040   ---
phpool-32           72          50    0           0    50     6      1      0     1   4096     1     0   inf    0 0x040  87.9%
phpool-64          104           0    0           0     0     0      0      0     0   4096     0     0   inf    0 0x040   ---
phpool-128         168           0    0           0     0     0      0      0     0   4096     0     0   inf    0 0x040   ---
pcgpool            140          67    0          46    21     7      1      0     1   4096     1     0   inf    0 0x040  71.8%
pmappl              76        3523    0        3494    29    24      1      0     1   4096     1     0   inf    0 0x040  53.8%
pdppl             4096          62    0          24    38     4     44      2    42   4096    42     0   inf    4 0x000  90.5%
	cache 0xc0363560: allocfrom 0xc067e08c freeto 0xc067e08c
	    hits 3461 misses 62 ngroups 1 nitems 9
kvakernel         4096        1289    0         142  1147     5     36      0    36 131072    36     0     0    0 0x600  99.6%
kvakmem           4096         548    0         104   444     4     14      0    14 131072    14     0     0    0 0x600  99.1%
igmppl              20           2    0           0     2   200      1      0     1   4096     1     0   inf    0 0x040   1.0%
inpcbpl            108       21508    0       21344   164   206     16      6    10   4096    12     0   inf    0 0x040  43.2%
inmltpl             32           3    0           0     3   123      1      0     1   4096     1     0   inf    0 0x040   2.3%
ipqepl              44         442    0         442     0     0      2      2     0   4096     1     0   inf    0 0x040   ---
tcpipqepl           44       18819    0       18819     0     0      1      1     0   4096     1     0   inf    0 0x040   ---
synpl              184        7003    0        7003     0     0      2      2     0   4096     1     0   inf    0 0x040   ---
sackholepl          20           0    0           0     0     0      0      0     0   4096     0     0   inf    0 0x040   ---
tcpcbpl            360       18387    0       18236   151   201     54     22    32   4096    37     0   inf    4 0x040  41.5%
in6pcbpl           156          26    0          16    10    16      1      0     1   4096     1     0   inf    0 0x040  38.1%
cd9660nopl         160           0    0           0     0     0      0      0     0   4096     0     0   inf    0 0x040   ---
nfsnodepl          308          35    0          27     8    18      2      0     2   4096     2     0   inf    1 0x040  30.1%
nfsvapl            108          35    0          27     8    29      1      0     1   4096     1     0   inf    0 0x040  21.1%
sdpcpool           124       19293    0       19292     1   351     29     18    11   4096    16     0   inf   10 0x040   0.3%
pagedeppl           68        1438    0        1437     1    58      2      1     1   4096     1     0   inf    0 0x040   1.7%
inodedeppl          88       15335    0       15334     1   321     24     17     7   4096    15     0   inf    6 0x040   0.3%
newblkpl            36       37156    0       37156     0   112      1      0     1   4096     1     0   inf    1 0x040   0.0%
bmsafemappl         36        2561    0        2560     1   111      1      0     1   4096     1     0   inf    0 0x040   0.9%
allocdirectpl       80       31863    0       31862     1   299     19     13     6   4096    11     0   inf    5 0x040   0.3%
indirdeppl          32         368    0         368     0   126      1      0     1   4096     1     0   inf    1 0x040   0.0%
allocindirpl        64        5293    0        5293     0   441     19     12     7   4096     7     0   inf    7 0x040   0.0%
freefragpl          40       14510    0       14510     0   202      4      2     2   4096     2     0   inf    2 0x040   0.0%
freeblkspl         172        6884    0        6884     0   138     22     16     6   4096    10     0   inf    6 0x040   0.0%
freefilepl          36        6918    0        6918     0   224      5      3     2   4096     3     0   inf    2 0x040   0.0%
diraddpl            36        8584    0        8583     1   223      4      2     2   4096     3     0   inf    1 0x040   0.4%
mkdirpl             32           2    0           2     0   126      1      0     1   4096     1     0   inf    1 0x040   0.0%
dirrempl            36        7180    0        7180     0   224      5      3     2   4096     2     0   inf    2 0x040   0.0%
newdirblkpl         20           1    0           1     0   202      1      0     1   4096     1     0   inf    1 0x040   0.0%
ffsinopl           196      248834    0      242744  6090   270    318      0   318   4096   318     0   inf    0 0x040  91.6%
dino1pl            128      248834    0      242744  6090   265    205      0   205   4096   205     0   inf    0 0x040  92.8%
dino2pl            256           0    0           0     0     0      0      0     0   4096     0     0   inf    0 0x000   ---
ufsdirpl           264        8554    0        8554     0    15      2      1     1   4096     1     0   inf    1 0x040   0.0%
rndsample          528        2324    0        2321     3     4      1      0     1   4096     1     0   inf    0 0x040  38.7%
filepl              56      697102    0      696911   191    97      6      2     4   4096     4     0   inf    0 0x040  65.3%
cwdipl              16        3524    0        3494    30   223      1      0     1   4096     1     0   inf    0 0x040  11.7%
fdescpl            284        3524    0        3494    30    12      3      0     3   4096     3     0   inf    0 0x040  69.3%
kqueuepl           260           2    0           0     2    13      1      0     1   4096     1     0   inf    0 0x040  12.7%
knotepl             72          13    0           0    13    43      1      0     1   4096     1     0   inf    0 0x040  22.9%
ktepl               56           0    0           0     0     0      0      0     0   4096     0     0   inf    0 0x040   ---
procpl             408        3539    0        3494    45    15      8      2     6   4096     6     0   inf    1 0x000  74.7%
lwppl              132        3539    0        3494    45    15      2      0     2   4096     2     0   inf    0 0x040  72.5%
lwpucpl            776           0    0           0     0     0      0      0     0   4096     0     0   inf    0 0x040   ---
pgrppl              16         221    0         201    20   233      1      0     1   4096     1     0   inf    0 0x040   7.8%
pcredpl             24        3539    0        3494    45   124      1      0     1   4096     1     0   inf    0 0x040  26.4%
plimitpl           176          50    0          45     5    18      1      0     1   4096     1     0   inf    0 0x040  21.5%
pstatspl           224        3539    0        3494    45    27      4      0     4   4096     4     0   inf    1 0x040  61.5%
rusgepl             72        3494    0        3494     0    56      3      2     1   4096     1     0   inf    1 0x040   0.0%
raspl               20          33    0          24     9   193      1      0     1   4096     1     0   inf    0 0x040   4.4%
sadatapl            40           0    0           0     0     0      0      0     0   4096     0     0   inf    0 0x040   ---
saupcpl           1612           0    0           0     0     0      0      0     0   4096     0     0   inf    0 0x040   ---
sastackpl           24           0    0           0     0     0      0      0     0   4096     0     0   inf    0 0x040   ---
savppl              52           0    0           0     0     0      0      0     0   4096     0     0   inf    0 0x040   ---
ptimerpl           124          62    0          58     4    28      1      0     1   4096     1     0   inf    0 0x040  12.1%
sessionpl           40         186    0         169    17    84      1      0     1   4096     1     0   inf    0 0x040  16.6%
credpl              84        1137    0        1120    17    79      2      0     2   4096     2     0   inf    0 0x040  17.4%
siginfo            128           0    0           0     0     0      0      0     0   4096     0     0   inf    0 0x040   ---
ksiginfo            44           2    0           2     0    92      1      0     1   4096     1     0   inf    1 0x040   0.0%
strprocpl          104           0    0           0     0     0      0      0     0   4096     0     0   inf    0 0x040   ---
strpolpl            28           0    0           0     0     0      0      0     0   4096     0     0   inf    0 0x040   ---
strmsgpl           300           0    0           0     0     0      0      0     0   4096     0     0   inf    0 0x040   ---
pipepl             144        4654    0        4646     8    20      1      0     1   4096     1     0   inf    0 0x040  28.1%
shmmp               12           0    0           0     0     0      0      0     0   4096     0     0   inf    0 0x040   ---
ttypl              264           7    0           0     7     8      1      0     1   4096     1     0   inf    0 0x040  45.1%
sockpl             216       24592    0       24272   320   184     43     15    28   4096    31     0   inf    3 0x040  60.3%
ncachepl            84      220970    0      214864  6106   278    155     22   133   4096   133     0   inf    0 0x040  94.2%
lockfpl             52         964    0         963     1    77      1      0     1   4096     1     0   inf    0 0x040   1.3%
vnodepl            168       13096    0        6986  6110   250    265      0   265   4096   265     0   inf    0 0x040  94.6%
rtentpl            132         290    0         242    48    72      4      0     4   4096     4     0   inf    0 0x040  38.7%
rttmrpl             32           0    0           0     0     0      0      0     0   4096     0     0   inf    0 0x040   ---
amappl              40       59293    0       58500   793   217     12      2    10   4096    10     0   inf    0 0x040  77.4%
uaoeltpl            84           8    0           3     5    43      1      0     1   4096     1     0   inf    0 0x040  10.3%
aobjpl              52           0    0           0     0     0      0      0     0   4096     0     0   inf    0 0x040   ---
vmsppl             224        3523    0        3494    29    25      5      2     3   4096     3     0   inf    1 0x040  52.9%
vmmpepl             88      293077    0      291630  1447   393     48      8    40   4096    40     0   inf    3 0x040  77.7%
swp vnx             20           0    0           0     0     0      0      0     0   4096     0     0   inf    0 0x040   ---
swp vnd            128           0    0           0     0     0      0      0     0   4096     0     0   inf    0 0x040   ---
ataspl              52      146592    0      146592     0    78      1      0     1   4096     1     0   inf    1 0x040   0.0%
bufpl              124      230221    0      227531  2690  3326    189      1   188   4096   188     0   inf    7 0x040  43.3%
buf1k             1024       14191    0       12036  2155   725    185    140    45  65536    78     1     1    0 0x000  74.8%
buf2k             2048         332    0         307    25     7      7      6     1  65536     4     1     1    0 0x000  78.1%
buf4k             4096         193    0         174    19    13      5      3     2  65536     3     1     1    0 0x000  59.4%
buf8k             8192        7973    0        7482   491    85    436    364    72  65536   229     1     1    0 0x000  85.2%
buf16k           16384           0    0           0     0     4      1      0     1  65536     1     1     1    1 0x000   0.0%
buf32k           32768           0    0           0     0     2      1      0     1  65536     1     1     1    1 0x000   0.0%
buf64k           65536           0    0           0     0     1      1      0     1  65536     1     1     1    1 0x000   0.0%
mbpl               256        1842    0         449  1393    31     90      1    89   4096    89     1   inf    1 0x000  97.8%
	cache 0xc03605c0: allocfrom 0xc067e460 freeto 0xc067e460
	    hits 2079559 misses 1842 ngroups 13 nitems 137
mclpl             2048        1100 100452       100  1000     0    507      7   500   4096   500     4   512    0 0x004 100.0%
	cache 0xc0360280: allocfrom 0x0 freeto 0xc067e71c
	    hits 778353 misses 101553 ngroups 6 nitems 0
pnbufpl           1024          12    0           8     4     0      1      0     1   4096     1     0   inf    0 0x000 100.0%
	cache 0xc0360bc0: allocfrom 0xc067e000 freeto 0xc067e000
	    hits 954678 misses 12 ngroups 1 nitems 4
extent              20       12524    0       11203  1321   295      8      0     8   4096     8     0   inf    0 0x040  80.6%
ccdpl              140           0    0           0     0     0      0      0     0   4096     0     0   inf    0 0x040   ---
raidpl             132      146532    0      146532     0   120      8      4     4   4096     4     2     4    4 0x040   0.0%
sigapl            2052        3524    0        3494    30    12     54     12    42   4096    42     0   inf   12 0x040  35.8%
rf_alloclist_pl    168       99305    0       99304     1    95      6      2     4   4096     4     3    11    3 0x040   1.0%
rf_asmhdr_pl        16       88741    0       88741     0   253      1      0     1   4096     1     1     2    1 0x040   0.0%
rf_asm_pl          288       99304    0       99304     0    98     11      4     7   4096     7     5    14    7 0x040   0.0%
rf_asmhle_pl         8           0    0           0     0   507      1      0     1   4096     1     1     2    1 0x040   0.0%
rf_pda_pl           40      198608    0      198608     0   202      4      2     2   4096     2     1     2    2 0x040   0.0%
rf_fss_pl           20           0    0           0     0   202      1      0     1   4096     1     1     2    1 0x040   0.0%
rf_vfple_pl          8           0    0           0     0   507      1      0     1   4096     1     1     2    1 0x040   0.0%
rf_vple_pl           8          30    0           0    30   477      1      0     1   4096     1     1     2    0 0x040   5.9%
rf_revent_pl        16           0    0           0     0   253      1      0     1   4096     1     1     2    1 0x040   0.0%
rf_callbackpl       20           0    0           0     0   202      1      0     1   4096     1     1     2    1 0x040   0.0%
rf_rad_pl          328       88741    0       88741     0    48      6      2     4   4096     4     3    11    4 0x040   0.0%
rf_stripelock_pl    28       99304    0       99304     0   144      1      0     1   4096     1     1     2    1 0x040   0.0%
rf_mcpair_pl        12           0    0           0     0   338      1      0     1   4096     1     1     2    1 0x040   0.0%
rf_dagnode_pl      444      444444    0      444444     0   279     61     30    31   4096    31    15    57   31 0x040   0.0%
rf_dagh_pl          76       99304    0       99304     0   106      4      2     2   4096     2     1     3    2 0x040   0.0%
rf_daglist_pl      236       99304    0       99304     0    68      8      4     4   4096     4     2     8    4 0x040   0.0%
rf_dagpcache_pl    712           0    0           0     0    10      2      0     2   4096     2     2    26    2 0x040   0.0%
rf_funclist_pl       8       99304    0       99304     0   507      1      0     1   4096     1     1     2    1 0x040   0.0%
rf_reconbuffer_pl   76           0    0           0     0    53      1      0     1   4096     1     1     2    1 0x040   0.0%
rf_dqd_pl          116      146532    0      146532     0   136      8      4     4   4096     4     2     8    4 0x040   0.0%
raidpsspl          128           0    0           0     0    31      1      0     1   4096     1     1     2    1 0x040   0.0%

In use 12968K, total allocated 16312K; utilization 79.5%

Thanks for any help.