Subject: Re: #32287 Processes hang in "mclpl" - feedback
To: Christos Zoulas <christos@astron.com>
From: Jorgen Lundman <lundman@lundman.net>
List: netbsd-users
Date: 08/28/2006 10:54:03
> 
> So we see from the data that you have the same problem like I do. We are
> leaking mbufs in the tcp recv. I've been reading the code, but I have not
> found a problem...
> 

They finally managed to make it happen:

Aug 26 14:28:16 mirror /netbsd: WARNING: mclpool limit reached; increase NMBCLUS
TERS
Aug 26 14:30:11 mirror /netbsd: WARNING: mclpool limit reached; increase NMBCLUS
TERS
Aug 26 14:31:14 mirror /netbsd: WARNING: mclpool limit reached; increase NMBCLUS
TERS
Aug 26 14:37:55 mirror last message repeated 5 times
Aug 26 15:03:28 mirror /netbsd: WARNING: mclpool limit reached; increase NMBCLUS
TERS
Aug 26 15:07:53 mirror last message repeated 3 times
Aug 26 15:23:00 mirror last message repeated 8 times
Aug 26 15:31:48 mirror last message repeated 5 times
Aug 26 15:47:12 mirror /netbsd: WARNING: mclpool limit reached; increase NMBCLUS
TERS


Crontab runs netstat every 5 minutes, which show just before:

Sat Aug 26 14:20:00 CEST 2006
33001 mbufs in use:
         32967 mbufs allocated to data
         34 mbufs allocated to packet headers 
  14666 calls to protocol drain routines
                                              small        ext    cluster
            route               inuse             0          0          0
                                claims           19          0          0
                                releases         19          0          0
              arp               inuse             0          0          0
                                claims          604        443        443
                                releases        604        443        443
             unix               inuse             0          0          0
                                claims       151316      14061       5504
                                releases     151316      14061       5504
              tcp               inuse            34          0          0
                                claims       541276          0          0
                                releases     541242          0          0
              tcp rx            inuse             0          0          0
                                claims    454399042  454302211  454302211
                                releases  454399042  454302211  454302211
              tcp tx            inuse         32838      17142      17142
                                claims   3235280646 1585116978 1585116792
                                releases 3235247808 1585099836 1585099650
              udp               inuse             0          0          0
                                claims        11362          0          0
                                releases      11362          0          0
              udp rx            inuse             0          0          0
                                claims        25640      16844      16844
                                releases      25640      16844      16844
                                              small        ext    cluster
              udp tx            inuse             0          0          0
                                claims        10356          0          0
                                releases      10356          0          0
         internet rx            inuse             0          0          0
                                claims    454403644  454310455  454310455
                                releases  454403644  454310455  454310455
         internet tx            inuse             0          0          0
                                claims    693325148          3          3
                                releases  693325148          3          3
              lo0               inuse             0          0          0
                                claims        93207          0          0
                                releases      93207          0          0
              ex0 rx            inuse             0          0          0
                                claims    454310898  454310898  454310898
                                releases  454310898  454310898  454310898
              ex0 tx            inuse             0          0          0
                                claims   2188204597 1076832568 1076832397
                                releases 2188204597 1076832568 1076832397
          unknown data          inuse           128        128        128
                                claims   2996472228  454311282  454311282
                                releases 2996472100  454311154  454311154
          unknown header        inuse             0          0          0
                                claims    693331839          0          0
                                releases  693331839          0          0
                                              small        ext    cluster
          unknown soname        inuse             0          0          0
                                claims       497540          0          0
                                releases     497540          0          0
          unknown soopts        inuse             0          0          0 
                                 claims        99586          0          0
                                releases      99586          0          0
          unknown control       inuse             0          0          0
                                claims            4          0          0
                                releases          4          0          0



And this mornings reading is:


Mon Aug 28 03:25:01 CEST 2006
53478 mbufs in use:
         53429 mbufs allocated to data
         49 mbufs allocated to packet headers
286534 calls to protocol drain routines
                                              small        ext    cluster
            route               inuse             0          0          0
                                claims           19          0          0
                                releases         19          0          0
              arp               inuse             0          0          0
                                claims          884        647        647
                                releases        884        647        647
             unix               inuse             0          0          0
                                claims       228266      20680       8191
                                releases     228266      20680       8191
              tcp               inuse            49          0          0
                                claims       736300          0          0
                                releases     736251          0          0
              tcp rx            inuse             0          0          0
                                claims    617399674  617271075  617271075
                                releases  617399674  617271075  617271075
              tcp tx            inuse         53301      27885      27885
                                claims    154404677 2183697636 2183141342
                                releases  154351376 2183669751 2183113457
              udp               inuse             0          0          0
                                claims        14544          0          0
                                releases      14544          0          0
              udp rx            inuse             0          0          0
                                claims        32344      21205      21205
                                releases      32344      21205      21205
                                              small        ext    cluster
              udp tx            inuse             0          0          0
                                claims        13095          0          0
                                releases      13095          0          0
         internet rx            inuse             0          0          0
                                claims    617440932  617317548  617317548
                                releases  617440932  617317548  617317548
         internet tx            inuse             0          0          0
                                claims    943883790      36582      36582
                                releases  943883790      36582      36582
         internet               inuse             0          0          0
                                claims          148         35         35
                                releases        148         35         35
              lo0               inuse             0          0          0
                                claims       123414          0          0
                                releases     123414          0          0
              ex0 rx            inuse             0          0          0
                                claims    617318195  617318195  617318195
                                releases  617318195  617318195  617318195
              ex0 tx            inuse             0          0          0
                                claims   3002895211 1481989909 1481482587
                                releases 3002895211 1481989909 1481482587
          unknown data          inuse           128        128        128
                                claims   4123143040  617318579  617318579
                                releases 4123142912  617318451  617318451
                                              small        ext    cluster
          unknown header        inuse             0          0          0
                                claims    943861192          0          0
                                releases  943861192          0          0
          unknown soname        inuse             0          0          0
                                claims       674196          0          0
                                releases     674196          0          0
          unknown soopts        inuse             0          0          0
                                claims       140343          0          0
                                releases     140343          0          0
          unknown control       inuse             0          0          0
                                claims           12          0          0
                                releases         12          0          0



              tcp tx            inuse         53301      27885      27885
                                claims    154404677 2183697636 2183141342
                                releases  154351376 2183669751 2183113457


tcp rx:
154351376  + 53301 =  154404677
2183669751 + 27885 = 2183697636
2183113457 + 27885 = 2183141342

Is it really leaking?

Lund

-- 
Jorgen Lundman       | <lundman@lundman.net>
Unix Administrator   | +81 (0)3 -5456-2687 ext 1017 (work)
Shibuya-ku, Tokyo    | +81 (0)90-5578-8500          (cell)
Japan                | +81 (0)3 -3375-1767          (home)