Subject: Re: kern/32409: NetBSD-3.0/i386 vnlock deadlock
To: YAMAMOTO Takashi <yamt@mwd.biglobe.ne.jp>
From: SODA Noriyuki <soda@sra.co.jp>
List: netbsd-bugs
Date: 12/29/2005 21:06:19
>>>>> On Thu, 29 Dec 2005 20:53:29 +0900,
      YAMAMOTO Takashi <yamt@mwd.biglobe.ne.jp> said:

>> 5145            ftpd   netbsd   17  36.7  65.1 biowait     0x65790428
>> 24882          rsync   netbsd    4 185.5 105.2 uvn_fp2     0x61641480

>> 8563            tcsh   netbsd   17   0.0   0.0 biowait     0x65158938

>> 145          syslogd   netbsd    4   2.0   6.6 uvn_fp2     0x61532480

> is their i/o making any progress?

Network I/O was working, for example, sshd echoed back keyboard input.

I'm not sure about disk I/O.
What is the good way to see that?


>> 6            ioflush   netbsd    4   0.0 250.1 genput      0x62538d80

> this thread might be interesting.

vnlock 0x71921a58 -> vnode 0x719219e8

db> x/lu 71921a58+0t24
0x71921a70:     6

db> trace/t 0t6
trace: pid 6  at 0x70e45b6c
ltsleep(62538d80,204,602df8ce,0,719219e8) at netbsd:ltsleep+0x355
genfs_putpages(70e45da4,8b6d667c,1fb94,66503494,66503494) at netbsd:genfs_putpages+0x756
ffs_full_fsync(70e45f44,6e710810,6e710824,0,6026d0f3) at netbsd:ffs_full_fsync+0x366
ffs_fsync(70e45f44,12,2,64,0) at netbsd:ffs_fsync+0x55
sched_sync(705eb294,388000,38f000,0,60100321) at netbsd:sched_sync+0x210

db> show vnode/f 719219e8    
OBJECT 0x719219e8: locked=0, pgops=0x603158d0, npages=356, refs=1
  PAGES <pg,offset>:
  <0x62af5580,0x248c000> <0x614235c0,0x248d000> <0x6202ee00,0x248e000>
  <0x62eeb440,0x248f000> <0x6105a200,0x2490000> <0x6089d440,0x2491000>
  <0x62e87280,0x2492000> <0x632684c0,0x2493000> <0x61cd6cc0,0x2494000>
  <0x6391b500,0x2495000> <0x612ad340,0x2496000> <0x63106d80,0x2497000>
  <0x61f49b40,0x2498000> <0x6219f980,0x2499000> <0x609bc3c0,0x249a000>
  <0x60d93200,0x249b000> <0x60a95e80,0x249c000> <0x60b786c0,0x249d000>
  <0x60a1bb00,0x249e000> <0x60c22f40,0x249f000> <0x608d4c40,0x24a0000>
  <0x62b39c80,0x24a1000> <0x624276c0,0x24a2000> <0x60440b00,0x24a3000>
  <0x61e05140,0x24a4000> <0x62412380,0x24a5000> <0x61321dc0,0x24a6000>
  <0x6110e400,0x24a7000> <0x62955100,0x24a8000> <0x63279d40,0x24a9000>
  <0x629d1380,0x24aa000> <0x63775fc0,0x24ab000> <0x62a8ee00,0x24ac000>
  <0x6099f840,0x24ad000> <0x609b2880,0x24ae000> <0x63253ec0,0x24af000>
  <0x6068fe80,0x24b0000> <0x60a9c6c0,0x24b1000> <0x63999700,0x24b2000>
  <0x60988740,0x24b3000> <0x61e78800,0x24b4000> <0x639ce240,0x24b5000>
  <0x6365d280,0x24b6000> <0x631d50c0,0x24b7000> <0x60a6d140,0x24b8000>
  <0x618dbb80,0x24b9000> <0x638dc9c0,0x24ba000> <0x61b0c000,0x24bb000>
  <0x604a08c0,0x24bc000> <0x6121c700,0x24bd000> <0x61bb8340,0x24be000>
  <0x61423380,0x24bf000> <0x61ad5f80,0x24c0000> <0x62a25bc0,0x24c1000>
  <0x62798200,0x24c2000> <0x6394dc40,0x24c3000> <0x6094e240,0x24c4000>
  <0x617ea880,0x24c5000> <0x618040c0,0x24c6000> <0x60e83f00,0x24c7000>
--db_more-- 
  <0x617ea880,0x24c5000> <0x618040c0,0x24c6000> <0x60e83f00,0x24c7000>
  <0x60de7e80,0x24c8000> <0x6247cec0,0x24c9000> <0x6317dd00,0x24ca000>
  <0x631bcb40,0x24cb000> <0x62e59d80,0x24cc000> <0x638163c0,0x24cd000>
  <0x62648c00,0x24ce000> <0x62165040,0x24cf000> <0x6174ed40,0x24d0000>
  <0x620bbb80,0x24d1000> <0x60afd5c0,0x24d2000> <0x6153a200,0x24d3000>
  <0x61d6d440,0x24d4000> <0x61de6880,0x24d5000> <0x61b1d4c0,0x24d6000>
  <0x61ac3500,0x24d7000> <0x613d5140,0x24d8000> <0x62f12580,0x24d9000>
  <0x61b63dc0,0x24da000> <0x6172a600,0x24db000> <0x62537000,0x24dc000>
  <0x61b3d640,0x24dd000> <0x6180b880,0x24de000> <0x623680c0,0x24df000>
  <0x60de1340,0x24e0000> <0x60f3bb80,0x24e1000> <0x6183d5c0,0x24e2000>
  <0x61547a00,0x24e3000> <0x608e5cc0,0x24e4000> <0x61457100,0x24e5000>
  <0x6071bf40,0x24e6000> <0x61aecd80,0x24e7000> <0x61eaf180,0x24e8000>
  <0x621fadc0,0x24e9000> <0x6394da00,0x24ea000> <0x62bd6040,0x24eb000>
  <0x62ff02c0,0x24ec000> <0x62927f00,0x24ed000> <0x631e5140,0x24ee000>
  <0x61f8d780,0x24ef000> <0x6275c200,0x24f0000> <0x60984c40,0x24f1000>
  <0x63393280,0x24f2000> <0x618950c0,0x24f3000> <0x637d39c0,0x24f4000>
  <0x61353e00,0x24f5000> <0x621f8240,0x24f6000> <0x6371e480,0x24f7000>
  <0x6323a580,0x24f8000> <0x63688fc0,0x24f9000> <0x61cefc00,0x24fa000>
  <0x61347e40,0x24fb000> <0x6059cf80,0x24fc000> <0x632553c0,0x24fd000>
  <0x61cad200,0x24fe000> <0x612b8240,0x24ff000> <0x6305d500,0x2500000>
  <0x6353e140,0x2501000> <0x62840d80,0x2502000> <0x609ea1c0,0x2503000>
  <0x6307efc0,0x2504000> <0x60843a00,0x2505000> <0x629dfa40,0x2506000>
  <0x6381bc80,0x2507000> <0x61c93c80,0x2508000> <0x6395b4c0,0x2509000>
  <0x61ae1700,0x250a000> <0x620dcd40,0x250b000> <0x61581ec0,0x250c000>
--db_more-- 
  <0x61ae1700,0x250a000> <0x620dcd40,0x250b000> <0x61581ec0,0x250c000>
  <0x636f0500,0x250d000> <0x631d9d40,0x250e000> <0x637a6980,0x250f000>
  <0x62491ac0,0x2510000> <0x615a7300,0x2511000> <0x60b84740,0x2512000>
  <0x61905980,0x2513000> <0x61aa1500,0x2514000> <0x61cff540,0x2515000>
  <0x6237e180,0x2516000> <0x632a85c0,0x2517000> <0x609a4380,0x2518000>
  <0x6050e1c0,0x2519000> <0x629e5200,0x251a000> <0x6218c640,0x251b000>
  <0x62498480,0x251c000> <0x62fde6c0,0x251d000> <0x62c65f00,0x251e000>
  <0x62f62740,0x251f000> <0x63026f00,0x2520000> <0x61f93f40,0x2521000>
  <0x60dce580,0x2522000> <0x6226cbc0,0x2523000> <0x6349ad40,0x2524000>
  <0x60a22980,0x2525000> <0x61fed9c0,0x2526000> <0x622c8600,0x2527000>
  <0x62c69800,0x2528000> <0x62260640,0x2529000> <0x624a0880,0x252a000>
  <0x62daeac0,0x252b000> <0x61a17800,0x252c000> <0x60d41a40,0x252d000>
  <0x63132680,0x252e000> <0x62460ac0,0x252f000> <0x60ce8e00,0x2530000>
  <0x62aada40,0x2531000> <0x612aa680,0x2532000> <0x6382c0c0,0x2533000>
  <0x60908b00,0x2534000> <0x61d40340,0x2535000> <0x61016580,0x2536000>
  <0x616e85c0,0x2537000> <0x61bb2f00,0x2538000> <0x61acbb40,0x2539000>
  <0x61306d80,0x253a000> <0x635f81c0,0x253b000> <0x61132b40,0x253c000>
  <0x62900980,0x253d000> <0x637ecbc0,0x253e000> <0x60d07e00,0x253f000>
  <0x62cf9f40,0x2540000> <0x63162780,0x2541000> <0x6080cbc0,0x2542000>
  <0x62097e00,0x2543000> <0x61477a40,0x2544000> <0x60513680,0x2545000>
  <0x61400ac0,0x2546000> <0x6384b900,0x2547000> <0x61575d40,0x2548000>
  <0x61cf0380,0x2549000> <0x62b987c0,0x254a000> <0x60dbfc00,0x254b000>
  <0x611405c0,0x254c000> <0x614f7e00,0x254d000> <0x61bbc040,0x254e000>
  <0x61348a80,0x254f000> <0x6164bb80,0x2550000> <0x620149c0,0x2551000>
--db_more-- 
  <0x61348a80,0x254f000> <0x6164bb80,0x2550000> <0x620149c0,0x2551000>
  <0x62bcfe00,0x2552000> <0x60ce2040,0x2553000> <0x62702680,0x2554000>
  <0x60fb7ac0,0x2555000> <0x61124100,0x2556000> <0x60550f40,0x2557000>
  <0x610b49c0,0x2558000> <0x61d95a00,0x2559000> <0x634dbc40,0x255a000>
  <0x6168ca80,0x255b000> <0x618dc500,0x255c000> <0x60da6940,0x255d000>
  <0x6338b380,0x255e000> <0x60c725c0,0x255f000> <0x6119f1c0,0x2560000>
  <0x6385aa00,0x2561000> <0x6046d040,0x2562000> <0x613c7680,0x2563000>
  <0x61cd8fc0,0x2564000> <0x60da2200,0x2565000> <0x627ffe40,0x2566000>
  <0x63340280,0x2567000> <0x60525e00,0x2568000> <0x61749040,0x2569000>
  <0x61047080,0x256a000> <0x623696c0,0x256b000> <0x6134cb80,0x256c000>
  <0x60f25bc0,0x256d000> <0x60a86800,0x256e000> <0x63234040,0x256f000>
  <0x62b79d00,0x2570000> <0x61bd3740,0x2571000> <0x60ff5580,0x2572000>
  <0x61a3a5c0,0x2573000> <0x607e0f40,0x2574000> <0x62e94f80,0x2575000>
  <0x62e8f7c0,0x2576000> <0x61f07800,0x2577000> <0x634b0000,0x2578000>
  <0x60fe8e40,0x2579000> <0x60715c80,0x257a000> <0x62325cc0,0x257b000>
  <0x62a985c0,0x257c000> <0x62655c00,0x257d000> <0x60e46240,0x257e000>
  <0x61fb0080,0x257f000> <0x62b3c500,0x2580000> <0x60fb8d40,0x2581000>
  <0x634d3580,0x2582000> <0x604dd1c0,0x2583000> <0x62347600,0x2584000>
  <0x62bf4440,0x2585000> <0x60975880,0x2586000> <0x617c32c0,0x2587000>
  <0x6336aac0,0x2588000> <0x6077a100,0x2589000> <0x6127c540,0x258a000>
  <0x60fb8780,0x258b000> <0x63430c00,0x258c000> <0x6253de40,0x258d000>
  <0x61eaa080,0x258e000> <0x60867ec0,0x258f000> <0x6395d880,0x2590000>
  <0x637950c0,0x2591000> <0x61c9ab00,0x2592000> <0x61521540,0x2593000>
  <0x62928400,0x2594000> <0x60496640,0x2595000> <0x61ae0c80,0x2596000>
--db_more-- 
  <0x62928400,0x2594000> <0x60496640,0x2595000> <0x61ae0c80,0x2596000>
  <0x62f67ec0,0x2597000> <0x60bb6300,0x2598000> <0x63367b40,0x2599000>
  <0x60b03980,0x259a000> <0x6077c9c0,0x259b000> <0x61b944c0,0x259c000>
  <0x62cd5b00,0x259d000> <0x62c48140,0x259e000> <0x62079580,0x259f000>
  <0x62382a80,0x25a0000> <0x63276980,0x25a1000> <0x60be35c0,0x25a2000>
  <0x6239a400,0x25a3000> <0x6241dcc0,0x25a4000> <0x61cefd00,0x25a5000>
  <0x60f30140,0x25a6000> <0x60bead80,0x25a7000> <0x634ca040,0x25a8000>
  <0x60577280,0x25a9000> <0x630114c0,0x25aa000> <0x612fd900,0x25ab000>
  <0x608d1d80,0x25ac000> <0x60755dc0,0x25ad000> <0x60e6fc00,0x25ae000>
  <0x6147ca40,0x25af000> <0x61530ec0,0x25b0000> <0x629d5700,0x25b1000>
  <0x60400940,0x25b2000> <0x605f9f80,0x25b3000> <0x62efca40,0x25b4000>
  <0x6119d280,0x25b5000> <0x6285b0c0,0x25b6000> <0x6047a700,0x25b7000>
  <0x61a031c0,0x25b8000> <0x619f0600,0x25b9000> <0x61432040,0x25ba000>
  <0x636d1c80,0x25bb000> <0x60439940,0x25bc000> <0x631fc780,0x25bd000>
  <0x63594dc0,0x25be000> <0x61c00a00,0x25bf000> <0x612dcf40,0x25c0000>
  <0x60504180,0x25c1000> <0x62538bc0,0x25c2000> <0x6383bc00,0x25c3000>
  <0x63213680,0x25c4000> <0x61cde2c0,0x25c5000> <0x61e29d00,0x25c6000>
  <0x613ad340,0x25c7000> <0x61a78540,0x25c8000> <0x61543f80,0x25c9000>
  <0x60ccf9c0,0x25ca000> <0x61a77a00,0x25cb000> <0x621be800,0x25cc000>
  <0x60573c40,0x25cd000> <0x6238f680,0x25ce000> <0x6171bcc0,0x25cf000>
  <0x62418d00,0x25d0000> <0x60838340,0x25d1000> <0x63364180,0x25d2000>
  <0x637eb9c0,0x25d3000> <0x623be500,0x25d4000> <0x60518540,0x25d5000>
  <0x629d2b80,0x25d6000> <0x611e8fc0,0x25d7000> <0x61e81fc0,0x25d8000>
  <0x63775000,0x25d9000> <0x61371240,0x25da000> <0x63126e80,0x25db000>
--db_more-- 
  <0x637eb9c0,0x25d3000> <0x623be500,0x25d4000> <0x60518540,0x25d5000>
  <0x629d2b80,0x25d6000> <0x611e8fc0,0x25d7000> <0x61e81fc0,0x25d8000>
  <0x63775000,0x25d9000> <0x61371240,0x25da000> <0x63126e80,0x25db000>
  <0x70e45cd4,0xffffffffffffffff> <0x62538d80,0x25dc000> <0x632921c0,0x25dd000>
  <0x62e8c200,0x25de000> <0x6133e240,0x25df000> <0x63909200,0x25e0000>
  <0x61abfc40,0x25e1000> <0x63287680,0x25e2000> <0x6213a2c0,0x25e3000>
  <0x60cf78c0,0x25e4000> <0x60f35300,0x25e5000> <0x621d2140,0x25e6000>
  <0x61c3bb80,0x25e7000> <0x624569c0,0x25e8000> <0x6339b200,0x25e9000>
  <0x62a66440,0x25ea000> <0x62787080,0x25eb000> <0x61469e40,0x25ec000>
  <0x6055e280,0x25ed000> <0x635a3ac0,0x25ee000> <0x63919900,0x25ef000>
  <0x70e45c94,0xffffffffffffffff>

VNODE flags 4080<VLOCKSWORK,ONWORKLIST>
mp 0x64842000 numoutput 2 size 0x25eff62
data 0x71920c84 usecount 1 writecount 1 holdcnt 2 numoutput 2
tag VT_UFS(1) type VREG(1) mount 0x64842000 typedata 0x0
clean bufs:
dirty bufs:
 bp 0x64c25614
  vp 0x719219e8 lblkno 0xfffffffffffffff4 blkno 0x10211a0 dev 0x1306
  error 0 flags 0x200a0<SCANNED,DELWRI,CACHE>
  bufsize 0x4000 bcount 0x4000 resid 0x0
  data 0x71f10000 saveaddr 0x0 dep 0x7185b060
  iodone 0x0
db>  
--
soda