serious performance regression in .41

[christos%zoulas.com@localhost (Christos Zoulas)]

Hi,

Since my last upgrade (I think the last working has been .24), I've been
getting very bad performance running "sync" and choppy interactive response.
It seems something bad happens with locking when there are too many objects
in the buffer cache, because rebooting makes the problem go away for a while.
Here's the lockstat output of sync (which in my 16GB machine takes ~11sec):

christos

$ lockstat sync
Elapsed time: 10.76 seconds.

-- Adaptive mutex spin

Total%  Count   Time/ms          Lock                       Caller
------ ------- --------- ---------------------- ------------------------------
 65.03  153803    270.17 mntvnode_lock          vfs_vnode_iterator_next+38
 34.90  148799    145.00 vnode_free_list_lock   <all>
 20.77   91575     86.28 vnode_free_list_lock   vrelel+3f0
 14.13   57224     58.72 vnode_free_list_lock   vremfree+2f
  0.06       1      0.26 fffffe841f7a2f00       do_sys_wait+8b
  0.00       4      0.00 uvm_pageqlock          <all>
  0.00       2      0.00 uvm_pageqlock          uvm_fault_internal+95b
  0.00       1      0.00 uvm_pageqlock          uvm_fault_internal+1345
  0.00       1      0.00 uvm_pageqlock          uvm_fault_internal+557

-- Adaptive mutex sleep

Total%  Count   Time/ms          Lock                       Caller
------ ------- --------- ---------------------- ------------------------------
 50.48      12      0.51 vnode_free_list_lock   <all>
 31.64       6      0.32 vnode_free_list_lock   vrelel+3f0
 18.84       6      0.19 vnode_free_list_lock   vremfree+2f
 38.14       9      0.38 mntvnode_lock          vfs_vnode_iterator_next+38
  8.07       2      0.08 fffffe841ef29000       dkstart+156
  3.31       1      0.03 fffffe841ef29080       dkstart+156

-- Spin mutex spin

Total%  Count   Time/ms          Lock                       Caller
------ ------- --------- ---------------------- ------------------------------
 45.80       6      0.03 fffffe841ef49000       <all>
 27.52       2      0.02 fffffe841ef49000       turnstile_block+322
 18.28       4      0.01 fffffe841ef49000       turnstile_lookup+22
 28.53       4      0.02 fffffe841ef298c0       <all>
 26.89       3      0.02 fffffe841ef298c0       sched_idle+215
  1.64       1      0.00 fffffe841ef298c0       sched_idle+112
 17.06       4      0.01 fffffe841ef498c0       turnstile_lookup+22
  3.47       1      0.00 fffffe841ef29ac0       sched_idle+112
  2.68       1      0.00 fffffe841ef29900       kpreempt+cb
  2.45       2      0.00 fffffe841ef49e00       turnstile_lookup+22

-- Kernel lock spin

Total%  Count   Time/ms          Lock                       Caller
------ ------- --------- ---------------------- ------------------------------
100.00     228      6.16 kernel_lock            <all>
 45.10     144      2.78 kernel_lock            sleepq_block+1c7
 37.73      39      2.33 kernel_lock            bdev_strategy+53
  9.18      22      0.57 kernel_lock            intr_biglock_wrapper+12
  7.99      23      0.49 kernel_lock            biodone2+6f

-- Kernel preemption defer

Total%  Count   Time/ms          Lock                       Caller
------ ------- --------- ---------------------- ------------------------------
100.00      16      0.07 cpu_kpreempt_enter_fail kpreempt_enable+43
TIME=0:10.77 CPU=99.4% (0.002u 10.714s) SWAPS=0 (0+672)pf (0i+71o) (0Kc+0Kd)