Subject: Re: LFS(?) ioflush/vnlock issues
To: Bill Stouder-Studenmund <wrstuden@netbsd.org>
From: Anil Gopinath <anil_public@yahoo.com>
List: tech-kern
Date: 03/09/2007 09:43:23
We are running into a similar issue with LFS where
ioflush is taking up a lot of cpu. I obtained the
following profiled information when this was
happening. Is there a  scope for optimization?

Note: the disk write cache is turned off

index % time    self  children    called     name
                                                
<spontaneous>
[1]     64.5   19.11    0.00                 Xspllower
[1]
-----------------------------------------------
                                                
<spontaneous>
[2]     28.3    0.02    8.37                
lfs_putpages [2]
                2.47    4.43 6893704/6893704    
check_dirty [3]
                0.01    1.11    2156/2273       
lfs_writeseg [15]
                0.06    0.18 6893704/6914521    
preempt [28]
                0.01    0.08    2156/2156       
genfs_putpages [43]
                0.00    0.02    2156/2273       
lfs_updatemeta [105]
                0.00    0.00    2156/2273       
lfs_release_finfo [665]
                0.00    0.00    2156/2273       
lfs_acquire_finfo [662]
-----------------------------------------------
                2.47    4.43 6893704/6893704    
lfs_putpages [2]
[3]     23.3    2.47    4.43 6893704        
check_dirty [3]
                1.52    0.14 74835280/75136474    
pmap_clear_attrs [12]
                1.39    0.00 81726828/81849018    
uvm_pagelookup [13]
                1.37    0.01 74835280/74835674    
pmap_page_remove [14]
-----------------------------------------------
[4]      7.8    0.02    2.29   10181+30552   <cycle 2
as a whole> [4]
                0.01    2.25    5092            
cgdstart <cycle 2> [5]
                0.00    0.03   10184            
spec_strategy <cycle 2> [88]
                0.01    0.00   10184            
VOP_STRATEGY <cycle 2> [171]
                0.00    0.00   10181            
dk_start <cycle 2> [569]
-----------------------------------------------
                                5092            
dk_start <cycle 2> [569]
[5]      7.6    0.01    2.25    5092         cgdstart
<cycle 2> [5]
                0.00    2.25    5092/5092       
cgd_cipher [8]
                0.00    0.00    2930/8764       
malloc [196]
                0.00    0.00    5092/10182      
iostat_busy [568]
                0.00    0.00    5092/10182      
disk_busy [567]
                0.00    0.00    5092/16110      
getiobuf1 [556]
                0.00    0.00    5092/5092       
getiobuf_nowait [612]
                0.00    0.00    2156/28410      
getmicrouptime [524]
                                5092            
VOP_STRATEGY <cycle 2> [171]
-----------------------------------------------
                0.00    2.25  260578/260578     
cgd_cipher [8]
[6]      7.6    0.00    2.25  260578        
cgd_cipher_aes_cbc [6]
                0.01    2.24  260578/260578     
cgd_cipher_uio_cbc [7]
-----------------------------------------------
                0.01    2.24  260578/260578     
cgd_cipher_aes_cbc [6]
[7]      7.6    0.01    2.24  260578        
cgd_cipher_uio_cbc [7]
                0.01    2.23  521156/521156     
aes_cbc_enc_int [9]
-----------------------------------------------
                0.00    2.25    5092/5092       
cgdstart <cycle 2> [5]
[8]      7.6    0.00    2.25    5092        
cgd_cipher [8]
                0.00    2.25  260578/260578     
cgd_cipher_aes_cbc [6]
                0.00    0.00  265670/407642     
memset [454]
-----------------------------------------------


thanks,
Anil

--- Bill Stouder-Studenmund <wrstuden@netbsd.org>
wrote:

> On Thu, Mar 08, 2007 at 08:34:21PM +1100, Paul Ripke
> wrote:
> > On Thu, Mar 01, 2007 at 08:19:49PM +0100, Edgar
> Fu? wrote:
> > > I've no idea whether this problem is actually
> LFS related or
> > > I was just hitting a coincidence.
> > > 
> > > We have performance problems on our mySQL
> server. As analysis
> > > showed that disk I/O, in particular seeks, were
> the problem,
> > > I thought it might be worth to try LFS. So we
> dumped the
> > > database, set up an LFS partition on a
> development server and
> > > fed the dump into mysql. First, everything went
> as expected
> > > (mysqld being CPU bound). Then, as most of the
> database seems
> > > to have been written, we observed ioflush
> consuming 100% CPU
> > > load. From then on, any attempt to access the
> LFS partition
> > > (ls, du) locked up in vnlock state.
> > 
> > IMHO: LFS really isn't suitable for databases -
> every random
> > write operation can fragment the file on disk. Do
> a few
> > thousand random writes and the database files will
> be very
> > fragmented.
> > 
> > While it'll still all work, I'm sure this can't be
> efficient.
> 
> Yes. FFS and LFS each have different "bad"
> workloads. For LFS, one is
> overwriting blocks in files. That's exactly what a
> database does a lot. 
> :-(
> 
> Take care,
> 
> Bill
> 



 
____________________________________________________________________________________
Don't pick lemons.
See all the new 2007 cars at Yahoo! Autos.
http://autos.yahoo.com/new_cars.html