Re: Poor disk performance in NetBSD DomU, again

["Michael L. Hitch" <mhitch%lightning.msu.montana.edu@localhost>]

On Wed, 14 May 2014, Yi Qiao wrote:

>  Thanks for the advise. I tried various tests (Dom0, lvm backed DomU,
> physical partition backed DomU), and the results are attached at the
> end of the message. In conclusion:
>
>  * There was no read activity during a write operation in DomU.
>  * physical partition (wd0e) backed DomU suffers from slow write too.

  This one is rather interesting.

>  * dd to a block lvm device is slower than dd to a raw lvm device

  This is to be expected.  Writing to a block device copies the data into 
a kernel buffer, and then writes the buffer to the disk device.  If the 
block device is not an FFS partition, the buffer size will be 2K.  So each 
write of 64K bytes is broken up into 32 separate operations, which can be 
much slower.

>  * In DomU, whenever the write MB/s is low, the w/s is also low

> Dom0, write to lvm volume directly (block device)
> -------------------------------------------------
> CMD: dd if=/dev/zero of=/dev/vg0/testlv bs=32k count=131072
> device  read KB/t    r/s   time     MB/s write KB/t    w/s   time     MB/s
> vg0-test     0.00      0   0.95     0.00       2.00  20142   0.95    39.34
> vg0-test     0.00      0   0.97     0.00       2.00  16881   0.97    32.97

  Note that the KB/transfer is 2KB here, representing the buffer size used 
by block devices.  Also, the time is quite close to 1, which I think 
indicates disk is nearly 100% busy.  That could show that the writes are 
backing up because the disk can't process them fast enough.  Hmm, if the 
disk has internal 4K sectors, writing 2K would require the disk to read a 
4K block, update the 2K data, and write the 4K back.

> DomU, write to root (backed by physical partition wd0e)
> -------------------------------------------------------
> CMD: dd if=/dev/zero of=testfile bs=32k count=131072
> ### iostat in Dom0, checking wd0
> device  read KB/t    r/s   time     MB/s write KB/t    w/s   time     MB/s
> wd0         16.00      0   0.94     0.01      32.01    327   0.94    10.23
> wd0         16.00      0   1.00     0.01      31.98    532   1.00    16.62
> wd0         16.00      0   0.93     0.01      32.19    407   0.93    12.78

  This is also showing the disk as nearly 100% busy.  What is the starting 
sector of the wd0e partition?  Does it start on a 4K boundary?

  I'm still not sure how to explain why using the lvm volume is slow 
though.

Mike

---
Michael L. Hitch                        mhitch%montana.edu@localhost
Operations Consulting,  Information Technology Center
Montana State University, Bozeman, MT     USA