Re: Poor disk performance in NetBSD DomU, again

[Yi Qiao <qiaoy%qiaoy.info@localhost>]

Hi Mike

On Wed, May 14, 2014 at 10:05 PM, Michael L. Hitch
<mhitch%lightning.msu.montana.edu@localhost> wrote:
> On Wed, 14 May 2014, Yi Qiao wrote:
>
>>  Thanks for the advise. I tried various tests (Dom0, lvm backed DomU,
>> physical partition backed DomU), and the results are attached at the
>> end of the message. In conclusion:
>>
>>  * There was no read activity during a write operation in DomU.
>>  * physical partition (wd0e) backed DomU suffers from slow write too.
>
>
>   This one is rather interesting.
>
>
>>  * dd to a block lvm device is slower than dd to a raw lvm device
>
>
>   This is to be expected.  Writing to a block device copies the data into a
> kernel buffer, and then writes the buffer to the disk device.  If the block
> device is not an FFS partition, the buffer size will be 2K.  So each write
> of 64K bytes is broken up into 32 separate operations, which can be much
> slower.
>
>
>>  * In DomU, whenever the write MB/s is low, the w/s is also low
>
>
>> Dom0, write to lvm volume directly (block device)
>> -------------------------------------------------
>> CMD: dd if=/dev/zero of=/dev/vg0/testlv bs=32k count=131072
>> device  read KB/t    r/s   time     MB/s write KB/t    w/s   time     MB/s
>> vg0-test     0.00      0   0.95     0.00       2.00  20142   0.95    39.34
>> vg0-test     0.00      0   0.97     0.00       2.00  16881   0.97    32.97
>
>
>   Note that the KB/transfer is 2KB here, representing the buffer size used
> by block devices.  Also, the time is quite close to 1, which I think
> indicates disk is nearly 100% busy.  That could show that the writes are
> backing up because the disk can't process them fast enough.  Hmm, if the
> disk has internal 4K sectors, writing 2K would require the disk to read a 4K
> block, update the 2K data, and write the 4K back.

I see!


>
>
>> DomU, write to root (backed by physical partition wd0e)
>> -------------------------------------------------------
>> CMD: dd if=/dev/zero of=testfile bs=32k count=131072
>> ### iostat in Dom0, checking wd0
>> device  read KB/t    r/s   time     MB/s write KB/t    w/s   time     MB/s
>> wd0         16.00      0   0.94     0.01      32.01    327   0.94    10.23
>> wd0         16.00      0   1.00     0.01      31.98    532   1.00    16.62
>> wd0         16.00      0   0.93     0.01      32.19    407   0.93    12.78
>
>
>   This is also showing the disk as nearly 100% busy.  What is the starting
> sector of the wd0e partition?  Does it start on a 4K boundary?

I think it does. The offset of wd0e is 32510016, and 32510016 * 512
mod 4096 = 0.


>
>   I'm still not sure how to explain why using the lvm volume is slow though.


I don't know how much is this relevant, but I performed the speed test
again on amazon aws, launching "NetBSD-x86_64-6.1.4-20140417-1511" on
a m3.medium (3.75G men, 4G SSD) instance, and the write speed is also
slow compared to read

ip-10-221-18-190# dd if=/dev/zero of=testfile bs=32k count=65536 &
ip-10-221-18-190# iostat -d -x xbd1 5
device  read KB/t    r/s   time     MB/s write KB/t    w/s   time     MB/s
xbd1         5.77      1   0.02     0.01      31.31     12   0.02     0.36
xbd1         0.00      0   1.00     0.00      32.00   1017   1.00    31.79
xbd1         0.00      0   1.00     0.00      31.34    965   1.00    29.54
xbd1         0.00      0   1.00     0.00      32.00   1017   1.00    31.77
xbd1         0.00      0   1.00     0.00      31.74    957   1.00    29.65
^C
ip-10-221-18-190#

65536+0 records in
65536+0 records out
2147483648 bytes transferred in 88.709 secs (24208182 bytes/sec)

ip-10-221-18-190# dd if=testfile of=/dev/null bs=32k
65536+0 records in
65536+0 records out
2147483648 bytes transferred in 3.740 secs (574193488 bytes/sec)

>
>
> Mike
>
> ---
> Michael L. Hitch                        mhitch%montana.edu@localhost
> Operations Consulting,  Information Technology Center
> Montana State University, Bozeman, MT     USA