Re: raw/block device disc troughput

[Eduardo Horvath <eeh%NetBSD.org@localhost>]

On Thu, 24 May 2012, Edgar Fuß wrote:

> > Keep in mind mpt uese a rather inefficient communication protocol and does
> > tagged queuing.
> You mean the protocol the main CPU uses to communicate with an MPT adapter is
> inefficient? Or do you mean SAS is inefficient?

The protocol used to communicate between the CPU and the adapter is 
inefficient.  Not well designed.  They redesigned it for SAS2.

> > The former means the overhead for each command is not so good, but the
> > latter means it can keep lots of commands in the air at the same time. 
> I'm sorry, I'm unable to conclude why this explains my results.

dd will send the kernel individual write operations.  sd and physio() will 
break them up into MAXPHYS chunks.  Each chunk will be queued at the 
HBA.  The HBA will dispatch them all as fast as it can.  Tagged queuing 
will overlap them.  

With smaller transfers, the setup overhead becomes significant and you see 
poor performance.

With large transfers (larger than MAXPHYS) the writes are split up into 
MAXPHYS chunks and the disk handles them in parallel, hence the 
performance increase even beyond MAXPHYS.

Also keep in mind:

When using the block device the data is copied from the process buffer 
into the buffer cache and the I/O happens from the buffer cache pages.

When using the raw device the I/O happens directly from process memory, no 
copying involved.

Eduardo