The speed with glusterfs: seems to be a combination of 74 MB/s and pauses with ttcp: seems to be 112 MB/s burst (for 0.2s) and some smaller pauses, which gut checks with 95 MB/s as reported by ttcp. So are you seeing high 40s MB/s out of glusterfs? What is between these two devices? Is this just a gigabit switch, or anything more complicated? We are seeing reordering which I would not expect on an ethernet. I wonder if the tso4 option is causing that. What if you turn off the offload options? (I realize it may slow down, but if both are then equal, that's a clue.) I wonder if the very large buffers get full and that causes cache thrashing. What happens if you change gluster to have smaller buffers (I don't understand why it's ok to have the FS change the tcp socket buffer options from system default)? Grab http://www.ir.bbn.com/~gdt/netbsd/gluster-xplot.tgz to look at the plots yourself (you'll need xplot, but the conversion From tcdpump is done already).
Attachment:
pgp5bX2Ekzz3I.pgp
Description: PGP signature