GigE Performance in Dom0

["Sam Fourman Jr." <sfourman%gmail.com@localhost>]

Hello List,

Inspired by this thread,
http://forums.citrix.com/thread.jspa?threadID=254104&start=0&tstart=0
I set out to reproduce this Xentools33 / no xentools33 problem in NetBSD HEAD

To my surprise, there isn't much of a disk I/O hit, but it is there...
but, the network i/o hit is large.

I have 2  8 core Opteron machines running the same Dom0 kernel

# uname -a
NetBSD  5.99.24 NetBSD 5.99.24 (XEN_DOM0_TEST) #7: Wed Mar 10 04:21:24
CST 2010  root@:/usr/objdir/sys/arch/amd64/compile/XEN_DOM0_TEST amd64


First machine does NOT have xentools33 installed the other does have xentools33.


***************************************
without xentools33 installed
***************************************


# bonnie -s 768
File './Bonnie.23441', size: 805306368
Writing with putc()...done
Rewriting...done
Writing intelligently...done
Reading with getc()...done
Reading intelligently...done
Seeker 1...Seeker 2...Seeker 3...start 'em...done...done...done...
              -------Sequential Output-------- ---Sequential Input-- --Random--
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU
          768 64196 34.2 64172 13.6 63994 18.8 135711 99.7 719943
100.0 19218.1 56.6
#
# iperf -s 192.168.12.117 -m
iperf: ignoring extra argument -- 192.168.12.117
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 32.0 KByte (default)
------------------------------------------------------------
[  4] local 192.168.12.160 port 5001 connected with 192.168.12.117 port 61379
[  4]  0.0-60.1 sec  6.50 GBytes    928 Mbits/sec
[  4] MSS size 1460 bytes (MTU 1500 bytes, ethernet)


# ifconfig bge0
bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        
capabilities=3f00<IP4CSUM_Rx,IP4CSUM_Tx,TCP4CSUM_Rx,TCP4CSUM_Tx,UDP4CSUM_Rx,UDP4CSUM_Tx>
        enabled=0
        address: 00:10:18:01:c4:eb
        media: Ethernet autoselect (1000baseT full-duplex,master)
        status: active
        inet 192.168.12.160 netmask 0xffffff00 broadcast 192.168.12.255
        inet6 fe80::210:18ff:fe01:c4eb%bge0 prefixlen 64 scopeid 0x3


***************************************
with xentools33 installed
***************************************

# bonnie -s 768
File './Bonnie.14718', size: 805306368
Writing with putc()...done
Rewriting...done
Writing intelligently...done
Reading with getc()...done
Reading intelligently...done
Seeker 1...Seeker 3...Seeker 2...start 'em...done...done...done...
              -------Sequential Output-------- ---Sequential Input-- --Random--
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU
          768 57334 30.3 52458 11.4 57833 17.7 131898 99.0 640324 99.0
10389.7 31.9
#


# iperf -s 192.168.12.66 -m
iperf: ignoring extra argument -- 192.168.12.66
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 32.0 KByte (default)
------------------------------------------------------------
[  4] local 192.168.12.66 port 5001 connected with 192.168.12.117 port 15343
[  4]  0.0-60.0 sec  1.59 GBytes    228 Mbits/sec
[  4] MSS size 1460 bytes (MTU 1500 bytes, ethernet)


wm0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        
capabilities=7ff80<TSO4,IP4CSUM_Rx,IP4CSUM_Tx,TCP4CSUM_Rx,TCP4CSUM_Tx,UDP4CSUM_Rx,UDP4CSUM_Tx,TCP6CSUM_Rx,TCP6CSUM_Tx,UDP6CSUM_Rx,UDP6CSUM_Tx,TSO6>
        enabled=0
        address: 00:1b:21:50:e7:bf
        media: Ethernet autoselect (1000baseT full-duplex)
        status: active
        inet 192.168.12.66 netmask 0xffffff00 broadcast 192.168.12.255
        inet6 fe80::21b:21ff:fe50:e7bf%wm0 prefixlen 64 scopeid 0x3


random tests I did are here
http://www.puffybsd.com/iperf.txt

is this normal behavior for Xentools33?
Feedback welcome, I will run more tests if asked

Sam Fourman Jr.
Fourman Networks