tech-net archive

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]

sluggish network performance in *one* domU



Hello,

I'm experiencing a rather strange network problem within a Xen guest
system.  The problem is bad network performance, it affects all tcp
connections and only appears in one domU.

The hardware is running Xen 3.1.2, NetBSD 4.0 dom0 on i386.  There are 3
domUs running: two nearly identical NetBSD copies and one Debian system.

The `problematic' domU is one of those two NetBSD domUs.  I checked lots
of stuff: the cpu is idle all the time, there is no disk or network I/O
and I couldn't find anything in the logs.

The layout is simple.  Here is a clumsy drawing:

   Linux Laptop
+------------------+
|.--------------.  |
|| host-eth     |  |                   AMD Athlon System
||(host.foo.net)|  |               +-----------------------+
|`-----.--------'  |               |.-------.              |
+------|-----------+           ,---.|dom0-re|              |
       |         ,-----.      /    |`---+---'              |
        \      ,'       `.    |    |    \                  |
         `.   /           \   /    |     |                 |
           `-(   Internet  )-'     |.----+----.            |
              \           /        ||dom0-xvif|            |
               `.       ,'         |`----`.---'   Virtual  |
                 '-----'           |       \      Machine  |
                                   |   +----`.------------+|
                                   |   |      \           ||
                                   |   | .-----`.-------. ||
                                   |   | | domU-xennet  | ||
                                   |   | |(domU.bar.net)| ||
                                   |   | `--------------' ||
                                   |   +------------------+|
                                   +-----------------------+

Based on that I tcpdumped and tcptraced the following ssh session from
my Linux Laptop at home (host.foo.net):

        ssh domU.bar.com dd if=/dev/urandom bs=1m count=5 >/dev/null

from four different interfaces:

        eth    @ host.foo.net
        re     @ dom0
        xvif   @ dom0
        xennet @ domU.bar.net

And it seems that I'm losing lots of data, since xennet, xvif and re
are all reporting the following:

        unique bytes sent:   5254577
        actual data pkts:       9346
        actual data bytes:   5512613
        rexmt data pkts:         369
        rexmt data bytes:     258036

(the complete tcptraces are attached)

The xennet interface in domU owns a public IPv4 address which gets
forwarded and filtered by dom0 (+ipfilter).

Now, it would be easy to blame the ISP but it seems strange that the
other domUs are not affected at all.  I tested the second NetBSD domU
and there were not one single `rexmt'.

Is there maybe something else I could try?

Thanks,



   Petar Bogdanovic



        host a:        host.foo.net:45866
        host b:        domU.bar.net:22
        complete conn: yes
        elapsed time:  0:02:14.672303
        total packets: 15436
   a->b:                              b->a:
     total packets:          6450           total packets:          8986
     ack pkts sent:          6449           ack pkts sent:          8986
     pure acks sent:         6394           pure acks sent:            5
     sack pkts sent:         1664           sack pkts sent:            2
     dsack pkts sent:           1           dsack pkts sent:           2
     max sack blks/ack:         3           max sack blks/ack:         1
     unique bytes sent:      4981           unique bytes sent:   5254577
     actual data pkts:         54           actual data pkts:       8979
     actual data bytes:      5821           actual data bytes:   5254701
     rexmt data pkts:           2           rexmt data pkts:           1
     rexmt data bytes:        840           rexmt data bytes:        124
     zwnd probe pkts:           0           zwnd probe pkts:           0
     zwnd probe bytes:          0           zwnd probe bytes:          0
     outoforder pkts:           0           outoforder pkts:         342
     pushed data pkts:         54           pushed data pkts:         14
     SYN/FIN pkts sent:       1/1           SYN/FIN pkts sent:       1/1
     req 1323 ws/ts:          Y/Y           req 1323 ws/ts:          Y/Y
     adv wind scale:            6           adv wind scale:            0
     req sack:                  Y           req sack:                  Y
     sacks sent:             1664           sacks sent:                2
     urgent data pkts:          0 pkts      urgent data pkts:          0 pkts
     urgent data bytes:         0 bytes     urgent data bytes:         0 bytes
     mss requested:          1460 bytes     mss requested:          1460 bytes
     max segm size:          1152 bytes     max segm size:          1448 bytes
     min segm size:            16 bytes     min segm size:            32 bytes
     avg segm size:           107 bytes     avg segm size:           585 bytes
     max win adv:           92672 bytes     max win adv:           33580 bytes
     min win adv:            5888 bytes     min win adv:           33532 bytes
     zero win adv:              0 times     zero win adv:              0 times
     avg win adv:           91439 bytes     avg win adv:           33579 bytes
     initial window:           21 bytes     initial window:           49 bytes
     initial window:            1 pkts      initial window:            1 pkts
     ttl stream length:      4981 bytes     ttl stream length:   5254577 bytes
     missed data:               0 bytes     missed data:               0 bytes
     truncated data:            0 bytes     truncated data:            0 bytes
     truncated packets:         0 pkts      truncated packets:         0 pkts
     data xmit time:      134.555 secs      data xmit time:      134.553 secs
     idletime max:         7532.9 ms        idletime max:         7532.9 ms
     throughput:               37 Bps       throughput:            39018 Bps
        host a:        host.foo.net:53062
        host b:        domU.bar.net:22
        complete conn: yes
        elapsed time:  0:02:14.603709
        total packets: 13656
   a->b:                              b->a:
     total packets:          4303           total packets:          9353
     ack pkts sent:          4302           ack pkts sent:          9353
     pure acks sent:         4247           pure acks sent:            5
     sack pkts sent:         1658           sack pkts sent:            2
     dsack pkts sent:           1           dsack pkts sent:           2
     max sack blks/ack:         3           max sack blks/ack:         1
     unique bytes sent:      4981           unique bytes sent:   5254577
     actual data pkts:         54           actual data pkts:       9346
     actual data bytes:      5821           actual data bytes:   5512613
     rexmt data pkts:           2           rexmt data pkts:         369
     rexmt data bytes:        840           rexmt data bytes:     258036
     zwnd probe pkts:           0           zwnd probe pkts:           0
     zwnd probe bytes:          0           zwnd probe bytes:          0
     outoforder pkts:           0           outoforder pkts:           0
     pushed data pkts:         54           pushed data pkts:         14
     SYN/FIN pkts sent:       1/1           SYN/FIN pkts sent:       1/1
     req 1323 ws/ts:          Y/Y           req 1323 ws/ts:          Y/Y
     adv wind scale:            6           adv wind scale:            0
     req sack:                  Y           req sack:                  Y
     sacks sent:             1658           sacks sent:                2
     urgent data pkts:          0 pkts      urgent data pkts:          0 pkts
     urgent data bytes:         0 bytes     urgent data bytes:         0 bytes
     mss requested:          1460 bytes     mss requested:          1460 bytes
     max segm size:          1152 bytes     max segm size:          1448 bytes
     min segm size:            16 bytes     min segm size:            32 bytes
     avg segm size:           107 bytes     avg segm size:           589 bytes
     max win adv:           92672 bytes     max win adv:           33580 bytes
     min win adv:            5888 bytes     min win adv:           33532 bytes
     zero win adv:              0 times     zero win adv:              0 times
     avg win adv:           91369 bytes     avg win adv:           33579 bytes
     initial window:           21 bytes     initial window:           49 bytes
     initial window:            1 pkts      initial window:            1 pkts
     ttl stream length:      4981 bytes     ttl stream length:   5254577 bytes
     missed data:               0 bytes     missed data:               0 bytes
     truncated data:            0 bytes     truncated data:            0 bytes
     truncated packets:         0 pkts      truncated packets:         0 pkts
     data xmit time:      134.497 secs      data xmit time:      134.487 secs
     idletime max:         7520.3 ms        idletime max:         3997.8 ms
     throughput:               37 Bps       throughput:            39037 Bps
        host a:        host.foo.net:53062
        host b:        domU.bar.net:22
        complete conn: yes
        elapsed time:  0:02:14.671055
        total packets: 13656
   a->b:                              b->a:
     total packets:          4303           total packets:          9353
     ack pkts sent:          4302           ack pkts sent:          9353
     pure acks sent:         4247           pure acks sent:            5
     sack pkts sent:         1658           sack pkts sent:            2
     dsack pkts sent:           1           dsack pkts sent:           2
     max sack blks/ack:         3           max sack blks/ack:         1
     unique bytes sent:      4981           unique bytes sent:   5254577
     actual data pkts:         54           actual data pkts:       9346
     actual data bytes:      5821           actual data bytes:   5512613
     rexmt data pkts:           2           rexmt data pkts:         369
     rexmt data bytes:        840           rexmt data bytes:     258036
     zwnd probe pkts:           0           zwnd probe pkts:           0
     zwnd probe bytes:          0           zwnd probe bytes:          0
     outoforder pkts:           0           outoforder pkts:           0
     pushed data pkts:         54           pushed data pkts:         14
     SYN/FIN pkts sent:       1/1           SYN/FIN pkts sent:       1/1
     req 1323 ws/ts:          Y/Y           req 1323 ws/ts:          Y/Y
     adv wind scale:            6           adv wind scale:            0
     req sack:                  Y           req sack:                  Y
     sacks sent:             1658           sacks sent:                2
     urgent data pkts:          0 pkts      urgent data pkts:          0 pkts
     urgent data bytes:         0 bytes     urgent data bytes:         0 bytes
     mss requested:          1460 bytes     mss requested:          1460 bytes
     max segm size:          1152 bytes     max segm size:          1448 bytes
     min segm size:            16 bytes     min segm size:            32 bytes
     avg segm size:           107 bytes     avg segm size:           589 bytes
     max win adv:           92672 bytes     max win adv:           33580 bytes
     min win adv:            5888 bytes     min win adv:           33532 bytes
     zero win adv:              0 times     zero win adv:              0 times
     avg win adv:           91369 bytes     avg win adv:           33579 bytes
     initial window:           21 bytes     initial window:           49 bytes
     initial window:            1 pkts      initial window:            1 pkts
     ttl stream length:      4981 bytes     ttl stream length:   5254577 bytes
     missed data:               0 bytes     missed data:               0 bytes
     truncated data:            0 bytes     truncated data:            0 bytes
     truncated packets:         0 pkts      truncated packets:         0 pkts
     data xmit time:      134.564 secs      data xmit time:      134.554 secs
     idletime max:         7524.1 ms        idletime max:         3999.8 ms
     throughput:               37 Bps       throughput:            39018 Bps
        host a:        host.foo.net:53062
        host b:        domU.bar.net:22
        complete conn: yes
        elapsed time:  0:02:14.671068
        total packets: 13656
   a->b:                              b->a:
     total packets:          4303           total packets:          9353
     ack pkts sent:          4302           ack pkts sent:          9353
     pure acks sent:         4247           pure acks sent:            5
     sack pkts sent:         1658           sack pkts sent:            2
     dsack pkts sent:           1           dsack pkts sent:           2
     max sack blks/ack:         3           max sack blks/ack:         1
     unique bytes sent:      4981           unique bytes sent:   5254577
     actual data pkts:         54           actual data pkts:       9346
     actual data bytes:      5821           actual data bytes:   5512613
     rexmt data pkts:           2           rexmt data pkts:         369
     rexmt data bytes:        840           rexmt data bytes:     258036
     zwnd probe pkts:           0           zwnd probe pkts:           0
     zwnd probe bytes:          0           zwnd probe bytes:          0
     outoforder pkts:           0           outoforder pkts:           0
     pushed data pkts:         54           pushed data pkts:         14
     SYN/FIN pkts sent:       1/1           SYN/FIN pkts sent:       1/1
     req 1323 ws/ts:          Y/Y           req 1323 ws/ts:          Y/Y
     adv wind scale:            6           adv wind scale:            0
     req sack:                  Y           req sack:                  Y
     sacks sent:             1658           sacks sent:                2
     urgent data pkts:          0 pkts      urgent data pkts:          0 pkts
     urgent data bytes:         0 bytes     urgent data bytes:         0 bytes
     mss requested:          1460 bytes     mss requested:          1460 bytes
     max segm size:          1152 bytes     max segm size:          1448 bytes
     min segm size:            16 bytes     min segm size:            32 bytes
     avg segm size:           107 bytes     avg segm size:           589 bytes
     max win adv:           92672 bytes     max win adv:           33580 bytes
     min win adv:            5888 bytes     min win adv:           33532 bytes
     zero win adv:              0 times     zero win adv:              0 times
     avg win adv:           91369 bytes     avg win adv:           33579 bytes
     initial window:           21 bytes     initial window:           49 bytes
     initial window:            1 pkts      initial window:            1 pkts
     ttl stream length:      4981 bytes     ttl stream length:   5254577 bytes
     missed data:               0 bytes     missed data:               0 bytes
     truncated data:            0 bytes     truncated data:            0 bytes
     truncated packets:         0 pkts      truncated packets:         0 pkts
     data xmit time:      134.564 secs      data xmit time:      134.554 secs
     idletime max:         7524.1 ms        idletime max:         3999.8 ms
     throughput:               37 Bps       throughput:            39018 Bps


Home | Main Index | Thread Index | Old Index