Re: XEN 3 DomU loses network and dies

To: port-xen%NetBSD.org@localhost
Subject: Re: XEN 3 DomU loses network and dies
From: Damian Lubosch <dl%xiqit.de@localhost>
Date: Thu, 24 Jan 2008 08:51:48 +0100
Hi Manuel!

Manuel Bouyer schrieb:
>
>> Because of the balloon driver I wondered because the amount of used
>> memory (in xm top) varies a little bit around the maximum.
>>     
>
> That's expected, it's the pages that are transfered between domains for I/O.
> It shouldn't change much.
>
>   
No, it does not change much, just around 0.5 MB or so.

>> When I set up the system  (dom0 + 5 domUs) xm info / top reported about
>> 50 MB free RAM for the hypervisor, and after the problems began I
>> switched off one domU to get about 108 MB of free RAM but the problems
>> still persists.
>>     
>
> 50MB may be a bit short, especially if you have lots of domUs. I would
> expect 100MB to be enough, but I've always had more than 200MB free on my
> Xen servers.
>
>   
On Xen2 I always had about 50 MB unallocated RAM in the hypervisor, and
there weren't problems like that.
The problem persisted even when I shut down one domU to having >100MB
free RAM.

>> I recompiled the dom0 kernel to having 64 vnd devices and the default of
>> xbd devices. In my /dev I generated 16 of both.
>>
>>     
>   
@Curt: Yes, if you start using many domUs with many "hard"-drives you
hit the problem of having only 4 (or 8?) vnd devices available. You
"just" need to recompile the XEN dom0 kernel, search for vnd and to
write e.g. 64 next to it.

> Next time could you look at the output of ifconfig, for the interfaces
> in the domU, and the corresponding xvif interfaces in the dom0 ?
>
>   

As it happened again, I caught the ifconfig messages:
(The external IP is fake)

After crash:

lo0: flags=8009<UP,LOOPBACK,MULTICAST> mtu 33192
        inet 127.0.0.1 netmask 0xff000000
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
xennet0: flags=8863<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST> mtu 
1500
        address: 00:16:3e:50:03:f7
        inet 10.0.1.5 netmask 0xffffff00 broadcast 10.0.1.255
        inet alias 12.23.34.45 netmask 0xfffffff8 broadcast 12.23.34.36
        inet6 fe80::216:3eff:fe50:3f7%xennet0 prefixlen 64 scopeid 0x2



# ifconfig -a
fxp0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
        address: 00:02:b3:1f:83:bf
        media: Ethernet autoselect (100baseTX 
full-duplex,flowcontrol,rxpause,txpause)
        status: active
        inet 10.0.1.6 netmask 0xffffff00 broadcast 10.0.1.255
        inet6 fe80::202:b3ff:fe1f:83bf%fxp0 prefixlen 64 scopeid 0x1
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 33192
        inet 127.0.0.1 netmask 0xff000000
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
bridge0: flags=41<UP,RUNNING> mtu 1500
xvif2.0: flags=8963<UP,BROADCAST,NOTRAILERS,RUNNING,PROMISC,SIMPLEX,MULTICAST> 
mtu 1500
        capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
        enabled=0
        address: 00:16:3e:51:03:f9
        inet6 fe80::216:3eff:fe51:3f9%xvif2.0 prefixlen 64 scopeid 0x6
xvif3.0: flags=8963<UP,BROADCAST,NOTRAILERS,RUNNING,PROMISC,SIMPLEX,MULTICAST> 
mtu 1500
        capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
        enabled=0
        address: 00:16:3e:51:03:f4
        inet6 fe80::216:3eff:fe51:3f4%xvif3.0 prefixlen 64 scopeid 0x7
xvif5.0: flags=8963<UP,BROADCAST,NOTRAILERS,RUNNING,PROMISC,SIMPLEX,MULTICAST> 
mtu 1500
        capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
        enabled=0
        address: 00:16:3e:51:03:f6
        inet6 fe80::216:3eff:fe51:3f6%xvif5.0 prefixlen 64 scopeid 0x9
xvif24.0: 
flags=8963<UP,BROADCAST,NOTRAILERS,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 
1500
        capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
        enabled=0
        address: 00:16:3e:51:03:a8
        inet6 fe80::216:3eff:fe51:3a8%xvif24.0 prefixlen 64 scopeid 0xc
xvif24.1: 
flags=8963<UP,BROADCAST,NOTRAILERS,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 
1500
        capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
        enabled=0
        address: 00:16:3e:20:81:76
        inet6 fe80::216:3eff:fe20:8176%xvif24.1 prefixlen 64 scopeid 0xd
xvif33.0: 
flags=8963<UP,BROADCAST,NOTRAILERS,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 
1500
        capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
        enabled=0
        address: 00:16:3e:51:03:f7
        inet6 fe80::216:3eff:fe51:3f7%xvif33.0 prefixlen 64 scopeid 0xe



And when the domU is working, ifconfig -a shows (I changed the external
IP here):
lo0: flags=8009<UP,LOOPBACK,MULTICAST> mtu 33192
        inet 127.0.0.1 netmask 0xff000000
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
xennet0: flags=8863<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST>
mtu 1500
        address: 00:16:3e:50:03:f7
        inet 10.0.1.5 netmask 0xffffff00 broadcast 10.0.1.255
        inet alias 12.23.34.45 netmask 0xfffffff8 broadcast 12.23.34.36
        inet6 fe80::216:3eff:fe50:3f7%xennet0 prefixlen 64 scopeid 0x2
 
 
the dom0 shows: (The domU uses xvif33 at that time)
 
fxp0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
        address: 00:02:b3:1f:83:bf
        media: Ethernet autoselect (100baseTX
full-duplex,flowcontrol,rxpause,tx                                              
                                              
pause)
        status: active
        inet 10.0.1.6 netmask 0xffffff00 broadcast 10.0.1.255
        inet6 fe80::202:b3ff:fe1f:83bf%fxp0 prefixlen 64 scopeid 0x1
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 33192
        inet 127.0.0.1 netmask 0xff000000
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
bridge0: flags=41<UP,RUNNING> mtu 1500
xvif2.0:
flags=8963<UP,BROADCAST,NOTRAILERS,RUNNING,PROMISC,SIMPLEX,MULTICAST>
m                                                                               
             
tu 1500
        capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
        enabled=0
        address: 00:16:3e:51:03:f9
        inet6 fe80::216:3eff:fe51:3f9%xvif2.0 prefixlen 64 scopeid 0x6
xvif3.0:
flags=8963<UP,BROADCAST,NOTRAILERS,RUNNING,PROMISC,SIMPLEX,MULTICAST>
m                                                                               
             
tu 1500
        capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
        enabled=0
        address: 00:16:3e:51:03:f4
        inet6 fe80::216:3eff:fe51:3f4%xvif3.0 prefixlen 64 scopeid 0x7
xvif5.0:
flags=8963<UP,BROADCAST,NOTRAILERS,RUNNING,PROMISC,SIMPLEX,MULTICAST>
m                                                                               
             
tu 1500
        capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
        enabled=0
        address: 00:16:3e:51:03:f6
        inet6 fe80::216:3eff:fe51:3f6%xvif5.0 prefixlen 64 scopeid 0x9
xvif24.0:
flags=8963<UP,BROADCAST,NOTRAILERS,RUNNING,PROMISC,SIMPLEX,MULTICAST>           
                                                                                
  
mtu 1500
        capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
        enabled=0
        address: 00:16:3e:51:03:a8
        inet6 fe80::216:3eff:fe51:3a8%xvif24.0 prefixlen 64 scopeid 0xc
xvif24.1:
flags=8963<UP,BROADCAST,NOTRAILERS,RUNNING,PROMISC,SIMPLEX,MULTICAST>           
                                                                                
  
mtu 1500
        capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
        enabled=0
        address: 00:16:3e:20:81:76
        inet6 fe80::216:3eff:fe20:8176%xvif24.1 prefixlen 64 scopeid 0xd
xvif33.0:
flags=8963<UP,BROADCAST,NOTRAILERS,RUNNING,PROMISC,SIMPLEX,MULTICAST>           
                                                                                
  
mtu 1500
        capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
        enabled=0
        address: 00:16:3e:51:03:f7
        inet6 fe80::216:3eff:fe51:3f7%xvif33.0 prefixlen 64 scopeid 0xe

Sorry for the line-breaks...

As mentioned, xvif 33 belongs to the problematic domU.

I hope it helps somehow.

Bye,
Damian
References:
- XEN 3 DomU loses network and dies
  - From: Damian Lubosch
- Re: XEN 3 DomU loses network and dies
  - From: Manuel Bouyer
- Re: XEN 3 DomU loses network and dies
  - From: Damian Lubosch
- Re: XEN 3 DomU loses network and dies
  - From: Manuel Bouyer
Prev by Date: Re: bouyer-xeni386 merge
Next by Date: Re: Xen 3.2.0 Packages
Previous by Thread: Re: XEN 3, memory and CPUs
Next by Thread: Modular X.org
Indexes:
Home | Main Index | Thread Index | Old Index