So, I bought another used Dell PE2950, and a disk in my old 2650 died,
which together gave me the impetus to upgrade both my 2950's to run Xen
(since I wanted to install and use Xen on the new one, and the existing
one wasn't running anything important, but I wanted it to take over from
the old 2650, and it would still have lots of left-over capacity that
could be used for more test hosts, and actually I wanted to install Xen
on the older one since I got it when I was working for Citrix!).
I've installed -current/amd64 built from sources updated on 2015/02/19,
and I've installed Xen-4.5 (xentools45 and xenkernel45, built on amd64).
(may I say xentools45 has an insane number of huge dependencies!!!!)
Almost all the kinks and wriggles have been ironed out and I'm typing
this in an emacs running in a NetBSD/amd64 5.2_STABLE domU, my new $HOME
server instance (hopefully soon to be upgraded to 6.x or -current too).
However there's one wee mystery remaining. rwhod on the dom0 systems is
not seeing the rwhod broadcasts from its own domU client(s). It does
see broadcasts from both the domU and dom0 running on the other machine
(as well as of course the other server on the network), and the domU
systems see everything, as expected.
If anyone has any clues as to why this might be happening, or
suggestions as to what other information might be useful beyond what's
provided below, please do let me know! Thanks!
(I'll send my install notes once I've cleaned them up -- there were tons
of confusing and misleading and outright wrong things in the NetBSD Xen
HowTo and hopefully my notes will help provide a better and more
complete guide, at least for anyone targeting a similar environment and
configuration as I have.)
xenful dom0 $ ruptime
building up 22:19, 0 users, load 0.00, 0.00, 0.00
once up 104+11:49, 0 users, load 0.01, 0.02, 0.00
xenful up 7+12:04, 2 users, load 0.02, 0.01, 0.00
xentastic up 1+16:13, 1 user, load 0.00, 0.00, 0.00
more domU $ ruptime
building up 22:22, 0 users, load 0.00, 0.00, 0.00
more up 5+00:27, 0 users, load 0.02, 0.03, 0.00
once up 104+11:52, 0 users, load 0.00, 0.00, 0.00
xenful up 7+12:04, 2 users, load 0.02, 0.01, 0.00
xentastic up 1+16:13, 1 user, load 0.00, 0.00, 0.00
xentastic dom0 $ ruptime
more up 5+00:30, 0 users, load 0.04, 0.04, 0.00
once up 104+11:52, 0 users, load 0.00, 0.00, 0.00
xenful up 7+12:04, 2 users, load 0.02, 0.01, 0.00
xentastic up 1+16:13, 1 user, load 0.00, 0.00, 0.00
building domU $ ruptime
building up 22:22, 0 users, load 0.00, 0.00, 0.00
more up 5+00:30, 0 users, load 0.04, 0.04, 0.00
once up 104+11:52, 0 users, load 0.00, 0.00, 0.00
xenful up 7+12:04, 2 users, load 0.02, 0.01, 0.00
xentastic up 1+16:16, 1 user, load 0.00, 0.00, 0.00
TCP connections and ping work fine from the domU to dom0.
UDP echo also seems to work fine from domU to dom0 (after enabling it in
inetd.conf of course):
more domU # hping -2 -p 7 xenful
HPING xenful (xennet0 10.0.1.139): udp mode set, 28 headers + 0 data bytes
len=28 ip=10.0.1.139 ttl=64 id=0 seq=0 rtt=0.2 ms
len=28 ip=10.0.1.139 ttl=64 id=0 seq=1 rtt=0.1 ms
len=28 ip=10.0.1.139 ttl=64 id=0 seq=2 rtt=0.1 ms
len=28 ip=10.0.1.139 ttl=64 id=0 seq=3 rtt=0.1 ms
len=28 ip=10.0.1.139 ttl=64 id=0 seq=4 rtt=0.1 ms
^?
--- xenful hping statistic ---
5 packets tramitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 0.1/0.1/0.2 ms
more domU # hping -c 2 -2 -e 128 -d 128 -p 7 xenful
HPING xenful (xennet0 10.0.1.139): udp mode set, 28 headers + 128 data bytes
len=156 ip=10.0.1.139 ttl=64 id=8142 seq=0 rtt=0.2 ms
len=156 ip=10.0.1.139 ttl=64 id=8143 seq=1 rtt=0.1 ms
--- xenful hping statistic ---
2 packets tramitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.1/0.2/0.2 ms
xenful dom0 # ps -u -p 11689
USER PID %CPU %MEM VSZ RSS TTY STAT STARTED TIME COMMAND
_rwhod 11689 0.0 0.0 1412 432 ? Ss 5:59PM 0:01.19 /usr/sbin/rwhod -u _rwhod
xenful dom0 # fstat -p 11689
USER CMD PID FD MOUNT INUM MODE SZ|DV R/W
_rwhod rwhod 11689 wd /var 135552 drwxr-xr-x 512 r
_rwhod rwhod 11689 0 / 768140 crw-rw-rw- null rw
_rwhod rwhod 11689 1 / 768140 crw-rw-rw- null rw
_rwhod rwhod 11689 2 / 768140 crw-rw-rw- null rw
_rwhod rwhod 11689 3* internet dgram udp *:login
_rwhod rwhod 11689 4 /var 67781 -rw-rw-r-- 7800 r
Note above that 'fstat' on -current incorrectly resolves the service
name for the "udp" services. It is actually listening on the correct
port, and obviously as seen above it is getting the right packets from
the real network (and localhost) just fine:
xenful dom0 # fstat -n -p 11689
USER CMD PID FD DEV INUM MODE SZ|DV R/W
_rwhod rwhod 11689 wd 4,4 135552 40755 512 r
_rwhod rwhod 11689 0 4,0 768140 20666 2,2 rw
_rwhod rwhod 11689 1 4,0 768140 20666 2,2 rw
_rwhod rwhod 11689 2 4,0 768140 20666 2,2 rw
_rwhod rwhod 11689 3* internet dgram udp *:513
_rwhod rwhod 11689 4 4,4 67781 100664 7800 r
xenful dom0 # fgrep \ 513/ /etc/services
login 513/tcp # priviledged port numbers and
who 513/udp # who's logged in to machines
xenful dom0 $ uname -a
NetBSD xenful 7.99.5 NetBSD 7.99.5 (XEN3_DOM0) #0: Fri Feb 20 18:12:09 PST 2015 woods@more:/build/woods/more/current-amd64-amd64-obj/once/rest/work/woods/m-NetBSD-current/sys/arch/amd64/compile/XEN3_DOM0 amd64
more domU $ uname -a
NetBSD more 5.2_STABLE NetBSD 5.2_STABLE (XEN3_DOMU) #0: Sat Feb 14 19:21:26 PST 2015 woods@more:/build/woods/more/netbsd-5-amd64-amd64-obj/once/rest/work/woods/m-NetBSD-5/sys/arch/amd64/compile/XEN3_DOMU amd64
I'm using a simple bridge config in the dom0:
xenful dom0 $ /sbin/brconfig -a
bridge0: flags=41<UP,RUNNING>
Configuration:
priority 32768 hellotime 2 fwddelay 15 maxage 20
ipfilter disabled flags 0x0
Interfaces:
xvif5i0 flags=3<LEARNING,DISCOVER>
port 5 priority 128
bnx1 flags=3<LEARNING,DISCOVER>
port 2 priority 128
[[ .... chopped .... ]]
xenful dom0 $ /sbin/ifconfig bnx1
bnx1: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX>bnx1: flags=8b43<MULTICAST> mtu 1500
capabilities=3f00<IP4CSUM_Rx,IP4CSUM_Tx,TCP4CSUM_Rx,TCP4CSUM_Tx>
capabilities=3f00<UDP4CSUM_Rx,UDP4CSUM_Tx>
enabled=0
ec_capabilities=7<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU>
ec_enabled=0
address: 00:1d:09:35:3c:09
media: Ethernet autoselect (1000baseT full-duplex)
status: active
inet 10.0.1.139 netmask 0xffffff00 broadcast 10.0.1.255
The domU's are using basic interface configs as well:
more domU $ /sbin/ifconfig xennet0
xennet0: flags=8963<UP,BROADCAST,NOTRAILERS,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
caps_enabled=0
address: 00:16:3e:21:3f:ed
inet 10.0.1.129 netmask 0xffffff00 broadcast 10.0.1.255
The domU config is:
xenful dom0 $ cat /usr/pkg/etc/xen/more.conf
name = "more"
uuid = "7d05d5e2-bf24-11e4-b2aa-00065bede2d1"
kernel = "/netbsd-5.2_STABLE-XEN3_DOMU"
#kernel = "/netbsd-5.2_STABLE-INSTALL_XEN3_DOMU"
extra = "root=xbd0"
#extra = "-s root=xbd0"
memory = 8000
maxmem = 16000
# xxx will only have effect with netbsd-6 or newer
vcpus = 4
vif = [ 'bridge=bridge0' ]
disk = [ 'format=raw, vdev=0x0, access=rw, target=/dev/mapper/vg0-lv0',
'format=raw, vdev=0x1, access=rw, target=/dev/mapper/vg0-lv1',
'format=raw, vdev=0x2, access=rw, target=/dev/mapper/vg0-lv2',
'format=raw, vdev=0x3, access=rw, target=/dev/mapper/vg0-lv3',
'format=raw, vdev=0x4, access=rw, target=/dev/mapper/vg0-lv4',
'format=raw, vdev=0x5, access=rw, target=/dev/mapper/vg0-lv5',
'format=raw, vdev=0x6, access=rw, target=/dev/sd1d'
#, 'format=raw, vdev=0x7, access=ro, devtype=cdrom, target=/build/woods/more/netbsd-5-amd64-release/images/NetBSD-5.2_STABLE-amd64.iso'
]
Both the dom0 and domU are seeing some UDP packets with bad checksums,
and lots of "broadcast/multicast datagrams dropped due to no socket",
but I think that's all "normal" on my network:
xenful dom0 $ netstat -s
icmp:
0 calls to icmp_error
0 errors not generated because old message was icmp
Output histogram:
echoreply: 8
0 messages with bad code fields
0 messages < minimum length
5 bad checksums
0 messages with bad length
29 multicast echo requests ignored
0 multicast timestamp requests ignored
Input histogram:
echoreply: 24
unreach: 4
echo: 37
8 message responses generated
0 path MTU changes
igmp:
1265 messages received
0 messages received with too few bytes
0 messages received with bad checksum
0 membership queries received
0 membership queries received with invalid field(s)
1265 membership reports received
0 membership reports received with invalid field(s)
1265 membership reports received for groups to which we belong
0 membership reports sent
tcp:
121465 packets sent
110584 data packets (97071854 bytes)
7 data packets (844 bytes) retransmitted
8495 ack-only packets (27663 delayed)
0 URG only packets
0 window probe packets
2265 window update packets
114 control packets
0 send attempts resulted in self-quench
120420 packets received
78357 acks (for 97071667 bytes)
225 duplicate acks
0 acks for unsent data
35632 packets (14706392 bytes) received in-sequence
17 completely duplicate packets (76 bytes)
0 old duplicate packets
4 packets with some dup. data (3168 bytes duped)
38 out-of-order packets (0 bytes)
0 packets (0 bytes) of data after window
0 window probes
8552 window update packets
2 packets received after close
0 discarded for bad checksums
0 discarded for bad header offset fields
0 discarded because packet too short
47 connection requests
41 connection accepts
87 connections established (including accepts)
164 connections closed (including 3 drops)
1 embryonic connection dropped
0 delayed frees of tcpcb
78403 segments updated rtt (of 57470 attempts)
7 retransmit timeouts
0 connections dropped by rexmit timeout
0 persist timeouts (resulting in 0 dropped connections)
197 keepalive timeouts
197 keepalive probes sent
0 connections dropped by keepalive
324 correct ACK header predictions
20545 correct data packet header predictions
101 PCB hash misses
7 dropped due to no socket
0 connections drained due to memory shortage
0 PMTUD blackholes detected
5 bad connection attempts
41 SYN cache entries added
0 hash collisions
41 completed
0 aborted (no space to build PCB)
0 timed out
0 dropped due to overflow
0 dropped due to bucket overflow
0 dropped due to RST
0 dropped due to ICMP unreachable
41 delayed free of SYN cache entries
0 SYN,ACKs retransmitted
0 duplicate SYNs received for entries already in the cache
0 SYNs dropped (no route or no space)
0 packets with bad signature
0 packets with good signature
0 successful ECN handshakes
0 packets with ECN CE bit
0 packets ECN ECT(0) bit
udp:
483110 datagrams received
0 with incomplete header
0 with bad data length field
25 with bad checksum
0 dropped due to no socket
187465 broadcast/multicast datagrams dropped due to no socket
0 dropped due to full socket buffers
295620 delivered
335602 PCB hash misses
224862 datagrams output
ip:
635056 total packets received
0 bad header checksums
0 with size smaller than minimum
0 with data size < data length
0 with length > max ip packet size
0 with header length < data size
0 with data length < header length
0 with bad options
0 with incorrect version number
19700 fragments received
0 fragments dropped (dup or out of space)
0 fragments dropped (out of ipqent)
0 malformed fragments dropped
0 fragments dropped after timeout
2076 packets reassembled ok
604866 packets for this host
0 packets for unknown/unsupported protocol
0 packets forwarded (0 packets fast forwarded)
12566 packets not forwardable
0 redirects sent
0 packets no matching gif found
346551 packets sent from this host
12 packets sent with fabricated ip header
0 output packets dropped due to no bufs, etc.
0 output packets discarded due to no route
8220 output datagrams fragmented
44740 fragments created
0 datagrams that can't be fragmented
0 datagrams with bad address in header
arp:
995 packets sent
905 reply packets
90 request packets
13257 packets received
88 reply packets
12850 valid request packets
12693 broadcast/multicast packets
0 packets with unknown protocol type
0 packets with bad (short) length
0 packets with null target IP address
60 packets with null source IP address
319 could not be mapped to an interface
0 packets sourced from a local hardware address
0 packets with a broadcast source hardware address
0 duplicates for a local IP address
0 attempts to overwrite a static entry
0 packets received on wrong interface
0 entrys overwritten
0 changes in hardware address length
8 packets deferred pending ARP resolution
7 sent
1 dropped
0 failures to allocate llinfo
ddp:
0 packets with short headers
0 packets with long headers
0 packets with no checksum
0 packets too short
0 packets with bad checksum
0 packets with not enough data
0 packets forwarded
0 packets encapsulated
0 packets rcvd for unreachable dest
0 packets dropped due to no socket space
--
Greg A. Woods
Planix, Inc.
<woods%planix.com@localhost> +1 250 762-7675 http://www.planix.com/
Attachment:
pgpDdYneGxgJG.pgp
Description: PGP signature