Re: multiple qemu networking together and with host (Linux host)

[Andrew Randrianasulu <randrianasulu%gmail.com@localhost>]

On Fri, Dec 12, 2025 at 5:31 AM Andrew Randrianasulu
<randrianasulu%gmail.com@localhost> wrote:
>
> It was harder than I thought.
>
> Main setup:
>
> https://fo.ax/kali-kit/html_docs/86box.html
>
> ip tuntap add dev tap-vde mode tap user "$USER"
> ip link set 86box0 up
> ip a add 10.1.0.1/16 dev tap-vde
>
>
>
> vde_switch --mode 666 --numports 8 --tap tap-vde --mgmt
> /tmp/vde.qemux0.mgmt --mgmtmode 666 -s /tmp/vde.qemux0 --daemon
>
> Next, because I use wifi
>
> sysctl -w net.ipv4.ip_forward=1
>
> iptables -t nat -A POSTROUTING -s 10.1.0.1/16 -o wlan0 -j MASQUERADE
>
> iptables -A FORWARD -i eth0 -o tap-vde -m state --state
> RELATED,ESTABLISHED -j ACCEPT
> iptables -A FORWARD -i 86box0 -o wlan0 -j ACCEPT
>
> Install dhcpd (DHCP daemon)
>
> config it like
>
> cat /etc/dhcpd.conf
> # dhcpd.conf
> #
> # Configuration file for ISC dhcpd (see 'man dhcpd.conf')
> #
> subnet 10.1.0.0 netmask 255.255.255.224 {
> option routers 10.1.0.1 ;
> range 10.1.0.10 10.1.0.15 ;
> }
>
> whole config (ifconfig -a)
>
> root@slax:/dev/shm# ifconfig -a
> eth7      Link encap:Ethernet  HWaddr A8:A1:59:9B:07:45
>           UP BROADCAST MULTICAST  MTU:1500  Metric:1
>           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:0 errors:0 dropped:3 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000
>           RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
>
> lo        Link encap:Local Loopback
>           inet addr:127.0.0.1  Mask:255.0.0.0
>           inet6 addr: ::1/128 Scope:Host
>           UP LOOPBACK RUNNING  MTU:65536  Metric:1
>           RX packets:340797 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:340797 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000
>           RX bytes:235555020 (224.6 Mb)  TX bytes:235555020 (224.6 Mb)
>
> tap-vde   Link encap:Ethernet  HWaddr 86:2F:0D:6D:AC:96
>           inet addr:10.1.0.1  Bcast:0.0.0.0  Mask:255.255.0.0
>           inet6 addr: fe80::842f:dff:fe6d:ac96/64 Scope:Link
>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>           RX packets:4709 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:4229 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000
>           RX bytes:305415 (298.2 Kb)  TX bytes:260279 (254.1 Kb)
>
> wlan0     Link encap:Ethernet  HWaddr 1C:BF:CE:C6:BE:00
>           inet addr:192.168.1.104  Bcast:192.168.1.255  Mask:255.255.255.0
>           inet6 addr: fe80::1ebf:ceff:fec6:be00/64 Scope:Link
>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>           RX packets:3233241 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:2858540 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000
>           RX bytes:1608028041 (1533.5 Mb)  TX bytes:845539101 (806.3 Mb)
>
> IPTABLES config:
>
> root@slax:/dev/shm# iptables -t nat -vnL
> Chain PREROUTING (policy ACCEPT 1343 packets, 142K bytes)
>  pkts bytes target     prot opt in     out     source               destination
>
> Chain INPUT (policy ACCEPT 1031 packets, 45964 bytes)
>  pkts bytes target     prot opt in     out     source               destination
>
> Chain OUTPUT (policy ACCEPT 1736 packets, 297K bytes)
>  pkts bytes target     prot opt in     out     source               destination
>
> Chain POSTROUTING (policy ACCEPT 1736 packets, 297K bytes)
>  pkts bytes target     prot opt in     out     source               destination
>    17  1052 MASQUERADE  all  --  *      wlan0   10.1.0.0/16          0.0.0.0/0
>     0     0 MASQUERADE  all  --  *      wlan0   10.1.0.0/16          0.0.0.0/0
>
> =================
>
> root@slax:/dev/shm# iptables  -vnL
> Chain INPUT (policy ACCEPT 86261 packets, 23M bytes)
>  pkts bytes target     prot opt in     out     source               destination
>
> Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
>  pkts bytes target     prot opt in     out     source               destination
>    56  8221 ACCEPT     all  --  tap-vde wlan0   0.0.0.0/0            0.0.0.0/0
>    66 29463 ACCEPT     all  --  wlan0  tap-vde  0.0.0.0/0
> 0.0.0.0/0            state RELATED,ESTABLISHED
>
> Chain OUTPUT (policy ACCEPT 82010 packets, 16M bytes)
>  pkts bytes target     prot opt in     out     source               destination
>
> =======================
>
> qemu 1;
> qemu-system-sparc -hda
> /mnt/zip/home/guest/QEMU/sparc-netbsd-15g-compressed-r1.qcow2   -smp 1
> -accel tcg,thread=multi,tb-size=256 -M SS-20 -net
> nic,macaddr=52:54:00:02:12:23 -net vde,sock=/tmp/vde.qemux0
>
> qemu 2:
> qemu-system-sparc -hda /dev/shm/sparc-netbsd-15g-compressed-r1.qcow2
> -smp 1 -accel tcg,thread=multi,tb-size=256 -M SS-20 -m 256 -net
> nic,macaddr=52:54:00:01:12:24 -net vde,sock=/tmp/vde.qemux0
>
> Both qemu NetBSD machines should get ip 10.1.0.xx
>
> You should be able to set /etc/resolv.conf to nameserver in your real
> network (192.168.1.1 in my case - wifi router) and ping sites like
> google.com etc
>
> I was able to get nc (netcat) sending and receiving across two qemu's
> and see host ftp/ssh services.
>
> Next step is configuring distcc itself ;)


I think I finally set it up, on server setting PATH to masquaraded compiler in
/usr/pkg/lib/distcc/bin eraly in PATH

then this command

distccd --listen 10.1.0.10 --allow 10.1.0.11 --enable-tcp-insecure
--log-stderr --verbose

on client I set /etc/mk.conf as was suggested by this hackaday article:

PKGSRC_COMPILER=distcc gcc
 MAKE_JOBS=3
 DISTCC_HOSTS=10.1.0.10:3632

and finally set DISTCC_HOSTS to same 10.1.0.10 (I launched distcc
server qemu first so it get IP earlier)

at some point I was getting "Failed to distribute" errors, I think
--enable-tcp-insecure, even if discouraged by man  was important?




>
> In theory vde2 should work on macos hosts, and may be freebsd ? Linux
> is easy mode
>
>
> It was bigger adventure than I initially hoped!