I run OpenWrt in QEMU on Arch Linux 5.1.2 with virtual tap network. The tap0 device (placed in default br-lan subnet) cannot reach other (guest) subnets beside its own (e.g. by ping). This is suspicious and only happens in QEMU (deployed on real hardware the clients can access other subnets). Arch Linux as guest works properly though (in QEMU).
This might not be a bug in OpenWrt but a security feature but i cannot figure out what exactly. I mainly need it for LXC containers that have their own subnet each. For demonstration purpose (below) i will use a dummy ethernet adapter as "target" interface/subnet.
To see what i see i think the best approach will be to set up a minimal system that will be affected by the problem:
- Download OpenWrt image: https://downloads.openwrt.org/releases/18.06.2/targets/x86/64/openwrt-18.06.2-x86-64-combined-ext4.img.gz and extract it into /tmp
- Create the host network environment:
#!/bin/bash
if [[ ! $(cat /proc/net/dev |grep br0) ]]; then
echo "Creating bridge"
#Make sure nothing interferes with bridges
#systemctl stop NetworkManager.service
sysctl net.ipv4.ip_forward=1 >/dev/null 2>&1
ip tuntap add tap0 mode tap user 1000
ip tuntap add tap1 mode tap user 1000
ip link set tap0 up
ip link set tap1 up
ip link add dev br0 type bridge
ip link add dev br1 type bridge
ip link set tap0 master br0
ip link set tap1 master br1
ip addr add 192.168.99.99/24 broadcast 192.168.99.255 dev br0
ip addr add 192.168.10.1/24 broadcast 192.168.10.255 dev br1
ip link set dev br0 up
ip link set dev br1 up
ip route add 192.168.2.0/24 dev br0
iptables -t nat -A POSTROUTING -s 192.168.10.0/24 ! -d 192.168.10.0/24 -j MASQUERADE
else
echo "Removing bridge"
ip tuntap del tap0 mode tap
ip tuntap del tap1 mode tap
ip link del dev br0 type bridge
ip link del dev br1 type bridge
sysctl net.ipv4.ip_forward=0 >/dev/null 2>&1
iptables -t nat -D POSTROUTING -s 192.168.10.0/24 ! -d 192.168.10.0/24 -j MASQUERADE
#systemctl start NetworkManager.service
fi
- Start QEMU:
#!/bin/bash
image=/tmp/openwrt-18.06.2-x86-64-combined-ext4.img
qemu="qemu-system-x86_64 \
-enable-kvm \
-cpu max \
-smp 4 \
-m 1G \
-boot c \
-drive id=disk0,if=none,format=raw,file=${image} \
-device virtio-blk-pci,drive=disk0 \
-device virtio-net,netdev=tap0 \
-device virtio-net,netdev=tap1 \
-netdev tap,id=tap0,ifname=tap0,script=no,downscript=no \
-netdev tap,id=tap1,ifname=tap1,script=no,downscript=no \
-serial mon:stdio \
-device qxl-vga \
-device piix3-usb-uhci \
-device usb-tablet \
"
${qemu}
echo -e "\n\033[1;35mBack at host!\033[0m\n"
- Press (Enter) on the OpenWrt text (not gfx) console and copy&paste:
uci set network.lan.ipaddr='192.168.99.1'
uci set network.lan.broadcast='192.168.99.255'
uci set network.wan.proto='static'
uci set network.wan.ipaddr='192.168.10.100'
uci set network.wan.netmask='255.255.255.0'
uci set network.wan.broadcast='192.168.10.255'
uci set network.wan.gateway='192.168.10.1'
uci set network.wan.dns='208.67.222.222'
uci commit
service network restart
opkg update
opkg install ip-full nano less-wide grep xz strace tar wget kmod-dummy ca-certificates
opkg find coreutils* |awk {'print $1}'| xargs opkg install --force-overwrite
opkg find procps-ng* |awk {'print $1}'| xargs opkg install --force-overwrite
ip addr add 192.168.2.1/24 broadcast 192.168.2.255 dev dummy0
ip link set dev dummy0 up
- Do some testing:
- On guest side:
ping -I 192.168.99.1 192.168.99.99 (ok)
ping -I 192.168.99.1 192.168.2.1 (ok)
ping -I 192.168.2.1 192.168.99.1 (ok)
ping -I 192.168.2.1 192.168.99.99 (failed)
- Do these when pinging from host:
tcpdump -n -i any -v (who-has 192.168.2.1 tell 192.168.99.99)
tcpdump -n -i br-lan -v (who-has 192.168.2.1 tell 192.168.99.99)
tcpdump -n -i dummy0 -v (nothing...)
- On host side:
ping -I 192.168.99.99 192.168.99.1 (ok)
ping -I 192.168.99.99 192.168.2.1 (failed)
The interface/subnet (192.168.2.1) is not reachable (from tap device) nor can it connect to the host bridge (192.168.99.99). I've checked/changed several values like 'arp_ignore' and 'rp_filter' but to no avail:
for i in /proc/sys/net/ipv4/conf/dummy0/*; do echo "${i} "$(cat ${i}); done
As mentioned above, this only happens in QEMU with OpenWrt and i have no clue why. I already flushed/cleaned all iptables/chains/rules, set all policies to ACCEPT, deleted all interfaces and created them manually (eth0, dummy0, bridges, eth1), checked routes and ARP entries and enabled promiscuous mode. Everything's looking good, but it doesn't work. Somehow OpenWrt doesn't like that virtual adapter. There are no 'martian' packets (if you enable that logging mode).
Any idea what's wrong here?
