I was recommended to move this to superuser from servervault, hopefully it's appropriate for here.
I'm a hobbyist / tinkerer using bhyve (in pfSense running on a physical machine) to start up a Linux VM (trying with both NixOS and Ubuntu Server). I followed instructions in this thread and think I might be running into the same issue as the linked comment -- specifically, the VM boots and runs, but my local DNS resolution (unbound, running on the pfSense host) is not working. Disabling pfblockerng doesn't seem to change anything. Firewall is disabled and rules flushed.
As per that thread, I have set up a TAP interface, a bridge between it and the LAN, and firewall rules on each to allow all traffic (for now).
From the VM, I can:
- get an IP address via DHCP (in the LAN subnet)
- ping both internal and external hosts by IP address, including the host pfsense machine at
- send and receive TCP and UDP data with netcat to both internal and external hosts
- resolve DNS using an external resolver (e.g.
host n8henrie.com 22.214.171.124or
126.96.36.199or what have you)
I have unbound running on pfsense and it's working great for over a year, is set to work on all interfaces. Using tcpdump, I can see that unbound is responding to DNS requests from the VM with a correct response, but for some reason they don't seem to be arriving. I've recorded a pcap file (from the ubuntu guest) to inspect these DNS responses to see if I can figure out why the packets are arriving but not working.
I don't see any relevant blocked traffic in
/var/log/filter.log, so I think the firewall rules are working.
watch host -4 -W1 n8henrie.com 192.168.0.2 running in the VM, which just shows
;; connection timed out; no servers could be reached . Simultaneously, I have
tcpdump -i tap0 -v host 192.168.0.202 and udp running on pfsense and I see appropriate resolution of the domain (repeating over time as expected):
[root@pfSense ~]# tcpdump -i tap0 -v host 192.168.0.202 and udp tcpdump: listening on tap0, link-type EN10MB (Ethernet), capture size 262144 bytes 11:25:34.981350 IP (tos 0x0, ttl 64, id 1081, offset 0, flags [none], proto UDP (17), length 58) 192.168.0.202.45334 > pfSense.home.arpa.domain: 14697+ A? n8henrie.com. (30) 11:25:34.994974 IP (tos 0x0, ttl 64, id 27251, offset 0, flags [none], proto UDP (17), length 90) pfSense.home.arpa.domain > 192.168.0.202.45334: 14697 2/0/0 n8henrie.com. A 188.8.131.52, n8henrie.com. A 184.108.40.206 (62) 11:25:35.814435 IP (tos 0x0, ttl 64, id 50026, offset 0, flags [none], proto UDP (17), length 58)
I can even see that the Running tcpdump in the guest shows the responses arriving...
root@ubuntu-server:/# tcpdump -vv -i enp0s2 host 192.168.0.2 and udp 11:46:49.569313 IP (tos 0x0, ttl 64, id 34767, offset 0, flags [none], proto UDP (17), length 58) 192.168.0.202.44462 > 192.168.0.2.domain: [udp sum ok] 59526+ A? n8henrie.com. (30) 11:46:49.576273 IP (tos 0x0, ttl 64, id 30262, offset 0, flags [none], proto UDP (17), length 90) 192.168.0.2.domain > 192.168.0.202.44462: [bad udp cksum 0x8274 -> 0x871c!] 59526 q: A? n8henrie.com. 2/0/0 n8henrie.com. A 220.127.116.11, n8henrie.com. A 18.104.22.168 (62)
So if the DNS responses can be seen in the client, why aren't they working?
resolv.conf from the NixOS guest, I can also post ubuntu if that helps:
# cat /etc/resolv.conf domain home.arpa nameserver 192.168.0.2 options edns0
From the host:
[root@pfSense /]# ifconfig tap0 tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: OPT1 options=80000<LINKSTATE> ether 58:9c:fc:10:88:32 groups: tap media: Ethernet autoselect status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> Opened by PID 76444 [root@pfSense /]# ifconfig bridge0 bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 ether 02:33:2d:65:15:00 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 10 priority 128 path cost 2000000 member: igb1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 2 priority 128 path cost 2000000 groups: bridge nd6 options=1<PERFORMNUD>
Other notes / things I've tried:
- As noted, firewall is disabled
rp_filterto 0 doesn't change anything
- Same behavior on both
- At the end of tcpdump from client I see e.g.
X packets captured, X packets received by filter, 0 packets dropped by kernel, so I guess they're not being dropped.
Thanks in advance for any suggestions!