0

We had email from Hetzner network team asking us not to use MAC addresses belonging to the Virtual Machines belonging to the subnet.

We configured Xen server host as a router using this guide.

After asking for additional details Hetzner support answered that normally, the network configuration of your hypervisor should only let packets exit the system with the MAC address of the real NIC. But if you do not find the issue, you might try to block those outgoing packets using IPtables.

So our questions:

If anyone had this kind of problem with Hetzner or other dedicated server provider.
How you solved it? Is blocking using iptables the only way?

Thank you

Configuration details

Host/Router configuration:

[root@xenserver-custom ~]# cat /etc/sysctl.conf

net.ipv4.ip_forward = 1  
net.ipv6.conf.all.forwarding=1

net.ipv4.conf.default.proxy_arp = 0  

net.ipv4.conf.all.send_redirects = 0  
net.ipv4.conf.default.send_redirects = 0  
net.ipv4.conf.lo.send_redirects = 0  
net.ipv4.conf.xenbr0.send_redirects = 0  

[root@xenserver-custom network-scripts]# ip addr add 85.91.107.177/28 dev xenbr0

[root@xenserver-custom ~]# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 0c:c4:7a:e7:dc:33  txqueuelen 1000  (Ethernet)
        RX packets 4704816217  bytes 6002063739181 (5.4 TiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 6294828922  bytes 7643975899027 (6.9 TiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        loop  txqueuelen 1  (Local Loopback)
        RX packets 518545683  bytes 6322784653872 (5.7 TiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 518545683  bytes 6322784653872 (5.7 TiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vifxxxx

.....................

xenbr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 115.35.61.184  netmask 255.255.255.192  broadcast 115.35.61.191
        ether 0c:c4:7a:e7:dc:33  txqueuelen 1  (Ethernet)
        RX packets 3070611738  bytes 8670969429554 (7.8 TiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2680055664  bytes 9822630727363 (8.9 TiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[root@xenserver-enginatics1 sysconfig]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP qlen 1000
    link/ether 0c:c4:7a:e7:dc:33 brd ff:ff:ff:ff:ff:ff
3: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1
    link/ether 72:ab:90:89:cf:cb brd ff:ff:ff:ff:ff:ff
4: xenbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN qlen 1
    link/ether 0c:c4:7a:e7:dc:33 brd ff:ff:ff:ff:ff:ff
    inet 115.35.61.184/26 brd 115.35.61.191 scope global dynamic xenbr0
       valid_lft 37373sec preferred_lft 37373sec
    inet 85.91.107.177/28 scope global xenbr0
       valid_lft forever preferred_lft forever
31: vif15.0: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP qlen 32
    link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
44: vif21.0: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP qlen 32
    link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
..........

VM Guest configuration

[root@r1213a network-scripts]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
TYPE=Ethernet
ONBOOT=yes
NM_CONTROLLED=yes
BOOTPROTO=static
IPADDR=85.91.107.184
PREFIX=28
GATEWAY=85.91.107.177
DNS1=213.133.98.98
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
IPV6INIT=no

[root@r1213a network-scripts]# ifconfig
eth0      Link encap:Ethernet  HWaddr B6:8F:14:74:A6:B6
          inet addr:85.91.107.184  Bcast:85.91.107.191  Mask:255.255.255.240
          inet6 addr: fe80::b48f:14ff:fe74:a6b6/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:27122939 errors:0 dropped:2 overruns:0 frame:0
          TX packets:2218911 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:5404322465 (5.0 GiB)  TX bytes:1061055301 (1011.9 MiB)

[root@r1213a ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether b6:8f:14:74:a6:b6 brd ff:ff:ff:ff:ff:ff
    inet 85.91.107.184/28 brd 85.91.107.191 scope global eth0
    inet6 fe80::b48f:14ff:fe74:a6b6/64 scope link
       valid_lft forever preferred_lft forever

[root@r1213a network-scripts]# ip route
default via 85.91.107.177 dev eth0
85.91.107.176/28 dev eth0  proto kernel  scope link  src 85.91.107.184

[root@r1213a network-scripts]# traceroute google.com
traceroute to google.com (172.217.18.110), 30 hops max, 60 byte packets
 1  xenserver.localdomain (85.91.107.177)  0.081 ms  0.029 ms  0.039 ms
 2  static.129.61.69.159.clients.your-server.de (159.69.61.129)  0.390 ms  0.410 ms  0.370 ms
 3  core22.fsn1.hetzner.com (213.239.245.121)  0.393 ms  0.416 ms  0.424 ms
 4  core0.fra.hetzner.com (213.239.252.33)  5.207 ms  5.184 ms core0.fra.hetzner.com (213.239.252.29)  5.049 ms
 5  72.14.218.94 (72.14.218.94)  5.273 ms  5.249 ms 72.14.218.176 (72.14.218.176)  4.990 ms
 6  108.170.251.193 (108.170.251.193)  5.139 ms *  5.019 ms
 7  209.85.241.75 (209.85.241.75)  5.834 ms 216.239.40.58 (216.239.40.58)  5.092 ms 172.253.64.119 (172.253.64.119)  5.707 ms
 8  108.170.251.144 (108.170.251.144)  15.292 ms zrh04s05-in-f110.1e100.net (172.217.18.110)  4.952 ms  4.903 ms
  • Can you show that the XenServer VMs are configured to use the host as a router, rather than being bridged directly to host eth0? – grawity Apr 2 at 6:10
  • Ok, tried to add as much configuration details as possible. Let me know if anything else required – Alex Apr 2 at 6:41
  • That guide only tells you how to enable routing in general. It’s missing the key parts about how to separate the internal (virtual) network from the external network, which is absolutely imperative. // It’s easier if you just generate a MAC address for your VM IPs and use a bridged setup. – Daniel B Apr 2 at 6:47
  • That's the problem - Hetzner doesn't allow to tie MAC address for IPs in subnet. It's allowed on their self-service webpage only for standalone IP addresses. Xen Center does generate MAC addresses, but they are illegal for use. I asked them for clear instructions or to update configuration guide, but the only advice I got is to use iptables to block outgoing packets with prohibited MACs, which doesn't seem to be a solution for me, but a workaround. Still, not sure how to implement even this. – Alex Apr 2 at 6:58
  • With a separate MAC address for your subnet you can at least properly set up a routing VM instead of messing with the host (which is bound to break on XenServer updates). – Daniel B Apr 2 at 7:10
1

I had also faced with these emails from Hetzner and was able to figure out the solution.

In Hetzner you have two types of additional IP-addresses which you can use with a dedicated server: a single IPv4 & an IPv4 subnet. For each single IP you get a MAC-address and you must use that MAC on the network interface of a new VM instance. For each additional subnet you must setup a new network and setup routing between that network and the default network (associated with eth0) of the server.

In XenServer this can be done using Linux console:

xe network-create name-label="Additional network" name-description="46.xx.yy.zz/28"

This will create a new network attached to a new bridge (xapi0 by default) in XenServer. Then assign the first usable IP-address of the network (according to it's netmask) to that bridge:

ip addr add 46.xx.yy.1/28 dev xapi0

Now you can add new VMs with autogenerated MACs attached to the newly created network instead of the default network. The traffic will be switched & routed inside XenServer.

Having that setup I got confirmation from Hetzner network team that only allowed MACs are visible on the switchport.

  • Hello! Thank you for answer, this solution works! I went to wrong direction, trying to create second xenbr in the same network and assign VIFs to it. But It's not working because after virtual machine restart VIF number is regenerated. So you saved me hours of additional research. – Alex Apr 9 at 7:37
  • Also after I created new network, xapi0 bridge didn't show up on host until I assigned one of VMs to new network in Xen Center. Just tip that may save time if it happens to anyone else. – Alex Apr 9 at 7:44
  • You can verify there is no offending traffic on eth0 anymore from your side at host: tcpdump -i eth0 -en|egrep -i 'mac1|mac2' It all should be seen on xapi0 instead tcpdump -i xapi0 -en|egrep -i 'mac1|mac2' Where mac1,mac2 - MAC addresses reported by Hetzner as prohibited. – Alex Apr 9 at 7:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.