Guest and host cannot see each other using linux-kvm and macvtap

Question

I'm migrating a kvm virtual machine from an old host (both hardware and OS) to a new one.

For networking, virt-manager proposed me a new option: macvtap. This looked a good alternative to setting up a bridge on eth0.

So now the guest boots just fine, gets an IP from my local network DHCP server, can reach the internet. The guest also sees other machines on the local network, I can ssh them, etc.

The problem is that the host and the guest do not see each other. I cannot reach the guest from the host using the guest IP, neither can I reach the host from the guest using the host IP. No ping, ssh, http, nothing.

Here is the route -n command from the host:

$ /sbin/route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.0.1     0.0.0.0         UG    0      0        0 eth0
192.168.0.0     0.0.0.0         255.255.255.0   U     1      0        0 eth0

(same output from the guest).

I could probably set up a new tun/tap interface dedicated to communication between host and guest but it looks a little bit overkill. Is there a way to make host and guest communicate?

Answer 1

I asked this question on IRC and it appears that macvtap

injects guest traffic into the network stack too low for that

The solution is then to add a network interface for the guest and the host to communicate, or stay with the old bridged solution...

Answer 2

virt-manager says explictly that macvtap does not work for host to guest network communications when you set it up. I simply added a second nat based interface, set it up in the guest, and use that to communicate with my host.

Answer 3

The solution is to configure a macvlan interface on the hypervisor, with the same IP address than the real hardware interface (very important), and to configure routing on the host to use it. In Qemu/KVM, use a macvtap interface on the hardware interface as usual.

For my config (192.168.1.0/24 network, p10p1 hardware interface, and 192.168.1.1 gateway), it gives (on the hypervisor):

ip link add link p10p1 address 00:19:d1:29:d2:58 macvlan0 type macvlan mode bridge
ip address add 192.168.1.100/24 dev macvlan0
ip link set dev macvlan0 up

ip route flush dev p10p1
ip route add default via 192.168.1.1 dev macvlan0 proto static

Answer 4

As has been mentioned in previous answers, a solution to this problem is to add a macvlan network adapter on the host. However, I felt that manually rewiring the routes to the macvlan adapter was kinda hacky, especially since I wanted IPv6 support and manually set routes may become a problem when the prefix changes. So here's my configuration which leaves the kernel in control of the routing table:

_{(The particular configuration here is Debian and Upstart specific, but the basic steps should work on any GNU/Linux.)}

Creating the macvlan adapter on boot

First, you need to select a MAC address for your adapter. You may be able to just use a random one, but I suggest you manually create a macvlan adapter and use its MAC. This way, the MAC obliges to any conventions that might be there.

Setting a fixed MAC is advisable, since otherwise there is no way for e.g. an DHCP server to recognize your machine after a reboot and assign it the same IP address as before.

So create an adapter and look up the MAC:

root@host:~# ip link add link eth0 macvlan0 type macvlan mode bridge
root@host:~# ip addr show dev macvlan0
#: macvlan0@eth0:  mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1
    link/ether 12:34:56:78:90:ab brd ff:ff:ff:ff:ff:ff
...

The highlighted hexadecimal number is your MAC address.

Now you create an init script - which must be run before the networking is initalized - to create the macvlan adapter each startup. The command to do this is:

ip link add link eth0 address <MACADDRESS> macvlan0 type macvlan mode bridge

An example Upstart init script for this purpose would be:

start on starting networking

script
    ip link add link eth0 address <MACADDRESS> macvlan0 type macvlan mode bridge
end script

Just put this in e.g. /etc/init/macvlan.conf.

Setting up the network configuration

In /etc/network/interfaces, set your physical network adapter to manual (but leave it auto) and move its previous configuration (typically DHCP or a static IP address) to your macvlan adapter. E.g.:

auto eth0
iface eth0 inet manual

auto macvlan0
iface macvlan0 inet dhcp

Disabling IPv6 for the physical adapter

Lastly, you do not want the physical adapter to obtain an IP address. For IPv4 setting the adapter to manual prevents it from obtaining an address. However, I have not found a configuration which stops the kernel from obtaining/assigning an IPv6 address for/to the adapter. When it does, it also adds routes for them, which may cause problems. So the best way seems to be to disable IPv6 for the physical adapter. You can do this by adding the line

net.ipv6.conf.eth0.disable_ipv6=1

to /etc/sysctl.conf, by creating a file in /etc/sysctl.d/ with this line, or by adding

sysctl -w net.ipv6.conf.eth0.disable_ipv6=1

to your init script.

When you now reboot your machine, communicating from host to guest should work with both IPv4 and IPv6.

---

Be aware, that if you make a mistake while setting this up, your host may become unreachable via network even after a reboot. Only do this if you have physical access to the machine or other safeguards are in place so you can fix potential problems.

Answer 5

You want to end up with something like this:

Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.0.1     0.0.0.0         UG    0      0        0 macvlan0
192.168.0.0     0.0.0.0         255.255.255.0   U     1      0        0 macvlan0

Note that the last column contains "macvlan0" instead of "eth0".

To achieve that, you can use these commands, assuming 192.168.0.42 as your IP address:

ip link add link eth0 macvlan0 type macvlan mode bridge
ip address add 192.168.0.42 dev macvlan0
ip link set dev macvlan0 up
ip route flush dev eth0
ip route add default via 192.168.0.1

Mostly similar to the solution of npen. If you want a more sophisticated script, see my webpage about this subject.

Answer 6

I used what Lazzaro wrote on Debian, but I want to add a few notes.

First of all, the guest interface needs be set to "Source device" --> "Host device eth0 : macvtap"; "Source mode" --> "Bridge".

Second, the init script you have didn't work for me (it seems that the macvlan0 interface was created too late during startup), so I used this:

nano /etc/init.d/macvlan

Change the mac to the one that was assigned when the interface was first created:

#!/bin/sh
### BEGIN INIT INFO
# Provides:          macvlan
# Required-Start:    $local_fs
# Required-Stop:     
# Default-Start:     S
# Default-Stop:      
# Short-Description: Set up macvlan
# Description:       This script sets up the macvlan interface
#                    before it can be used by networking.
# X-Start-Before:    networking
### END INIT INFO


PATH=/sbin:/usr/sbin:/bin:/usr/bin
. /lib/init/vars.sh


do_start () {
        ip link add link eth0 address 00:11:22:aa:bb:cc macvlan0 type macvlan mode bridge
###      REPLACE MAC ON EACH HOST ->  ^^ ^^ ^^ ^^ ^^ ^^
}

case "$1" in
  start|"")
        do_start
        ;;
  *)
        echo "Sorry, this only sets up the macvlan interface." >&2
        exit 3
        ;;
esac

Save and set the execute bit:

chmod +x /etc/init.d/macvlan

and add all links for init:

update-rc.d macvlan defaults

Answer 7

Problem.

As it has been already said in previous answers, to enable communication between host and guest, the proposed solution is to add macvlan network adapter on the host. Above given examples explain how to do it from command line/scripts.

If you are using NetworkManager to manage your connections, macvlan adapter created as stated above, will not cooperate nicely with NetwokManager. On official libvirt wiki MacvtapHostFail it is also suggested that added macvlan does not cooperate well with NetworkManager.

Just as NetworkManager currently doesn't understand bridge devices, it also doesn't understand macvtap devices, so NetworkManager would be unable to monitor the online state of the macvtap interface, and would give erroneous reports about the online status of the host. In other words, it's really no better than just using a traditional host bridge (with the added problem that even the traditional methods of network configuration (e.g. initscripts on Fedora and RHEL) don't support configuration of a macvtap device).

It seems that that is not the case any more (since end of 2015 if I understood correctly).

The solution is to add macvlan adapter using NetworkManager.

What to do?

Basicall, we have to replace our ethernet connection with macvlan connection in NetworkManager. First we have to delete ethernet connection using GUI or nmcli, then we have to add macvlan connection which will replace the functionality of existing/previous ethernet conenction. The catch is, we can only add and edit macvlan connection using nmcli.

GUI does not support adding/editing macvlan connection. Adding macvlan connection in GUI is not possible at all. Editing on other side is possible, since macvlan connection will be visible in connection list, but when saved, connection will be saved as ethernet connection, thus destroying macvlan connection (albeit preserving the name). Do not edit your macvlan connection in GUI!

How to do it?

1. Delete ethernet connection using GUI or nmcli (to list connections run: nmcli con show, and to delete nmcli con del conn-name).

2. Add macvlan connection using nmcli:

    nmcli connection add \
        con-name host-macvlan \
        type macvlan \
        mode bridge  \
        ifname macX  \
        dev eno1     \
        mac aa:eb:b5:e6:df:fd \
        ip4 192.168.1.100/24 \
        gw4 192.168.1.1      \
        ipv4.dns 192.168.1.1,192.168.1.2 \
        ipv4.dns-search home.lan,example.com
   

   This will create macvlan interface named macX (you can use any name for interface, like mac0 or macvlan0 or any string), via device eno1, and NetworkManager connection named host-macvlan. Connection will be automatically saved, and automatically connected, at the moment of creation and on each boot.

   mac option is optional. If given (with appropriate mac address) created interface will have fix mac address.

Notes:

1. Do not edit macvlan connection in GUI!

2. You can safely delete macvlan connection in GUI.

3. If you have any openVPN connections in NetworkManager, they will cooperate nicely with macvlan, and use it as an interface to create vpn connection.

4. If you are using firewall, and you have defined rules referencing ethernet device eno1 as interface, replace the interface name with macX.

   For example:

   itpables -A INPUT -i eno1 -j ACCEPT
   

   into

   iptables -A INPUT -i macX -j ACCEPT
   

   While traffic will be visible in interface eno1 (for example using tcpdump -ni eno1), in kernel, traffic filtering is done in relation with device/interface macX.