7

I have a KVM server (host) with multiple virtual machines (guests).

My goal is my host forward port 222 to port 22 of a guest running an ssh service.

This works...

iptables -I OUTPUT -d 0.0.0.0/0 -j ACCEPT
iptables -I FORWARD -d 0.0.0.0/0 -j ACCEPT
iptables -I INPUT -d 0.0.0.0/0 -j ACCEPT
iptables -t nat -I PREROUTING -d 0.0.0.0/0 -p tcp --dport 222 -j DNAT --to-destination 10.1.0.9:22

This don't work...

firewall-cmd --permanent --zone=public --direct --add-rule ipv4 filter OUTPUT 0 -d 0.0.0.0/0 -j ACCEPT
firewall-cmd --permanent --zone=public --direct --add-rule ipv4 filter FORWARD 0 -d 0.0.0.0/0 -j ACCEPT
firewall-cmd --permanent --zone=public --direct --add-rule ipv4 filter INPUT 0 -d 0.0.0.0/0 -j ACCEPT
firewall-cmd --permanent --zone=public --direct --add-rule ipv4 nat PREROUTING 0 -d 0.0.0.0/0 -p tcp --dport 222 -j DNAT --to-destination 10.1.0.9:22
firewall-cmd --reload

This don't work too...

firewall-cmd --permanent --zone=public --add-forward-port=port=222:proto=tcp:toport=22:toaddr=10.1.0.9
firewall-cmd --reload

QUESTION: Why rules set with firewall-cmd don't work?

NOTE I: The firewall-cmd is the CentOS 7's default firewall service. This seems to me to be a problem with no solution! I searched many, many forums and nothing works! I'm starting to believe this is a limitation or bug in firewall-cmd...

NOTE II: I know that ssh itself provides the means to make this possible, but I really want this process to be "transparent" for the user to access the guest directly.

[Refs.: https://serverfault.com/q/915257/276753 , https://serverfault.com/q/980223/276753 , https://sebastianblade.com/how-to-modify-ssh-port-in-centos7/ , https://www.rootusers.com/how-to-use-firewalld-rich-rules-and-zones-for-filtering-and-nat/ , https://www.centos.org/forums/viewtopic.php?f=50&t=71454 ]

SYMPTOM:

The command...

ssh root@[HOST_IP] -p 222

... returns me the following error...

ssh: connect to host 172.16.13.8 port 222: Connection refused

UPDATE I:

@mwfearnley iptables-save output...

iptables-save - WORKS...

[root@localhost ~]# iptables-save
# Generated by iptables-save v1.4.21 on Thu Aug 22 11:59:17 2019
*nat
:PREROUTING ACCEPT [1:70]
:INPUT ACCEPT [1:70]
:OUTPUT ACCEPT [2:146]
:POSTROUTING ACCEPT [3:206]
:OUTPUT_direct - [0:0]
:POSTROUTING_ZONES - [0:0]
:POSTROUTING_ZONES_SOURCE - [0:0]
:POSTROUTING_direct - [0:0]
:POST_public - [0:0]
:POST_public_allow - [0:0]
:POST_public_deny - [0:0]
:POST_public_log - [0:0]
:PREROUTING_ZONES - [0:0]
:PREROUTING_ZONES_SOURCE - [0:0]
:PREROUTING_direct - [0:0]
:PRE_public - [0:0]
:PRE_public_allow - [0:0]
:PRE_public_deny - [0:0]
:PRE_public_log - [0:0]
-A PREROUTING -p tcp -m tcp --dport 222 -j DNAT --to-destination 10.1.0.9:22
-A PREROUTING -j PREROUTING_direct
-A PREROUTING -j PREROUTING_ZONES_SOURCE
-A PREROUTING -j PREROUTING_ZONES
-A OUTPUT -j OUTPUT_direct
-A POSTROUTING -s 10.1.0.0/24 -d 224.0.0.0/24 -j RETURN
-A POSTROUTING -s 10.1.0.0/24 -d 255.255.255.255/32 -j RETURN
-A POSTROUTING -s 10.1.0.0/24 ! -d 10.1.0.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 10.1.0.0/24 ! -d 10.1.0.0/24 -p udp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 10.1.0.0/24 ! -d 10.1.0.0/24 -j MASQUERADE
-A POSTROUTING -j POSTROUTING_direct
-A POSTROUTING -j POSTROUTING_ZONES_SOURCE
-A POSTROUTING -j POSTROUTING_ZONES
-A POSTROUTING_ZONES -o ens33 -g POST_public
-A POSTROUTING_ZONES -o br0 -g POST_public
-A POSTROUTING_ZONES -g POST_public
-A POST_public -j POST_public_log
-A POST_public -j POST_public_deny
-A POST_public -j POST_public_allow
-A PREROUTING_ZONES -i ens33 -g PRE_public
-A PREROUTING_ZONES -i br0 -g PRE_public
-A PREROUTING_ZONES -g PRE_public
-A PRE_public -j PRE_public_log
-A PRE_public -j PRE_public_deny
-A PRE_public -j PRE_public_allow
COMMIT
# Completed on Thu Aug 22 11:59:17 2019
# Generated by iptables-save v1.4.21 on Thu Aug 22 11:59:17 2019
*mangle
:PREROUTING ACCEPT [672:77587]
:INPUT ACCEPT [610:68993]
:FORWARD ACCEPT [58:7886]
:OUTPUT ACCEPT [655:151604]
:POSTROUTING ACCEPT [713:159490]
:FORWARD_direct - [0:0]
:INPUT_direct - [0:0]
:OUTPUT_direct - [0:0]
:POSTROUTING_direct - [0:0]
:PREROUTING_ZONES - [0:0]
:PREROUTING_ZONES_SOURCE - [0:0]
:PREROUTING_direct - [0:0]
:PRE_public - [0:0]
:PRE_public_allow - [0:0]
:PRE_public_deny - [0:0]
:PRE_public_log - [0:0]
-A PREROUTING -j PREROUTING_direct
-A PREROUTING -j PREROUTING_ZONES_SOURCE
-A PREROUTING -j PREROUTING_ZONES
-A INPUT -j INPUT_direct
-A FORWARD -j FORWARD_direct
-A OUTPUT -j OUTPUT_direct
-A POSTROUTING -j POSTROUTING_direct
-A PREROUTING_ZONES -i ens33 -g PRE_public
-A PREROUTING_ZONES -i br0 -g PRE_public
-A PREROUTING_ZONES -g PRE_public
-A PRE_public -j PRE_public_log
-A PRE_public -j PRE_public_deny
-A PRE_public -j PRE_public_allow
COMMIT
# Completed on Thu Aug 22 11:59:17 2019
# Generated by iptables-save v1.4.21 on Thu Aug 22 11:59:17 2019
*security
:INPUT ACCEPT [609:68793]
:FORWARD ACCEPT [58:7886]
:OUTPUT ACCEPT [660:152010]
:FORWARD_direct - [0:0]
:INPUT_direct - [0:0]
:OUTPUT_direct - [0:0]
-A INPUT -j INPUT_direct
-A FORWARD -j FORWARD_direct
-A OUTPUT -j OUTPUT_direct
COMMIT
# Completed on Thu Aug 22 11:59:17 2019
# Generated by iptables-save v1.4.21 on Thu Aug 22 11:59:17 2019
*raw
:PREROUTING ACCEPT [672:77587]
:OUTPUT ACCEPT [655:151604]
:OUTPUT_direct - [0:0]
:PREROUTING_ZONES - [0:0]
:PREROUTING_ZONES_SOURCE - [0:0]
:PREROUTING_direct - [0:0]
:PRE_public - [0:0]
:PRE_public_allow - [0:0]
:PRE_public_deny - [0:0]
:PRE_public_log - [0:0]
-A PREROUTING -j PREROUTING_direct
-A PREROUTING -j PREROUTING_ZONES_SOURCE
-A PREROUTING -j PREROUTING_ZONES
-A OUTPUT -j OUTPUT_direct
-A PREROUTING_ZONES -i ens33 -g PRE_public
-A PREROUTING_ZONES -i br0 -g PRE_public
-A PREROUTING_ZONES -g PRE_public
-A PRE_public -j PRE_public_log
-A PRE_public -j PRE_public_deny
-A PRE_public -j PRE_public_allow
COMMIT
# Completed on Thu Aug 22 11:59:17 2019
# Generated by iptables-save v1.4.21 on Thu Aug 22 11:59:17 2019
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:FORWARD_IN_ZONES - [0:0]
:FORWARD_IN_ZONES_SOURCE - [0:0]
:FORWARD_OUT_ZONES - [0:0]
:FORWARD_OUT_ZONES_SOURCE - [0:0]
:FORWARD_direct - [0:0]
:FWDI_public - [0:0]
:FWDI_public_allow - [0:0]
:FWDI_public_deny - [0:0]
:FWDI_public_log - [0:0]
:FWDO_public - [0:0]
:FWDO_public_allow - [0:0]
:FWDO_public_deny - [0:0]
:FWDO_public_log - [0:0]
:INPUT_ZONES - [0:0]
:INPUT_ZONES_SOURCE - [0:0]
:INPUT_direct - [0:0]
:IN_public - [0:0]
:IN_public_allow - [0:0]
:IN_public_deny - [0:0]
:IN_public_log - [0:0]
:OUTPUT_direct - [0:0]
-A INPUT -j ACCEPT
-A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -j INPUT_direct
-A INPUT -j INPUT_ZONES_SOURCE
-A INPUT -j INPUT_ZONES
-A INPUT -m conntrack --ctstate INVALID -j DROP
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j ACCEPT
-A FORWARD -d 10.1.0.0/24 -o virbr0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -s 10.1.0.0/24 -i virbr0 -j ACCEPT
-A FORWARD -i virbr0 -o virbr0 -j ACCEPT
-A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i lo -j ACCEPT
-A FORWARD -j FORWARD_direct
-A FORWARD -j FORWARD_IN_ZONES_SOURCE
-A FORWARD -j FORWARD_IN_ZONES
-A FORWARD -j FORWARD_OUT_ZONES_SOURCE
-A FORWARD -j FORWARD_OUT_ZONES
-A FORWARD -m conntrack --ctstate INVALID -j DROP
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
-A OUTPUT -j ACCEPT
-A OUTPUT -o virbr0 -p udp -m udp --dport 68 -j ACCEPT
-A OUTPUT -j OUTPUT_direct
-A FORWARD_IN_ZONES -i ens33 -g FWDI_public
-A FORWARD_IN_ZONES -i br0 -g FWDI_public
-A FORWARD_IN_ZONES -g FWDI_public
-A FORWARD_OUT_ZONES -o ens33 -g FWDO_public
-A FORWARD_OUT_ZONES -o br0 -g FWDO_public
-A FORWARD_OUT_ZONES -g FWDO_public
-A FWDI_public -j FWDI_public_log
-A FWDI_public -j FWDI_public_deny
-A FWDI_public -j FWDI_public_allow
-A FWDI_public -p icmp -j ACCEPT
-A FWDO_public -j FWDO_public_log
-A FWDO_public -j FWDO_public_deny
-A FWDO_public -j FWDO_public_allow
-A INPUT_ZONES -i ens33 -g IN_public
-A INPUT_ZONES -i br0 -g IN_public
-A INPUT_ZONES -g IN_public
-A IN_public -j IN_public_log
-A IN_public -j IN_public_deny
-A IN_public -j IN_public_allow
-A IN_public -p icmp -j ACCEPT
-A IN_public_allow -p tcp -m tcp --dport 22 -m conntrack --ctstate NEW -j ACCEPT
COMMIT
# Completed on Thu Aug 22 11:59:17 2019

iptables-save - DON'T WORK...

[root@localhost ~]# iptables-save
# Generated by iptables-save v1.4.21 on Thu Aug 22 11:59:16 2019
*nat
:PREROUTING ACCEPT [1:72]
:INPUT ACCEPT [1:72]
:OUTPUT ACCEPT [5:371]
:POSTROUTING ACCEPT [5:371]
:OUTPUT_direct - [0:0]
:POSTROUTING_ZONES - [0:0]
:POSTROUTING_ZONES_SOURCE - [0:0]
:POSTROUTING_direct - [0:0]
:POST_public - [0:0]
:POST_public_allow - [0:0]
:POST_public_deny - [0:0]
:POST_public_log - [0:0]
:PREROUTING_ZONES - [0:0]
:PREROUTING_ZONES_SOURCE - [0:0]
:PREROUTING_direct - [0:0]
:PRE_public - [0:0]
:PRE_public_allow - [0:0]
:PRE_public_deny - [0:0]
:PRE_public_log - [0:0]
-A PREROUTING -j PREROUTING_direct
-A PREROUTING -j PREROUTING_ZONES_SOURCE
-A PREROUTING -j PREROUTING_ZONES
-A OUTPUT -j OUTPUT_direct
-A POSTROUTING -s 10.1.0.0/24 -d 224.0.0.0/24 -j RETURN
-A POSTROUTING -s 10.1.0.0/24 -d 255.255.255.255/32 -j RETURN
-A POSTROUTING -s 10.1.0.0/24 ! -d 10.1.0.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 10.1.0.0/24 ! -d 10.1.0.0/24 -p udp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 10.1.0.0/24 ! -d 10.1.0.0/24 -j MASQUERADE
-A POSTROUTING -j POSTROUTING_direct
-A POSTROUTING -j POSTROUTING_ZONES_SOURCE
-A POSTROUTING -j POSTROUTING_ZONES
-A POSTROUTING_ZONES -o ens33 -g POST_public
-A POSTROUTING_ZONES -o br0 -g POST_public
-A POSTROUTING_ZONES -g POST_public
-A POST_public -j POST_public_log
-A POST_public -j POST_public_deny
-A POST_public -j POST_public_allow
-A PREROUTING_ZONES -i ens33 -g PRE_public
-A PREROUTING_ZONES -i br0 -g PRE_public
-A PREROUTING_ZONES -g PRE_public
-A PREROUTING_direct -p tcp -m tcp --dport 222 -j DNAT --to-destination 10.1.0.9:22
-A PRE_public -j PRE_public_log
-A PRE_public -j PRE_public_deny
-A PRE_public -j PRE_public_allow
COMMIT
# Completed on Thu Aug 22 11:59:16 2019
# Generated by iptables-save v1.4.21 on Thu Aug 22 11:59:16 2019
*mangle
:PREROUTING ACCEPT [12:1319]
:INPUT ACCEPT [11:1259]
:FORWARD ACCEPT [1:60]
:OUTPUT ACCEPT [12:1070]
:POSTROUTING ACCEPT [12:1070]
:FORWARD_direct - [0:0]
:INPUT_direct - [0:0]
:OUTPUT_direct - [0:0]
:POSTROUTING_direct - [0:0]
:PREROUTING_ZONES - [0:0]
:PREROUTING_ZONES_SOURCE - [0:0]
:PREROUTING_direct - [0:0]
:PRE_public - [0:0]
:PRE_public_allow - [0:0]
:PRE_public_deny - [0:0]
:PRE_public_log - [0:0]
-A PREROUTING -j PREROUTING_direct
-A PREROUTING -j PREROUTING_ZONES_SOURCE
-A PREROUTING -j PREROUTING_ZONES
-A INPUT -j INPUT_direct
-A FORWARD -j FORWARD_direct
-A OUTPUT -j OUTPUT_direct
-A POSTROUTING -j POSTROUTING_direct
-A PREROUTING_ZONES -i ens33 -g PRE_public
-A PREROUTING_ZONES -i br0 -g PRE_public
-A PREROUTING_ZONES -g PRE_public
-A PRE_public -j PRE_public_log
-A PRE_public -j PRE_public_deny
-A PRE_public -j PRE_public_allow
COMMIT
# Completed on Thu Aug 22 11:59:16 2019
# Generated by iptables-save v1.4.21 on Thu Aug 22 11:59:16 2019
*security
:INPUT ACCEPT [11:1259]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [12:1070]
:FORWARD_direct - [0:0]
:INPUT_direct - [0:0]
:OUTPUT_direct - [0:0]
-A INPUT -j INPUT_direct
-A FORWARD -j FORWARD_direct
-A OUTPUT -j OUTPUT_direct
COMMIT
# Completed on Thu Aug 22 11:59:16 2019
# Generated by iptables-save v1.4.21 on Thu Aug 22 11:59:16 2019
*raw
:PREROUTING ACCEPT [12:1319]
:OUTPUT ACCEPT [12:1070]
:OUTPUT_direct - [0:0]
:PREROUTING_ZONES - [0:0]
:PREROUTING_ZONES_SOURCE - [0:0]
:PREROUTING_direct - [0:0]
:PRE_public - [0:0]
:PRE_public_allow - [0:0]
:PRE_public_deny - [0:0]
:PRE_public_log - [0:0]
-A PREROUTING -j PREROUTING_direct
-A PREROUTING -j PREROUTING_ZONES_SOURCE
-A PREROUTING -j PREROUTING_ZONES
-A OUTPUT -j OUTPUT_direct
-A PREROUTING_ZONES -i ens33 -g PRE_public
-A PREROUTING_ZONES -i br0 -g PRE_public
-A PREROUTING_ZONES -g PRE_public
-A PRE_public -j PRE_public_log
-A PRE_public -j PRE_public_deny
-A PRE_public -j PRE_public_allow
COMMIT
# Completed on Thu Aug 22 11:59:16 2019
# Generated by iptables-save v1.4.21 on Thu Aug 22 11:59:16 2019
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:FORWARD_IN_ZONES - [0:0]
:FORWARD_IN_ZONES_SOURCE - [0:0]
:FORWARD_OUT_ZONES - [0:0]
:FORWARD_OUT_ZONES_SOURCE - [0:0]
:FORWARD_direct - [0:0]
:FWDI_public - [0:0]
:FWDI_public_allow - [0:0]
:FWDI_public_deny - [0:0]
:FWDI_public_log - [0:0]
:FWDO_public - [0:0]
:FWDO_public_allow - [0:0]
:FWDO_public_deny - [0:0]
:FWDO_public_log - [0:0]
:INPUT_ZONES - [0:0]
:INPUT_ZONES_SOURCE - [0:0]
:INPUT_direct - [0:0]
:IN_public - [0:0]
:IN_public_allow - [0:0]
:IN_public_deny - [0:0]
:IN_public_log - [0:0]
:OUTPUT_direct - [0:0]
-A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -j INPUT_direct
-A INPUT -j INPUT_ZONES_SOURCE
-A INPUT -j INPUT_ZONES
-A INPUT -m conntrack --ctstate INVALID -j DROP
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -d 10.1.0.0/24 -o virbr0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -s 10.1.0.0/24 -i virbr0 -j ACCEPT
-A FORWARD -i virbr0 -o virbr0 -j ACCEPT
-A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i lo -j ACCEPT
-A FORWARD -j FORWARD_direct
-A FORWARD -j FORWARD_IN_ZONES_SOURCE
-A FORWARD -j FORWARD_IN_ZONES
-A FORWARD -j FORWARD_OUT_ZONES_SOURCE
-A FORWARD -j FORWARD_OUT_ZONES
-A FORWARD -m conntrack --ctstate INVALID -j DROP
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
-A OUTPUT -o virbr0 -p udp -m udp --dport 68 -j ACCEPT
-A OUTPUT -j OUTPUT_direct
-A FORWARD_IN_ZONES -i ens33 -g FWDI_public
-A FORWARD_IN_ZONES -i br0 -g FWDI_public
-A FORWARD_IN_ZONES -g FWDI_public
-A FORWARD_OUT_ZONES -o ens33 -g FWDO_public
-A FORWARD_OUT_ZONES -o br0 -g FWDO_public
-A FORWARD_OUT_ZONES -g FWDO_public
-A FORWARD_direct -j ACCEPT
-A FWDI_public -j FWDI_public_log
-A FWDI_public -j FWDI_public_deny
-A FWDI_public -j FWDI_public_allow
-A FWDI_public -p icmp -j ACCEPT
-A FWDO_public -j FWDO_public_log
-A FWDO_public -j FWDO_public_deny
-A FWDO_public -j FWDO_public_allow
-A INPUT_ZONES -i ens33 -g IN_public
-A INPUT_ZONES -i br0 -g IN_public
-A INPUT_ZONES -g IN_public
-A INPUT_direct -j ACCEPT
-A IN_public -j IN_public_log
-A IN_public -j IN_public_deny
-A IN_public -j IN_public_allow
-A IN_public -p icmp -j ACCEPT
-A IN_public_allow -p tcp -m tcp --dport 22 -m conntrack --ctstate NEW -j ACCEPT
-A OUTPUT_direct -j ACCEPT
COMMIT
# Completed on Thu Aug 22 11:59:16 2019

iptables-save - DON'T WORK TOO...

[root@localhost ~]# iptables-save
# Generated by iptables-save v1.4.21 on Thu Aug 22 11:59:36 2019
*nat
:PREROUTING ACCEPT [5:371]
:INPUT ACCEPT [1:67]
:OUTPUT ACCEPT [2:134]
:POSTROUTING ACCEPT [2:134]
:OUTPUT_direct - [0:0]
:POSTROUTING_ZONES - [0:0]
:POSTROUTING_ZONES_SOURCE - [0:0]
:POSTROUTING_direct - [0:0]
:POST_public - [0:0]
:POST_public_allow - [0:0]
:POST_public_deny - [0:0]
:POST_public_log - [0:0]
:PREROUTING_ZONES - [0:0]
:PREROUTING_ZONES_SOURCE - [0:0]
:PREROUTING_direct - [0:0]
:PRE_public - [0:0]
:PRE_public_allow - [0:0]
:PRE_public_deny - [0:0]
:PRE_public_log - [0:0]
-A PREROUTING -j PREROUTING_direct
-A PREROUTING -j PREROUTING_ZONES_SOURCE
-A PREROUTING -j PREROUTING_ZONES
-A OUTPUT -j OUTPUT_direct
-A POSTROUTING -s 10.1.0.0/24 -d 224.0.0.0/24 -j RETURN
-A POSTROUTING -s 10.1.0.0/24 -d 255.255.255.255/32 -j RETURN
-A POSTROUTING -s 10.1.0.0/24 ! -d 10.1.0.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 10.1.0.0/24 ! -d 10.1.0.0/24 -p udp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 10.1.0.0/24 ! -d 10.1.0.0/24 -j MASQUERADE
-A POSTROUTING -j POSTROUTING_direct
-A POSTROUTING -j POSTROUTING_ZONES_SOURCE
-A POSTROUTING -j POSTROUTING_ZONES
-A POSTROUTING_ZONES -o ens33 -g POST_public
-A POSTROUTING_ZONES -o br0 -g POST_public
-A POSTROUTING_ZONES -g POST_public
-A POST_public -j POST_public_log
-A POST_public -j POST_public_deny
-A POST_public -j POST_public_allow
-A PREROUTING_ZONES -i ens33 -g PRE_public
-A PREROUTING_ZONES -i br0 -g PRE_public
-A PREROUTING_ZONES -g PRE_public
-A PRE_public -j PRE_public_log
-A PRE_public -j PRE_public_deny
-A PRE_public -j PRE_public_allow
-A PRE_public_allow -p tcp -m mark --mark 0x64 -j DNAT --to-destination 10.1.0.9:22
COMMIT
# Completed on Thu Aug 22 11:59:36 2019
# Generated by iptables-save v1.4.21 on Thu Aug 22 11:59:36 2019
*mangle
:PREROUTING ACCEPT [17:1649]
:INPUT ACCEPT [12:1285]
:FORWARD ACCEPT [5:364]
:OUTPUT ACCEPT [10:3037]
:POSTROUTING ACCEPT [14:3341]
:FORWARD_direct - [0:0]
:INPUT_direct - [0:0]
:OUTPUT_direct - [0:0]
:POSTROUTING_direct - [0:0]
:PREROUTING_ZONES - [0:0]
:PREROUTING_ZONES_SOURCE - [0:0]
:PREROUTING_direct - [0:0]
:PRE_public - [0:0]
:PRE_public_allow - [0:0]
:PRE_public_deny - [0:0]
:PRE_public_log - [0:0]
-A PREROUTING -j PREROUTING_direct
-A PREROUTING -j PREROUTING_ZONES_SOURCE
-A PREROUTING -j PREROUTING_ZONES
-A INPUT -j INPUT_direct
-A FORWARD -j FORWARD_direct
-A OUTPUT -j OUTPUT_direct
-A POSTROUTING -j POSTROUTING_direct
-A PREROUTING_ZONES -i ens33 -g PRE_public
-A PREROUTING_ZONES -i br0 -g PRE_public
-A PREROUTING_ZONES -g PRE_public
-A PRE_public -j PRE_public_log
-A PRE_public -j PRE_public_deny
-A PRE_public -j PRE_public_allow
-A PRE_public_allow -p tcp -m tcp --dport 222 -j MARK --set-xmark 0x64/0xffffffff
COMMIT
# Completed on Thu Aug 22 11:59:36 2019
# Generated by iptables-save v1.4.21 on Thu Aug 22 11:59:36 2019
*security
:INPUT ACCEPT [12:1285]
:FORWARD ACCEPT [4:304]
:OUTPUT ACCEPT [10:3037]
:FORWARD_direct - [0:0]
:INPUT_direct - [0:0]
:OUTPUT_direct - [0:0]
-A INPUT -j INPUT_direct
-A FORWARD -j FORWARD_direct
-A OUTPUT -j OUTPUT_direct
COMMIT
# Completed on Thu Aug 22 11:59:36 2019
# Generated by iptables-save v1.4.21 on Thu Aug 22 11:59:36 2019
*raw
:PREROUTING ACCEPT [17:1649]
:OUTPUT ACCEPT [10:3037]
:OUTPUT_direct - [0:0]
:PREROUTING_ZONES - [0:0]
:PREROUTING_ZONES_SOURCE - [0:0]
:PREROUTING_direct - [0:0]
:PRE_public - [0:0]
:PRE_public_allow - [0:0]
:PRE_public_deny - [0:0]
:PRE_public_log - [0:0]
-A PREROUTING -j PREROUTING_direct
-A PREROUTING -j PREROUTING_ZONES_SOURCE
-A PREROUTING -j PREROUTING_ZONES
-A OUTPUT -j OUTPUT_direct
-A PREROUTING_ZONES -i ens33 -g PRE_public
-A PREROUTING_ZONES -i br0 -g PRE_public
-A PREROUTING_ZONES -g PRE_public
-A PRE_public -j PRE_public_log
-A PRE_public -j PRE_public_deny
-A PRE_public -j PRE_public_allow
COMMIT
# Completed on Thu Aug 22 11:59:36 2019
# Generated by iptables-save v1.4.21 on Thu Aug 22 11:59:36 2019
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [8:2813]
:FORWARD_IN_ZONES - [0:0]
:FORWARD_IN_ZONES_SOURCE - [0:0]
:FORWARD_OUT_ZONES - [0:0]
:FORWARD_OUT_ZONES_SOURCE - [0:0]
:FORWARD_direct - [0:0]
:FWDI_public - [0:0]
:FWDI_public_allow - [0:0]
:FWDI_public_deny - [0:0]
:FWDI_public_log - [0:0]
:FWDO_public - [0:0]
:FWDO_public_allow - [0:0]
:FWDO_public_deny - [0:0]
:FWDO_public_log - [0:0]
:INPUT_ZONES - [0:0]
:INPUT_ZONES_SOURCE - [0:0]
:INPUT_direct - [0:0]
:IN_public - [0:0]
:IN_public_allow - [0:0]
:IN_public_deny - [0:0]
:IN_public_log - [0:0]
:OUTPUT_direct - [0:0]
-A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -j INPUT_direct
-A INPUT -j INPUT_ZONES_SOURCE
-A INPUT -j INPUT_ZONES
-A INPUT -m conntrack --ctstate INVALID -j DROP
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -d 10.1.0.0/24 -o virbr0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -s 10.1.0.0/24 -i virbr0 -j ACCEPT
-A FORWARD -i virbr0 -o virbr0 -j ACCEPT
-A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i lo -j ACCEPT
-A FORWARD -j FORWARD_direct
-A FORWARD -j FORWARD_IN_ZONES_SOURCE
-A FORWARD -j FORWARD_IN_ZONES
-A FORWARD -j FORWARD_OUT_ZONES_SOURCE
-A FORWARD -j FORWARD_OUT_ZONES
-A FORWARD -m conntrack --ctstate INVALID -j DROP
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
-A OUTPUT -o virbr0 -p udp -m udp --dport 68 -j ACCEPT
-A OUTPUT -j OUTPUT_direct
-A FORWARD_IN_ZONES -i ens33 -g FWDI_public
-A FORWARD_IN_ZONES -i br0 -g FWDI_public
-A FORWARD_IN_ZONES -g FWDI_public
-A FORWARD_OUT_ZONES -o ens33 -g FWDO_public
-A FORWARD_OUT_ZONES -o br0 -g FWDO_public
-A FORWARD_OUT_ZONES -g FWDO_public
-A FWDI_public -j FWDI_public_log
-A FWDI_public -j FWDI_public_deny
-A FWDI_public -j FWDI_public_allow
-A FWDI_public -p icmp -j ACCEPT
-A FWDI_public_allow -m conntrack --ctstate NEW -m mark --mark 0x64 -j ACCEPT
-A FWDO_public -j FWDO_public_log
-A FWDO_public -j FWDO_public_deny
-A FWDO_public -j FWDO_public_allow
-A INPUT_ZONES -i ens33 -g IN_public
-A INPUT_ZONES -i br0 -g IN_public
-A INPUT_ZONES -g IN_public
-A IN_public -j IN_public_log
-A IN_public -j IN_public_deny
-A IN_public -j IN_public_allow
-A IN_public -p icmp -j ACCEPT
-A IN_public_allow -p tcp -m tcp --dport 22 -m conntrack --ctstate NEW -j ACCEPT
COMMIT
# Completed on Thu Aug 22 11:59:36 2019

UPDATE II:

The first rule in FORWARD of "works" is ACCEPT. That allows every packet to be forwarded. The others have rules to accept DNATted packets, but later in the chain. So... We may be able to solve the problem if we find out why this works...

iptables -I FORWARD -d 0.0.0.0/0 -j ACCEPT
firewall-cmd --permanent --zone=public --add-forward-port=port=222:proto=tcp:toport=22:toaddr=10.1.0.9
firewall-cmd --reload

... and why this not...

firewall-cmd --permanent --zone=public --direct --add-rule ipv4 filter FORWARD 0 -d 0.0.0.0/0 -j ACCEPT
firewall-cmd --permanent --zone=public --add-forward-port=port=222:proto=tcp:toport=22:toaddr=10.1.0.9
firewall-cmd --reload
Improve this question
0

1 Answer 1

Reset to default
13

THE REAL PROBLEM:

The command...

firewall-cmd --permanent --zone=public --direct --add-rule ipv4 filter FORWARD 0 -d 0.0.0.0/0 -j ACCEPT
firewall-cmd --permanent --zone=public --add-forward-port=port=222:proto=tcp:toport=22:toaddr=10.1.0.9
firewall-cmd --reload

... described in "UPDATE II" works just as well as firewalld (used by CentOS 7).

The problem is in the KVM (libvirt) which when starting the "network" virbr0 injects rules (probably via firewalld) which prevents (first match principle) any attempt to port forwarding by conventional ways using firewall-cmd. The KVM injected (libvirt) rules are at the top of FORWARD and POSTROUTING, before regular firewalld ruleset. Since the netfilter operates on first match principle, these rules.

SOLUTION:

Due to the problems generated by the rules injected by KVM (libvirt) into the firewall the solution is use a hack/workaround called libvirt-hook-qemu as we explained below...

Download and install libvirt-hook-qemu...

yum -y install git-core
cd /usr/local/src
git clone https://github.com/saschpe/libvirt-hook-qemu.git
cd libvirt-hook-qemu
make install
cd ..
rm -rf libvirt-hook-qemu

List your virtual machines (guests)...

virsh list --all

... and copy the name of the target virtual machine.

Turn off the virtual machine...

virsh shutdown "<VM_NAME>" --mode acpi

Open the file "hooks.json", delete its contents...

vi /etc/libvirt/hooks/hooks.json

... and enter your rules as the template below (other ways of setting up are possible)...

{
    "<VM_NAME>": {
        "interface": "virbr0",
        "private_ip": "<VM_IP>",
        "port_map": {
            "tcp": [[<PORT_ON_HOST>, <PORT_ON_GUEST>]]
        }
    }
}

Restart libvirtd...

systemctl restart libvirtd.service

... and start the virtual machine...

virsh start "<VM_NAME>"

Done! =D

FURTHER:

This KVM (libvirt) behavior - injects rules which prevents any attempt to port forwarding - is IMPOSITIVE, INVASIVE AND UNNECESSARY... First because since there is no port forwarding on the host no virtual machine will be reachable on the default network (virbr0) and second because it forces us to use a hack/workaround (or "gato"/"gambiarra" as we call it in Brazil =D ).

In our view makes no sense impose the user how he should build his infrastructure or not... This makes no sense! To put it bluntly, we think it would be best if these settings were optional (injects rules...) as other leading hypervisors do. And we go further... If is to KVM (libvirt) controlling this kind of thing then it should also offer ways to map ports to virtual machines.

The KVM (libvirt) is a GREAT tool, but not all users are willing - for various reasons - to place their machines directly on the host machine's network just as not all users are willing to give up all the flexibility/accessibility that the default network (virbr0) offers.

Thanks!

[Refs.: https://bugzilla.redhat.com/show_bug.cgi?id=846810 , https://libvirt.org/firewall.html , https://libvirt.org/formatnetwork.html#examplesRoute, https://saschpe.wordpress.com/2014/03/06/dynamic-iptables-port-forwarding-for-nat-ed-libvirt-networks/ , https://serverfault.com/q/170079/276753 , https://serverfault.com/q/915257/276753 , https://wiki.libvirt.org/page/VirtualNetworking#Routed_mode , https://www.centos.org/forums/viewtopic.php?f=50&t=71454&sid=7f5190a29eebbf755235ec3dc404a47f , https://www.cyberciti.biz/faq/kvm-forward-ports-to-guests-vm-with-ufw-on-linux/ ]

UPDATE:

injects rules which prevents any attempt to port forwarding

-> The current upstream of KVM (libvirt) has resolved the issue!

See this: ...current upstream of KVM (libvirt) has resolved the issue...

Improve this answer
2
  • unfortunately I received a Error starting stopped VM: virError(Code=70, Domain=34, Message='Hook script execution failed: internal error: Child process (LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin /etc/libvirt/hooks/qemu crc prepare begin -) unexpected exit status 127: libvirt: error : cannot execute binary /etc/libvirt/hooks/qemu: No such file or directory ') after following your SOLUTION after executing command virsh start
    – Michel
    Dec 16, 2019 at 20:30
  • Hook script execution failed -> I've had this kind of problem because of a JSON formatting error on /etc/libvirt/hooks/hooks.json. =D
    – Eduardo Lucio
    Dec 17, 2019 at 13:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.