1

Linux Gentoo 3.13.6-hardened-r3 #1 SMP Sat Apr 12 09:17:25 EDT 2014 x86_64 Intel(R) Core(TM)2 CPU 6300 @ 1.86GHz GenuineIntel GNU/Linux hardened with grsecurity and selinux with a strict policy in enforcing mode

These problems existed on Slackware 64 14.0, I moved to gentoo because Slackware doesn't support many security features, like selinux, app armor, grsecurity, etc.

The internal box is Windows 7.

I have a problem with UDP and NAT specifically with MAQUERADING AND FORWARDING. Whenever I have UDP traffic to a specific port, if I shut down the service or close the port too long when I bring the service back up I get floods of connections coming into port >=1024 instead of the port that the service runs on. One packet goes out the original port from the masqueraded ip and returns on a different port on the external ip but the packet also somehow gets routed to the correct port on the masqueraded ip.

There are udp connections coming from Windows 7 on both the original service port and port >=1024 but this may be because the bad connections/configuration from the linux box poisoned the service on the windows box. So, I've blocked all outgoing traffic from the service on the windows box on port 1024/1025 because that's where the bulk of the traffic lies.

But the problem still exists. I've been trying to hunt this down for over a month. I posted this on serverfault.com where my question might have been answered. But they transferred it to superuser.com where it never got answered.

https://superuser.com/questions/718860/why-does-iptables-suddenly-stop-properly-forwarding-packets

Then tcpdump reported the connections were being sent from my external ip on port >=1024 when they should have been getting sent from the original port. However some of the connections to different ips were being sent out on the correct port. I can only guess these ip's that were working correctly were new connections that came in after the service was restarted.

These are the relevant iptables rules:

$IPT -t nat -A PREROUTING -i $EXTIF -p udp -m udp --dport 5555 -j DNAT --to-destination $WIN_IP:5555
$IPT -t nat -A POSTROUTING -o $EXTIF -j MASQUERADE

$IPT -A FORWARD -i $EXTIF -o $INTIF -m state --state ESTABLISHED,RELATED -j ACCEPT
$IPT -A FORWARD -i $INTIF -o $EXTIF -j DEFAULT_FWD_OUT
$IPT -A FORWARD -i $EXTIF -o $INTIF -j DEFAULT_FWD_IN

# To and from my internal masqueraded ip

$IPT -A DEFAULT_FWD_IN -p icmp -m icmp --icmp-type 12 -j ACCEPT
$IPT -A DEFAULT_FWD_IN -p icmp -m icmp --icmp-type 3 -j ACCEPT
$IPT -A DEFAULT_FWD_IN -p icmp -m icmp --icmp-type 0/0 -j ACCEPT
$IPT -A DEFAULT_FWD_IN -p icmp -m icmp --icmp-type 11/0 -j ACCEPT
$IPT -A DEFAULT_FWD_IN -p icmp -m icmp --icmp-type 11/1 -j ACCEPT
$IPT -A DEFAULT_FWD_IN -p udp -m udp --dport 5555 -j ACCEPT

$IPT -A DEFAULT_FWD_OUT -p icmp -m icmp --icmp-type 8/0 -j ACCEPT
$IPT -A DEFAULT_FWD_OUT -p tcp -m tcp -m multiport --dports 21,80,8080,443,143,993,110,995,25,465 -j ACCEPT
$IPT -A DEFAULT_FWD_OUT -p udp -m udp -m multiport --sports 5555,123,53 -j ACCEPT

# To and from my external ip

$IPT -A DEFAULT_IN -p icmp -m icmp --icmp-type 12 -j ACCEPT
$IPT -A DEFAULT_IN -p icmp -m icmp --icmp-type 3 -j ACCEPT
$IPT -A DEFAULT_IN -p icmp -m icmp --icmp-type 0/0 -j ACCEPT
$IPT -A DEFAULT_IN -p icmp -m icmp --icmp-type 11/0 -j ACCEPT
$IPT -A DEFAULT_IN -p icmp -m icmp --icmp-type 11/1 -j ACCEPT

$IPT -A DEFAULT_OUT -p icmp -m icmp --icmp-type 8/0 -j ACCEPT
$IPT -A DEFAULT_OUT -p tcp -m tcp -m multiport --dports 21,80,443,143,993,110,995,25,465 -j ACCEPT
$IPT -A DEFAULT_OUT -p udp -m udp -m multiport --dports 53,123 -j ACCEPT

Conntrack shows the outgoing source port as port 5555 and expects it to return on port 1025.

[UPDATE] udp      17 30 src=MASQUERADED_INTERNAL_IP dst=EXTERNAL_DST_IP sport=5555 dport=39363 src=EXTERNAL_DST_IP dst=MY_EXTERNAL_IP sport=39363 dport=1025
[UPDATE] udp      17 180 src=MASQUERADED_INTERNAL_IP dst=EXTERNAL_DST_IP sport=5555 dport=39363 src=EXTERNAL_DST_IP dst=MY_EXTERNAL_IP sport=39363 dport=1025 [ASSURED]

Tcpdump confirms that there are outgoing connections from my ip with a source of 1025 when the source should be port 1640. So it's understandle the remote ip's would want to communicate on port 1025.

21:13:49.191821 IP (tos 0x0, ttl 63, id 4600, offset 0, flags [none], proto UDP (17), length 230)
MY_EXTERNAL_IP.1025 > EXTERNAL_DST_IP.39363: [udp sum ok] UDP, length 202
21:13:51.726037 IP (tos 0x0, ttl 63, id 4679, offset 0, flags [none], proto UDP (17), length 354)
MY_EXTERNAL_IP.1025 > EXTERNAL_DST_IP.39363: [udp sum ok] UDP, length 326
21:13:51.930653 IP (tos 0x0, ttl 63, id 4686, offset 0, flags [none], proto UDP (17), length 538)
MY_EXTERNAL_IP.1025 > EXTERNAL_DST_IP.39363: [udp sum ok] UDP, length 510
21:13:52.193349 IP (tos 0x0, ttl 63, id 4694, offset 0, flags [none], proto UDP (17), length 515)
MY_EXTERNAL_IP.1025 > EXTERNAL_DST_IP.39363: [udp sum ok] UDP, length 487
21:13:53.415424 IP (tos 0x0, ttl 63, id 4724, offset 0, flags [none], proto UDP (17), length 539)
MY_EXTERNAL_IP.1025 > EXTERNAL_DST_IP.39363: [udp sum ok] UDP, length 511
21:13:53.686204 IP (tos 0x0, ttl 63, id 4793, offset 0, flags [none], proto UDP (17), length 106)
MY_EXTERNAL_IP.1025 > EXTERNAL_DST_IP.39363: [udp sum ok] UDP, length 78
21:13:56.713590 IP (tos 0x0, ttl 63, id 4847, offset 0, flags [none], proto UDP (17), length 105)
MY_EXTERNAL_IP.1025 > EXTERNAL_DST_IP.39363: [udp sum ok] UDP, length 77
21:13:58.097788 IP (tos 0x0, ttl 63, id 4935, offset 0, flags [none], proto UDP (17), length 107)
MY_EXTERNAL_IP.1025 > EXTERNAL_DST_IP.39363: [udp sum ok] UDP, length 79
21:13:59.754290 IP (tos 0x0, ttl 63, id 4992, offset 0, flags [none], proto UDP (17), length 210)
MY_EXTERNAL_IP.1025 > EXTERNAL_DST_IP.39363: [udp sum ok] UDP, length 182
21:14:01.644835 IP (tos 0x0, ttl 63, id 5024, offset 0, flags [none], proto UDP (17), length 97)
MY_EXTERNAL_IP.1025 > EXTERNAL_DST_IP.39363: [udp sum ok] UDP, length 69
21:14:01.860478 IP (tos 0x0, ttl 63, id 5062, offset 0, flags [none], proto UDP (17), length 104)
MY_EXTERNAL_IP.1025 > EXTERNAL_DST_IP.39363: [udp sum ok] UDP, length 76
21:14:05.633698 IP (tos 0x0, ttl 63, id 5154, offset 0, flags [none], proto UDP (17), length 111)
MY_EXTERNAL_IP.1025 > EXTERNAL_DST_IP.39363: [udp sum ok] UDP, length 83
21:14:14.950748 IP (tos 0x0, ttl 63, id 5309, offset 0, flags [none], proto UDP (17), length 100)
MY_EXTERNAL_IP.1025 > EXTERNAL_DST_IP.39363: [udp sum ok] UDP, length 72
21:14:16.370384 IP (tos 0x0, ttl 63, id 5377, offset 0, flags [none], proto UDP (17), length 115)
MY_EXTERNAL_IP.1025 > EXTERNAL_DST_IP.39363: [udp sum ok] UDP, length 87
21:14:20.720215 IP (tos 0x0, ttl 63, id 5542, offset 0, flags [none], proto UDP (17), length 111)
MY_EXTERNAL_IP.1025 > EXTERNAL_DST_IP.39363: [udp sum ok] UDP, length 83
21:14:24.528689 IP (tos 0x0, ttl 63, id 5624, offset 0, flags [none], proto UDP (17), length 174)
MY_EXTERNAL_IP.1025 > EXTERNAL_DST_IP.39363: [udp sum ok] UDP, length 146
21:14:28.783134 IP (tos 0x0, ttl 63, id 5760, offset 0, flags [none], proto UDP (17), length 123)
MY_EXTERNAL_IP.1025 > EXTERNAL_DST_IP.39363: [udp sum ok] UDP, length 95
21:14:30.276222 IP (tos 0x0, ttl 63, id 5823, offset 0, flags [none], proto UDP (17), length 108)
MY_EXTERNAL_IP.1025 > EXTERNAL_DST_IP.39363: [udp sum ok] UDP, length 80
21:14:33.646507 IP (tos 0x0, ttl 63, id 5989, offset 0, flags [none], proto UDP (17), length 106)
MY_EXTERNAL_IP.1025 > EXTERNAL_DST_IP.39363: [udp sum ok] UDP, length 78
21:14:34.826793 IP (tos 0x0, ttl 63, id 6024, offset 0, flags [none], proto UDP (17), length 95)
MY_EXTERNAL_IP.1025 > EXTERNAL_DST_IP.39363: [udp sum ok] UDP, length 67
21:14:36.203849 IP (tos 0x0, ttl 63, id 6039, offset 0, flags [none], proto UDP (17), length 239)
MY_EXTERNAL_IP.1025 > EXTERNAL_DST_IP.39363: [udp sum ok] UDP, length 211
21:14:37.693874 IP (tos 0x0, ttl 63, id 6073, offset 0, flags [none], proto UDP (17), length 207)
MY_EXTERNAL_IP.1025 > EXTERNAL_DST_IP.39363: [udp sum ok] UDP, length 179
21:14:37.935895 IP (tos 0x0, ttl 63, id 6086, offset 0, flags [none], proto UDP (17), length 103)
MY_EXTERNAL_IP.1025 > EXTERNAL_DST_IP.39363: [udp sum ok] UDP, length 75
21:14:39.445114 IP (tos 0x0, ttl 63, id 6100, offset 0, flags [none], proto UDP (17), length 95)
MY_EXTERNAL_IP.1025 > EXTERNAL_DST_IP.39363: [udp sum ok] UDP, length 67
21:14:39.554406 IP (tos 0x0, ttl 63, id 6114, offset 0, flags [none], proto UDP (17), length 123)
MY_EXTERNAL_IP.1025 > EXTERNAL_DST_IP.39363: [udp sum ok] UDP, length 95
21:14:39.778376 IP (tos 0x0, ttl 63, id 6132, offset 0, flags [none], proto UDP (17), length 97)
MY_EXTERNAL_IP.1025 > EXTERNAL_DST_IP.39363: [udp sum ok] UDP, length 69
21:14:46.593156 IP (tos 0x0, ttl 63, id 6329, offset 0, flags [none], proto UDP (17), length 1212)
MY_EXTERNAL_IP.1025 > EXTERNAL_DST_IP.39363: [udp sum ok] UDP, length 1184
21:14:46.595169 IP (tos 0x0, ttl 63, id 6330, offset 0, flags [none], proto UDP (17), length 1212)
MY_EXTERNAL_IP.1025 > EXTERNAL_DST_IP.39363: [udp sum ok] UDP, length 1184
21:14:46.597708 IP (tos 0x0, ttl 63, id 6331, offset 0, flags [none], proto UDP (17), length 1212)
MY_EXTERNAL_IP.1025 > EXTERNAL_DST_IP.39363: [udp sum ok] UDP, length 1184
21:14:46.600152 IP (tos 0x0, ttl 63, id 6332, offset 0, flags [none], proto UDP (17), length 969)
MY_EXTERNAL_IP.1025 > EXTERNAL_DST_IP.39363: [udp sum ok] UDP, length 941
21:14:48.049739 IP (tos 0x0, ttl 63, id 6375, offset 0, flags [none], proto UDP (17), length 1212)
MY_EXTERNAL_IP.1025 > EXTERNAL_DST_IP.39363: [udp sum ok] UDP, length 1184
21:14:48.052057 IP (tos 0x0, ttl 63, id 6376, offset 0, flags [none], proto UDP (17), length 1212)
MY_EXTERNAL_IP.1025 > EXTERNAL_DST_IP.39363: [udp sum ok] UDP, length 1184
21:14:48.054117 IP (tos 0x0, ttl 63, id 6377, offset 0, flags [none], proto UDP (17), length 1212)
MY_EXTERNAL_IP.1025 > EXTERNAL_DST_IP.39363: [udp sum ok] UDP, length 1184
21:14:48.056132 IP (tos 0x0, ttl 63, id 6378, offset 0, flags [none], proto UDP (17), length 866)
MY_EXTERNAL_IP.1025 > EXTERNAL_DST_IP.39363: [udp sum ok] UDP, length 838
21:14:48.239566 IP (tos 0x0, ttl 63, id 6400, offset 0, flags [none], proto UDP (17), length 1212)
MY_EXTERNAL_IP.1025 > EXTERNAL_DST_IP.39363: [udp sum ok] UDP, length 1184
21:14:48.241738 IP (tos 0x0, ttl 63, id 6401, offset 0, flags [none], proto UDP (17), length 1212)
MY_EXTERNAL_IP.1025 > EXTERNAL_DST_IP.39363: [udp sum ok] UDP, length 1184
21:14:48.244361 IP (tos 0x0, ttl 63, id 6402, offset 0, flags [none], proto UDP (17), length 1212)
MY_EXTERNAL_IP.1025 > EXTERNAL_DST_IP.39363: [udp sum ok] UDP, length 1184
21:14:48.247134 IP (tos 0x0, ttl 63, id 6403, offset 0, flags [none], proto UDP (17), length 1212)
MY_EXTERNAL_IP.1025 > EXTERNAL_DST_IP.39363: [udp sum ok] UDP, length 1184
21:14:48.249168 IP (tos 0x0, ttl 63, id 6404, offset 0, flags [none], proto UDP (17), length 737)
MY_EXTERNAL_IP.1025 > EXTERNAL_DST_IP.39363: [udp sum ok] UDP, length 709
21:14:48.258415 IP (tos 0x0, ttl 63, id 6405, offset 0, flags [none], proto UDP (17), length 158)
21:14:49.816682 IP (tos 0x0, ttl 63, id 6429, offset 0, flags [none], proto UDP (17), length 1212)
MY_EXTERNAL_IP.1025 > EXTERNAL_DST_IP.39363: [udp sum ok] UDP, length 1184
21:14:49.819281 IP (tos 0x0, ttl 63, id 6430, offset 0, flags [none], proto UDP (17), length 1212)
MY_EXTERNAL_IP.1025 > EXTERNAL_DST_IP.39363: [udp sum ok] UDP, length 1184
21:14:49.821505 IP (tos 0x0, ttl 63, id 6431, offset 0, flags [none], proto UDP (17), length 1212)
MY_EXTERNAL_IP.1025 > EXTERNAL_DST_IP.39363: [udp sum ok] UDP, length 1184
21:14:49.823571 IP (tos 0x0, ttl 63, id 6432, offset 0, flags [none], proto UDP (17), length 1212)
MY_EXTERNAL_IP.1025 > EXTERNAL_DST_IP.39363: [udp sum ok] UDP, length 1184
21:14:49.825720 IP (tos 0x0, ttl 63, id 6433, offset 0, flags [none], proto UDP (17), length 737)
MY_EXTERNAL_IP.1025 > EXTERNAL_DST_IP.39363: [udp sum ok] UDP, length 709
21:14:50.214646 IP (tos 0x0, ttl 63, id 6458, offset 0, flags [none], proto UDP (17), length 1212)
MY_EXTERNAL_IP.1025 > EXTERNAL_DST_IP.39363: [udp sum ok] UDP, length 1184
21:14:50.217102 IP (tos 0x0, ttl 63, id 6459, offset 0, flags [none], proto UDP (17), length 1212)
MY_EXTERNAL_IP.1025 > EXTERNAL_DST_IP.39363: [udp sum ok] UDP, length 1184
21:14:50.219476 IP (tos 0x0, ttl 63, id 6460, offset 0, flags [none], proto UDP (17), length 1212)
MY_EXTERNAL_IP.1025 > EXTERNAL_DST_IP.39363: [udp sum ok] UDP, length 1184
21:14:50.222532 IP (tos 0x0, ttl 63, id 6461, offset 0, flags [none], proto UDP (17), length 1212)
MY_EXTERNAL_IP.1025 > EXTERNAL_DST_IP.39363: [udp sum ok] UDP, length 1184
21:14:50.224647 IP (tos 0x0, ttl 63, id 6462, offset 0, flags [none], proto UDP (17), length 749)
MY_EXTERNAL_IP.1025 > EXTERNAL_DST_IP.39363: [udp sum ok] UDP, length 721




21:14:51.629383 IP (tos 0x0, ttl 63, id 6485, offset 0, flags [none], proto UDP (17), length 1212)
MY_EXTERNAL_IP.1025 > EXTERNAL_DST_IP.39363: [udp sum ok] UDP, length 1184
21:14:51.631733 IP (tos 0x0, ttl 63, id 6486, offset 0, flags [none], proto UDP (17), length 1212)
MY_EXTERNAL_IP.1025 > EXTERNAL_DST_IP.39363: [udp sum ok] UDP, length 1184
21:14:51.634302 IP (tos 0x0, ttl 63, id 6487, offset 0, flags [none], proto UDP (17), length 1212)
MY_EXTERNAL_IP.1025 > EXTERNAL_DST_IP.39363: [udp sum ok] UDP, length 1184
21:14:51.638055 IP (tos 0x0, ttl 63, id 6488, offset 0, flags [none], proto UDP (17), length 1212)
MY_EXTERNAL_IP.1025 > EXTERNAL_DST_IP.39363: [udp sum ok] UDP, length 1184
21:14:51.640863 IP (tos 0x0, ttl 63, id 6489, offset 0, flags [none], proto UDP (17), length 1212)
MY_EXTERNAL_IP.1025 > EXTERNAL_DST_IP.39363: [udp sum ok] UDP, length 1184
21:14:51.643918 IP (tos 0x0, ttl 63, id 6490, offset 0, flags [none], proto UDP (17), length 631)
MY_EXTERNAL_IP.1025 > EXTERNAL_DST_IP.39363: [udp sum ok] UDP, length 603
21:14:51.998529 IP (tos 0x0, ttl 63, id 6515, offset 0, flags [none], proto UDP (17), length 1212)
MY_EXTERNAL_IP.1025 > EXTERNAL_DST_IP.39363: [udp sum ok] UDP, length 1184
21:14:53.796813 IP (tos 0x0, ttl 63, id 6601, offset 0, flags [none], proto UDP (17), length 1212)
MY_EXTERNAL_IP.1025 > EXTERNAL_DST_IP.39363: [udp sum ok] UDP, length 1184
21:14:56.770203 IP (tos 0x0, ttl 63, id 6714, offset 0, flags [none], proto UDP (17), length 994)
MY_EXTERNAL_IP.1025 > EXTERNAL_DST_IP.39363: [udp sum ok] UDP, length 966

I get "many" UNREPLIED messages in conntrack

~# contrack -E
[DESTROY] udp      17 src=MASQUERADED_INTERNAL_IP dst=EXTERNAL_DST_IP sport=5555 dport=41575 [UNREPLIED] src=EXTERNAL_DST_IP dst=MY_EXTERNAL_IP sport=41575 dport=1025
[DESTROY] udp      17 src=EXTERNAL_DST_IP dst=MY_EXTERNAL_IP sport=16942 dport=5555 [UNREPLIED] src=MASQUERADED_INTERNAL_IP dst=EXTERNAL_DST_IP sport=5555 dport=1026

The masqueraded ip's ports should be the same in both the sport of the sent packet and the expected reply's dport. In this case the sport is 5555 but the connections are coming back on port >=1024

I set the mtu to 1300 on both Windows and Linux. That didn't stop it.

I'd expected that it was iptables or netfilter's innerworkings within the way it handles NAT and Masquerading and that iptables or netfilter forgot how to route the connections. But after thousands of google searches these two sites lead me to believe conntrack was the culprit.

http://www.linksysinfo.org/index.php?threads/vpn-build-with-web-gui.27511/page-21#post-131548

http://permalink.gmane.org/gmane.comp.file-systems.openafs.general/30256

This page seemed to confirm a suggestion that the UDP timeouts needed to be adjusted:

http://en.it-usenet.org/thread/11955/543/

But I tried changing the UDP timeouts:

~ # cat /proc/sys/net/netfilter/nf_conntrack_udp_timeout
30
~ # cat /proc/sys/net/netfilter/nf_conntrack_udp_timeout_stream
180


sysctl net.netfilter.nf_conntrack_udp_timeout_stream=28800
sysctl net.netfilter.nf_conntrack_udp_timeout=28800

This link suggested keepalives might also be part of the cause if udp timeouts are less than the keepalives there will be problems with the connection.

http://www.linksysinfo.org/index.php?threads/vpn-build-with-web-gui.27511/page-21

I haven't found anything yet on keepalives but I suspected the conntrack tables might have been dirty so I flushed the conntrack tables

conntrack -F

and rebooted. That stopped the flood of connections to port 1024/1025 but the connections were broken. So after getting everything working again. The problem resurfaced. The only thing that stopped iptables or conntrack or whatever handles outgoing connections from translating the original port to port >=1024 was to incessantly flush conntrack tables, and restart all network interfaces as just restarting eth0 causes failures on eth1 on Gentoo. I have to do this numerous amounts of times on the off chance that on one of the restarts everything will be just right. Additionaly, on Gentoo,

ifconfig eth0 down
ifconfig eth0 up

apparently destroys functionality in another service(s) I've been unable to hunt down. My guess is that on Gentoo it triggers dhcp to shut down. The proper way to restart an interface on Gentoo is to run a script which after looking at it I wasn't in the mood to decipher.

/etc/init.d/net.eth0 restart

This forces many of my services, some which take forever to start, to be stopped because of some dependency in the above script. Troubleshooting this is very time-consuming because of that script or should I say because of the services which depend on that script, like snort and barnyard2, both of which take forever to both stop and start.

On Slackware I could just do, ifdown eth0, ifup eth0 done. No need to restart eth1 or reboot the PC or restart snort or barnyard.

While I sort of solved my own problem the problem will return when I decide its OK to run utorrent again. I stopped running utorrent because of the problems I tried to resolve here:

http://www.bleepingcomputer.com/forums/t/526523/linux-pc-is-under-some-kind-of-whois-dos-attack/

http://www.bleepingcomputer.com/forums/t/526524/utorrent-seems-to-be-vulnerable-to-dos-attacks/

Essentially when I run it on the Windows PC there are attempts to connect to IANA addresses. And after stopping utorrent for a while attempts are coming in to port 1024/1025, etc which the entire internet universe seems to think that the bittorrent protocol or the software uses that port. While that may be true, there are "also" connections which are supposed to be going to the port that the service for bittorrent runs on and while restarting bittorrent to resume the connections should start routing the connections back to the desired port only the new connections get routed to it and all of the old connections are stuck in a NAT blackhole.

There is either something wrong with my setup or there is something wrong with NAT/Conntrack. My guess is both. That's not to mention the plethora of destination unreachables I've been plagued with. Perhaps it's related. Can anyone see what I might be doing wrong with iptables? Perhaps opening up all icmp will solve the problem or a specific type and code?

1
  • Such issue can happen in possible race condition in conntrack module. To confirm one needs to monitor conntrack table enteries for the first second issue is seen. Alternatively issue can occur due to some dead entry in conntrack. search tupple clash fro more data on this
    – Vivek
    Apr 30, 2014 at 4:59

0

You must log in to answer this question.

Browse other questions tagged .