1

I am using socat to forward UDP traffic from one port to another with the following command:

socat -T5 UDP4-LISTEN:12345,reuseaddr,fork UDP4:127.0.0.1:23456,bind=127.0.0.2,so-bindtodevice=lo

Using netcat to test this, I ran nc -u -l -p 23456 -vv and echo test | nc -u 127.0.0.1 12345. This gave me the following unexpected output:

listening on [any] 23456 ...
connect to [<myPublicIp>] from <hostname> [<myPublicIp>] 53995
test

Instead of seeing <myPublicIp> as the source address, I would have expected to see 127.0.0.2. Am I misunderstanding something about networking, or am I using socat incorrectly?

The affected server is running Debian 10, Linux kernel version is 4.19.0-13-amd64, socat version is 1.7.3.2.

EDIT: When using TCP, the output is slightly different:

listening on [any] 23456 ...
connect to [127.0.0.1] from <hostname> [<myPublicIp>] 48520
test

As requested, the relevant output of netstat -aun

Proto Recv-Q Send-Q Local Address           Foreign Address         State  
udp        0      0 0.0.0.0:12345           0.0.0.0:*
udp        0      0 <myPublicIp>:23456      <myPublicIp>:54299      ESTABLISHED
udp        0      0 127.0.0.1:33156         127.0.0.1:12345         ESTABLISHED

EDIT 2: I can confirm that it doesn't happen on a fresh Debian install. Additionally, I've tried a few other things:

Creating another interface using tunctl, and making socat use that interface. The commands executed are:

  1. tunctl
  2. ifconfig tap1 192.168.10.1 netmask 255.255.255.255 up
  3. socat -T5 UDP4-LISTEN:12345,reuseaddr,fork UDP4:127.0.0.1:23456,bind=192.168.10.1,so-bindtodevice=tap1

The behavior I observed is exactly as the UDP example above; all requests appear to originate from <myPublicIp>.

Additionally, I tried using this approach to create more loopback addresses.

  1. ifconfig lo:10 192.168.10.1 netmask 255.255.255.0 up
  2. socat -T5 UDP4-LISTEN:12345,reuseaddr,fork UDP4:127.0.0.1:23456,bind=192.168.10.1

This actually worked, requests seem to originate from 192.168.10.1. Using

socat -T5 UDP4-LISTEN:12345,reuseaddr,fork UDP4:127.0.0.1:23456,bind=192.168.10.2

(different bind address) doesn't work, requests now originate from 192.168.10.1.

Creating multiple aliases for lo doesn't work either.

  1. ifconfig lo:1 192.168.10.1 netmask 255.255.255.255 up
  2. ifconfig lo:2 192.168.10.2 netmask 255.255.255.255 up
  3. ifconfig lo:3 192.168.10.3 netmask 255.255.255.255 up

Trying to use either of these interfaces results in all requests coming from the first of these that was created. In this case, 192.168.10.1. Removing lo:1 would then make all requests come from 192.168.10.2. After readding lo:1, requests would still come from 192.168.10.2, as right now, lo:2 is the "oldest" interface alias.

EDIT 3: Firewall rules (as requested). enp35s0 is my public interface.

# Generated by xtables-save v1.8.2 on Fri Jul 23 12:15:33 2021
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-USER - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
-A INPUT -i enp35s0 -p udp -m state --state NEW -m udp --dport 1195 -j ACCEPT
-A INPUT -i tun+ -j ACCEPT
-A INPUT -i enp35s0 -p udp -m state --state NEW -m udp --dport 1195 -j ACCEPT
-A INPUT -i tun+ -j ACCEPT
-A INPUT -i enp35s0 -p udp -m state --state NEW -m udp --dport 1195 -j ACCEPT
-A INPUT -i tun+ -j ACCEPT
-A INPUT -i enp35s0 -p udp -m state --state NEW -m udp --dport 1195 -j ACCEPT
-A INPUT -i tun+ -j ACCEPT
-A INPUT -i enp35s0 -p udp -m state --state NEW -m udp --dport 1195 -j ACCEPT
-A INPUT -i tun+ -j ACCEPT
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A FORWARD -o br-d83e4273341d -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-d83e4273341d -j DOCKER
-A FORWARD -i br-d83e4273341d ! -o br-d83e4273341d -j ACCEPT
-A FORWARD -i br-d83e4273341d -o br-d83e4273341d -j ACCEPT
-A FORWARD -o br-adf3fed056db -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-adf3fed056db -j DOCKER
-A FORWARD -i br-adf3fed056db ! -o br-adf3fed056db -j ACCEPT
-A FORWARD -i br-adf3fed056db -o br-adf3fed056db -j ACCEPT
-A FORWARD -i tun+ -j ACCEPT
-A FORWARD -i tun+ -o eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i enp35s0 -o tun+ -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i tun+ -j ACCEPT
-A FORWARD -i tun+ -o eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i enp35s0 -o tun+ -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i tun+ -j ACCEPT
-A FORWARD -i tun+ -o eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i enp35s0 -o tun+ -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i tun+ -j ACCEPT
-A FORWARD -i tun+ -o eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i enp35s0 -o tun+ -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i tun+ -j ACCEPT
-A FORWARD -i tun+ -o enp35s0 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i enp35s0 -o tun+ -m state --state RELATED,ESTABLISHED -j ACCEPT
-A DOCKER -d 172.18.0.4/32 ! -i br-adf3fed056db -o br-adf3fed056db -p tcp -m tcp --dport 8048 -j ACCEPT
-A DOCKER -d 172.18.0.4/32 ! -i br-adf3fed056db -o br-adf3fed056db -p tcp -m tcp --dport 8008 -j ACCEPT
-A DOCKER -d 172.18.0.5/32 ! -i br-adf3fed056db -o br-adf3fed056db -p tcp -m tcp --dport 9005 -j ACCEPT
-A DOCKER -d 172.18.0.8/32 ! -i br-adf3fed056db -o br-adf3fed056db -p tcp -m tcp --dport 8080 -j ACCEPT
-A DOCKER -d 172.18.0.10/32 ! -i br-adf3fed056db -o br-adf3fed056db -p tcp -m tcp --dport 8080 -j ACCEPT
-A DOCKER -d 172.18.0.11/32 ! -i br-adf3fed056db -o br-adf3fed056db -p tcp -m tcp --dport 8090 -j ACCEPT
-A DOCKER -d 172.19.0.2/32 ! -i br-d83e4273341d -o br-d83e4273341d -p udp -m udp --dport 49172 -j ACCEPT
-A DOCKER -d 172.19.0.2/32 ! -i br-d83e4273341d -o br-d83e4273341d -p udp -m udp --dport 49171 -j ACCEPT
-A DOCKER -d 172.19.0.2/32 ! -i br-d83e4273341d -o br-d83e4273341d -p udp -m udp --dport 49170 -j ACCEPT
-A DOCKER -d 172.19.0.2/32 ! -i br-d83e4273341d -o br-d83e4273341d -p udp -m udp --dport 49169 -j ACCEPT
-A DOCKER -d 172.19.0.2/32 ! -i br-d83e4273341d -o br-d83e4273341d -p udp -m udp --dport 49168 -j ACCEPT
-A DOCKER -d 172.19.0.2/32 ! -i br-d83e4273341d -o br-d83e4273341d -p udp -m udp --dport 49167 -j ACCEPT
-A DOCKER -d 172.19.0.2/32 ! -i br-d83e4273341d -o br-d83e4273341d -p udp -m udp --dport 49166 -j ACCEPT
-A DOCKER -d 172.19.0.2/32 ! -i br-d83e4273341d -o br-d83e4273341d -p udp -m udp --dport 49165 -j ACCEPT
-A DOCKER -d 172.19.0.2/32 ! -i br-d83e4273341d -o br-d83e4273341d -p udp -m udp --dport 49164 -j ACCEPT
-A DOCKER -d 172.19.0.2/32 ! -i br-d83e4273341d -o br-d83e4273341d -p udp -m udp --dport 49163 -j ACCEPT
-A DOCKER -d 172.19.0.2/32 ! -i br-d83e4273341d -o br-d83e4273341d -p udp -m udp --dport 49162 -j ACCEPT
-A DOCKER -d 172.19.0.2/32 ! -i br-d83e4273341d -o br-d83e4273341d -p udp -m udp --dport 49161 -j ACCEPT
-A DOCKER -d 172.19.0.2/32 ! -i br-d83e4273341d -o br-d83e4273341d -p udp -m udp --dport 49160 -j ACCEPT
-A DOCKER -d 172.19.0.2/32 ! -i br-d83e4273341d -o br-d83e4273341d -p udp -m udp --dport 49159 -j ACCEPT
-A DOCKER -d 172.19.0.2/32 ! -i br-d83e4273341d -o br-d83e4273341d -p udp -m udp --dport 49158 -j ACCEPT
-A DOCKER -d 172.19.0.2/32 ! -i br-d83e4273341d -o br-d83e4273341d -p udp -m udp --dport 49157 -j ACCEPT
-A DOCKER -d 172.19.0.2/32 ! -i br-d83e4273341d -o br-d83e4273341d -p udp -m udp --dport 49156 -j ACCEPT
-A DOCKER -d 172.19.0.2/32 ! -i br-d83e4273341d -o br-d83e4273341d -p udp -m udp --dport 49155 -j ACCEPT
-A DOCKER -d 172.19.0.2/32 ! -i br-d83e4273341d -o br-d83e4273341d -p udp -m udp --dport 49154 -j ACCEPT
-A DOCKER -d 172.19.0.2/32 ! -i br-d83e4273341d -o br-d83e4273341d -p udp -m udp --dport 49153 -j ACCEPT
-A DOCKER -d 172.19.0.2/32 ! -i br-d83e4273341d -o br-d83e4273341d -p udp -m udp --dport 49152 -j ACCEPT
-A DOCKER -d 172.19.0.2/32 ! -i br-d83e4273341d -o br-d83e4273341d -p tcp -m tcp --dport 5349 -j ACCEPT
-A DOCKER -d 172.19.0.2/32 ! -i br-d83e4273341d -o br-d83e4273341d -p udp -m udp --dport 5349 -j ACCEPT
-A DOCKER -d 172.19.0.2/32 ! -i br-d83e4273341d -o br-d83e4273341d -p tcp -m tcp --dport 3478 -j ACCEPT
-A DOCKER -d 172.19.0.2/32 ! -i br-d83e4273341d -o br-d83e4273341d -p udp -m udp --dport 3478 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-d83e4273341d ! -o br-d83e4273341d -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-adf3fed056db ! -o br-adf3fed056db -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-USER -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o br-d83e4273341d -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o br-adf3fed056db -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
COMMIT
# Completed on Fri Jul 23 12:15:33 2021
# Generated by xtables-save v1.8.2 on Fri Jul 23 12:15:33 2021
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:DOCKER - [0:0]
-A PREROUTING -p tcp -m tcp --dport 12348 -j DNAT --to-destination 10.9.0.11:62128
-A PREROUTING -p tcp -m tcp --dport 12348 -j DNAT --to-destination 10.9.0.11:62128
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A PREROUTING -p tcp -m tcp --dport 12348 -j DNAT --to-destination 10.9.0.11:62128
-A PREROUTING -p tcp -m tcp --dport 12348 -j DNAT --to-destination 10.9.0.11:62128
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 172.19.0.0/16 ! -o br-d83e4273341d -j MASQUERADE
-A POSTROUTING -s 172.18.0.0/16 ! -o br-adf3fed056db -j MASQUERADE
-A POSTROUTING -s 10.8.0.0/24 -o enp35s0 -j MASQUERADE
-A POSTROUTING -s 10.9.0.0/24 -o enp35s0 -j MASQUERADE
-A POSTROUTING -j MASQUERADE
-A POSTROUTING -s 10.8.0.0/24 -o enp35s0 -j MASQUERADE
-A POSTROUTING -s 10.9.0.0/24 -o enp35s0 -j MASQUERADE
-A POSTROUTING -j MASQUERADE
-A POSTROUTING -s 172.18.0.4/32 -d 172.18.0.4/32 -p tcp -m tcp --dport 8048 -j MASQUERADE
-A POSTROUTING -s 172.18.0.4/32 -d 172.18.0.4/32 -p tcp -m tcp --dport 8008 -j MASQUERADE
-A POSTROUTING -s 172.18.0.5/32 -d 172.18.0.5/32 -p tcp -m tcp --dport 9005 -j MASQUERADE
-A POSTROUTING -s 172.18.0.8/32 -d 172.18.0.8/32 -p tcp -m tcp --dport 8080 -j MASQUERADE
-A POSTROUTING -s 172.18.0.10/32 -d 172.18.0.10/32 -p tcp -m tcp --dport 8080 -j MASQUERADE
-A POSTROUTING -s 172.18.0.11/32 -d 172.18.0.11/32 -p tcp -m tcp --dport 8090 -j MASQUERADE
-A POSTROUTING -s 172.19.0.2/32 -d 172.19.0.2/32 -p udp -m udp --dport 49172 -j MASQUERADE
-A POSTROUTING -s 172.19.0.2/32 -d 172.19.0.2/32 -p udp -m udp --dport 49171 -j MASQUERADE
-A POSTROUTING -s 172.19.0.2/32 -d 172.19.0.2/32 -p udp -m udp --dport 49170 -j MASQUERADE
-A POSTROUTING -s 172.19.0.2/32 -d 172.19.0.2/32 -p udp -m udp --dport 49169 -j MASQUERADE
-A POSTROUTING -s 172.19.0.2/32 -d 172.19.0.2/32 -p udp -m udp --dport 49168 -j MASQUERADE
-A POSTROUTING -s 172.19.0.2/32 -d 172.19.0.2/32 -p udp -m udp --dport 49167 -j MASQUERADE
-A POSTROUTING -s 172.19.0.2/32 -d 172.19.0.2/32 -p udp -m udp --dport 49166 -j MASQUERADE
-A POSTROUTING -s 172.19.0.2/32 -d 172.19.0.2/32 -p udp -m udp --dport 49165 -j MASQUERADE
-A POSTROUTING -s 172.19.0.2/32 -d 172.19.0.2/32 -p udp -m udp --dport 49164 -j MASQUERADE
-A POSTROUTING -s 172.19.0.2/32 -d 172.19.0.2/32 -p udp -m udp --dport 49163 -j MASQUERADE
-A POSTROUTING -s 172.19.0.2/32 -d 172.19.0.2/32 -p udp -m udp --dport 49162 -j MASQUERADE
-A POSTROUTING -s 172.19.0.2/32 -d 172.19.0.2/32 -p udp -m udp --dport 49161 -j MASQUERADE
-A POSTROUTING -s 172.19.0.2/32 -d 172.19.0.2/32 -p udp -m udp --dport 49160 -j MASQUERADE
-A POSTROUTING -s 172.19.0.2/32 -d 172.19.0.2/32 -p udp -m udp --dport 49159 -j MASQUERADE
-A POSTROUTING -s 172.19.0.2/32 -d 172.19.0.2/32 -p udp -m udp --dport 49158 -j MASQUERADE
-A POSTROUTING -s 172.19.0.2/32 -d 172.19.0.2/32 -p udp -m udp --dport 49157 -j MASQUERADE
-A POSTROUTING -s 172.19.0.2/32 -d 172.19.0.2/32 -p udp -m udp --dport 49156 -j MASQUERADE
-A POSTROUTING -s 172.19.0.2/32 -d 172.19.0.2/32 -p udp -m udp --dport 49155 -j MASQUERADE
-A POSTROUTING -s 172.19.0.2/32 -d 172.19.0.2/32 -p udp -m udp --dport 49154 -j MASQUERADE
-A POSTROUTING -s 172.19.0.2/32 -d 172.19.0.2/32 -p udp -m udp --dport 49153 -j MASQUERADE
-A POSTROUTING -s 172.19.0.2/32 -d 172.19.0.2/32 -p udp -m udp --dport 49152 -j MASQUERADE
-A POSTROUTING -s 172.19.0.2/32 -d 172.19.0.2/32 -p tcp -m tcp --dport 5349 -j MASQUERADE
-A POSTROUTING -s 172.19.0.2/32 -d 172.19.0.2/32 -p udp -m udp --dport 5349 -j MASQUERADE
-A POSTROUTING -s 172.19.0.2/32 -d 172.19.0.2/32 -p tcp -m tcp --dport 3478 -j MASQUERADE
-A POSTROUTING -s 172.19.0.2/32 -d 172.19.0.2/32 -p udp -m udp --dport 3478 -j MASQUERADE
-A POSTROUTING -s 10.8.0.0/24 -o enp35s0 -j MASQUERADE
-A POSTROUTING -s 10.9.0.0/24 -o enp35s0 -j MASQUERADE
-A POSTROUTING -s 10.8.0.0/24 -o enp35s0 -j MASQUERADE
-A POSTROUTING -s 10.9.0.0/24 -o enp35s0 -j MASQUERADE
-A POSTROUTING -j MASQUERADE
-A POSTROUTING -s 10.8.0.0/24 -o enp35s0 -j MASQUERADE
-A POSTROUTING -s 10.9.0.0/24 -o enp35s0 -j MASQUERADE
-A POSTROUTING -j MASQUERADE
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A DOCKER -i docker0 -j RETURN
-A DOCKER -i br-d83e4273341d -j RETURN
-A DOCKER -i br-adf3fed056db -j RETURN
-A DOCKER -d 127.0.0.1/32 ! -i br-adf3fed056db -p tcp -m tcp --dport 8048 -j DNAT --to-destination 172.18.0.4:8048
-A DOCKER -d 127.0.0.1/32 ! -i br-adf3fed056db -p tcp -m tcp --dport 8008 -j DNAT --to-destination 172.18.0.4:8008
-A DOCKER -d 127.0.0.1/32 ! -i br-adf3fed056db -p tcp -m tcp --dport 9005 -j DNAT --to-destination 172.18.0.5:9005
-A DOCKER -d 127.0.0.1/32 ! -i br-adf3fed056db -p tcp -m tcp --dport 9006 -j DNAT --to-destination 172.18.0.8:8080
-A DOCKER -d 127.0.0.1/32 ! -i br-adf3fed056db -p tcp -m tcp --dport 8765 -j DNAT --to-destination 172.18.0.10:8080
-A DOCKER -d 127.0.0.1/32 ! -i br-adf3fed056db -p tcp -m tcp --dport 8090 -j DNAT --to-destination 172.18.0.11:8090
-A DOCKER ! -i br-d83e4273341d -p udp -m udp --dport 49172 -j DNAT --to-destination 172.19.0.2:49172
-A DOCKER ! -i br-d83e4273341d -p udp -m udp --dport 49171 -j DNAT --to-destination 172.19.0.2:49171
-A DOCKER ! -i br-d83e4273341d -p udp -m udp --dport 49170 -j DNAT --to-destination 172.19.0.2:49170
-A DOCKER ! -i br-d83e4273341d -p udp -m udp --dport 49169 -j DNAT --to-destination 172.19.0.2:49169
-A DOCKER ! -i br-d83e4273341d -p udp -m udp --dport 49168 -j DNAT --to-destination 172.19.0.2:49168
-A DOCKER ! -i br-d83e4273341d -p udp -m udp --dport 49167 -j DNAT --to-destination 172.19.0.2:49167
-A DOCKER ! -i br-d83e4273341d -p udp -m udp --dport 49166 -j DNAT --to-destination 172.19.0.2:49166
-A DOCKER ! -i br-d83e4273341d -p udp -m udp --dport 49165 -j DNAT --to-destination 172.19.0.2:49165
-A DOCKER ! -i br-d83e4273341d -p udp -m udp --dport 49164 -j DNAT --to-destination 172.19.0.2:49164
-A DOCKER ! -i br-d83e4273341d -p udp -m udp --dport 49163 -j DNAT --to-destination 172.19.0.2:49163
-A DOCKER ! -i br-d83e4273341d -p udp -m udp --dport 49162 -j DNAT --to-destination 172.19.0.2:49162
-A DOCKER ! -i br-d83e4273341d -p udp -m udp --dport 49161 -j DNAT --to-destination 172.19.0.2:49161
-A DOCKER ! -i br-d83e4273341d -p udp -m udp --dport 49160 -j DNAT --to-destination 172.19.0.2:49160
-A DOCKER ! -i br-d83e4273341d -p udp -m udp --dport 49159 -j DNAT --to-destination 172.19.0.2:49159
-A DOCKER ! -i br-d83e4273341d -p udp -m udp --dport 49158 -j DNAT --to-destination 172.19.0.2:49158
-A DOCKER ! -i br-d83e4273341d -p udp -m udp --dport 49157 -j DNAT --to-destination 172.19.0.2:49157
-A DOCKER ! -i br-d83e4273341d -p udp -m udp --dport 49156 -j DNAT --to-destination 172.19.0.2:49156
-A DOCKER ! -i br-d83e4273341d -p udp -m udp --dport 49155 -j DNAT --to-destination 172.19.0.2:49155
-A DOCKER ! -i br-d83e4273341d -p udp -m udp --dport 49154 -j DNAT --to-destination 172.19.0.2:49154
-A DOCKER ! -i br-d83e4273341d -p udp -m udp --dport 49153 -j DNAT --to-destination 172.19.0.2:49153
-A DOCKER ! -i br-d83e4273341d -p udp -m udp --dport 49152 -j DNAT --to-destination 172.19.0.2:49152
-A DOCKER ! -i br-d83e4273341d -p tcp -m tcp --dport 5349 -j DNAT --to-destination 172.19.0.2:5349
-A DOCKER ! -i br-d83e4273341d -p udp -m udp --dport 5349 -j DNAT --to-destination 172.19.0.2:5349
-A DOCKER ! -i br-d83e4273341d -p tcp -m tcp --dport 3478 -j DNAT --to-destination 172.19.0.2:3478
-A DOCKER ! -i br-d83e4273341d -p udp -m udp --dport 3478 -j DNAT --to-destination 172.19.0.2:3478
COMMIT
# Completed on Fri Jul 23 12:15:33 2021
# Generated by xtables-save v1.8.2 on Fri Jul 23 12:15:33 2021
*mangle
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
COMMIT
# Completed on Fri Jul 23 12:15:33 2021

Output of ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp35s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether a8:a1:59:48:f5:da brd ff:ff:ff:ff:ff:ff
    inet <myPublicIPv4>/26 brd 135.181.209.63 scope global enp35s0
       valid_lft forever preferred_lft forever
    inet6 <myPublicIPv6>/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::aaa1:59ff:fe48:f5da/64 scope link 
       valid_lft forever preferred_lft forever
5: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:f4:2e:c5:95 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:f4ff:fe2e:c595/64 scope link 
       valid_lft forever preferred_lft forever
6: br-adf3fed056db: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:4b:fd:ae:6f brd ff:ff:ff:ff:ff:ff
    inet 172.18.0.1/16 brd 172.18.255.255 scope global br-adf3fed056db
       valid_lft forever preferred_lft forever
    inet6 fe80::42:4bff:fefd:ae6f/64 scope link 
       valid_lft forever preferred_lft forever
216865: veth225712d@if216864: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-d83e4273341d state UP group default 
    link/ether 32:65:78:a0:be:47 brd ff:ff:ff:ff:ff:ff link-netnsid 10
    inet6 fe80::3065:78ff:fea0:be47/64 scope link 
       valid_lft forever preferred_lft forever
61: br-d83e4273341d: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:c4:38:c9:c5 brd ff:ff:ff:ff:ff:ff
    inet 172.19.0.1/16 brd 172.19.255.255 scope global br-d83e4273341d
       valid_lft forever preferred_lft forever
    inet6 fe80::42:c4ff:fe38:c9c5/64 scope link 
       valid_lft forever preferred_lft forever
202134: tap0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 100
    link/ether 36:e7:1a:1f:b5:c8 brd ff:ff:ff:ff:ff:ff
    inet 10.9.0.1/24 brd 10.9.0.255 scope global tap0
       valid_lft forever preferred_lft forever
    inet6 fe80::34e7:1aff:fe1f:b5c8/64 scope link 
       valid_lft forever preferred_lft forever
202135: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 100
    link/none 
    inet 10.8.0.1 peer 10.8.0.2/32 scope global tun0
       valid_lft forever preferred_lft forever
    inet6 fe80::cc13:3dec:ab8c:2e3b/64 scope link stable-privacy 
       valid_lft forever preferred_lft forever
88033: veth0949050@if88032: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-adf3fed056db state UP group default 
    link/ether 1e:e0:14:92:f3:ad brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::1ce0:14ff:fe92:f3ad/64 scope link 
       valid_lft forever preferred_lft forever
88035: veth5621bc2@if88034: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-adf3fed056db state UP group default 
    link/ether 66:a7:77:bd:d0:ca brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet6 fe80::64a7:77ff:febd:d0ca/64 scope link 
       valid_lft forever preferred_lft forever
88037: veth00b963e@if88036: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-adf3fed056db state UP group default 
    link/ether e6:18:ec:30:5c:a6 brd ff:ff:ff:ff:ff:ff link-netnsid 3
    inet6 fe80::e418:ecff:fe30:5ca6/64 scope link 
       valid_lft forever preferred_lft forever
88041: vethda3acd6@if88040: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-adf3fed056db state UP group default 
    link/ether 3a:ad:d7:6c:fa:cb brd ff:ff:ff:ff:ff:ff link-netnsid 4
    inet6 fe80::38ad:d7ff:fe6c:facb/64 scope link 
       valid_lft forever preferred_lft forever
88043: veth2f937b7@if88042: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-adf3fed056db state UP group default 
    link/ether ea:9e:65:d9:9a:f3 brd ff:ff:ff:ff:ff:ff link-netnsid 5
    inet6 fe80::e89e:65ff:fed9:9af3/64 scope link 
       valid_lft forever preferred_lft forever
88045: vethac132e2@if88044: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-adf3fed056db state UP group default 
    link/ether e2:46:7e:4f:71:52 brd ff:ff:ff:ff:ff:ff link-netnsid 6
    inet6 fe80::e046:7eff:fe4f:7152/64 scope link 
       valid_lft forever preferred_lft forever
88049: vethe8f0958@if88048: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-adf3fed056db state UP group default 
    link/ether 9a:e9:40:f1:4d:6b brd ff:ff:ff:ff:ff:ff link-netnsid 7
    inet6 fe80::98e9:40ff:fef1:4d6b/64 scope link 
       valid_lft forever preferred_lft forever
88053: vethf57e351@if88052: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-adf3fed056db state UP group default 
    link/ether 6a:52:e2:51:c5:7d brd ff:ff:ff:ff:ff:ff link-netnsid 2
    inet6 fe80::6852:e2ff:fe51:c57d/64 scope link 
       valid_lft forever preferred_lft forever
88055: vethec5ddd0@if88054: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-adf3fed056db state UP group default 
    link/ether 76:69:ec:ce:e7:f8 brd ff:ff:ff:ff:ff:ff link-netnsid 8
    inet6 fe80::7469:ecff:fece:e7f8/64 scope link 
       valid_lft forever preferred_lft forever
88057: veth473fda8@if88056: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-adf3fed056db state UP group default 
    link/ether 36:48:f4:47:26:81 brd ff:ff:ff:ff:ff:ff link-netnsid 9
    inet6 fe80::3448:f4ff:fe47:2681/64 scope link 
       valid_lft forever preferred_lft forever

EDIT 4:

Output based on Trolzen's answer

# socat -d -d UDP-RECVFROM:23456,fork -2021/08/12 12:12:07 socat[4506] N receiving on AF=2 0.0.0.0:23456
2021/08/12 12:12:11 socat[4506] N receiving packet from AF=2 <myPublicIp>:47228
2021/08/12 12:12:11 socat[4506] N forked off child process 4510
2021/08/12 12:12:11 socat[4510] N reading from and writing to stdio
2021/08/12 12:12:11 socat[4510] N starting data transfer loop with FDs [5,5] and [0,1]
2021/08/12 12:12:11 socat[4510] N received packet with 5 bytes from AF=2 <myPublicIp>:47228
test
2021/08/12 12:12:11 socat[4510] N socket 1 (fd 5) is at EOF
2021/08/12 12:12:11 socat[4506] N receiving on AF=2 0.0.0.0:23456
2021/08/12 12:12:12 socat[4510] N exiting with status 0
2021/08/12 12:12:12 socat[4506] N receiving on AF=2 0.0.0.0:23456

# socat -d -d -T.5 UDP-LISTEN:23456,fork -
2021/08/12 12:15:20 socat[4811] N listening on UDP AF=2 0.0.0.0:23456
2021/08/12 12:15:23 socat[4811] N accepting UDP connection from AF=2 <myPublicIp>:55887
2021/08/12 12:15:23 socat[4811] N forked off child process 4817
2021/08/12 12:15:23 socat[4811] N listening on UDP AF=2 0.0.0.0:23456
2021/08/12 12:15:23 socat[4817] N reading from and writing to stdio
2021/08/12 12:15:23 socat[4817] N starting data transfer loop with FDs [5,5] and [0,1]
test
2021/08/12 12:15:23 socat[4817] N inactivity timeout triggered
2021/08/12 12:15:23 socat[4817] N exiting with status 0
2021/08/12 12:15:23 socat[4811] N childdied(): handling signal 17

# socat -d -d UDP-RECV:23456 -
2021/08/12 12:16:12 socat[4891] N reading from and writing to stdio
2021/08/12 12:16:12 socat[4891] N starting data transfer loop with FDs [5,5] and [0,1]
2021/08/12 12:16:15 socat[4891] N received packet with 5 bytes from AF=2 <myPublicIp>:42481
test

Output of the "Main command"

2021/08/12 12:22:37 socat[5435] N accepting UDP connection from AF=2 <myPublicIp>:40659
2021/08/12 12:22:37 socat[5435] N forked off child process 5482
2021/08/12 12:22:37 socat[5435] N listening on UDP AF=2 0.0.0.0:12345
2021/08/12 12:22:37 socat[5482] N opening connection to AF=2 127.0.0.1:23456
2021/08/12 12:22:37 socat[5482] N successfully connected from local address AF=2 127.0.0.2:36305
2021/08/12 12:22:37 socat[5482] N starting data transfer loop with FDs [5,5] and [6,6]
2021/08/12 12:22:42 socat[5482] N inactivity timeout triggered
2021/08/12 12:22:42 socat[5482] N exiting with status 0
2021/08/12 12:22:42 socat[5435] N childdied(): handling signal 17

My netcat version is not the issue. Netcat is just an example application I am using to test this, and I can see this problem in packet dumps too.

Improve this question
20
  • Does this also occur with TCP? Do you have any NAT rules in iptables or nft?
    – user1686
    Jul 16, 2021 at 9:49
  • It also occurs with TCP (although it's slightly different); I've updated my question. I do not have any NAT rules in iptables and I'm not using nft.
    – Joba
    Jul 16, 2021 at 10:04
  • Tried your commands, and I cannot reproduce this behavior. Can you provide an output of netstat -utn?
    – trolzen
    Jul 16, 2021 at 13:04
  • You are right, I can't reproduce this issue on a fresh Ubuntu. I've updated the post with the netstat output
    – Joba
    Jul 16, 2021 at 13:42
  • @Joba, sorry, just realised that it should be netstat -aun to include listening sockets and not include tcp sockets. Could you please update?
    – trolzen
    Jul 16, 2021 at 16:11

2 Answers 2

Reset to default
0

Try changing the lo interface scope - I think you need to change it from host to global.

You can do this in [/etc/network/interfaces][1] if you're using Debian (need to do ifdown lo and ifup lo to make it go into effect) or use the ip addr set dev lo scope global command (I think).

Details

I had to do this to get DNS queries originating from localhost to have the source address set correctly so it could receive replies from a DNS server hosted on localhost, where the DNS server was also accessible on other interfaces.

Improve this answer
3
  • That doesn't seem to fix it. Changing the scope of lo had no effect (according to ip addr). Creating a new interface with a global scope is possible, but the problem persists. The fresh Debian installation that doesn't have this problem also has lo with scope host, so setting it to global shouldn't be necessary.
    – Joba
    Jul 22, 2021 at 8:54
  • 1
    lo interface is host loopback and should never be global. Wouldn't change the scope myself.
    – Jari Turkia
    Jul 23, 2021 at 6:28
  • @JariTurkia It was the only way I could get BIND on my PC-based router to send DNS queries originating from itself (127.0.0.1) to actually come from 127.0.0.1 as a source IP and not from eth0's IP address which it was doing for some reason I can't fathom--needed because BIND introduced access lists so the source IP had to be correct and I don't want to serve DNS over my WAN interface. Anyway, worth a shot, sorry it didn't work. :(
    – LawrenceC
    Jul 26, 2021 at 12:58
0
+50

Since, as you confirm, this behavior could not be reproduced, it's not your misunderstanding or incorrect usage of socat. It's something in your particular setup: either misconfiguration or incorrect diagnostics by nc. Here are suggestions for troubleshooting.

I'll be using following terms to reference the commands:

  1. Generator, command 1 (I've added the -q0 so nc doesn't "hang"): 
    echo test | nc -u -q0 127.0.0.1 12345
  2. Main command, command 2: 
    socat -T5 UDP4-LISTEN:12345,reuseaddr,fork UDP4:127.0.0.1:23456,bind=127.0.0.2,so-bindtodevice=lo
  3. Receiver, receiving command, command 3: 
    nc -u -l -p 23456 -vv

1. More diagnostics.

Replace the receiver command with one of the following (they are almost equivalent):

socat -d -d UDP-RECVFROM:23456,fork -
socat -d -d -T.5 UDP-LISTEN:23456,fork -
socat -d -d UDP-RECV:23456 -

Also add -d -d to the main command.

This will let you get more info about what's going on.

2. Analyze packets.

Use some packet dumper and analyzer (Wireshark, tcpdump) on all interfaces to capture packets and figure out what actual packets are sent.

3. Check the packet filter rules.

Besides the packet filter rules you provided there are three more tables to look at:

sudo iptables -S -t nat
sudo iptables -S -t mangle
sudo iptables -S -t raw

4. Netcat version

What's the version of nc? Maybe it is buggy? Also it turns out that there are two netcats: traditional and OpenBSD version.

$ nc -h 2>&1 | head -n 1
[v1.10-41.1]

Proper behavior (for reference)

Command 1:

$ echo test | nc -u -q0 127.0.0.1 12345
$

Command 2:

$ sudo socat -T5 -d -d UDP4-LISTEN:12345,reuseaddr,fork UDP4:127.0.0.1:23456,bind=127.0.0.2,so-bindtodevice=lo
2021/07/23 01:49:48 socat[4617] N listening on UDP AF=2 0.0.0.0:12345
2021/07/23 01:51:12 socat[4617] N accepting UDP connection from AF=2 127.0.0.1:59877
2021/07/23 01:51:12 socat[4617] N forked off child process 4622
2021/07/23 01:51:12 socat[4617] N listening on UDP AF=2 0.0.0.0:12345
2021/07/23 01:51:12 socat[4622] N opening connection to AF=2 127.0.0.1:23456
2021/07/23 01:51:12 socat[4622] N successfully connected from local address AF=2 127.0.0.2:36338
2021/07/23 01:51:12 socat[4622] N starting data transfer loop with FDs [5,5] and [6,6]
2021/07/23 01:51:17 socat[4622] N inactivity timeout triggered
2021/07/23 01:51:17 socat[4622] N exiting with status 0
2021/07/23 01:51:17 socat[4617] N childdied(): handling signal 17

Command 3:

vagrant@packer-debian-10-amd64:~$ socat -d -d UDP-RECVFROM:23456,fork -
2021/07/23 01:50:54 socat[4619] N receiving on AF=2 0.0.0.0:23456
2021/07/23 01:51:12 socat[4619] N receiving packet from AF=2 127.0.0.2:36338
2021/07/23 01:51:12 socat[4619] N forked off child process 4623
2021/07/23 01:51:12 socat[4623] N reading from and writing to stdio
2021/07/23 01:51:12 socat[4623] N starting data transfer loop with FDs [5,5] and [0,1]
2021/07/23 01:51:12 socat[4623] N received packet with 5 bytes from AF=2 127.0.0.2:36338
2021/07/23 01:51:12 socat[4619] N receiving on AF=2 0.0.0.0:23456
test
2021/07/23 01:51:12 socat[4623] N socket 1 (fd 5) is at EOF
2021/07/23 01:51:13 socat[4623] N exiting with status 0
2021/07/23 01:51:13 socat[4619] N receiving on AF=2 0.0.0.0:23456

Side notes.

  1. I think the so-bindtodevice=lo option in the main command is not necessary, since you are already binding to ip address with bind=127.0.0.2.

  2. If your task is not about learning every bit of socat behavior, but more about getting things done (like forwarding specific udp traffic between subnets), I suggest you to use packet filter and its NAT capabilities for this task. It would be proper way to do that, because socat works on application level (L7), and packet filter works on the network level (L3).

Improve this answer
3
  • Using the other socat commands gives me the same output you see, but with my external ip instead of 127.0.0.2. tcpdump confirms, that the source ip is not 127.0.0.2.
    – Joba
    Jul 25, 2021 at 15:53
  • Also, I'm awarding the bounty to this question. Even though it didn't fix my problem, you did a lot to try to figure this out. Thank you.
    – Joba
    Jul 25, 2021 at 15:54
  • But still what are the outputs of the main socat (with -d -d added) and the receiving socat. Comparing those can be interesting.
    – trolzen
    Jul 26, 2021 at 15:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.