I have a box with two ethernet cards, eth0 and eth1, having different ip-addresses.
Connections can come in on both these interfaces. I need the return packets of a TCP connection to go out on the same interface as it came in. That is, a connection coming in on eth1 should have its reply packets coming from that card too, and not follow the default route which is eth0. At the same time, I want all connections that originate from the server, to appear from eth0.
One way of looking at it, might be that I want the default route to change based on whether the packet about to be routed belongs to a connection that came in on eth1. In that case, the default route should go to eth1. In all other cases, the default route should be eth0.
(Bacground: The eth1 card is reachable from a specific external IP-address, reachable from the world. Eth0 have an internal IP-address, which also have internet connectivity through NAT on another external IP. Due to legacy ip-based firewall openings previously made for us on some external service providers, I need all outgoing connections from the box in question to originate from this NAT'ed address, while the box is reachable as a website on the specific external address.)
I believe this could be done with iptables - but whatever fixes the original issue is just fine!