0

Given an extra loopback interface in the 127.0.0.X (X>1) range on BoxA (which can be running either OSX or Linux), I want to bind port 22 of this extra loopback interface to a forward SSH tunnel (ie. local port forward) that is pointed at BoxB.

On OSX this works fine (strangely, in retrospect). [Taking X=2] after bringing up the loopback alias with ifconfig lo0 alias 127.0.0.2 up, SSH can establish a tunnel with ssh -NfL 127.0.0.2:22:localhost:22 BoxB. Then in a new shell on BoxA, ssh 127.0.0.2 logs me into BoxB.

On Ubuntu, I can bring up the loopback alias on BoxA, but when trying to establish the SSH tunnel, ssh complains about not being able to bind (and hence forward) BoxA's port 22. The subsequent ssh 127.0.0.2 (in a new shell on BoxA) gives a fingerprint warning, which if bypassed, logs me back into BoxA. Makes sense - sshd on BoxA is listening to all interfaces.

Looking at the sshd_config in each, both are configured to listen on 0.0.0.0 (and :: for IPv6).

lsof for OSX gives:

launchd       1           root   40u  IPv6 0xddfcabed61001f0d      0t0  TCP *:ssh (LISTEN)
launchd       1           root   41u  IPv4 0xddfcabed6100413d      0t0  TCP *:ssh (LISTEN)
launchd       1           root   43u  IPv6 0xddfcabed61001f0d      0t0  TCP *:ssh (LISTEN)
launchd       1           root   44u  IPv4 0xddfcabed6100413d      0t0  TCP *:ssh (LISTEN)

and for Ubuntu:

sshd       1287              0    3u     IPv4           21903340        0t0        TCP *:ssh (LISTEN)

So both are listening on all interfaces, though I'm not sure why OSX uses 4 processes. In any case, Ubuntu gives the expected behaviour. Why does OSX behave differently?

The follow up question of course, is how to make Ubuntu behave like OSX in this regard.

While I wish for the sshd_config to have state, wildcards and/or logical operators (e.g. "do not listen on 127.0.0.*; listen on 127.0.0.1") like iptables, it doesn't seem to be the case...

  • No need to play with X. Just use ssh -NfL 22:localhost:22 BoxB and connect to tunneled port as localhost:22 – Alex Aug 2 '18 at 12:21
  • It gives a channel_setup_fwd_listener_tcpip failure, with "bind: Address already in use" for good measure (on Ubuntu). – Congee Aug 2 '18 at 12:50
  • That's means you already bind something to the same port. If you having multiple tunnels to other boxes or some local service already used it for some purpose, simply use different local port that not in use: ssh -NfL 20001:localhost:22 BoxA, ssh -NfL 20002:localhost:22 BoxB, ssh -NfL 20003:localhost:22 BoxC and so on... Use netstat -lntp to spot free/buzy local ports – Alex Aug 2 '18 at 12:56
  • Yes, sshd on BoxA is using port 22, clearly visible in the lsof output. The entire question is why OSX allows forwarding :22 on new loopback interfaces, and Ubuntu does not... – Congee Aug 2 '18 at 13:24
0

That's reproducible using socat (1.7.3.2-2, not 1.7.3.1-2+deb9u1 which ignores reuseport) on Debian (so similar for Ubuntu):

term1$ socat -d -d  TCP4-LISTEN:5555,reuseaddr,fork -
2018/08/03 08:20:43 socat[25084] N listening on AF=2 0.0.0.0:5555

term2$ socat -d -d TCP4-LISTEN:5555,bind=127.0.0.2,reuseaddr,fork -
2018/08/03 08:21:00 socat[25085] E bind(5, {AF=2 127.0.0.2:5555}, 16): Address already in use
2018/08/03 08:21:00 socat[25085] N exit(1)

Now if you add reuseport on both:

term1$ socat -d -d  TCP4-LISTEN:5555,reuseaddr,reuseport,fork -
2018/08/03 08:21:28 socat[25086] N listening on AF=2 0.0.0.0:5555

term2$ socat -d -d TCP4-LISTEN:5555,bind=127.0.0.2,reuseaddr,reuseport,fork -
2018/08/03 08:21:56 socat[25092] N listening on AF=2 127.0.0.2:5555


otherterm$ netstat -tnlp 2>/dev/null|grep :5555 
tcp        0      0 127.0.0.2:5555          0.0.0.0:*               LISTEN      25092/socat         
tcp        0      0 0.0.0.0:5555            0.0.0.0:*               LISTEN      25086/socat         

Using an other socat or netcat to connect will show it's properly routed to the correct listening socat depending on the IP address.

So the behaviour difference is most certainly on how the bind and setsockopt calls are done on MacOS and Ubuntu for ssh (maybe launchd has a role to play too?). Using strace, this is added in the working socat case:

 setsockopt(5, SOL_SOCKET, SO_REUSEPORT, [1], 4) = 0

UPDATE:
While I will address the letter of the OP's question: make it work, I also think the question doesn't tell about the broader goal and this goal might be achieved with an entirely different solution.

So to alter sshd and ssh behaviour when they bind a socket, it's possible to use a dlsym() wrapper that will alter bind(3)'s function behaviour before the real bind(2) syscall.

compile the file reuseport-wrapper.c below with:

gcc -shared -fPIC -o reuseport-wrapper.so reuseport-wrapper.c -ldl

#define _GNU_SOURCE
#include <dlfcn.h>

#include <sys/socket.h>
#include <stddef.h> /* NULL */

int bind(int socket, const struct sockaddr *address, socklen_t address_len) {
    static const int optval=1;
    static int (*orig_bind)(int, const struct sockaddr *,socklen_t)=NULL;

    if (orig_bind == NULL && (orig_bind=dlsym(RTLD_NEXT,"bind")) == NULL)
        return -1;
    if (address != NULL && (address->sa_family == AF_INET || address->sa_family == AF_INET6)) {
        if (setsockopt(socket, SOL_SOCKET, SO_REUSEPORT, &optval, sizeof optval) != 0) {
            return -1;
        }
    }
    return orig_bind(socket, address, address_len);
}

This will put a wrapper on bind(3) to always set SO_REUSEPORT before binding an inet or inet6 socket.

Place it somewhere (eg as /usr/local/lib/reuseport-wrapper.so but it really doesn't matter since it's explicitely loaded like would a plugin).

Edit /etc/default/ssh and add:

LD_PRELOAD=/usr/local/lib/reuseport-wrapper.so

restart the ssh server (eg service ssh restart)

And run any other tool (including ssh) as root (see below why) with LD_PRELOAD=/usr/local/lib/reuseport-wrapper.so exported. For example:

$ sudo su -
[...]
# LD_PRELOAD=/usr/local/lib/reuseport-wrapper.so ssh -NfL 127.0.0.2:22:localhost:22 userB@BoxB

This command has to be started as root for two reasons: port 22 requires root to bind to it, and even with some additional capability, there's an additional restriction described in socket(7):

SO_REUSEPORT (since Linux 3.9)
Permits multiple AF_INET or AF_INET6 sockets to be bound to an identical socket address. This option must be set on each socket (including the first socket) prior to calling bind(2) on the socket. To prevent port hijacking, all of the processes binding to the same address must have the same effective UID. This option can be employed with both TCP and UDP sockets.

Once this is done:

# netstat -tnlp |grep -w 22
tcp        0      0 127.0.0.2:22            0.0.0.0:*               LISTEN      157/ssh             
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      143/sshd            
tcp6       0      0 :::22                   :::*                    LISTEN      143/sshd            

And the tunnel will work as expected.

  • Good information, thanks. Not sure if I accept it as an answer per se, as it doesn't fix the problem (assuming its fixable at all), but provides useful information... – Congee Aug 5 '18 at 12:33
  • Added a "working" solution just to underline something else should be sought. It's possible the question is hiding an XY problem. – A.B Aug 5 '18 at 20:04
  • @Congee so did you try my update? That makes it working as per the question, doesn't it? I answered why there is a difference, and how to make it work on Ubuntu. – A.B Aug 11 '18 at 23:14

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.