23

Context

I have a personal server that I use for the web. I sometimes need to SSH/SFTP to it.

Disclamer: I have very little experience with nginx internals.

Problem

This morning, I figured out that the free wifi in a well-know cafe chain was blocking SSH (actually, they are blocking anything that's not on 80/443). But when I need SSH, I need it, so I looked for ways to share SSH and HTTPS on the same port.

What I looked at

I have looked at a few possible solutions that can run on port 443:

  • SSHL: a SSH/OpenVPN/HTTPS multiplexer;
  • OpenVPN: a VPN solution has a built-in multiplexer for OpenVPN and HTTPS;
  • HAProxy: a webserver/load balancer can also multiplex everything.

All of these seem pretty straight-forward but I don't really like the fact of adding layers and complexity and possibly slowing things down just in the unlikely event that I need to SSH on 443.

Putting nginx into the mix

I know that nginx already supports raw TCP streams handling. So I was wondering if I could use that on port 443 too directly in nginx. The idea being that nginx could choose to use the http module if it recognizes HTTP(S) or stream for everything else.

Questions

In that context, I have two questions:

  • Is nginx even capable of doing such a distinction? (I am not even sure I would be able to listen on port 443 in both the http and the stream block at the same time.)
  • If so, would there be any blatant performance issue with that setup? (I am thinking about transfer speed with SFTP for instance, not really SSH per se.)
5
  • I know enough about those protocols to explain to what extent such protocol detection is possible. However I do not know what Nginx is capable of doing. The name stream does not sound like a feature capable of doing any protocol detection. Regardless of what you use on the server side, you will need to either encapsulate the SSH traffic inside another protocol or use an SSH client which is recent enough to be capable of speaking before the server. Relying on the server to detect the protocol of a client which hasn't spoken yet would be fragile.
    – kasperd
    Oct 8, 2016 at 17:14
  • nginx support seems quite small indeed and it seems that it is only relying on the port number to "route" the traffic. From what I've understood, the http bloc is merely a way to load the right module for the right corresponding server listening to a given port. That module, in turn, will do all the SNI magic. But I am no expert and I just wanted to make sure I understood that correctly.
    – JohnW
    Oct 8, 2016 at 17:21
  • 1
    It is not clear to me why you are even asking about Nginx. It seems you have no experience with Nginx and have no evidence suggesting it should be capable of doing what you are asking for in the first place. If you made it clear what your current setup looks like, you could probably get a much better answer.
    – kasperd
    Oct 8, 2016 at 17:58
  • That being said, SNI is occurring at the TLS level: if there is no SNI, nginx may default to stream instead.
    – JohnW
    Oct 8, 2016 at 18:07
  • Seeing as this is about blocked ports more than anything, somebody in the same situation might port check a connection to see whats possible for outbound services, it could just be that the free wifi has blocked common ports. so if you used the free wifi often and needed the ssh access you could just change the the port that the ssh server is listening on to one that is free on your system and open on the network you are using most. Sep 28, 2019 at 18:43

9 Answers 9

21

Since nginx version 1.15.2 added new variable $ssl_preread_protocol. And in official blog added post about how to use this variable for multiplexing HTTPS and SSH on the same port https://www.nginx.com/blog/running-non-ssl-protocols-over-ssl-port-nginx-1-15-2/

Example of configuring SSH(by default) and HTTPS:

stream {
    upstream ssh {
        server 192.0.2.1:22;
    }

    upstream web {
        server 192.0.2.2:443;
    }

    map $ssl_preread_protocol $upstream {
        default ssh;
        "TLSv1.2" web;
    }

    # SSH and SSL on the same port
    server {
        listen 443;

        proxy_pass $upstream;
        ssl_preread on;
    }
}
3
  • 2
    Is it possible to have both the ssh and (nginx) https servers be on the same host? I assume you'd need to make https 'listen' on port 444 or something, and direct the ssh preread that listens on 443 to port 444 for https? Aug 13, 2018 at 21:48
  • @starbeamrainbowlabs You can't put both on 443, but you can use unix domain socket so you don't need to run two nginx processes. You can do listen unix:/run/nginx.sock in http > server blocks.
    – kissgyorgy
    Aug 21, 2018 at 22:23
  • Note: stream { ...} is a section in /etc/nginx/nginx.conf
    – Quandary
    Feb 21, 2020 at 18:45
11

I have a working configuration tunneling ssh over tls on port 443, using the nginx stream module. I also do XMPP over TLS and normal HTTP on the same port. I do the mutliplexing via ALPN.

(You need nginx > 1.13.10 to use the ssl_preread module with alpn http://nginx.org/en/docs/stream/ngx_stream_ssl_preread_module.html#ssl_preread)

My configuration uses the docker version of nginx, but it should also work without docker.

stream {
    # check ALPN for xmpp client or server and redirect to local ssl termination endpoints
    map $ssl_preread_alpn_protocols $ssl_multiplexer {
        "xmpp-client"     127.0.0.1:5422;
        "xmpp-server"     127.0.0.1:5469;
        "identifyssh"     127.0.0.1:8822;
        default           127.0.0.1:8443;
    }

    server {
        listen 443;
        ssl_preread on;
        proxy_pass $ssl_multiplexer;
        proxy_protocol on;
        set_real_ip_from  172.18.0.0/32;
    }

    # ssl termination for c2s connections
    server {
        listen 5422 ssl proxy_protocol;
        # ... <- tls keys and options here
        proxy_ssl off;
        proxy_pass ejabberd:5222;
    }

    # ssl termination for s2s connections
    server {
        listen 5469 ssl proxy_protocol;
        # ... <- tls keys and options here
        proxy_ssl off;
        proxy_pass ejabberd:5269;
    }

    # ssl termination for ssh connections
    server {
        listen 8822 ssl proxy_protocol;
        # ... <- tls keys and options here
        proxy_ssl off;
        proxy_pass yourserver:22;
    }
}

If you want to use the XMPP stuff you have to add some SRV records to point to your servers 443 port, see https://xmpp.org/extensions/xep-0368.html

If you want to connect to your ssh server you have to wrap your ssh session in a ssl session that sends the ALPN string you defined in your stream config. I used "identifyssh" in the example above. You can use anything, but try to not collide with the official defined names: https://www.iana.org/assignments/tls-extensiontype-values/tls-extensiontype-values.xhtml#alpn-protocol-ids

To start the ssh session from your client to your prepared server use:

ssh you@yourserver -o "ProxyCommand openssl s_client -alpn identifyssh -ign_eof -connect yourserver:443"

And you should be connected.

I should also note, that I use the proxy_protocol to keep the clients headers and IP-address while passing to my backends.

Your normal http server configured in the http {} section should take care of this:

server {
  listen 8443 ssl proxy_protocol;
  # ...
}

The best thing is, that you don't need tools like sslh, stunnel, proxytunnel or others to make this work. You only need a newer nginx and openssl. Hope this helps somebody. It would helped me digging into that stuff.

2
  • Great information, I now wonder about termination purpose of SSL? is it cutting off all that connected chains from the establishment? thus improve performance?
    – Seandex
    May 18, 2020 at 5:36
  • @tux Can i use ssl_preread to redirect http, https and SSH ? all three? Can you please add this to your example if possible?
    – Lonko
    Sep 19, 2021 at 11:30
5

Or, you could proxytunnel ssh through your nginx with HTTP/S connection

Take a look: proxytunnel

2

This question is slightly related to another one I've answered a while ago:

https://stackoverflow.com/questions/34741571/nginx-tcp-forwarding-based-on-hostname/34958192#34958192

Yes, it is technically possible to differentiate between ssh and https traffic, and route the connection appropriately; however, nginx currently doesn't have such support, to my knowledge.

However, what you could do is simply run sshd directly on the https port in addition to the ssh one (/usr/sbin/sshd -p 22 -p 443), and/or use the firewall and/or port knocking in order to differentiate where connections to port 443 get routed to.

1
2

You could setup an iptables rule to forward connections from your well known cafe on port 443 to port 22. Alternatively bounce the traffic off a port relay elsewhere on the internet, changing the pirt number.

Just trying to solve the problem with nginx isn't going to work.

2

There exists unmaintained patch for nginx to support protocol multiplexing for SSH and HTTPS: https://github.com/shawnl/nginx-ssh

However it is not maintained anymore, non-nginx project which supports what you are looking for is: https://github.com/yrutschle/sslh

I suggest you use SSLH infront of your nginx and ssh server.

1

As far as I know, currently nginx doesn't support it.

In SSH-2, client will send a hello message to server:

When the connection has been established, both sides MUST send an identification string. This identification string MUST be

 SSH-protoversion-softwareversion SP comments CR LF

In TLS 1.2, clients need to send first:

  Client                                                Server

  ClientHello                   -------->
                                                   ServerHello
                                            [ChangeCipherSpec]
                                <--------             Finished
  [ChangeCipherSpec]
  Finished                      -------->
  Application Data              <------->     Application Data

Clients can use this information for implementation.

2
  • 1
    This answer is a bit inaccurate. Historically the SSH protocol had the server speak first. But in SSH-2 it is allowed for the client to speak without waiting for the server to speak first. Any OpenSSH client less than a couple of years old will speak without waiting for the server to speak first - as long as you don't enable SSH-1 support in the client.
    – kasperd
    Oct 8, 2016 at 18:29
  • Yes you are right, I will fix my answer. Oct 8, 2016 at 18:32
1

Since Nginx Version 1.9.0,NGINX support ngx_stream_core_module module, it should be enabled with the --with-stream. When stream module is enable they are possible to ssh protocol tcp proxy

stream {
upstream ssh {
    server 192.168.1.12:22;
}
    server {
    listen        12345;
    proxy_pass    ssh;

} }

https://www.nginx.com/resources/admin-guide/tcp-load-balancing/

1
  • 2
    This doesn't let you multiplex. Oct 25, 2017 at 10:29
0

Maybe you can use nginScript to implement your own multiplexer? See here for an introduction: https://www.nginx.com/blog/introduction-nginscript/

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .