0

I have a web server which is sat behind Carrier Grade NAT and therefore needs an external server to give me a stable IP address. To this end I have an AWS EC2 instance running SLES, and I have a reverse forwarded ssh tunnel initiated from the web server to serve up incoming requests on port 80. Though I don’t think it is relevant to the problem per se, it has been helpful for troubleshooting, so for context I should state that the web server is listening on a non-standard port, lets say 8080, and a prerouting firewall rule on the AWS instance redirects requests on port 80 to 8080. The ssh tunnel uses autossh and takes it’s configuration from a config file, but is equivalent to the following ssh command.

ssh -N -R 8080:localhost:8080 -i ~/.fwd/portal.uk.pem [email protected]

The firewall redirect rule looks like this

iptables -A PREROUTING -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 8080

There are also firewall rules on the input chain that open up ports 80 and 8080 from all sources, and I have configured a security group in the EC2 management console to similarly allow traffic into ports 80 and 8080 from all sources. With all this in place everything works just as intended, pages are served to the outside world.

I’ve come to the point where I need to start serving up pages over https and gone about creating the same config for port 443, but I keep coming up against a failure to connect to port 443 on the AWS instance. First I’ve tried a straight-through tunnel for port 443. A firewall input rule and security group rule is in place to allow connections to 443, but attempting to set up the tunnel results in a listening port warning.

ssh -N -R 443:localhost:443 -i ~/.fwd/portal.uk.pem [email protected]
Warning: remote port forwarding failed for listen port 443

The ssh process stays alive on the web server but seems to be dead for all intents and purposes, any attempt to connect to 443 on AWS with telnet gets refused. To be clear, the web server is definitely listening on port 443 because I can connect with https requests locally. All the info I’ve found on this warning suggests this is likely caused by a process that still has a grip on the port and can be found using either netstat -plant or similar, or lsof -i -t. Neither of these are showing any process active on 443 on AWS and in any case, I’ve tried rebooting the AWS instances to clear any persistent processes and still get the same symptoms when it comes back up.

I’ve then tried to replicate the config for port 80 by changing the web server listening port to something non-standard, say 8443, and adding another prerouting rule, input rule and security group exactly as before. I can now set up the tunnel on port 8443 successfully without receiving the warning but I still get connection refused on AWS port 443 with telnet and https requests are not getting through. However, if I telnet on the AWS machine to port 8443 it is successful. To me this suggests the tunnel and web server config is working just fine and it is definitely something blocking port 443 on AWS but I’m at a loss as to where to turn next. So, specifically, my question is if there are no listening processes shown by netstat or lsof and the firewall is open what else should I be looking for?

2
  • Non-root users typically can't bind to ports below 1024 on unix systems. So sshd running under the ec2-user ID probably can't bind to port 443. I don't know the simplest way to deal with that on ec2.
    – Kenster
    Feb 17, 2019 at 23:17
  • That might explain why the straight-through reverse forward didn't bind and is good to know, however I haven't had the same problem on port 80, plus I (think) I have successfully set up the tunnel on port 8443 and so nothing is bound to port 443, any traffic arriving at 443 should be getting rerouted to port 8443 due to the prerouting rule, which was setup as root using sudo iptables ...
    – hobrob
    Feb 18, 2019 at 13:05

0

You must log in to answer this question.