New Linux install, can't get ssh or http connections working, connections are made but drop?

Question

So, I / we have brand new Fedora Core 28 installation we've been trying to bring online. The installation went perfectly so far as we could tell. It has two network cards, one for an internal net, one with a fixed IP address. It's a server, so its primary job is to run a web server, httpd. And it has to be accessible from other than the console, so it's running sshd.

At first, everything seems OK. The first net use was to use yum to install many packages, there was no problem. We added gnome to get a GUI, added firefox to get a web browser and then used that to help get data for troubleshooting. Connections on the inner net are fine. Connecting with SSH is not a problem and at least one account was set up to use encryption keys to avoid a password. Connecting to the new web server seems fine too, using both internal and external IPs. So, we felt we had confirmed the ssh and web server were configured properly. ...Initially, we didn't know anything was wrong.

When we got around to connecting from the outside world is when things "went sideways." Attempts to connect with SSH just failed outright and connections to the web server also appeared to fail outright as well.

FIRST attempt was with the firewall. How's it configured? Is it iptables or firewalld? How do we ensure we can get through from the outside? We found that somehow we'd configured the exception in /etc/firewalld/zones - the default zone is "FedoraServer.xml" in our case - as httpd, and we noticed on a reboot that it complained about that and changed it to http, with which we no longer noticed any complaint. Still no joy.

Then there's nmap! This is a port scanner for Linux systems to look at any network target's open ports. This seemed to show we were not blocked by the firewall. We confirmed iptables was not in use and it was firewalld, so we turned it off and still no joy.

Then we considered the older equipment. Twisted pair ethernet ports are bi-directional devices so was it possible inbound on the exterior port was good while outbound was bad? OR, maybe, there was a bad port on the switch? So, we tried swapping them, internal and external. No change.

We then considered that MAYBE the router was filtering, even though it shouldn't. The new box was a replacement for an older box that was retired, and the old one was fully functional, but who knows? Maybe the router has gone crazy? So, we went to log into it and see. But, unfortunately, nobody could find the router's password.

Instead, since the router only knows the systems by their external IPs, we shut down a fully-functional system and had the new one pick up the shut-down system's IP address. This eliminates the router as a potential cause, at least we thought so. However, the problems persisted.

It was suggested that maybe the router is at fault as perhaps it was grabbing MAC addresses, and so to eliminate that possibility, we rebooted it. Again, no improvement.

I then had the insight to ask our external tester to please increase the verbosity of ssh connection diagnostic data by adding, increasingly, -v to the command line (you can do up to three), and our internal sys-admin to please check the Apache logs. And this was very helpful. We found that the external perception was ssh was getting to our sshd daemon but the connections were being dropped and also the web server logs were showing correct inbound connections from just the places we expected, and logging "200" as the result, which means httpd thought the web clients were getting their pages. But it wasn't true.

OK, so now what?

Answer 1

After great agony, as documented above, I felt a thorough, pedantically detailed survey was in order, and it was done. We found the solution, and it was exceptionally simple!

It's important to understand that several long-time experts with more than 30 years in the field each missed this!

The cause was simply that the default route on the box had a one character flaw; it was addressing the router, but the wrong one of the two IP addresses the router responds to! One address is for manipulating the router via a port 80 connection and the other is for routing outbound traffic.

It's helpful to remember that inbound connections are good until some certain point when the return effort switches focus, and then, when the response is initiated from the inside and not outside, the route has to be correct, and it wasn't. This is why it failed.

I'm not entirely sure why the outbound-initiated connections worked as well as they did but a reasonable guess is that the router realized the packets it was getting on the wrong IP address were still headed outbound and routed accordingly. Hmmm. Input on that is appreciated.

Simply, just because you're talking to the router doesn't mean you have the outbound route address correctly! At least, this was true for us.