0

I have a java program running on a Centos7 server that continuously "tails" a file on another Centos7 server using the JSch library to run an ssh exec process to run a shell script on the target machine.

Recently we had an issue where it stopped getting data from the file without any kind of disconnection showing up in any of the logs (it was a silent issue, so it looks the same as if there was no data to get).

Eventually I found that the issue was caused when the firewalld daemon was stopped and started, after starting it again the data stops flowing. If you then stop the firewalld daemon the flow of data resumes, but otherwise the only solution was to restart the Java program itself after the firewalld daemon was started.

To reproduce the issue without the Java code:

  • Have two Centos7 instances where one is able to ssh to the other
  • Have two terminals open on the first machine (the one that would be running the software)
  • Make sure the firewalld daemon is already running (systemctl status firewalld / systemctl start firewalld)
  • Run an ssh exec command on the second terminal like this (using whatever credentials needed): ssh -t user@second-server watch -n 1 date
  • You will now be seeing the current time updating every second on that terminal, but being run on the target server
  • Stop firewalld on the first terminal (systemctl stop firewalld)
  • Start firewalld again on the first terminal (systemctl start firewalld)
  • You will note that the date/time has stopped updating in the second terminal
  • Stop firewalld again on the first terminal (systemctl stop firewalld)
  • You should see the date/time updates resuming on the second terminal

I have tried turning on the highest level of debug output in firewalld (--debug=10 in /etc/sysconfig/firewalld config file) and also enabling logging of all denied requests (firewall-cmd --set-log-denied=all) however there is nothing in the logs to indicate what it is doing that interrupts the in-progress ssh exec session.

Is there any way to configure firewalld so that it does not interrupt the in-progress ssh exec session?

8
  • why is firewalld being stopped and started? firewall-cmd --reload should keep the connections after a config change. Mar 5, 2021 at 14:55
  • First noticed the issue as part of an ansible script for building and configuring the machine instance, did not know this was the cause at the time and added step to restart software at end and thought that would be end of it. Then later the issue recurred on one of two instances when an attempt to install machine monitoring software automatically restarted the firewall.
    – Ross
    Mar 6, 2021 at 8:33
  • Yes, now that we know that it was firewalld we could avoid restarting it and/or keep it in mind, but because it happens in such a way that the software does not detect that it is disconnected the system still shows as "healthy" when it is not. So in the event that someone in the future NOT knowing this restarts firewalld for any reason, I would like it if it did not cause this issue, or at the very least was more violent in the suppression so the software can detect the issue and reconnect automatically.
    – Ross
    Mar 6, 2021 at 8:35
  • @strobelight Sorry, new at this and didn't know how to do a reply comment, see above responses.
    – Ross
    Mar 7, 2021 at 22:16
  • I tried your steps on my Centos 8, and the watch of date kept on going after restarting firewalld on the VM running the date. You might try the ssh config option TCPKeepAlive. Mar 8, 2021 at 14:24

0

You must log in to answer this question.

Browse other questions tagged .