I have a java program running on a Centos7 server that continuously "tails" a file on another Centos7 server using the JSch library to run an ssh exec process to run a shell script on the target machine.
Recently we had an issue where it stopped getting data from the file without any kind of disconnection showing up in any of the logs (it was a silent issue, so it looks the same as if there was no data to get).
Eventually I found that the issue was caused when the firewalld daemon was stopped and started, after starting it again the data stops flowing. If you then stop the firewalld daemon the flow of data resumes, but otherwise the only solution was to restart the Java program itself after the firewalld daemon was started.
To reproduce the issue without the Java code:
- Have two Centos7 instances where one is able to ssh to the other
- Have two terminals open on the first machine (the one that would be running the software)
- Make sure the firewalld daemon is already running (
systemctl status firewalld
/systemctl start firewalld
) - Run an ssh exec command on the second terminal like this (using whatever credentials needed):
ssh -t user@second-server watch -n 1 date
- You will now be seeing the current time updating every second on that terminal, but being run on the target server
- Stop firewalld on the first terminal (
systemctl stop firewalld
) - Start firewalld again on the first terminal (
systemctl start firewalld
) - You will note that the date/time has stopped updating in the second terminal
- Stop firewalld again on the first terminal (
systemctl stop firewalld
) - You should see the date/time updates resuming on the second terminal
I have tried turning on the highest level of debug output in firewalld (--debug=10
in /etc/sysconfig/firewalld
config file) and also enabling logging of all denied requests (firewall-cmd --set-log-denied=all
) however there is nothing in the logs to indicate what it is doing that interrupts the in-progress ssh exec session.
Is there any way to configure firewalld so that it does not interrupt the in-progress ssh exec session?
firewall-cmd --reload
should keep the connections after a config change.