6

I am working on a Linux embedded distribution and I need to monitor incoming and outcoming connections (if they are still actives etc.) and I am experiencing issues because I noticed that, even if the remote host turn off the connection, I can see the connection as ESTABLISHED through netstat.

I have two different versions of this Linux embedded system: one which uses an old 2.6.34 kernel and it works fine, while the one which is failing uses a kernel 3.18.18.

Looking for more information I suspect I had to set/change TCP timeouts, so I configured them through syctl.conf but doesn't change anything. The system which uses 2.6.34 kernel doesn't have TCP timeouts configured, so it makes me think that TCP are not the root of my problem.

Do you have an idea about what could be the reason of the fact that the connection remain as ESTABLISHED?

|improve this question
  • This is just the way TCP works. A connection can remain established for days or even weeks without a single packet being exchanged. If you don't send, there's no way to know what will happen if you do send. Your question is too vague to give you a helpful answer to though because you don't explain your outer problem, just ask for help with a proposed solution. Are you writing application code that actually implements the TCP endpoint? If so, what protocol does it use on top of TCP? Or are you trying to monitor some other application's connections? Or what? – David Schwartz Jan 5 '16 at 9:15
  • I am trying to monitor some other application's connections. If I was writing code I would have specified it. I have a remote host (Windows client) which connects to my Linux system (server) and once the client is connected it starts to retrieve data from server through TCP protocol. If the client disconnects, the connection on the server does not disappear. It remains as established. The same softwares are used on a old Linux system (as specified in the question) and it works all fine. – simo-zz Jan 5 '16 at 9:43
  • That's all normal. The disconnection should be detected as soon as the connected side tries to send data. Obviously, a disconnected device can't send any data, so there's no way to magically know that it's disconnected until it fails to respond to a query. – David Schwartz Jan 5 '16 at 9:45
  • 1
    The timeout only works if the connected side is trying to send data. Otherwise, there's nothing to time out. A TCP connection can remain established for days without a single packet being sent if neither side tries to send any data. In general, the only way to detect a disconnected device is to try to send data to it. If you don't ever send any data, you're simply not assured of detecting it. What application protocol (on top of TCP) are we talking about here? – David Schwartz Jan 5 '16 at 9:50
  • 1
    Well then the sending of data should cause the connection loss to be detected. The application on the other side needs to send data periodically too, or detect the absence of received data. It has to do something if it needs to detect connection loss. – David Schwartz Jan 5 '16 at 10:23
2

If a host disconnects from the other and you still see its connection as ESTABLISHED, it's probably related to the fact it's not honoring the TCP protocol and not closing the connection cleanly.

The netstat output is an interpreter of the current state of TCP connections. If a client wants to disconnect/close the socket that has previously been open and established, they should notify this to the remote system. This is done sending the FIN request to the other node (more info here), in this case, the server.

It they fail to do so, the client indeed disconnects, but the remote server keeps thinking that the client is still connected and thus keeping their state as ESTABLISHED, and that's where the tcp_keepalive_time parameter join the equation. As no further packets will be received, the kernel will wait the specified time to this parameter to time-out the connection and forcibly close it.

You can debug this issue using the tcpdump tool. You could simply listen to the connection from the client host on the server side and check if it fails to send the FIN request.

tcpdump host X.X.X.X and port Y
|improve this answer
  • OK. But I set the parameter tcp_keepalive_time many times, so there should be something else. – simo-zz Jan 5 '16 at 9:26
  • Did you run sysctl -p after modifying the file? – nKn Jan 5 '16 at 9:29
  • Yes, I also executed this command. It's all as I have configured. – simo-zz Jan 5 '16 at 9:36
  • Then either the client has been not closed, or for some reason there still are packets being received on the server side. Note that ACK are not considered "idle-breaking" packets in this case. Note as well that any modifications made to the file do not affect any previously ESTABLISHED sockets, so it just affects newly established ones. – nKn Jan 5 '16 at 9:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.