0

I have a setup wherein one "master" Linux system communicates with 3 "slave" systems also running Linux on a dedicated Ethernet interface(just the master & the 3 slaves). The slaves send data to the master via UDP every 5 ms or so. In addition, the master has apps that continuously pull files from all 3 slaves via FTP, SCP, etc. protocols.

The UDP packets need to be collected by the master as fast as possible, preferably within 3-4 ms. When I run the setup with just the UDP reception app running on the master, I see that this condition is easily met. However, when the FTP/SCP/etc. apps are also left running, there are spikes in the reception time. The size of the files being transferred is pretty less but a new file is retrieved from each slave about every second or so.

The fact that the results are good when running the setup without the file transfer apps active tells that Linux network "queuing/scheduling" seems to be giving similar priority to both UDP & the other protocols. Maybe it's even holding off UDP if a FTP is going on?

Is there a way to tell Linux (programmatically/commands) to give highest priority to UDP communication & "pause" other things like file transfer when a UDP message is ready for reception?

Edit 1: I added these to control the traffic based on the protocol type (UDP is protocol 17)

tc qdisc add dev eth0 root handle 10: prio
tc filter add dev eth0 parent 10: protocol ip prio 1 u32 match ip protocol 17 0xff flowid 10:1
tc filter add dev eth0 parent 10: prio 3 protocol all u32 match u32 0 0 flowid 10:3

The 3rd one is what I think is an "all match filter". However, this made no difference. I still get the same spikes.

  • 1
    That's not how UDP works, it's not 'collected' from the slave, it's sent blindly by the slave and the master receives it. if you're seeing delays it's either because the networking queue is full on the slaves, or the master, or because the master isn't scheduling correctly for your case. – djsmiley2kStaysInside Oct 19 '17 at 15:56
  • Does 5ms imply 5milliseconds or did you mean 5 mins. If the former you are on a fools errand because Linux is not a real time OS. You can prioritise traffic by adding in rate limiters/queues - you will want to do that on the senders. Google "advanced packet control" for howtos on traffic management. Also look at reducing the MTU sizes of all devices on your lan. This will decrease throughput but also decrease latency. – davidgo Oct 19 '17 at 18:34
  • Yes, I meant 5 milliseconds & I do realise that Linux isn't an RTOS. That said, I've also mentioned that when running the UDP app alone, the delays are well within limits. It's only when the FTP app too becomes active that the problem starts. This is a gigabit network reserved just for these 4 boards with no added hops, etc. I looked up into Linux traffic control & created 2 filters. However, they don't seem to be taking effect. Please look at the edit in the question. – John Smith Oct 22 '17 at 8:50
1

First: Linux is not a real-time operating system. There's no way to tell the kernel that some applications have to react to some event in a given timeframe, like 3-4 ms, and to give a guarantee that this happens. So whenever the system is loaded, you'll have to assume that there will be delays.

That said, you can tune things in favour of your UDP-packet receiving application:

  • Linux has network traffic control (see e.g. Traffic Control HOWTO, example script for DSL with limited bandwidth) which you can use to set up different queues with different priorities e.g. for your UDP packets and for large packets.

  • Linux has cgroups (control groups), which you can use to assign different I/O priorities and limits for your UDP-receiving process and the ftpd/sshd etc. processes, because my guess is that disk I/O by other processes can also stall your UDP-packet receiving application (experiment to find out if that is true).

Again: There is no way to guarantee anything.

  • I looked up traffic control ran these. "tc qdisc add dev eth0 root handle 10: prio" , "tc filter add dev eth0 parent 10: protocol ip prio 1 u32 match ip protocol 17 0xff flowid 10:1" , "tc filter add dev eth0 parent 10: prio 3 protocol all u32 match u32 0 0 flowid 10:3".The 3rd one is what I think is an "all match filter". However, this made no difference. I still get the same spikes. – John Smith Oct 22 '17 at 8:42
  • I can't remotely debug your problem, sorry. The first step would be to find out if the packet queuing is actually related to your spikes, i.e. get some logging about how your queues are used. And if it's the I/O scheduler which is causing the spikes, then of course traffic control won't help ... – dirkt Oct 22 '17 at 8:46
  • I definitely appreciate your help! Do those filters (I've now added them into the question itself. Properly formatted) look correct to you? How do I check if the 3 prio bands are being used as expected? I can't find any reference on that. – John Smith Oct 22 '17 at 8:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.