1

I am trying to log data from a serial device onto an OpenWrt router.

The data is all ASCII based, 115200 baud, 8N1, but only transmit, receive and ground are connected (no flow control). The device sends 50 lines of text per second, of 76 characters per line (plus CR+LF this would be 50x78 = 3900 char/s, and considering 1 start bit (?), 8 data bits, and 1 stop bit per character this would seem to need 10 bits per char, so 39,000 bit/s of the available 115200 -- it should not be maxed out right?)

The problem is data is being dropped, lines are being truncated, merged, etc in what I am saving to NVRAM. And if I run this through head or cut to stop it after say exactly 30000 lines, which should take exactly 10 minutes, it takes 11 or 12 minutes to get to that number of lines because of the lost/truncated/merged lines. For example the following should show exactly the same number of fields on every line but shows 3 examples of damaged lines:

$label,014704260,000000000,000000000,000000000,000000000,000000000,453266711                                                                                                                                        
$label,014704280,000000000,000000000,000000000,000000000,000000000,498035313                                                                                                                                        
$label,014704300,000000000,000000000,000000000,000000000,000000000,564242105                                                                                                                                        
$label,014704320,190418.30,000000000,000000000,000000000,000000000,458805062                                                                                                                                        
$label,014704340,000000000,000000000,000000000,000000000,000000000,447439419                                                                                                                                        
$label,014704360,000000000,000000000,000000000,000000000,000000000,515812674                                                                                                                                        
$label,014704380,000000$label,014704500,000000000,000000000,000000000,000000000,000000000,430368215                                                                                                                 
$label,014704520,190418.50,000000000,000000000,000000000,000000000,474793672                                                                                                                                        
$label,014704540,000000000,000000000,000000000,000000000,000000000,514101937                                                                                                                                        
$label,014704560,000000000,000000000,000000000,000000000,000000000,498089337                                                                                                                                        
$label,014704580,000000000,000000000,000000000,000000000,000000000,479420800                                                                                                                                        
$label,014704600,000000000,000000000,000000000,000000000,000000000,568108911                                                                                                                                        
$label,014704620,000000000,000000000,00000000000,497468975                                                                                                                                                          
$label,014704640,190418.60,000000000,000000000,000000000,000000000,509747997                                                                                                                                        
$label,014704660,000000000,000000000,000000000,000000000,000000000,441899024                                                                                                                                        
$label,014704680,000000000,000000000,000000000,000000000,000000000,543482880                                                                                                                                        
$label,014704700,000000000,000000000,000000000,000000000,000000000,445069837                                                                                                                                        
$label,014704720,190418.70,000000000,000000000,000000000,000000000,517975535                                                                                                                                        
$label,014704740,000000000,000000000,000000000,000000000,000000000,477103089                                                                                                                                        
$label,014704760,000000000,000000,000000000,483954418                                                                                                                                                               
$label,014704860,000000000,000000000,000000000,000000000,000000000,484600994                                                                                                                                        
$label,014704880,000000000,000000000,000000000,000000000,000000000,489675319

Also field 2 is a millisecond counter increasing by 20 per line, and field 3 when present (every 5th line) is hhmmss.ss. This shows that at the same times that lines are truncated/merged, there are also 3-5 lines (0.1 s) lost.

Clearly something is bottlenecked, falling behind, getting out of sync, buffer overrunning, etc.

I can even be satisfied with discarding 80% or 98% of the data (keeping only 1 line every 5, or 1 line every 50, as long as its consistent) but I can't find a low-level enough way to drop them without suffering from the same problem. I've tried stuff like awk 'NR % 5 == 0' /dev/ttyUSB0, with or without piping that to a fifo and then only logging from there to NVRAM, etc. Also tried various combinations of nice -n -19 and ionice -c 1 -n 0 which should be "real time" class on my first (awk) command that is reading from ttyUSB0, then I use lower priority on the filtered (smaller) data stream.

Is there a lower-level way or less CPU-intensive way to log this data or drop lines, say within the stty handler or elsewhere? Set a larger buffer on the tty? or what is my best approach here to either keep all data or predictably drop some without this random lost/merged line issue?

  • First try to verify that all the text is properly received. Take a look at the files in /proc/tty/driver/ for your serial port statistics (e.g. are snapshots of the receive byte count close to 3900 char per sec?) and any overrun byte count. If you're willing to post your code, then ask at stackoverflow.com – sawdust Jan 2 at 23:06
0

The premise of the question, that CPU was the bottleneck, was wrong.

The problem appeared and disappeared consistently as I added and removed a USB hub from the set up. It was Dodoro branded, but also happened with another hub I had handy, Qicent branded, both had 3xUSB3 ports plus gigabit ethernet, or from lsusb:

Bus 001 Device 002: ID 05e3:0610 Genesys Logic, Inc. 4-port hub
Bus 001 Device 002: ID 2109:2813 VIA Labs, Inc.

So it might be similar to other hub questions like hubs are acting like bottleneck, where getting multi-TT hubs seems to be the solution. However in my case the issue happens with only 2 total devices, the hub and the FTDI USB-to-RS232 adapter, so it seems odd that a multi-TT hub would be needed to talk to only one device. Unless talking to the upstream (router's) USB controller counts as another device.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.