I am trying to log data from a serial device onto an OpenWrt router.
The data is all ASCII based, 115200 baud, 8N1, but only transmit, receive and ground are connected (no flow control). The device sends 50 lines of text per second, of 76 characters per line (plus CR+LF this would be 50x78 = 3900 char/s, and considering 1 start bit (?), 8 data bits, and 1 stop bit per character this would seem to need 10 bits per char, so 39,000 bit/s of the available 115200 -- it should not be maxed out right?)
The problem is data is being dropped, lines are being truncated, merged, etc in what I am saving to NVRAM. And if I run this through head or cut to stop it after say exactly 30000 lines, which should take exactly 10 minutes, it takes 11 or 12 minutes to get to that number of lines because of the lost/truncated/merged lines. For example the following should show exactly the same number of fields on every line but shows 3 examples of damaged lines:
$label,014704260,000000000,000000000,000000000,000000000,000000000,453266711
$label,014704280,000000000,000000000,000000000,000000000,000000000,498035313
$label,014704300,000000000,000000000,000000000,000000000,000000000,564242105
$label,014704320,190418.30,000000000,000000000,000000000,000000000,458805062
$label,014704340,000000000,000000000,000000000,000000000,000000000,447439419
$label,014704360,000000000,000000000,000000000,000000000,000000000,515812674
$label,014704380,000000$label,014704500,000000000,000000000,000000000,000000000,000000000,430368215
$label,014704520,190418.50,000000000,000000000,000000000,000000000,474793672
$label,014704540,000000000,000000000,000000000,000000000,000000000,514101937
$label,014704560,000000000,000000000,000000000,000000000,000000000,498089337
$label,014704580,000000000,000000000,000000000,000000000,000000000,479420800
$label,014704600,000000000,000000000,000000000,000000000,000000000,568108911
$label,014704620,000000000,000000000,00000000000,497468975
$label,014704640,190418.60,000000000,000000000,000000000,000000000,509747997
$label,014704660,000000000,000000000,000000000,000000000,000000000,441899024
$label,014704680,000000000,000000000,000000000,000000000,000000000,543482880
$label,014704700,000000000,000000000,000000000,000000000,000000000,445069837
$label,014704720,190418.70,000000000,000000000,000000000,000000000,517975535
$label,014704740,000000000,000000000,000000000,000000000,000000000,477103089
$label,014704760,000000000,000000,000000000,483954418
$label,014704860,000000000,000000000,000000000,000000000,000000000,484600994
$label,014704880,000000000,000000000,000000000,000000000,000000000,489675319
Also field 2 is a millisecond counter increasing by 20 per line, and field 3 when present (every 5th line) is hhmmss.ss. This shows that at the same times that lines are truncated/merged, there are also 3-5 lines (0.1 s) lost.
Clearly something is bottlenecked, falling behind, getting out of sync, buffer overrunning, etc.
I can even be satisfied with discarding 80% or 98% of the data (keeping only 1 line every 5, or 1 line every 50, as long as its consistent) but I can't find a low-level enough way to drop them without suffering from the same problem. I've tried stuff like awk 'NR % 5 == 0' /dev/ttyUSB0, with or without piping that to a fifo and then only logging from there to NVRAM, etc. Also tried various combinations of nice -n -19 and ionice -c 1 -n 0 which should be "real time" class on my first (awk) command that is reading from ttyUSB0, then I use lower priority on the filtered (smaller) data stream.
Is there a lower-level way or less CPU-intensive way to log this data or drop lines, say within the stty handler or elsewhere? Set a larger buffer on the tty? or what is my best approach here to either keep all data or predictably drop some without this random lost/merged line issue?
/proc/tty/driver/for your serial port statistics (e.g. are snapshots of the receive byte count close to 3900 char per sec?) and any overrun byte count. If you're willing to post your code, then ask at stackoverflow.com