Buffer can underrun, or connections can be closed if data is piped / uploaded to a server.

Let's call that hipothetical program "lb" (large buffer).

cat data.iso | dvd-writer-app /dev/sr0

DVD writer buffer can underrun, and then DVD writing fails. However when using our "lb" app:

cat data.iso | lb --keep-buffer 32M | dvd-writer-app /dev/sr0

In this case it can prevent the underrun, if it reserves a 32M buffer internally.

Sometimes I do this:

(for $(seq -w 1 10);do wget -q "http://example.com/file.$i";patch_that_file "file.$i";cat "file.$i";rm "file.$i";done)|curl -T - ftp://yaexample.com/big_concatenated_file.bin

I fetch several files, one after another, patch them somehow locally on the disk (which needs random seeks, I can't do it while downloading), and concatenate them together, then finally upload their concatenated content to a server. I do this when I have low disk space for downloading all the files. So I download them one by one, patch them locally as needed, and their concatenated content will leave the for-loop. When a file was processed, it is deleted with "rm" immediately, to release disk space for the next one, and so on.

Unfortunately the remote server is configured in a way, that it closes the connection, when no data is received for at least 20 seconds. However fetching a file with "wget" takes some time (1-2 minutes or even more per file), patching them also takes some time (several seconds). That is a hitch, because connection is closed by the remote server between two consecutive file fetches/patches if it takes more than 20 seconds.

If I had a filter, called "lb", which maintains a large enough internal buffer, then I could do:

(for $(seq -w 1 10);do wget -q "http://example.com/file.$i";patch_that_file "file.$i";cat "file.$i";rm "file.$i";done) | lb --maxtime $[ 4*60 ] --min-bandwidth 50K | curl -T - ftp://yaexample.com/concatenated_file.bin

It would be said to it: please reserve an internal buffer, which is enough for 4*60 seconds to ensure a minimal bandwidth of at least 50 KB/s. It is only an about 12 MB buffer till it runs out, if I do the math properly.

Filter should be working in a way, it lets through and transmits the data with full bandwith when there is incoming data available on its standard input (STDIN). When it receives no data on its STDIN, then it must turn to its internal buffer, (which is continuously maintained of course) slowing down, and provide only a minimal (here 50K) bandwidth on its standard output (STDOUT). In this case the remote server won't close the connection, and I could upload the large file in order.

Does such a filter exists, or should I code it for myself in an appropriate programming language?

PS.: Fetching the first file, and preloading the "lb" internal buffer is not a problem for me, at least in this particular case, because there is a complex uploading app at the end of the piping chain, and I can modify it to start/open the connection right after the incoming data becomes available.

  • If you download them one by one you're still downloading all of them. What you're asking for would just be an upload speed limit or alternatively an FTP client that sends NOOP periodically. – Seth Mar 1 at 13:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.