2

I have a somewhat tricky question. I want to pipe result from one command to another, but since I'm working on system with limited memory, I want to make sure that the pipe won't use too much of that. But I don't want it to break when it hits the limit, just switch to using the disk as a temporary file.

The use case is as follows. I download a large file using curl or wget. I pipe the result to another program (to named pipe, actually, but that named pipe is immediately fed to another command). If everything goes well, the second command is able to consume the input faster than curl can output it (download is slower than the processing of the second command).

But sometimes things go wrong and the second command needs some time to start consuming it. It probably will, eventually, but since that second command needs a bit of RAM and I have that resource limited, I want to switch to writing to disk if the pipe starts to use more than, let's say, 200 MB of RAM.

That second command may even need more time to start than the time needed to download the file. In that case the downloaded file should be completely written to disk, so that the second process could consume it later.

Is there any solution for such a silly problem?

Improve this question
0

1 Answer 1

Reset to default
1

Your question is similar to this other one: Use disk-backed buffer between pipes, where the answer is mbuffer -T /path/to/file. The difference is in you idea to combine buffers:

I want to switch to writing to disk if the pipe starts to use more than, let's say, 200 MB of RAM.

Connect two buffers like this:

feeder | mbuffer -T /path/to/file -m 2G | mbuffer -m 200M | consumer

Data will flow as far as it can, so the 200 MiB memory buffer will be filled first (if ever). Only then the 2 GiB disk-based buffer will start to hold data.

Notes:

  • It seems the file gets allocated in its full size at the very beginning. If there's no space left on the device, the whole pipe will fail early. This introduces some initial delay though.

  • In my tests the first mbuffer writes to the file even when the second one reads from the pipe immediately. I think the first one might pass the data without touching the file; but it's not the case and the entire pipe is limited by the disk speed even if the disk-based buffer is not in use at all.

Improve this answer
6
  • Won't that always write to the disk, even for a brief while, since the disk-based buffer is between feeder and the memory-based one? Or maybe mbuffer only writes to disk if nothing receives from it? I don't completely understand how this works.
    – Anonymouse
    Jan 6, 2019 at 20:34
  • But maybe it doesn't always write to disk. “While ZFS receive can’t receive, mbuffer buffers, when zfs receive can receive, mbuffer sends it data as fast as it can” – that's what I found here. If mbuffers has a way to know if something receives from it, it could behave that way. Maybe someone else here knows anything about this? What else bothers me is using two pipes, I wonder if it affects RAM usage considerably.
    – Anonymouse
    Jan 6, 2019 at 21:03
  • @Anonymouse Unfortunately my recent tests show your doubts are justified. I read /dev/zero via two mbuffer-s to /dev/null. With two memory-based buffers it was fast. With a disk-based buffer and a memory-based one, the throughput was limited and it matched the speed of the disk.
    – Kamil Maciorowski
    Jan 6, 2019 at 21:11
  • That's unfortunate. But what may help in my case is that the sending process will be usually slower than disk speed, so the reading process will have time to catch-up. I would still leave that answer as it may help someone. I would accept yours, since it really could be helpful in my case, but I don't want to discourage others from posting something that could be better. Anyway, thanks!
    – Anonymouse
    Jan 6, 2019 at 21:17
  • 1
    Great, we'll wait together then. :-)
    – Anonymouse
    Jan 6, 2019 at 21:24

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .