I have a somewhat tricky question. I want to pipe result from one command to another, but since I'm working on system with limited memory, I want to make sure that the pipe won't use too much of that. But I don't want it to break when it hits the limit, just switch to using the disk as a temporary file.
The use case is as follows. I download a large file using curl or wget. I pipe the result to another program (to named pipe, actually, but that named pipe is immediately fed to another command). If everything goes well, the second command is able to consume the input faster than curl can output it (download is slower than the processing of the second command).
But sometimes things go wrong and the second command needs some time to start consuming it. It probably will, eventually, but since that second command needs a bit of RAM and I have that resource limited, I want to switch to writing to disk if the pipe starts to use more than, let's say, 200 MB of RAM.
That second command may even need more time to start than the time needed to download the file. In that case the downloaded file should be completely written to disk, so that the second process could consume it later.
Is there any solution for such a silly problem?