1

I'd like to pipe a command into a slower one, with a rather big (~200GB) buffer in between. Here is an example use-case:

command1 | buffer -f file.buffer | command2

Is there a command to do that? The ones I saw only buffered to memory.

Now that I think about it, would this achieve the desired result?

#!/bin/bash
# buffer.sh
FILE="$1"
tail -F "$FILE" 2> /dev/null &
cat > "$FILE"

Though I'm not sure how to stop the tail once it has read everything...

3

I just found mbuffer.

Apparently the option -t uses a temporary file for huge buffer, which is exactly what I was looking for. Alternatively with -T /path/to/file I can choose which of my mounted filesystems will hold the file.

I also note the file gets deleted just after it's opened, so ls doesn't show it. Still ls -l /proc/<PID of mbuffer>/fd confirms it's there.

So my example becomes:

command1 | mbuffer -T file.buffer -m 200G | command2

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.