When re-directing to a file in linux tcsh, how can I change the write buffer size to be larger?

Question

In linux tcsh, if I am creating a log file like:

CMD > cmd.log

How can I make the write buffer, which caches the data before sending it to disk, be larger than the default? How can I buffer more write data going to cmd.log and decrease the frequency of writes to that file handle?

EDIT: I still haven't figured out how to do this, but we found the issue with our system. Someone was thrashing an NFS disk filer with multiple write of the same data. They weren't even using the local disks to incrementally generate the file and then copying the results to NFS.

Also, you can use

CMD_BLAH | gzip --stdout > log.gz

Which will do some caching in main memory as well as reducing the final output file size ( usually).

Answer 1

you can use 'pv' to 'throttle' whats beeing piped through.

-B BYTES, --buffer-size BYTES
   Use a transfer buffer size of BYTES bytes. A suffix of 
   "k", "m", "g", or "t" can be added to denote kilobytes (*1024),
   megabytes, and so on. The default buffer size is the block
   size of the input file's filesystem multiplied by 32 (512kb max),
   or 400kb if the block size cannot be determined.

applied to your given example it would look like:

% CMD | pv -B 1m > cmd.log

Answer 2

You could try setting up a named pipe (man mkfifo). Write the output of your command to the named pipe. Setup another process to read periodically. This won't change the read/write buffers, but it will control the flow of data flushed to disk based upon a time interval that you specify. If you want to control your write buffer size you can write a simple C program that uses setvbuf.