Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

In linux tcsh, if I am creating a log file like:

CMD > cmd.log

How can I make the write buffer, which caches the data before sending it to disk, be larger than the default? How can I buffer more write data going to cmd.log and decrease the frequency of writes to that file handle?

EDIT: I still haven't figured out how to do this, but we found the issue with our system. Someone was thrashing an NFS disk filer with multiple write of the same data. They weren't even using the local disks to incrementally generate the file and then copying the results to NFS.

Also, you can use

CMD_BLAH | gzip --stdout > log.gz

Which will do some caching in main memory as well as reducing the final output file size ( usually).

share|improve this question
The answer, of course, it to recompile tcsh linked to a hacked version of the standard library. I didn't post this answer because a.) it wouldn't solve any problem (then unknown to us) he had, and b.) nobody should be using tcsh anyway, since bash has such a superior command line history, and posix-like compatibility as well. – kmarsh Dec 7 '09 at 16:55
I work in a corporate environment. We use tcsh. I use bash on my home machines. – Ross Rogers Dec 14 '09 at 17:41
up vote 3 down vote accepted

you can use 'pv' to 'throttle' whats beeing piped through.

-B BYTES, --buffer-size BYTES
   Use a transfer buffer size of BYTES bytes. A suffix of 
   "k", "m", "g", or "t" can be added to denote kilobytes (*1024),
   megabytes, and so on. The default buffer size is the block
   size of the input file's filesystem multiplied by 32 (512kb max),
   or 400kb if the block size cannot be determined.

applied to your given example it would look like:

% CMD | pv -B 1m > cmd.log
share|improve this answer
This is beautiful and exactly what I hoped for. Thank you! – Ross Rogers Dec 14 '09 at 17:41

You could try setting up a named pipe (man mkfifo). Write the output of your command to the named pipe. Setup another process to read periodically. This won't change the read/write buffers, but it will control the flow of data flushed to disk based upon a time interval that you specify. If you want to control your write buffer size you can write a simple C program that uses setvbuf.

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .