Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

In linux tcsh, if I am creating a log file like:

CMD > cmd.log

How can I make the write buffer, which caches the data before sending it to disk, be larger than the default? How can I buffer more write data going to cmd.log and decrease the frequency of writes to that file handle?

EDIT: I still haven't figured out how to do this, but we found the issue with our system. Someone was thrashing an NFS disk filer with multiple write of the same data. They weren't even using the local disks to incrementally generate the file and then copying the results to NFS.

Also, you can use

CMD_BLAH | gzip --stdout > log.gz

Which will do some caching in main memory as well as reducing the final output file size ( usually).

share|improve this question
    
The answer, of course, it to recompile tcsh linked to a hacked version of the standard library. I didn't post this answer because a.) it wouldn't solve any problem (then unknown to us) he had, and b.) nobody should be using tcsh anyway, since bash has such a superior command line history, and posix-like compatibility as well. –  kmarsh Dec 7 '09 at 16:55
    
I work in a corporate environment. We use tcsh. I use bash on my home machines. –  Ross Rogers Dec 14 '09 at 17:41
add comment

2 Answers

up vote 3 down vote accepted

you can use 'pv' to 'throttle' whats beeing piped through.

-B BYTES, --buffer-size BYTES
   Use a transfer buffer size of BYTES bytes. A suffix of 
   "k", "m", "g", or "t" can be added to denote kilobytes (*1024),
   megabytes, and so on. The default buffer size is the block
   size of the input file's filesystem multiplied by 32 (512kb max),
   or 400kb if the block size cannot be determined.

applied to your given example it would look like:

% CMD | pv -B 1m > cmd.log
share|improve this answer
    
This is beautiful and exactly what I hoped for. Thank you! –  Ross Rogers Dec 14 '09 at 17:41
add comment

You could try setting up a named pipe (man mkfifo). Write the output of your command to the named pipe. Setup another process to read periodically. This won't change the read/write buffers, but it will control the flow of data flushed to disk based upon a time interval that you specify. If you want to control your write buffer size you can write a simple C program that uses setvbuf.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.