1

I have a process that outputs a line for every progress update (sidenote: it does clear/replace the line, no pure newline break).

I want to save the latest line of that process to an output file or truncate the output file to keep the size manageable.

At the moment I have genrtr > genrtr.log and with a cron I tried to use > genrtr.log but it doesn't work. Also rm genrtr.log doesn't help because then the process stops updating the file.

I understand why those don't work, but wonder how to restructure it so it fits my needs.

Tried genrtr | sed -ne '$w genrtr.log' but then it waits for the process to end before writing to the file.

Clarifications: The process produces output every 1 second and unless the server crashes the process will keep running for ever.

3
  • What exactly is genrtr? Can you edit it?
    – terdon
    Apr 13, 2013 at 15:01
  • It's a program written in C that I can edit but have almost no knowledge on C and I would rather not mess with that for maintenance purposes.
    – Sev
    Apr 13, 2013 at 16:54
  • Maybe logrotate can help you? I do not have enough experience with that, so I don't know whether that's what you are looking for.
    – ignis
    Apr 19, 2013 at 18:31

4 Answers 4

0

i was wondering whats the frequency of the run ? ....

try tee to redirect output to the logfile , its a utility often used to make a copy of the strem and redirect that copy into the output file

use : 
  command | tee command_result.log 

you could have a monitor function in a wrapper script that invokes this program which will delete the top 10 lines after some intreval ....... this way your log file wont get more than a few KB

also if there are useless spaces in between you could use translate "tr" utility to squeeze ... example :

Nitin@Kaizen ~
$ df -h | head -1
    Filesystem      Size  Used Avail Use% Mounted on

Nitin@Kaizen ~
$ df -h | head -1 | tr -s ' '
    Filesystem Size Used Avail Use% Mounted on   *** note the squeeze in space 

hope this helps

2
  • I didn't know of tee, but it seems to have the same effect as > while it adds data while the process is running, when I > command_result.log the file is unaffected.
    – Sev
    Apr 13, 2013 at 18:24
  • what tee does is that it duplicates the stream , but here it might help :: command | tee lofile 2>&1 > /dev/null .... this way you get a seperate copy of the log that you can manipulate in your wrapper script to extract usefiull info .... or whatever , the dev/null is a special file that dosent grow .... so your space issue is also managed this way.
    – Nitin4873
    Apr 18, 2013 at 5:40
0

There are a few things you could try. The easiest is ti just print the last lines of the file:

tail genrtr.log

Then, once the process finished delete the log file. Another option, would be to periodically overwrite the file.

  • Launch the process in the background:

    genrtr > genrtr.log &
    
  • Overwrite the contents of the logfile:

    echo > genrtr.log
    

The file is now truncated but will continue to be updated by gentr so the next update report will be written to it. You could automate this, for example, truncating the file if it gets larger than 1MB:

while true; 
  do if [ $(stat -c%s genrtr.log) -gt 1000000 ]; then 
    tail genrt.log > /tmp/foo && cat /tmp/foo > genrt.log; 
     fi;
done

That little scriptlet will run until you stop it (while true;), and every time that genrtr.log is larger than one MB, it will keep the last few lines and delete the rest of the file.


UPDATE:

As Scott very correctly pointed out below, if your output contains \r to clear the line, tail will not work as expected. This, however, should:

while true; 
  do if [ $(stat -c%s genrtr.log) -gt 1000000 ]; then 
    tail genrt.log | perl -pe 's/.+\r(.+)/$1\n/' > /tmp/foo && cat /tmp/foo > genrt.log; 
     fi;
done

The perl command deletes everything before the last \r and prints the last "line" (the data after the \r and a newline. The result should be that the list line is kept, the rest of the file is cleared and the file continues to be populated.

7
  • Unfortunately the ` > ` redirect doesn't seem to allow any modification to the target file while the process is running and genrtr will run for ever.
    – Sev
    Apr 13, 2013 at 18:53
  • 1
    @terdon: The question says that the program doesn’t end the update messages with newlines, but rather with escape sequences to erase and re-use the same line on the screen. Do you believe that tail will behave intelligently with that input? (Maybe with tail -400c, to get the last 400 characters and hope that that is at least the last five lines.) Apr 15, 2013 at 21:12
  • 1
    @Sev: We seem to be having a failure to communicate. Are you saying that, if you type genrtr > genrtr.log, the output still comes out on the screen, and genrtr.log is empty? If so, what happens if you type genrtr 2> genrtr.log? Apr 15, 2013 at 21:14
  • genrtr > genrtr.log doesn't show output on the screen, it redirects the output correctly to the file, but while the process is running, you can't amend the file successfully (tried > genrtr.log and similar) I tried genrtr 2> genrtr.log and it saved the top info text of the initial output on the file, but the main repeating progress update text shows up on the screen instead.
    – Sev
    Apr 16, 2013 at 9:13
  • @Sev: Belatedly, welcome to Super User. For your information, when you respond to a comment (in a new comment), it’s conventional to mention the author’s name, preceded by “@”, as in “@Scott”. That way he gets notified. You can abbreviate, and you can mention multiple names, as in “@George, @Scott”. See the Replying in comments paragraphs of the Comment formatting section of the Markdown Editing Help page. Apr 17, 2013 at 22:23
0

I believe that this is going to be very tricky to do without either modifying genrtrc or writing a new program.  If you’re more comfortable doing the latter, I suggest this outline:

int   c;
FILE  *fp;

fp = fopen(log_file, "w");
if (fp == NULL) (Handle errors)
while ((c = getc()) != EOF)
{
    putchar(c);
    if (c ==escape sequence that ends a line)
    {
        fflush(fp);     //You should probably check for errors here, too.
        rewind(fp);
    }
    else
        putc(c, fp);
}

This acts like a combination of tee and tail -- reading standard input, and writing it to standard output and a file -- with the difference that it keeps only the last line in the file.  Then you would run

genrtr |the_above_program

0

Solution

At the moment I have genrtr > genrtr.log and with a cron I tried to use > genrtr.log but it doesn't work.

This approach can easily be fixed by using genrtr >> … instead of genrtr > … (while still using > in cron to truncate the file).

The difference is explained in this another answer of mine which can be summarized by the following statement:

>> is essentially "always seek to end of file" while > maintains a pointer to the last written location.

Read the linked answer, it totally matches your case.


Side note

Also rm genrtr.log doesn't help because then the process stops updating the file.

Strictly it does not stop updating the file. The file gets unlinked from the directory, but it's still open, still written to, it still consumes more and more space in the filesystem.

Any new file is a different file even if it takes the same pathname. The process does not update the new file because it never opens it, it doesn't even notice it. The file descriptor used by the process still leads to the old (deleted) file.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .