74

I have a Linux program which can write information to stdout and stderr.

I have a shell script which redirects that output to a file in /var/log. (Via >> and 2>&1.)

Is there a way to make that log file rotate? (max size, then switch to a different file, keep only a limited number of files)

I've seen a few answers which talk about the logrotate program, which sounds good, but they also seem to be focused on programs which are generating log files internally and handle HUP signals. Is there a way to make this work with a basic output redirection script?

4
  • 1
    Why can't you just modify the script that redirects the output to contain the logic for the rotation?
    – MaQleod
    Jun 1, 2011 at 5:05
  • I could, if someone could tell me how to detect the size of a logfile and rotate it out from under the stdout of a process without disturbing that process. I don't have to use logrotate if there's a better option, that just sounded like a convenient starting point for discussion.
    – Miral
    Jun 1, 2011 at 5:12
  • 2
    You don't have to use logrotate, but using logrotate just saves time... There is usually little point reinventing the wheel.
    – bubu
    Jun 1, 2011 at 5:43
  • Exactly my point. So is there a way to make logrotate work with an ongoing process's redirected stdout?
    – Miral
    Jun 1, 2011 at 6:12

11 Answers 11

64

As an alternative, you could pipe the output through tools designed with the primary purpose of maintaining size-capped, automatically rotated, log file sets, such as:

Tools to then process multilog-format log file sets include, amongst others:

Further reading

4
  • 2
    Thanks, multilog looks like just what I needed.
    – Miral
    Jun 3, 2011 at 1:36
  • multilog seems to be the only plug-and-play solution in debian (daemontools has an official package). But in my particular case, where I wanted to store the logs on a fat32 partition, the rotating does not work, since multilog wants to use a symlink. No plug and play for me:)
    – Arnout
    Apr 11, 2018 at 12:58
  • That cannot be true, as multilog nowhere creates or demands symbolic links. It is entirely neutral with respect to them.
    – JdeBP
    Sep 23, 2018 at 9:38
  • URL of "Don't use logrotate or newsyslog in this century" has extra dot
    – duyue
    Aug 3, 2019 at 6:09
19

the rotatelogs tool shipped with apache (in the bin dir) (see docs) takes input from stdin and rotates the log after some specific amount of time

1
  • I tried rotatelogs with truncation and file size limit (| rotatelogs -t $LOGFILE 10M) , but it does not append to the existing log file but overwrites it each time. Any idea what's missing?
    – not2savvy
    Sep 22, 2022 at 23:09
15

I had similar problem and had initially discard logrotate but it turned out logrotate can actually do this well, the key directive is "copytruncate". For some reason that term didn't come up on any of the googling I did, so I am adding this answer to clarify exactly how to use it for this case.

The trick is this only works if the redirect is done with ">>" (append) instead of ">" (create).

Config File (truncate.cfg):

/tmp/temp.log {
    size 10M
    copytruncate
    rotate 4
    maxage 100
}

Test Program (never gives up file). You can watch it filling disk and though deleting logfile will appear to work it will not actually free up any space on the disk:

cat /dev/urandom >> /tmp/temp.log

Running log rotate:

logrotate truncate.cfg
5
  • It's a nice theory, but it doesn't actually work on any system I've tried it on. The file does not actually get truncated and the program continues to append to it as before. (And yes, that's even with the redirection done via >>.) ((BTW, this answer was already given previously.))
    – Miral
    Oct 24, 2014 at 0:10
  • 1
    … as discussed in logrotate won’t truncate original file (on our Unix&Linux site). Also, echo /dev/urandom >> /tmp/temp.log will write 13 deterministic characters to /tmp/temp.log and then immediately exit. Did you mean cat /dev/urandom? Oct 24, 2014 at 0:36
  • 2
    Just tested here, and it seems to work. Content of file is copied to new log file. Original file is kept open by process and is truncated (size now shows 0).
    – Philipp
    Jan 21, 2015 at 9:09
  • 1
    Be careful about the possible data loss with copytruncate.
    – wanghq
    Oct 25, 2016 at 22:55
  • 1
    +1 although "Note that there is a very small time slice between copying the file and truncating it, so some logging data might be lost."
    – Tagar
    May 27, 2017 at 7:09
14

If you can have it go to one of the standard log streams (syslog, daemon, cron, user, security, mail, etc.) you can use the logger command and pipe to it instead.

echo "Hello." | logger -p daemon.info

Otherwise, you may be better off piping your logged content to a custom program or script to handle it, or look at setting up the logrotate configuration.

EDIT: JdeBP's answer seems to have what you may be looking for.

1
  • 2
    +1 for simplicity. BTW, you can also configure a custom facility (local0) instead of the standard ones (daemon in your example) Oct 9, 2012 at 7:56
5

I like multilog for my use case, but my use case is so trivial/simple that it is not laid out very simply in the docs/examples I found. Here is a simple multilog rotate example:

mkdir /tmp/myapp
./myapp | multilog t s10000 n5 '!tai64nlocal' /tmp/myapp 2>&1

Some notes:

  • this dumps logs into that /tmp/myapp/ directory
  • the s10000 represents 10,000 bytes*
  • the n5 represents 5 files.* The 'current' log counts as one of the files, so this includes 4 older logs + 'current'
  • this is based on, adapted from the examples provided by François Beausoleil at: http://blog.teksol.info/pages/daemontools/best-practices
  • I don't understand many of the options - I refer you to the various documentation to extend this...
  • The docs warn that: "Note that running processor may block any program feeding input to multilog." where 'processor' is the '!tai64nlocal' portion of the command

*For many applications, these are poor choices for long term use. They do allow you to observe the behavior of filling and rotating the logs more quickly than large logs do.

Finally, don't forget to nohup if required! With nohup, you do not need the 2>&1 (s=10e6 and n=30 here):

mkdir -p /tmp/myapp
nohup ./myapp | multilog t s10000000 n30 '!tai64nlocal' /tmp/myapp &

That command should get you started.

1
3

So is there a way to make logrotate work with an ongoing process's redirected stdout?

Yes! Check out the "copytruncate" directive offered by logrotate. Specifying that instructs logrotate to handle this very situation: a simple program that keeps its log file open indefinitely.

One caveat may or may not be a problem in your situation:

Note that there is a very small time slice between copying the file and truncating it, so some logging data might be lost.

Anecdotally, I've seen some "real world" log sources that do encourage users to apply this directive. There's some discussion of this option here.

3

Use split, it's part of coreutils. It can take stdin and split it into chunks (based on chunk size, or number of lines, etc.).

Example:

app | split --bytes 1G - /var/logs/put-prefix-here

Note dash (-) instructs "split" to use stdin instead of file.

6
  • Can you expand your answer to describe how to do that? Thanks.
    – fixer1234
    Jun 11, 2015 at 18:30
  • just updated my reply with example.
    – Nazar
    Jun 11, 2015 at 18:49
  • The 1G is an arbitrary size, after which it starts a new file?
    – fixer1234
    Jun 11, 2015 at 19:17
  • 1
    This isn't a particualrly good solution to the problem because it means you can end up with half a message in one file and half in the next. There's also the risk of data loss if the machine crashes while split has data in what could be a large buffer. Given that there are multiple tools that solve this problem properly, I don't think this kind of roll-your-own solution can be at all recommended. Jun 11, 2015 at 19:28
  • 1
    @David Richerby- how about adding -u for unbuffered?
    – Nick
    Dec 15, 2015 at 20:55
1

I just wanted to add to Sam Hendley's comment above:

The trick is this only works if the redirect is done with >> (append) instead of > (create).

I ran into the same problem where the original file just keeps growing if you use > (create) but if you use >> (append) Logrotate copytruncate works beautifully and as expected. The original file goes back down to zero bytes and the program continues writing.

Redirect STDOUT and STDERR to a rotating logfile:

  1. some-program.sh >> /tmp/output.txt 2>&1 &
  2. Create a logrotate config file under /etc/logrotate.d called whatever, output_roll in my case.

    Sample config for my case:

    /tmp/output.txt {
        notifempty
        missingok
        size 1G
        copytruncate
        start 0
        rotate 15
        compress
    }
    
  3. Setup your cron job inside the /etc/crontab file

    *  *  *  *  * root /usr/sbin/logrotate /etc/logrotate.d/output_roll
    

    This will check the file every minute. You can adjust to suit your needs.

  4. Start it up:

    $> service crond restart
    
  5. That's it

Note: I also had a problem with SELinux being set to SELINUX=enforcing so I set it to SELINUX=disabled.

1

I wrote a logrotee this weekend. I probably wouldn't if I've read @JdeBP's great answer and multilog.

I focused on it being lightweight and being able to bzip2 its output chunks like:

verbosecommand | logrotee \
  --compress "bzip2 {}" --compress-suffix .bz2 \
  /var/log/verbosecommand.log

There's a lot of to be done and tested yet, though.

1

you can use such a simple bash scripts: ./my_app|./log_rotate.sh log.txt 5 mylogdir 10 1000

log named as log.txt ,rotate 5 times,stored in mylogdir, rotate 10 times,log rotate on 1000 lines, dir rotate on each start up of my_app

log_rotate.sh:

#!/bin/sh
logname=$1
lognumber=$2
dirname=$3
dirnumber=$4
line=$5
date
mkdir -p ${dirname}
cd ${dirname}
rm "data${dirnumber}" -rfv
for i in $(seq $((dirnumber - 1)) -1 0)
do
        mv "data$i" "data$(($i+1))" 2>/dev/null && echo mv "data$i" "data$(($i+1))"
done
mkdir -p data0
cd data0
dd of="${logname}" bs=2048 count=${line} 2>/dev/null
date
mv "${logname}" "${logname}.first" 2>/dev/null && echo mv "${logname}" "${logname}.first"
while true
do
        for i in $(seq $((lognumber - 1)) -1 1)
        do
                mv "${logname}.$i" "${logname}.$(($i+1))" 2>/dev/null && echo mv "${logname}.$i" "${logname}.$(($i+1))"
        done
        mv "${logname}" "${logname}.1" 2>/dev/null && echo mv "${logname}" "${logname}.1"
        dd of="${logname}" bs=2048 count=${line} 2>/dev/null
        date
done
0

As we were not fully satisfied with listed tools here, we wrote another one (sorry!) called log_proxy.

Main features:

  • usable as a pipe (myapp myapp_arg1 myapp_arg2 |log_proxy /log/myapp.log)
  • configurable log rotation suffix with stftime placeholders (for example: .%Y%m%d%H%M%S)
  • can limit the number of rotated files (and delete oldest)
  • can rotate files depending on their size (in bytes)
  • can rotate files depending on their age (in seconds)
  • does not need a specific log directory for a given app (you can have one directory with plenty of different log files from different apps)
  • several instances of the same app can log to the same file without issue (example: myapp arg1 |log_proxy --use-locks /log/myapp.log and myapp arg2 |log_proxy --use-locks /log/myapp.log can run at the same time)
  • implemented in C (fast and do not eat a lot of memory)
  • configurable with CLI options as well with env variables
  • usable as a wrapper to capture stdout and stderr (log_proxy_wrapper --stdout=/log/myapp.stdout --stderr=/log/myapp.stderr -- myapp myapp_arg1 myapp_arg2)
  • binary releases with no dependency, even on a very old distribution like CentOS 6 (2011!)

Comments, issues and PRs welcome on: https://github.com/metwork-framework/log_proxy

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .