Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a Linux program which can write information to stdout and stderr.

I have a shell script which redirects that output to a file in /var/log. (Via >> and 2>&1.)

Is there a way to make that log file rotate? (max size, then switch to a different file, keep only a limited number of files)

I've seen a few answers which talk about the logrotate program, which sounds good, but they also seem to be focused on programs which are generating log files internally and handle HUP signals. Is there a way to make this work with a basic output redirection script?

share|improve this question
Why can't you just modify the script that redirects the output to contain the logic for the rotation? – MaQleod Jun 1 '11 at 5:05
I could, if someone could tell me how to detect the size of a logfile and rotate it out from under the stdout of a process without disturbing that process. I don't have to use logrotate if there's a better option, that just sounded like a convenient starting point for discussion. – Miral Jun 1 '11 at 5:12
You don't have to use logrotate, but using logrotate just saves time... There is usually little point reinventing the wheel. – bubu Jun 1 '11 at 5:43
Exactly my point. So is there a way to make logrotate work with an ongoing process's redirected stdout? – Miral Jun 1 '11 at 6:12
up vote 14 down vote accepted

As an alternative, you could pipe the output through tools designed with the primary purpose of maintaining size-capped, automatically rotated, log file sets, such as:

Tools to then process multilog-format log file sets include, amongst others:

share|improve this answer
Thanks, multilog looks like just what I needed. – Miral Jun 3 '11 at 1:36

the rotatelogs tool shipped with apache (in the bin dir) (see docs) takes input from stdin and rotates the log after some specific amount of time

share|improve this answer

If you can have it go to one of the standard log streams (syslog, daemon, cron, user, security, mail, etc.) you can use the logger command and pipe to it instead.

echo "Hello." | logger -p

Otherwise, you may be better off piping your logged content to a custom program or script to handle it, or look at setting up the logrotate configuration.

EDIT: JdeBP's answer seems to have what you may be looking for.

share|improve this answer
+1 for simplicity. BTW, you can also configure a custom facility (local0) instead of the standard ones (daemon in your example) – Roger Keays Oct 9 '12 at 7:56

I had similar problem and had initially discard logrotate but it turned out logrotate can actually do this well, the key directive is "copytruncate". For some reason that term didn't come up on any of the googling I did, so I am adding this answer to clarify exactly how to use it for this case.

The trick is this only works if the redirect is done with ">>" (append) instead of ">" (create).

Config File (truncate.cfg):

/tmp/temp.log {
    size 10M
    rotate 4
    maxage 100

Test Program (never gives up file). You can watch it filling disk and though deleting logfile will appear to work it will not actually free up any space on the disk:

cat /dev/urandom >> /tmp/temp.log

Running log rotate:

logrotate truncate.cfg
share|improve this answer
It's a nice theory, but it doesn't actually work on any system I've tried it on. The file does not actually get truncated and the program continues to append to it as before. (And yes, that's even with the redirection done via >>.) ((BTW, this answer was already given previously.)) – Miral Oct 24 '14 at 0:10
… as discussed in logrotate won’t truncate original file (on our Unix&Linux site). Also, echo /dev/urandom >> /tmp/temp.log will write 13 deterministic characters to /tmp/temp.log and then immediately exit. Did you mean cat /dev/urandom? – G-Man Oct 24 '14 at 0:36
Just tested here, and it seems to work. Content of file is copied to new log file. Original file is kept open by process and is truncated (size now shows 0). – Philipp Jan 21 '15 at 9:09
Fixed example, thanks G-Man – Sam Hendley Jan 20 at 14:50

So is there a way to make logrotate work with an ongoing process's redirected stdout?

Yes! Check out the "copytruncate" directive offered by logrotate. Specifying that instructs logrotate to handle this very situation: a simple program that keeps its log file open indefinitely.

One caveat may or may not be a problem in your situation:

Note that there is a very small time slice between copying the file and truncating it, so some logging data might be lost.

Anecdotally, I've seen some "real world" log sources that do encourage users to apply this directive. There's some discussion of this option here.

share|improve this answer

Use split, it's part of coreutils. It can take stdin and split it into chunks (based on chunk size, or number of lines, etc.).


app | split --bytes 1G - /var/logs/put-prefix-here

Note dash (-) instructs "split" to use stdin instead of file.

share|improve this answer
Can you expand your answer to describe how to do that? Thanks. – fixer1234 Jun 11 '15 at 18:30
just updated my reply with example. – Nazar Jun 11 '15 at 18:49
The 1G is an arbitrary size, after which it starts a new file? – fixer1234 Jun 11 '15 at 19:17
This isn't a particualrly good solution to the problem because it means you can end up with half a message in one file and half in the next. There's also the risk of data loss if the machine crashes while split has data in what could be a large buffer. Given that there are multiple tools that solve this problem properly, I don't think this kind of roll-your-own solution can be at all recommended. – David Richerby Jun 11 '15 at 19:28
All tools have their use.. I don't think anything that works with pipes will "guarantee" no data loss. This is quick and dirty solution that would work just fine for lots of jobs. Also "split" has option to split by number of lines (instead of number of bytes). It's a good-old unix tool which I think still has it's uses. – Nazar Jun 16 '15 at 18:44

I made simple script. Which write to 1 file and rotate it every day and every 5M

[note to moderators: do not inline my work without copyright notices]

# echo test 2>&1 | log ./run.log
-rw-r--r-- 1 axet axet     850176 Feb 22 12:28 run.log
-rw-r--r-- 1 axet axet    1048595 Feb 22 09:00 run.log.1
-rw-r--r-- 1 axet axet    1319926 Feb 22 04:57 run.log.2
-rw-r--r-- 1 axet axet    2400918 Feb 22 01:17 run.log.3
share|improve this answer
Nice script. But I don't like the fact that the file is reopened and closed for each line it writes – Daniel Alder Jun 8 '15 at 14:30

I like multilog for my use case, but my use case is so trivial/simple that it is not laid out very simply in the docs/examples I found. Here is a simple multilog rotate example:

mkdir /tmp/myapp
./myapp | multilog t s10000 n5 '!tai64nlocal' /tmp/myapp 2>&1

Some notes:

  • this dumps logs into that /tmp/myapp/ directory
  • the s10000 represents 10,000 bytes*
  • the n5 represents 5 files.* The 'current' log counts as one of the files, so this includes 4 older logs + 'current'
  • this is based on, adapted from the examples provided by François Beausoleil at:
  • I don't understand many of the options - I refer you to the various documentation to extend this...
  • The docs warn that: "Note that running processor may block any program feeding input to multilog." where 'processor' is the '!tai64nlocal' portion of the command

*For many applications, these are poor choices for long term use. They do allow you to observe the behavior of filling and rotating the logs more quickly than large logs do.

Finally, don't forget to nohup if required! With nohup, you do not need the 2>&1 (s=10e6 and n=30 here):

mkdir -p /tmp/myapp
nohup ./myapp | multilog t s10000000 n30 '!tai64nlocal' /tmp/myapp &

That command should get you started.

share|improve this answer

I just wanted to add to Sam Hendley's comment above:

The trick is this only works if the redirect is done with >> (append) instead of > (create).

I ran into the same problem where the original file just keeps growing if you use > (create) but if you use >> (append) Logrotate copytruncate works beautifully and as expected. The original file goes back down to zero bytes and the program continues writing.

Redirect STDOUT and STDERR to a rotating logfile:

  1. >> /tmp/output.txt 2>&1 &
  2. Create a logrotate config file under /etc/logrotate.d called whatever, output_roll in my case.

    Sample config for my case:

    /tmp/output.txt {
        size 1G
        start 0
        rotate 15
  3. Setup your cron job inside the /etc/crontab file

    *  *  *  *  * root /usr/sbin/logrotate /etc/logrotate.d/output_roll

    This will check the file every minute. You can adjust to suit your needs.

  4. Start it up:

    $> service crond restart
  5. That's it

Note: I also had a problem with SELinux being set to SELINUX=enforcing so I set it to SELINUX=disabled.

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .