1

I'm running a process on Linux (CentOS 7) which produces a lot of output on stdout. Sending it all to a file will result in a file size greater than 1 TB. But happily the output is quite compressible - it consists of lots of quite repetitive text.

My thought is, can I use some kind of log rotation program like those described here: ( Log rotation of stdout? ), but where once a log is no longer current, it is compressed (and never deleted)? So I end up writing, say, a couple of GB to the file, compress it, and keep writing to a fresh file?

5

You can use logrotate for this. CentOS 7 is using systemd so you can configure a scheduled logrotate task with something like this:

  • Create a logrotate config file in /etc/systemd/system called mylogrotate.config with the following content:

    /path/to/your/logfile/*.log {
        compress
        copytruncate
        delaycompress
        minsize 1G
        missingok
        nomail
        notifempty
        rotate 30
    }
    

This will tell logrotate to do the following:

  • Old versions of log files are compressed with gzip by default.
  • Truncate the original log file in place after creating a copy, instead of moving the old log file and optionally creating a new one.
  • Postpone compression of the previous log file to the next rotation cycle so you're going to have two uncompressed log files.
  • Log files are rotated when they grow bigger than 1Gbytes.
  • If the log file is missing, go on to the next one without issuing an error message.
  • Don't mail old log files to any address.
  • Log files are rotated 30 times before being removed so you'll have 30 old log files. Anything older than that will be deleted.

Create a systemd service. Create a file in /etc/systemd/system called mylogrotate.service with the following:

[Unit]
Description=Rotate My Log

[Service]
Type=oneshot
ExecStart=/usr/sbin/logrotate /etc/systemd/system/mylogrotate.config --state /etc/systemd/system/mylogrotate.state --verbose

Let's schedule this task. Create a third file in /etc/systemd/system called mylogrotate.timer with the following:

[Unit]
Description=Rotate My Log Timer

[Timer]
OnCalendar=*:00:00
Persistent=true

[Install]
WantedBy=timers.target

This will run logrotate every hour.

All has been set up. Now enable the scheduled task.

systemctl enable mylogrotate.timer
systemctl start mylogrotate.timer

Of course do all this as root.

(I'm on Ubuntu but I hope this will work on CentOS without any issues)

  • 1
    copytruncate seems useful, but I think it will work if the redirection to the log file is done with >>; only then truncating the file causes the new content to appear from the beginning. Redirection with > keeps its (growing) offset, so any new data after the file is truncated causes the file to grow (with leading zeros) to the old size. – Kamil Maciorowski Jul 20 '18 at 18:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.