0

I have a fairly stock, minimal Debian Jessie distro on which log files under /var/log are being rotated (seemingly arbitrarily) but never deleted:

$ sudo du -sk /var/log/* | sort -rn
4167364 /var/log/daemon.log.1
1710535 /var/log/daemon.log
596240  /var/log/syslog.1
521091  /var/log/syslog
19905   /var/log/daemon.log.2.gz
15803   /var/log/syslog.4.gz
15187   /var/log/syslog.5.gz
14598   /var/log/syslog.7.gz
14595   /var/log/syslog.6.gz
14545   /var/log/syslog.3.gz
14496   /var/log/syslog.2.gz
1365    /var/log/messages.1
476 /var/log/kern.log.1
403 /var/log/messages
286 /var/log/lastlog
229 /var/log/auth.log.1
128 /var/log/syslog.1.gz
95  /var/log/auth.log
56  /var/log/wtmp
35  /var/log/debug.1
24  /var/log/faillog
19  /var/log/messages.2.gz
11  /var/log/kern.log.2.gz
3   /var/log/fsck
3   /var/log/debug.2.gz
3   /var/log/auth.log.2.gz
3   /var/log/apt
2   /var/log/kern.log
1   /var/log/sysstat
1   /var/log/dpkg.log
1   /var/log/dmesg
1   /var/log/alternatives.log
0   /var/log/debug
0   /var/log/btmp

Now, from my reading of journald.conf documentation, the default SystemKeepFree is 15% and SystemMaxuse is 10% so I don't understand why these are growing so large. My root mount is actually only 8GB so I discovered this because my rootfs usage actually hit 100%:

$ df
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/root        7529518 7525422         0 100% /
....

It occurred to me that rsyslog might be the one writing these files under /var/log and there is a stock /etc/ryslog.conf. But then why does it appear files are being rotated in some cases but never deleted?

/etc/rsyslog.conf for reference:

$ cat /etc/rsyslog.conf 
#  /etc/rsyslog.conf    Configuration file for rsyslog.
#
#           For more information see
#           /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html


#################
#### MODULES ####
#################

$ModLoad imuxsock # provides support for local system logging
$ModLoad imklog   # provides kernel logging support
#$ModLoad immark  # provides --MARK-- message capability

# provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514

# provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514


###########################
#### GLOBAL DIRECTIVES ####
###########################

#
# Use traditional timestamp format.
# To enable high precision timestamps, comment out the following line.
#
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

#
# Set the default permissions for all log files.
#
$FileOwner root
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022

#
# Where to place spool and state files
#
$WorkDirectory /var/spool/rsyslog

#
# Include all config files in /etc/rsyslog.d/
#
$IncludeConfig /etc/rsyslog.d/*.conf


###############
#### RULES ####
###############

#
# First some standard log files.  Log by facility.
#
auth,authpriv.*         /var/log/auth.log
*.*;auth,authpriv.none      -/var/log/syslog
#cron.*             /var/log/cron.log
daemon.*            -/var/log/daemon.log
kern.*              -/var/log/kern.log
lpr.*               -/var/log/lpr.log
mail.*              -/var/log/mail.log
user.*              -/var/log/user.log

#
# Logging for the mail system.  Split it up so that
# it is easy to write scripts to parse these files.
#
mail.info           -/var/log/mail.info
mail.warn           -/var/log/mail.warn
mail.err            /var/log/mail.err

#
# Logging for INN news system.
#
news.crit           /var/log/news/news.crit
news.err            /var/log/news/news.err
news.notice         -/var/log/news/news.notice

#
# Some "catch-all" log files.
#
*.=debug;\
  auth,authpriv.none;\
  news.none;mail.none   -/var/log/debug
*.=info;*.=notice;*.=warn;\
  auth,authpriv.none;\
  cron,daemon.none;\
  mail,news.none        -/var/log/messages

#
# Emergencies are sent to everybody logged in.
#
*.emerg             :omusrmsg:*

#
# I like to have messages displayed on the console, but only on a virtual
# console I usually leave idle.
#
#daemon,mail.*;\
#   news.=crit;news.=err;news.=notice;\
#   *.=debug;*.=info;\
#   *.=notice;*.=warn   /dev/tty8

# The named pipe /dev/xconsole is for the `xconsole' utility.  To use it,
# you must invoke `xconsole' with the `-file' option:
# 
#    $ xconsole -file /dev/xconsole [...]
#
# NOTE: adjust the list below, or you'll go crazy if you have a reasonably
#      busy site..
#
daemon.*;mail.*;\
  news.err;\
  *.=debug;*.=info;\
  *.=notice;*.=warn |/dev/xconsole

(nothing under /etc/rsyslog.d/)

Stock conf at /etc/systemd/journald.conf:

#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See journald.conf(5) for details.

[Journal]
#Storage=auto
#Compress=yes
#Seal=yes
#SplitMode=uid
#SyncIntervalSec=5m
#RateLimitIntervalSec=30s
#RateLimitBurst=1000
#SystemMaxUse=
#SystemKeepFree=
#SystemMaxFileSize=
#SystemMaxFiles=100
#RuntimeMaxUse=
#RuntimeKeepFree=
#RuntimeMaxFileSize=
#RuntimeMaxFiles=100
#MaxRetentionSec=
#MaxFileSec=1month
#ForwardToSyslog=yes
#ForwardToKMsg=no
#ForwardToConsole=no
#ForwardToWall=yes
#TTYPath=/dev/console
#MaxLevelStore=debug
#MaxLevelSyslog=debug
#MaxLevelKMsg=notice
#MaxLevelConsole=info
#MaxLevelWall=emerg

And I actually added one file under journald.conf.d:

$ cat /etc/systemd/journald.conf.d/custom.conf 
# See: https://www.freedesktop.org/software/systemd/man/journald.conf.html
# vi: ft=dosini

[Journal]
MaxFileSec=0
SystemMaxUse=5M
Compress=yes
# Uncomment this to never write to FS:
#Storage=volatile

So, some rotation is happening, but not consistently and it's not being capped at what journald says it should be.

Note I'm running systemd-230 from jessie-backports.

Questions

  • What's already rotating the files?
  • Why is total usage not being capped as expected?
  • Is journald actually writing these files or is it rsyslog? (I can't actually see any other location under /var where journald is storing any data, binary or otherwise.)
  • Can I set file size caps in rsyslog config?
  • Do I need logrotate to do this for me? Would prefer if journald or rsyslogd could handle this directly.

Thanks!

Update Although the journald.conf documentation suggests ForwardToSyslog=no by default, looking closer at the stock journald.conf I posted, it says the compiled-in default is ForwardToSyslog=yes. So it looks like rsyslog is doing it's thing under /var/log

Improve this question
1

logrotated does indeed manage those logs (for rsyslog and others). But... you are seeing the same file names constantly so that may be what you are thinking. Default config is to keep 4 or 7 old logs and current log. Here's the default config - you can see that /var/log/syslog will keep 7 plus current log, the others will keep 4. 

debian@templatevm:~$ cat /etc/logrotate.d/rsyslog 
/var/log/syslog
{
    rotate 7
    daily
    missingok
    notifempty
    delaycompress
    compress
    postrotate
        invoke-rc.d rsyslog rotate > /dev/null
    endscript
}

/var/log/mail.info
/var/log/mail.warn
/var/log/mail.err
/var/log/mail.log
/var/log/daemon.log
/var/log/kern.log
/var/log/auth.log
/var/log/user.log
/var/log/lpr.log
/var/log/cron.log
/var/log/debug
/var/log/messages
{
    rotate 4
    weekly
    missingok
    notifempty
    compress
    delaycompress
    sharedscripts
    postrotate
        invoke-rc.d rsyslog rotate > /dev/null
    endscript
}

If you want to rotate based on size alone, you change the daily parameter to size NNNN where the N is a size with measurement like 100M - so size 100M to limit to 100mb

Then, if your log files tend to grow rapidly, you can change the frequency of how often logrotate checks by placing the symbolic link to it in /etc/cron.hourly instead of /etc/cron.daily

Here's what your new /etc/logrotate.d/rsyslog file might look like

/var/log/syslog
{
    rotate 7
    daily
    missingok
    notifempty
    delaycompress
    compress
    postrotate
        invoke-rc.d rsyslog rotate > /dev/null
    endscript
}

/var/log/mail.info
/var/log/mail.warn
/var/log/mail.err
/var/log/mail.log
/var/log/daemon.log
/var/log/kern.log
/var/log/auth.log
/var/log/user.log
/var/log/lpr.log
/var/log/cron.log
/var/log/debug
/var/log/messages
{
    rotate 4
    size 25M
    missingok
    notifempty
    compress
    delaycompress
    sharedscripts
    postrotate
        invoke-rc.d rsyslog rotate > /dev/null
    endscript
}
Improve this answer
3
  • Ah somehow I missed that; which logrotate let me to believe it was not installed because I did not do sudo which ..! Thanks. So the default logrotate config is not doing any size-based limitation - but at least I know where to add it. Thanks! – Thom Nichols Jun 28 '17 at 17:23
  • 1
    Yes, you can specify a max size before rotating, etc. if you want to do that, you may want to step up how often logrotate is called... I taught a PHP class and of course students would have errors in unending loops which would fill the drive with log files and make the class server unusable, so I started doing size limits and rotating/checking for need of rotation every hour via cron. I'll go back and edit my answer to include info on that... – ivanivan Jun 28 '17 at 17:25
  • Great! This is all really straightforward, my main point of confusion was thinking logrotate was not installed and I didn't notice the config files already in /etc/logrotate.d/ - I guess the default assumption is your files are not going to fill up so quickly that you can't handle 7 days of logs! – Thom Nichols Jun 29 '17 at 18:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.