I have a fairly stock, minimal Debian Jessie distro on which log files under /var/log
are being rotated (seemingly arbitrarily) but never deleted:
$ sudo du -sk /var/log/* | sort -rn
4167364 /var/log/daemon.log.1
1710535 /var/log/daemon.log
596240 /var/log/syslog.1
521091 /var/log/syslog
19905 /var/log/daemon.log.2.gz
15803 /var/log/syslog.4.gz
15187 /var/log/syslog.5.gz
14598 /var/log/syslog.7.gz
14595 /var/log/syslog.6.gz
14545 /var/log/syslog.3.gz
14496 /var/log/syslog.2.gz
1365 /var/log/messages.1
476 /var/log/kern.log.1
403 /var/log/messages
286 /var/log/lastlog
229 /var/log/auth.log.1
128 /var/log/syslog.1.gz
95 /var/log/auth.log
56 /var/log/wtmp
35 /var/log/debug.1
24 /var/log/faillog
19 /var/log/messages.2.gz
11 /var/log/kern.log.2.gz
3 /var/log/fsck
3 /var/log/debug.2.gz
3 /var/log/auth.log.2.gz
3 /var/log/apt
2 /var/log/kern.log
1 /var/log/sysstat
1 /var/log/dpkg.log
1 /var/log/dmesg
1 /var/log/alternatives.log
0 /var/log/debug
0 /var/log/btmp
Now, from my reading of journald.conf documentation, the default SystemKeepFree
is 15% and SystemMaxuse
is 10% so I don't understand why these are growing so large. My root mount is actually only 8GB so I discovered this because my rootfs usage actually hit 100%:
$ df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/root 7529518 7525422 0 100% /
....
It occurred to me that rsyslog might be the one writing these files under /var/log
and there is a stock /etc/ryslog.conf
. But then why does it appear files are being rotated in some cases but never deleted?
/etc/rsyslog.conf for reference:
$ cat /etc/rsyslog.conf
# /etc/rsyslog.conf Configuration file for rsyslog.
#
# For more information see
# /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html
#################
#### MODULES ####
#################
$ModLoad imuxsock # provides support for local system logging
$ModLoad imklog # provides kernel logging support
#$ModLoad immark # provides --MARK-- message capability
# provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514
# provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514
###########################
#### GLOBAL DIRECTIVES ####
###########################
#
# Use traditional timestamp format.
# To enable high precision timestamps, comment out the following line.
#
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
#
# Set the default permissions for all log files.
#
$FileOwner root
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
#
# Where to place spool and state files
#
$WorkDirectory /var/spool/rsyslog
#
# Include all config files in /etc/rsyslog.d/
#
$IncludeConfig /etc/rsyslog.d/*.conf
###############
#### RULES ####
###############
#
# First some standard log files. Log by facility.
#
auth,authpriv.* /var/log/auth.log
*.*;auth,authpriv.none -/var/log/syslog
#cron.* /var/log/cron.log
daemon.* -/var/log/daemon.log
kern.* -/var/log/kern.log
lpr.* -/var/log/lpr.log
mail.* -/var/log/mail.log
user.* -/var/log/user.log
#
# Logging for the mail system. Split it up so that
# it is easy to write scripts to parse these files.
#
mail.info -/var/log/mail.info
mail.warn -/var/log/mail.warn
mail.err /var/log/mail.err
#
# Logging for INN news system.
#
news.crit /var/log/news/news.crit
news.err /var/log/news/news.err
news.notice -/var/log/news/news.notice
#
# Some "catch-all" log files.
#
*.=debug;\
auth,authpriv.none;\
news.none;mail.none -/var/log/debug
*.=info;*.=notice;*.=warn;\
auth,authpriv.none;\
cron,daemon.none;\
mail,news.none -/var/log/messages
#
# Emergencies are sent to everybody logged in.
#
*.emerg :omusrmsg:*
#
# I like to have messages displayed on the console, but only on a virtual
# console I usually leave idle.
#
#daemon,mail.*;\
# news.=crit;news.=err;news.=notice;\
# *.=debug;*.=info;\
# *.=notice;*.=warn /dev/tty8
# The named pipe /dev/xconsole is for the `xconsole' utility. To use it,
# you must invoke `xconsole' with the `-file' option:
#
# $ xconsole -file /dev/xconsole [...]
#
# NOTE: adjust the list below, or you'll go crazy if you have a reasonably
# busy site..
#
daemon.*;mail.*;\
news.err;\
*.=debug;*.=info;\
*.=notice;*.=warn |/dev/xconsole
(nothing under /etc/rsyslog.d/)
Stock conf at /etc/systemd/journald.conf
:
# This file is part of systemd.
#
# systemd is free software; you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation; either version 2.1 of the License, or
# (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See journald.conf(5) for details.
[Journal]
#Storage=auto
#Compress=yes
#Seal=yes
#SplitMode=uid
#SyncIntervalSec=5m
#RateLimitIntervalSec=30s
#RateLimitBurst=1000
#SystemMaxUse=
#SystemKeepFree=
#SystemMaxFileSize=
#SystemMaxFiles=100
#RuntimeMaxUse=
#RuntimeKeepFree=
#RuntimeMaxFileSize=
#RuntimeMaxFiles=100
#MaxRetentionSec=
#MaxFileSec=1month
#ForwardToSyslog=yes
#ForwardToKMsg=no
#ForwardToConsole=no
#ForwardToWall=yes
#TTYPath=/dev/console
#MaxLevelStore=debug
#MaxLevelSyslog=debug
#MaxLevelKMsg=notice
#MaxLevelConsole=info
#MaxLevelWall=emerg
And I actually added one file under journald.conf.d:
$ cat /etc/systemd/journald.conf.d/custom.conf
# See: https://www.freedesktop.org/software/systemd/man/journald.conf.html
# vi: ft=dosini
[Journal]
MaxFileSec=0
SystemMaxUse=5M
Compress=yes
# Uncomment this to never write to FS:
#Storage=volatile
So, some rotation is happening, but not consistently and it's not being capped at what journald says it should be.
Note I'm running systemd-230 from jessie-backports.
Questions
- What's already rotating the files?
- Why is total usage not being capped as expected?
- Is journald actually writing these files or is it rsyslog? (I can't actually see any other location under
/var
where journald is storing any data, binary or otherwise.) - Can I set file size caps in rsyslog config?
- Do I need logrotate to do this for me? Would prefer if journald or rsyslogd could handle this directly.
Thanks!
Update
Although the journald.conf documentation suggests ForwardToSyslog=no
by default, looking closer at the stock journald.conf I posted, it says the compiled-in default is ForwardToSyslog=yes
. So it looks like rsyslog is doing it's thing under /var/log