1

I discovered that journald only has about one month of data, despite the system having run for much longer than that. What could cause this?

  • I did not change any journald settings or manually messed with the persistent journal files
  • There is not much data overall, journalctl | wc -l gives 241313 lines.
  • The journal files are at the configured limit (~300 MB), but there is no way that 241k lines take 300 MB of space.
  • I am using default settings in /etc/systemd/journald.conf, with the exception of SystemMaxUse=300M.
1

There is not much data overall, journalctl | wc -l gives 241313 lines.

The thing is that the journal does not consist of lines, so this command does not actually show you how much data is there "overall".

The entire reason journald is journald (and not simply another syslogd writing to plain-text .log files) is that each .journal file is database where each entry consists of various "KEY=value" pairs:

  • In addition to the actual message, it also stores various process information (the PID, UID, command line, cgroup, systemd unit name...).

  • Messages received using the "/dev/log" syslog API generate some hidden fields one of which contains the full copy of the unparsed message buffer – which basically duplicates data in the parsed 'MESSAGE' field. (This is apparently done to aid debugging, as the syslog message format is very loosely defined.)

  • Programs using the "native" journald API can attach their own fields, such as the event information (address, device name, unit name, whatever is relevant to the message). Often the API automatically attaches the source code file:line:function which generated a particular message.

  • Fields may be multi-line, and even hold binary data. For example, a journal entry describing that a process dumped core will have a multi-line stack trace in its "message". In certain older systemd versions, such entries actually contained the whole multi-megabyte coredump! (This was disabled later.)

Try journalctl -o export or journalctl -o verbose to get a better idea of what a systemd-journal message looks like. For example, a message that your 'wc -l' saw like this:

Nov 02 21:30:15 somehost ldap_child[2103938]: Failed to initialize credentials using keytab [MEMORY:/etc/krb5.keytab]: Cannot contact any KDC for realm 'EXAMPLE.COM'. Unable to create GSSAPI-encrypted LDAP connection.

is really stored more like this:

__MONOTONIC_TIMESTAMP=5009050061377
__REALTIME_TIMESTAMP=1604345415827892
_BOOT_ID=50352af2d4aa4bc698628ee4e53e062a
_CAP_EFFECTIVE=3fffffffff
_COMM=ldap_child
_GID=0
_HOSTNAME=somehost
_MACHINE_ID=d7d18a5b04bbda0e66e8be55367049e2
_PID=2103938
_SOURCE_REALTIME_TIMESTAMP=1604345415825424
_SYSTEMD_CGROUP=/system.slice/sssd.service
_SYSTEMD_INVOCATION_ID=5407b2ce221b4abb8307f0664436fc5b
_SYSTEMD_SLICE=system.slice
_SYSTEMD_UNIT=sssd.service
_TRANSPORT=journal
_UID=0
CODE_FILE=src/providers/ldap/ldap_child.c
CODE_FUNC=ldap_child_get_tgt_sync
CODE_LINE=543
MESSAGE=Failed to initialize credentials using keytab [MEMORY:/etc/krb5.keytab]: Cannot contact any KDC for realm 'AD.EXAMPLE.COM'. Unable to create GSSAPI-encrypted LDAP connection.
PRIORITY=7
SSSD_DEBUG_LEVEL=10
SSSD_DOMAIN=ad.example.com
SSSD_PRG_NAME=ldap_child[2103938]
SYSLOG_IDENTIFIER=ldap_child

Journal files are stored in binary format, not in plain ASCII text, so there will still be some differences. On the one hand, a few universally-present fields (timestamps, boot ID) are stored as packed 64-bit values, and other fields are deduplicated and lz4-compressed to some extent.

On the other hand, all fields are indexed for efficient lookup by value (e.g. it is possible to search the journal by "SYSLOG_IDENTIFIER=ldap_child") and these hash table indexes will occupy additional space that the text-based dumps don't account for.

For a specific example, a journal containing 260 MB of text messages might easily hold 15 times as much data (give or take a few MB for the binary timestamps):

# journalctl -a | wc -l
2103207

# journalctl -a | wc -c | numfmt --to=iec
258M

$ journalctl -o export | grep -av ^__CURSOR | wc -c | numfmt --to=iec
1.6G

$ journalctl --header | awk '/Arena size:/ {a += $3} END {print a}' | numfmt --to=iec
2.4G

# du -hs /var/log/journal
2.4G    /var/log/journal

Finally, if I remember correctly, journald also pre-allocates space for new entries (the "arena"), causing files to be larger than the actual amount of data stored.

I'm not entirely sure how to check how much actual data is stored in a journal file, but take the following example where at least half of user.journal is completely empty and just pre-allocated for future writes:

# du -hs user-2001.journal
8.1M    user-2001.journal

$ tail -c 4M user-2001.journal | hexdump -C
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00400000

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.