Huge log files consuming entire hard drive

Question

My hard drive keeps becoming full, and I thought it was just normal usage, but now that I pay attention, rsyslogd is hogging the CPU, and I think the drive is being filled by log files:

> la -S
total 28G
-rw-r-----  1 syslog            adm          10G 2012-08-13 13:45 kern.log
-rw-r-----  1 syslog            adm         6.9G 2012-08-13 07:57 syslog.1
-rw-r-----  1 syslog            adm         6.8G 2012-08-12 07:56 kern.log.1
-rw-r-----  1 syslog            adm         3.4G 2012-08-13 13:45 syslog
-rw-r-----  1 syslog            adm         583M 2012-08-05 07:52 kern.log.2.gz
-rw-r-----  1 syslog            adm         190M 2012-08-11 07:59 syslog.3.gz
-rw-r-----  1 syslog            adm          77M 2012-08-07 07:35 syslog.7.gz
-rw-r-----  1 syslog            adm          37M 2012-08-09 07:45 syslog.5.gz
-rw-r-----  1 syslog            adm          24M 2012-08-10 01:08 syslog.4.gz
-rw-r-----  1 syslog            adm         9.3M 2012-08-12 07:46 syslog.2.gz
-rw-r--r--  1 root              root        565k 2012-04-27 01:24 Xorg.1.log

It's just logging errors repeatedly?

> tail kern.log
Aug 13 13:50:33 optiplex kernel: [ 6530.465782] attempt to access beyond end of device
Aug 13 13:50:33 optiplex kernel: [ 6530.465785] loop3: rw=0, want=12919461, limit=8388607
Aug 13 13:50:33 optiplex kernel: [ 6530.465788] attempt to access beyond end of device
Aug 13 13:50:33 optiplex kernel: [ 6530.465791] loop3: rw=0, want=12919462, limit=8388607
Aug 13 13:50:33 optiplex kernel: [ 6530.465794] attempt to access beyond end of device
Aug 13 13:50:33 optiplex kernel: [ 6530.465797] loop3: rw=0, want=12919463, limit=8388607
Aug 13 13:50:33 optiplex kernel: [ 6530.465800] attempt to access beyond end of device
Aug 13 13:50:33 optiplex kernel: [ 6530.465803] loop3: rw=0, want=12919464, limit=8388607
Aug 13 13:50:33 optiplex kernel: [ 6530.465807] attempt to access beyond end of device
Aug 13 13:50:33 optiplex kernel: [ 6530.465810] loop3: rw=0, want=12919465, limit=8388607

syslog is full of the same garbage

1. How can I prevent Linux from suffocating itself with log files?
2. Why is this error occurring? I guess it started when I put some dd drive images in the fstab?

Answer 1

loop3 is a device node, which tells me that when you "put some dd drive images in the fstab", something is weird about the dd images.

1. How can I prevent Linux from suffocating itself with log files?

Don't do something that causes an error ;-). No, seriously, that's the best way, because whatever is causing the error is going to hog 100% CPU even if you prevent the log files from being written to. Something is very frequently trying to access beyond the bounds of the /dev/loop3 block device, so you must have mounted it with the wrong size, or the size changed after it was mounted, or something.

2. Why is this error occurring? I guess it started when I put some dd drive images in the fstab?

Yep, it's definitely related to that. It's trying to read past the end of the file that the loopback device points to. Maybe the image, or the filesystem within the image, declares itself as being bigger than it actually is?

Answer 2

To answer the first question, I remember back when each of the root folders were "mounted" to their own location or device. Having said that, I remember that we had, for example, /dev/hda1 for the root / folder, /dev/hda2 for /var, /dev/hda3 for /home, and /dev/hda4 for the /swap. I'm wondering if you have a setup like that, and if so, is it possible to redirect the output of /log to another device, maybe a removable drive or a network share

Answer 3

How can I prevent Linux from suffocating itself with log files?

• You can edit syslog.conf and change the way that those events are logged. You may be able to suppress them temporarily or write those specific events to a non-critical disk (e.g. a large USB drive)

• you can rotate the log files more frequently and aggressively. Your system may be using logrotate. If so edit it's configuration.

• There are tools that periodically prune log files to a specific size rather than just rotating them at a fixed schedule.

• Fix whatever problem is causing the log records to be written. (i.e. check fstab)