0

Problem

Apparently I am running low on disk space on my root partition. At the time of installing my OS (openSUSE Leap 15 on a VM) I chose the default partitioning. Now I get the warning/error Low Disk Space on "Filesystem root". It warns me when I start the system, and when I try to compile a large project it throws an error.

Analysis

Let's check the storage situation:

report file system disk space usage:

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs         13G     0   13G   0% /dev
tmpfs            13G   34M   13G   1% /dev/shm
tmpfs            13G   82M   13G   1% /run
tmpfs            13G     0   13G   0% /sys/fs/cgroup
/dev/sda2        40G   38G  2.2G  95% /
/dev/sda2        40G   38G  2.2G  95% /root
/dev/sda2        40G   38G  2.2G  95% /boot/grub2/i386-pc
/dev/sda3       204G  165G   40G  81% /home
/dev/sda2        40G   38G  2.2G  95% /boot/grub2/x86_64-efi
/dev/sda1       500M  5.0M  495M   1% /boot/efi
/dev/sda2        40G   38G  2.2G  95% /usr/local
/dev/sda2        40G   38G  2.2G  95% /srv
/dev/sda2        40G   38G  2.2G  95% /opt
/dev/sda2        40G   38G  2.2G  95% /.snapshots
/dev/sda2        40G   38G  2.2G  95% /tmp
/dev/sda2        40G   38G  2.2G  95% /var
tmpfs           2.6G   20K  2.6G   1% /run/user/462
tmpfs           2.6G   48K  2.6G   1% /run/user/1000

Estimate file space usage:

$ sudo du -hsx /* | sort -rh | head -n 40
[sudo] password for root: 
du: cannot access '/proc/8809/task/8809/fd/4': No such file or directory
du: cannot access '/proc/8809/task/8809/fdinfo/4': No such file or directory
du: cannot access '/proc/8809/fd/4': No such file or directory
du: cannot access '/proc/8809/fdinfo/4': No such file or directory
51G /home
5.5G    /usr
972M    /opt
894M    /var
792M    /lib
63M /boot
38M /tmp
24M /etc
18M /run
11M /sbin
11M /lib64
2.1M    /bin
320K    /root
0   /sys
0   /srv
0   /selinux
0   /proc
0   /mnt
0   /dev

$ sudo du -hsx /.snapshots
2.2M    /.snapshots

$ sudo du -hs /.snapshots
129G    /.snapshots

Listing snapshots as @Kamil Maciorowsk suggested:

$ sudo snapper list
 Type   | #   | Pre # | Date                             | User | Cleanup | Description           | Userdata     
-------+-----+-------+----------------------------------+------+---------+-----------------------+--------------
single | 0   |       |                                  | root |         | current               |              
single | 1   |       | Tue 02 Oct 2018 02:42:20 PM CEST | root |         | first root filesystem |              
pre    | 74  |       | Mon 08 Oct 2018 03:25:32 PM CEST | root | number  | zypp(zypper)          | important=yes
post   | 75  | 74    | Mon 08 Oct 2018 03:27:17 PM CEST | root | number  |                       | important=yes
pre    | 82  |       | Tue 16 Oct 2018 09:11:33 AM CEST | root | number  | zypp(zypper)          | important=yes
post   | 83  | 82    | Tue 16 Oct 2018 09:12:04 AM CEST | root | number  |                       | important=yes
pre    | 108 |       | Thu 01 Nov 2018 01:25:41 PM CET  | root | number  | zypp(zypper)          | important=yes
post   | 109 | 108   | Thu 01 Nov 2018 01:27:12 PM CET  | root | number  |                       | important=yes
pre    | 122 |       | Thu 08 Nov 2018 09:26:09 AM CET  | root | number  | zypp(zypper)          | important=yes
post   | 123 | 122   | Thu 08 Nov 2018 09:27:40 AM CET  | root | number  |                       | important=yes
pre    | 128 |       | Mon 12 Nov 2018 08:40:03 AM CET  | root | number  | zypp(zypper)          | important=yes
post   | 129 | 128   | Mon 12 Nov 2018 08:41:36 AM CET  | root | number  |                       | important=yes
pre    | 144 |       | Mon 19 Nov 2018 09:52:15 AM CET  | root | number  | zypp(zypper)          | important=no 
post   | 145 | 144   | Mon 19 Nov 2018 09:54:33 AM CET  | root | number  |                       | important=no 
pre    | 146 |       | Wed 21 Nov 2018 11:07:33 AM CET  | root | number  | zypp(zypper)          | important=no 
post   | 147 | 146   | Wed 21 Nov 2018 11:07:56 AM CET  | root | number  |                       | important=no 
pre    | 148 |       | Thu 22 Nov 2018 09:19:51 AM CET  | root | number  | zypp(zypper)          | important=no 
post   | 149 | 148   | Thu 22 Nov 2018 09:19:54 AM CET  | root | number  |                       | important=no 
pre    | 150 |       | Mon 26 Nov 2018 09:12:02 AM CET  | root | number  | zypp(zypper)          | important=no 
post   | 151 | 150   | Mon 26 Nov 2018 09:12:19 AM CET  | root | number  |                       | important=no 
pre    | 152 |       | Thu 29 Nov 2018 09:34:37 AM CET  | root | number  | zypp(zypper)          | important=no 
post   | 153 | 152   | Thu 29 Nov 2018 09:35:22 AM CET  | root | number  |                       | important=no

I have also heard about old unused Kernels, so checked it out and found this:

$ ls -la /lib/modules
total 0
drwxr-xr-x 1 root root 108 Nov  8 09:29 .
drwxr-xr-x 1 root root  78 Oct  4 16:13 ..
drwxr-xr-x 1 root root 354 Oct 16 09:11 4.12.14-lp150.12.22-default
drwxr-xr-x 1 root root 354 Nov  8 09:26 4.12.14-lp150.12.25-default

Ideas for a Solution:

  1. Resize root partition. (giving root 10 more gigs would be nice)
  2. Deleting the old kernel version and hope I dont break things and the freed up 245 MB will be enough for now.

I really favourise just give root more space, but have no clue how to do that or if it's a good idea to mess with it at all. What solution would you propose and how can I do that?

0

First thing to do is make a backup of anything important. Going further down this path involves doing things that can result in data loss. Some options below:

  • buy a new USB SATA hard drive. Swap the USB SATA drive with your old drive in your case. Re-install Linux to the new SATA drive. Whenever you need to access your old files, plug in the USB drive and there they are.

  • If you have partitioned using LVM (which SuSE probably hasn't) then see if you can extend (lvmresize -L+10G /dev/mapper/whatever) your slash partition and then resize (resize2fs /dev/mappper/whatever). This is the easiest fix.

  • if you have hard partitions (eg: root is on /dev/sda1) then you can try booting up with Gparted Live (https://gparted.org/livecd.php) and monkey around trying to extend your hard partition. Generally, success here depends how much slack space is left on your drive and how you've partitioned things

  • buy a new hard drive. Same capacity or bigger. plug it in and create larger partitions (use LVM if possible). First partition on the new disk should be 1G in size (can be smaller, just being brief) and is there for Grub compatibility. Afterwards, boot into your old disk and create directories/mount the new disk partitions on /mnt/new_disk/; rsync all the old partitions onto the new disk. (eg: rsync -av / /mnt/new_disk/slash/; rsync -av /usr /mnt/new_disk/usr/; ...). After you have finished you will need to get grub installed on the new disk somehow. I usually do this using a chroot into /mnt/new_disk/slash/ but it can be daunting. Usually grub.cfg gets confused about things. There have got to be easier ways to do it.

0

/dev/sda2 mounted to different mountpoints (where you're supposed to have different content) makes me believe you're using Btrfs. You also have /.snapshots mounted which indicates the presence of Snapper. It's highly probable that /.snapshots takes the majority of used space.

Note your analysis with du -hsx /* didn't even take /.snapshots into consideration because * glob doesn't expand to names starting with ..

I have no experience with Snapper. I suspect there are Btrfs subvolumes (shapshots) inside /.snapshots. The command du -hsx /.snapshots may even return 0 because of -x option used. On the other hand du -hs /.snapshots will probably return a huge value. It may be way bigger than the size of /dev/sda2 (which is 40GiB) because you probably have multiple snapshots that share disk space (this is how Btrfs works), still du will consider them to be separate filesystems.

To analyze further you need Btrfs-specific and/or Snapper tools. I have some experience with Btrfs but not with Snapper. I cannot tell if you can confuse Snapper by manually mangling snapshots it created, it may be possible; but I'm sure you cannot break Btrfs by using Snapper. Therefore the safe approach is to handle the situation with whatever Snapper provides, not directly with Btrfs tools.

The already mentioned tutorial gives us some hints on what you can do:

  • List all snapshots for the default configuration (root)

    snapper list
    
  • Deleting Snapshots

    Delete snapshot 65 for the default (root) configuration:

    snapper delete 65
    

But also:

Automatic Snapshots Cleanup Mechanisms

To prevent disk space full, Snapper periodically cleans snapshots up. By default, this period = 1 day.

Automatic snapshots cleanup task can be managed via 2 ways:

  1. cron scheduler (/etc/cron.daily/suse.de-snapper).
  2. systemd timer scheduler (snapper-cleanup.timer and snapper-cleanup.service systemd units).

By default, cron mechanism is in use.

Maybe something failed with this cleanup mechanism?

As I said I have no experience with Snapper, so I cannot help you more. However, if I'm right, now you know what to investigate. To summarize:

  • you totally missed /.snapshots directory, most likely it's responsible for the majority of used space;
  • Btrfs snapshots are probably involved;
  • Snapper is probably involved.
  • Did not do the snapper delete 65 , since snapper list did not show any element with a 65. Should'nt the 65 be listed? – Human Nov 29 '18 at 19:10
  • @Human It's just an example… – Kamil Maciorowski Nov 29 '18 at 19:40
  • I'm sorry I couldn't test your solution on my broken system due to another problem. I had to do a complete reinstall, but you were mostly right, I had a ton of btrfs snapshots. But I couldn't find out their real size. Is there an btrfs command or something? Also in my cron/daily there was no snapper-job, but I did find one snapper-cleanup in systemd time scheduler with systemctl list-timers. – Human Dec 6 '18 at 15:15
  • Do you have an idea how I could rephrase my question so it might become more helpful for other users who have a full root partition and don't know why. I mean a full nice analysis that works with btrfs partitions and hidden folders could be helpfull right? – Human Dec 6 '18 at 15:19
  • @Human I don't know a reliable way to predict how much space will be freed if you delete a snapshot. Note the values (if you knew them) do not add (imagine two huge identical snapshots as the only content; deletion of any single one changes nothing, but if you delete both then you will free a lot). I'm not sure how to rephrase the question, especially since you lost the setup. I think it's hard to write "a full nice analysis" without any "guinea pig OS". If I had any experience with Snapper then maybe I would be able to help in this matter. The question (as it is) is not useless though. – Kamil Maciorowski Dec 7 '18 at 8:30

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.