1

OK this is driving me nuts. Running zfs on a system, and my root partition has been shrinking with seeming no explanation. I have now run out of space and can't find where it all went.

A simple df shows the following:

[root@SV02 /]# df -h
Filesystem             Size   Used  Available Capacity  Mounted on
rpool/ROOT/opus-2457409-2017-11-07-release
                        90G   6.6G        35G    16%    /
swap                    11G   404K        11G     1%    /etc/svc/volatile
...
/usr/lib/libc/libc_hwcap1.so.1
                        41G   6.6G        35G    16%    /lib/libc.so.1
...
swap                    11G     4K        11G     1%    /tmp
swap                    11G    60K        11G     1%    /var/run
rpool/export            90G    32K        35G     1%    /export
rpool/export/home       90G    86K        35G     1%    /export/home
rpool/export/home/open  90G   404M        35G     2%    /export/home/open
rpool                   90G    43K        35G     1%    /rpool

Used/available don't seem consistent... Trying to track it down:

[root@SV02 /]# du -sh *  | sort -h | tail -10
  10M   root
  48M   lib
  58M   etc
 141M   kernel
 174M   platform
 405M   export
 1.5G   opt
 1.5G   usr
 3.1G   var
 3.6G   proc

OK, only a few gigabytes of data there... thought maybe a snapshot, but that doesn't seem to be the case:

[root@SV02 /]# zfs list -t all
NAME                                                             USED  AVAIL  REFER  MOUNTPOINT
rpool                                                           55.5G  34.6G  43.5K  /rpool
rpool/ROOT                                                      38.6G  34.6G    31K  legacy
rpool/ROOT/C_Backup1                                            1.24M  34.6G  5.61G  /
rpool/ROOT/C_backup2                                            1.33M  34.6G  5.61G  /
rpool/ROOT/napp-it-0.8l3                                        3.00M  34.6G  2.10G  /
rpool/ROOT/napp-it-0.9e1                                        1.66M  34.6G  5.61G  /
rpool/ROOT/nfsv4                                                  54K  34.6G  2.11G  /
rpool/ROOT/openindiana                                          18.3M  34.6G  2.02G  /
rpool/ROOT/opus-2457044-2015-01-31-install                      65.4M  34.6G  31.6G  /a
rpool/ROOT/opus-2457044-2015-01-31-preinstall                      1K  34.6G  5.61G  /
rpool/ROOT/opus-2457044-2015-05-19-pre15                          45K  34.6G  6.32G  /
rpool/ROOT/opus-2457044-2016-10-03-backup                         52K  34.6G  31.3G  /
rpool/ROOT/opus-2457409-2016-10-04-install                      15.5M  34.6G  31.7G  /a
rpool/ROOT/opus-2457409-2016-10-04-preinstall                     52K  34.6G  31.5G  /
rpool/ROOT/opus-2457409-2017-11-07-release                      38.5G  34.6G  6.57G  /
rpool/ROOT/opus-2457409-2017-11-07-release@install              4.40M      -  1.56G  -
rpool/ROOT/opus-2457409-2017-11-07-release@2014-02-25-22:01:25  6.64M      -  1.58G  -
rpool/ROOT/opus-2457409-2017-11-07-release@2014-02-25-22:05:33  84.6M      -  2.02G  -
rpool/ROOT/opus-2457409-2017-11-07-release@2014-03-19-23:25:59  58.0M      -  2.11G  -
rpool/ROOT/opus-2457409-2017-11-07-release@2014-04-12-17:55:40      0      -  2.10G  -
rpool/ROOT/opus-2457409-2017-11-07-release@2014-04-12-17:55:48      0      -  2.10G  -
rpool/ROOT/opus-2457409-2017-11-07-release@2014-04-12-18:13:10  1.44M      -  2.10G  -
rpool/ROOT/opus-2457409-2017-11-07-release@2014-04-12-18:14:31  2.17M      -  2.12G  -
rpool/ROOT/opus-2457409-2017-11-07-release@2015-01-31-21:55:47  15.3M      -  5.61G  -
rpool/ROOT/opus-2457409-2017-11-07-release@2015-01-31-22:34:52  9.18M      -  5.61G  -
rpool/ROOT/opus-2457409-2017-11-07-release@2015-01-31-22:43:54   362K      -  5.61G  -
rpool/ROOT/opus-2457409-2017-11-07-release@2015-01-31-22:44:05   362K      -  5.61G  -
rpool/ROOT/opus-2457409-2017-11-07-release@2015-05-19-21:49:36   227M      -  6.32G  -
rpool/ROOT/opus-2457409-2017-11-07-release@2016-10-04-16:11:30  3.15M      -  31.3G  -
rpool/ROOT/opus-2457409-2017-11-07-release@2016-10-04-16:34:30   936K      -  31.5G  -
rpool/ROOT/opus-2457409-2017-11-07-release@2016-10-04-16:34:42   986K      -  31.5G  -
rpool/ROOT/opus-2457409-2017-11-07-release@2016-10-04-16:53:00   112M      -  31.8G  -
rpool/ROOT/pre_napp-it-0.8l3                                      35K  34.6G  1.58G  /
rpool/ROOT/pre_napp-it-0.9e1                                      71K  34.6G  2.10G  /
rpool/ROOT/sv02-4-10-14                                           70K  34.6G  2.10G  /
rpool/ROOT/sv02-4-10-14-v2                                      4.17M  34.6G  2.12G  /
rpool/dump                                                      8.00G  34.6G  8.00G  -
rpool/export                                                     404M  34.6G    32K  /export
rpool/export/home                                                404M  34.6G  86.5K  /export/home
rpool/export/home/open                                           404M  34.6G   404M  /export/home/open
rpool/swap                                                      8.50G  43.0G   132M  -

34 gigs available but where, I do not know? At a loss and unfortunately I don't quite have a good enough grasp of zfs to properly troubleshoot. It's persistent after reboots and I haven't deleted any large files or anything recently, so don't think it's a process holding on to anything... Any direction would be appreciated.

ADDITION:

[root@SV02 /]# zfs list -t all
NAME                                         USED  AVAIL  REFER  MOUNTPOINT
rpool                                       23.5G  66.6G  43.5K  /rpool
rpool/ROOT                                  6.58G  66.6G    31K  legacy
rpool/ROOT/opus-2457409-2017-11-07-release  6.58G  66.6G  6.58G  /
rpool/dump                                  8.00G  66.6G  8.00G  -
rpool/export                                 404M  66.6G    32K  /export
rpool/export/home                            404M  66.6G  86.5K  /export/home
rpool/export/home/open                       404M  66.6G   404M  /export/home/open
rpool/swap                                  8.50G  75.0G   132M  -

1 Answer 1

1

It looks like there's a few "leaf node" datasets (no sub-datasets) that make up the majority of your disk usage on rpool:

rpool/ROOT/opus-2457409-2017-11-07-release    38.5G
rpool/dump                                    8.00G
rpool/swap                                    8.50G

That sums up to 55G, which is quite close to the reported 55.5G for rpool's USED. Presumably you want to keep rpool/dump and rpool/swap defined the way they are now, so that just leaves rpool/ROOT/opus-...-11-07-release as the thing you can potentially delete stuff from to free up space.

For a reason I don't understand, in your output the mountpoint for that dataset is listed as /, but it's the same across a bunch of other datasets too. You can't actually mount multiple filesystems to the same directory, so clearly something is messed up there. Try creating a new directory, zfs mounting the dataset there, identifying stuff to delete from it with du, and then deleting some files.

Unfortunately, as you delete stuff there, you'll quickly realize that it's not freeing any space, and this is because you have a bunch of snapshots of that dataset too:

rpool/ROOT/opus-2457409-2017-11-07-release                      38.5G  34.6G  6.57G  /
rpool/ROOT/opus-2457409-2017-11-07-release@install              4.40M      -  1.56G  -
rpool/ROOT/opus-2457409-2017-11-07-release@2014-02-25-22:01:25  6.64M      -  1.58G  -
rpool/ROOT/opus-2457409-2017-11-07-release@2014-02-25-22:05:33  84.6M      -  2.02G  -
rpool/ROOT/opus-2457409-2017-11-07-release@2014-03-19-23:25:59  58.0M      -  2.11G  -
rpool/ROOT/opus-2457409-2017-11-07-release@2014-04-12-17:55:40      0      -  2.10G  -
rpool/ROOT/opus-2457409-2017-11-07-release@2014-04-12-17:55:48      0      -  2.10G  -
rpool/ROOT/opus-2457409-2017-11-07-release@2014-04-12-18:13:10  1.44M      -  2.10G  -
rpool/ROOT/opus-2457409-2017-11-07-release@2014-04-12-18:14:31  2.17M      -  2.12G  -
rpool/ROOT/opus-2457409-2017-11-07-release@2015-01-31-21:55:47  15.3M      -  5.61G  -
rpool/ROOT/opus-2457409-2017-11-07-release@2015-01-31-22:34:52  9.18M      -  5.61G  -
rpool/ROOT/opus-2457409-2017-11-07-release@2015-01-31-22:43:54   362K      -  5.61G  -
rpool/ROOT/opus-2457409-2017-11-07-release@2015-01-31-22:44:05   362K      -  5.61G  -
rpool/ROOT/opus-2457409-2017-11-07-release@2015-05-19-21:49:36   227M      -  6.32G  -
rpool/ROOT/opus-2457409-2017-11-07-release@2016-10-04-16:11:30  3.15M      -  31.3G  -
rpool/ROOT/opus-2457409-2017-11-07-release@2016-10-04-16:34:30   936K      -  31.5G  -
rpool/ROOT/opus-2457409-2017-11-07-release@2016-10-04-16:34:42   986K      -  31.5G  -
rpool/ROOT/opus-2457409-2017-11-07-release@2016-10-04-16:53:00   112M      -  31.8G  -

Specifically, check out the last one, @2016-10-04-16:53:00 -- 31.8G out of the total 38.5G is going to be retained by that snapshot. It looks like most of the usage goes back to at least @2016-10-04-16:11:30, so you'll want to delete (at least) these snapshots as well to reclaim the bulk of your data:

rpool/ROOT/opus-2457409-2017-11-07-release@2016-10-04-16:11:30  3.15M      -  31.3G  -
rpool/ROOT/opus-2457409-2017-11-07-release@2016-10-04-16:34:30   936K      -  31.5G  -
rpool/ROOT/opus-2457409-2017-11-07-release@2016-10-04-16:34:42   986K      -  31.5G  -
rpool/ROOT/opus-2457409-2017-11-07-release@2016-10-04-16:53:00   112M      -  31.8G  -

If you don't need the even older snapshots, you might as well delete them too. As a general recommendation, it's usually best to only keep snapshots around as long as you need them for, rather than indefinitely, because you'll eventually capture something big in one of them accidentally and get into a situation like this.

7
  • OK deleted ALL the snapshots. Unneeded. I expected that to free up some space but it didn't? This is kinda where I'm foggy though... rpool/ROOT/opus-2457409-2017-11-07-release is mounted at /, but that says there's 34.6G available. whereas the df says it's full. The du shows very minimal usage so I really don't have anything to delete. WHAT is occupying that 38.5G I guess is what I need to figure out, but how? Nov 3, 2018 at 0:29
  • @ChristopherGlennSchlägel it’s not just the snapshots — the mountpoint doesn’t seem correct to me, like I mentioned above. Did you try mounting that FS somewhere else and looking in it? If so, what did you find?
    – Dan
    Nov 3, 2018 at 0:35
  • Will try now... if I mount that to a different directory it won't interrupt the current mounts? If this is the active root filesystem don't want to risk bringing things down as this is a production server... Nov 3, 2018 at 0:44
  • @ChristopherGlennSchlägel You could also try making a clone first and mounting that somewhere, so you can compare its contents to /. If they’re the same, I think you’re going to have to update your question with the new output from zfs list -t all.
    – Dan
    Nov 3, 2018 at 0:49
  • just added the output for that to the original post Nov 3, 2018 at 0:54

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .