0

I'm encountering extremely slow deletion of files on a Linux machine with ZFS filesystem:

$ ls -l
-rwxr-xr-x 1 root nfsnobody 297056 Jul 17  2018 somefile
$ time rm -f somefile
real    1m30.117s
user    0m0.000s
sys     0m0.705s

Most cases take almost exactly 1.5 minutes. This does not happen with every file: At the time I'm trying to delete files that are withing an installation of Python and these files have been there for a while (days to months). Also, if I create a file in the same location and then try to delete it (seconds to minutes later), the problem does not occur and takes milliseconds.

$ dd if=/dev/zero of=file.txt count=1024 bs=1024
1024+0 records in
1024+0 records out
1048576 bytes (1.0 MB) copied, 0.0191559 s, 54.7 MB/s

$ time rm -f file.txt
real    0m0.014s
user    0m0.002s
sys     0m0.001s

Any ideas on how to start debugging this? Are there any logs that would contain a record of filesystem operations? Is there a way to enable some logging to get lower level information during a specific rm operation?

The machine seems to be operating normally with low cpu and ram usage, no swapping and the disks have plenty of space left. This is the kernel in the machine:

$ uname -a
Linux servername 3.10.0-327.28.3.el7.x86_64 #1 SMP Thu Aug 18 19:05:49 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Additional information about the file system:

$ df .
Filesystem       1K-blocks        Used   Available Use% Mounted on
mypool/wrk     67737890560 51963463040 15774427520  77% /wrk

$ zpool list
NAME     SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
mypool    98T  75.7T  22.3T         -    28%    77%  1.00x  ONLINE  -
3
  • 2
    The nfsnobody vaguely suggests that you might not be on a local disk at all. Does df . identify this as a local ZFS partition?
    – tripleee
    Jul 31, 2019 at 16:03
  • @triplee, I edited the question adding details to answer your question. It is a local disk. There is an NFS server serving this mount, with about 20-30 clients, though, and I suspect that is the origin of the group name.
    – jpcgt
    Jul 31, 2019 at 21:34
  • You could see if you are getting IO errors in /var/log/syslog.
    – gmatht
    Jun 12, 2020 at 8:56

0

You must log in to answer this question.

Browse other questions tagged .