I'm encountering extremely slow deletion of files on a Linux machine with ZFS filesystem:
$ ls -l
-rwxr-xr-x 1 root nfsnobody 297056 Jul 17 2018 somefile
$ time rm -f somefile
real 1m30.117s
user 0m0.000s
sys 0m0.705s
Most cases take almost exactly 1.5 minutes. This does not happen with every file: At the time I'm trying to delete files that are withing an installation of Python and these files have been there for a while (days to months). Also, if I create a file in the same location and then try to delete it (seconds to minutes later), the problem does not occur and takes milliseconds.
$ dd if=/dev/zero of=file.txt count=1024 bs=1024
1024+0 records in
1024+0 records out
1048576 bytes (1.0 MB) copied, 0.0191559 s, 54.7 MB/s
$ time rm -f file.txt
real 0m0.014s
user 0m0.002s
sys 0m0.001s
Any ideas on how to start debugging this? Are there any logs that would contain a record of filesystem operations? Is there a way to enable some logging to get lower level information during a specific rm
operation?
The machine seems to be operating normally with low cpu and ram usage, no swapping and the disks have plenty of space left. This is the kernel in the machine:
$ uname -a
Linux servername 3.10.0-327.28.3.el7.x86_64 #1 SMP Thu Aug 18 19:05:49 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
Additional information about the file system:
$ df .
Filesystem 1K-blocks Used Available Use% Mounted on
mypool/wrk 67737890560 51963463040 15774427520 77% /wrk
$ zpool list
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
mypool 98T 75.7T 22.3T - 28% 77% 1.00x ONLINE -
nfsnobody
vaguely suggests that you might not be on a local disk at all. Doesdf .
identify this as a local ZFS partition?/var/log/syslog
.