I was running a Python process which creates a huge number of files under a single directory (I should have been smarter and bucketed them into multiple directories, but that's another story).
After a while, I noticed that I couldn't go further and the script started giving me an error:
IOError: [Errno 28] No space left on device: /scr1/data/somefile_00023532.txt
Upon researching I've found that the most common reasons are (1) the partition really does not have space left and (2) all inodes are used up. However, df
shows
$ df -h
...
/dev/sdb1 2.7T 1.2T 1.4T 46% /scr1
...
(/dev/sdb1 is the partition in question) and
$ df -i
...
/dev/sdb1 183148544 17725595 165422949 10% /scr1
...
so both space and inodes are indeed still available.
There may be too many files:
$ ls /scr1/data | wc
6468500 6468500 349747747
but ext4 should be able to handle this.
What could be causing Linux to think the space is used up?
UPDATE 1
It appears with problem happens with a specific file name. For example,
$ touch /scr1/data/somefilewithproblem.txt
touch: cannot touch ‘/scr1/data/somefilewithproblem.txt‘: No space left on device
but other files with the same file name pattern (I'm using something benign, like a hash in hex, numeric ID number, etc.) do not have the same issue.
UPDATE 2
DUH!! It looks like the filesystem was corrupt somehow, and after running fsck.ext4
on the partition in question, the issue went away.
Thank you very much for those who offered me help!