I was running a Python process which creates a huge number of files under a single directory (I should have been smarter and bucketed them into multiple directories, but that's another story).
After a while, I noticed that I couldn't go further and the script started giving me an error:
IOError: [Errno 28] No space left on device: /scr1/data/somefile_00023532.txt
Upon researching I've found that the most common reasons are (1) the partition really does not have space left and (2) all inodes are used up. However,
$ df -h ... /dev/sdb1 2.7T 1.2T 1.4T 46% /scr1 ...
(/dev/sdb1 is the partition in question) and
$ df -i ... /dev/sdb1 183148544 17725595 165422949 10% /scr1 ...
so both space and inodes are indeed still available.
There may be too many files:
$ ls /scr1/data | wc 6468500 6468500 349747747
but ext4 should be able to handle this.
What could be causing Linux to think the space is used up?
It appears with problem happens with a specific file name. For example,
$ touch /scr1/data/somefilewithproblem.txt touch: cannot touch ‘/scr1/data/somefilewithproblem.txt‘: No space left on device
but other files with the same file name pattern (I'm using something benign, like a hash in hex, numeric ID number, etc.) do not have the same issue.
DUH!! It looks like the filesystem was corrupt somehow, and after running
fsck.ext4 on the partition in question, the issue went away.
Thank you very much for those who offered me help!