3

I was running a Python process which creates a huge number of files under a single directory (I should have been smarter and bucketed them into multiple directories, but that's another story).

After a while, I noticed that I couldn't go further and the script started giving me an error:

IOError: [Errno 28] No space left on device: /scr1/data/somefile_00023532.txt

Upon researching I've found that the most common reasons are (1) the partition really does not have space left and (2) all inodes are used up. However, df shows

$ df -h
...
/dev/sdb1       2.7T  1.2T  1.4T  46% /scr1
...

(/dev/sdb1 is the partition in question) and

$ df -i 
...
/dev/sdb1      183148544 17725595 165422949   10% /scr1
...

so both space and inodes are indeed still available.

There may be too many files:

$ ls /scr1/data | wc
6468500 6468500 349747747

but ext4 should be able to handle this.

What could be causing Linux to think the space is used up?

UPDATE 1

It appears with problem happens with a specific file name. For example,

$ touch /scr1/data/somefilewithproblem.txt
touch: cannot touch ‘/scr1/data/somefilewithproblem.txt‘: No space left on device

but other files with the same file name pattern (I'm using something benign, like a hash in hex, numeric ID number, etc.) do not have the same issue.

UPDATE 2

DUH!! It looks like the filesystem was corrupt somehow, and after running fsck.ext4 on the partition in question, the issue went away.

Thank you very much for those who offered me help!

  • 1
    6 million files is WAY too many to have on a single directory; you need to break that up. What you are probably seeing is a hash collision: the name you are trying to use has the same has as one of the other 6 million files. – psusi Nov 4 '14 at 20:31
  • That's indeed an interesting hypothesis, and I certainly screwed up in not breaking them up into multiple directories. But I somehow would be worried if the hashing mechanism actually causes collisions after just 6 million files. Anyways, it turned out there was some issue with the partition, which was the real culprit. Thanks for your insight. – Taro Sato Nov 5 '14 at 2:17
  • I have 10 million+ files needed in a single directory. I'm running text analytics on users of reddit. All users who comment in a single day is in the millions, and I have two months of comments to analyze. Suffice to say I actually need this many files. So is it a hash collision? After my python prog gave me OSError, I copied a big file manually to the "full disk" and it copied just fine, so it's not actually full. – Geoffrey Anderson Dec 11 '18 at 19:51
3

"No space left on device" can be a very missleading error. It can pop up on all sorts of write conditions other than having no disk space available.

Does that file already exist and is owned by someone else?

Do you have quotas enabled on this filesystem?

If you try to create another file in that filesystem manually (ie touch /src1/data/testfile) do you get the same error, or does it create the file?

Are you able to try creating a file as root?

  • There is no quota set on the partition. New files can be created via touch. No other process is writing to the partition... really confused by what I am seeing. – Taro Sato Nov 3 '14 at 21:44
  • How is your process opening the files? Does it close them once it creates them, or does it keep them open? You could be hitting a limit on allowed number of open files. – Omnipresence Nov 4 '14 at 13:46
  • 1
    I am running multiple Python processes with multiprocessing.Pool, but it happens when I limit to a single process so that's not an issue. I use with open and wrap around only the write operation, so there shouldn't be too many connections left open. What's weird, though, is that upon further investigation this problem seems to happen with a specific file name; I cannot create a file with the same name with touch on the shell, for example. – Taro Sato Nov 4 '14 at 19:03

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.