5

I was running a Python process which creates a huge number of files under a single directory (I should have been smarter and bucketed them into multiple directories, but that's another story).

After a while, I noticed that I couldn't go further and the script started giving me an error:

IOError: [Errno 28] No space left on device: /scr1/data/somefile_00023532.txt

Upon researching I've found that the most common reasons are (1) the partition really does not have space left and (2) all inodes are used up. However, df shows

$ df -h
...
/dev/sdb1       2.7T  1.2T  1.4T  46% /scr1
...

(/dev/sdb1 is the partition in question) and

$ df -i 
...
/dev/sdb1      183148544 17725595 165422949   10% /scr1
...

so both space and inodes are indeed still available.

There may be too many files:

$ ls /scr1/data | wc
6468500 6468500 349747747

but ext4 should be able to handle this.

What could be causing Linux to think the space is used up?

UPDATE 1

It appears with problem happens with a specific file name. For example,

$ touch /scr1/data/somefilewithproblem.txt
touch: cannot touch ‘/scr1/data/somefilewithproblem.txt‘: No space left on device

but other files with the same file name pattern (I'm using something benign, like a hash in hex, numeric ID number, etc.) do not have the same issue.

UPDATE 2

DUH!! It looks like the filesystem was corrupt somehow, and after running fsck.ext4 on the partition in question, the issue went away.

Thank you very much for those who offered me help!

3
  • 1
    6 million files is WAY too many to have on a single directory; you need to break that up. What you are probably seeing is a hash collision: the name you are trying to use has the same has as one of the other 6 million files.
    – psusi
    Nov 4, 2014 at 20:31
  • That's indeed an interesting hypothesis, and I certainly screwed up in not breaking them up into multiple directories. But I somehow would be worried if the hashing mechanism actually causes collisions after just 6 million files. Anyways, it turned out there was some issue with the partition, which was the real culprit. Thanks for your insight.
    – Taro Sato
    Nov 5, 2014 at 2:17
  • I have 10 million+ files needed in a single directory. I'm running text analytics on users of reddit. All users who comment in a single day is in the millions, and I have two months of comments to analyze. Suffice to say I actually need this many files. So is it a hash collision? After my python prog gave me OSError, I copied a big file manually to the "full disk" and it copied just fine, so it's not actually full. Dec 11, 2018 at 19:51

2 Answers 2

4

"No space left on device" can be a very missleading error. It can pop up on all sorts of write conditions other than having no disk space available.

Does that file already exist and is owned by someone else?

Do you have quotas enabled on this filesystem?

If you try to create another file in that filesystem manually (ie touch /src1/data/testfile) do you get the same error, or does it create the file?

Are you able to try creating a file as root?

3
  • There is no quota set on the partition. New files can be created via touch. No other process is writing to the partition... really confused by what I am seeing.
    – Taro Sato
    Nov 3, 2014 at 21:44
  • How is your process opening the files? Does it close them once it creates them, or does it keep them open? You could be hitting a limit on allowed number of open files. Nov 4, 2014 at 13:46
  • 1
    I am running multiple Python processes with multiprocessing.Pool, but it happens when I limit to a single process so that's not an issue. I use with open and wrap around only the write operation, so there shouldn't be too many connections left open. What's weird, though, is that upon further investigation this problem seems to happen with a specific file name; I cannot create a file with the same name with touch on the shell, for example.
    – Taro Sato
    Nov 4, 2014 at 19:03
0

sync command is used to copy all contents from cache to memory. caching does make some processes faster, but swap memory space reduces, causing the "no space left" error to come up.

run the sync command as root user.

$ sync; echo 1 > /proc/sys/vm/drop_caches
$ sync; echo 2 > /proc/sys/vm/drop_caches
$ sync; echo 3 > /proc/sys/vm/drop_caches

After running these commands, run the $ free -g command to check the free, used and total available memory, cache and swap spaces.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .