0

EDIT:

Turns out it is hash collision from the ext4 filesystem.

Im writing a lot of files to a folder, no subdirectories, however after 2.8 million files this error shows up:

with open(bottleneck_path, 'w') as save_file:
OSError: [Errno 28] No space left on device: '/home/user/path/redacted'

I've checked the length of the filename which is only 149 characters long. The file size is supposed to be around 18K.

My operating system is Linux.

df -i:

Filesystem                               Inodes   IUsed    IFree IUse% Mounted on
udev                                    4106923     544  4106379    1% /dev
tmpfs                                   4113022     836  4112186    1% /run
/dev/sda2                              28401664 9008557 19393107   32% /
tmpfs                                   4113022      40  4112982    1% /dev/shm
tmpfs                                   4113022       5  4113017    1% /run/lock
tmpfs                                   4113022      16  4113006    1% /sys/fs/cgroup
/dev/sda1                                     0       0        0     - /boot/efi

df -T

Filesystem                             Type      1K-blocks       Used  Available Use% Mounted on
udev                                   devtmpfs   16427692          0   16427692   0% /dev
tmpfs                                  tmpfs       3290420      22136    3268284   1% /run
/dev/sda2                              ext4      447088512  355325584   69029056  84% /
tmpfs                                  tmpfs      16452088      82448   16369640   1% /dev/shm
tmpfs                                  tmpfs          5120          4       5116   1% /run/lock
tmpfs                                  tmpfs      16452088          0   16452088   0% /sys/fs/cgroup
/dev/sda1                              vfat         523248       3684     519564   1% /boot/efi

du -sh .

56G

ls | wc -l

2892084

From my understanding ext4 should be able to handle this just fine.

EDIT:

tune2fs -l /dev/sda2

tune2fs 1.42.13 (17-May-2015)
Filesystem volume name:   <none>
Last mounted on:          /
Filesystem UUID:          cd620466-1f88-400b-acf5-457a9c9544cf
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Filesystem flags:         signed_directory_hash 
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              28401664
Block count:              113587456
Reserved block count:     5679372
Free blocks:              82864623
Free inodes:              25755495
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      996
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8192
Inode blocks per group:   512
Flex block group size:    16
Filesystem created:       Wed Mar  1 15:14:22 2017
Last mount time:          Mon Mar 27 13:20:00 2017
Last write time:          Mon Mar 27 13:20:00 2017
Mount count:              35
Maximum mount count:      -1
Last checked:             Wed Mar  1 15:14:22 2017
Check interval:           0 (<none>)
Lifetime writes:          1813 GB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:           256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
First orphan inode:       1312056
Default directory hash:   half_md4
Directory Hash Seed:      e186507d-32b5-49c0-8ce1-09bf2a75d816
Journal backup:           inode blocks

Using

touch /home/user/path/redacted/somefile_1

throws an error but the same filename structure with a different name works fine. eg:

touch /home/user/path/redacted/somefile_2

EDIT: Turns out it is hash collision from the ext4 filesystem.

  • Have you try touch /home/user/path/redacted/filename to see if the problem persist? – Romeo Ninov Mar 29 '17 at 7:06
  • That command throws No space left on device – Wesley Mar 29 '17 at 7:07
  • You can make one more test like dd if=/dev/zero of=/home/user/path/redacted/filename count=10 bs=1024 to make 10k file. But it seems like a python bug – Romeo Ninov Mar 29 '17 at 7:09
  • I just tested it with another filename and that works fine. Its a structure filename consisting of the path and some number. – Wesley Mar 29 '17 at 7:10
  • Do the path where you try to write a file exist? Also check if by accident you do not try to write in file, but you have already directory with this name – Romeo Ninov Mar 29 '17 at 7:12
0

Can you test if it's really OS related?

Since you didn't put your python script code to the question, here is a script which creates 3mio files with 150 character filenames each. On my end this is running fine without any errors.

After running the script successfully :

  • Move this question to stackoverflow to get help with your code.

if the script fails (ie. after 2.8 mio files):

  • Move this question to unix/linux to get linux OS related help.

You need to set the target_dir to an existing, empty directory

#!/usr/bin/env python3
# vim:fenc=utf-8 ts=4 sw=4 sts=4 et:
# -*- coding: utf-8 -*-
import sys

target_dir = "/home/user/sandbox/test/"
no_files = 3000000

sys.stdout.write("start")


def pretty_no(nr, l):
    #  create filename string
    res = str(nr)
    l -= len(res)
    return "0"*l+res

for i in range(0, no_files):
    fn = target_dir+pretty_no(i, 150)
    f = open(fn, "w")
    f.write("hello world!")
    f.close()
    sys.stdout.write("\r%s files" % i)

print("\ndone")
| improve this answer | |
  • I should've updated my post but it turns out that it is due to hash collision from the ext4 filesystem. – Wesley Mar 29 '17 at 11:42
  • Interesting. I'm using btrfs. It's possible to convert ext4 to it with btrfs-convert but maybe only for SSD drives. – Michael D. Mar 29 '17 at 11:55
  • Its an SSD actually. But I've already resolved the issue by writing data to separate directories. – Wesley Mar 29 '17 at 12:15
0

The problem isn't that there are hash collisions --- htree can support collisions just fine. The problem is that ext4 currently only supports a two level deep hash tree. There are patches to lift this limit:

https://www.spinics.net/lists/linux-ext4/msg55729.html

Note though that performance when you have two million directories is not going to be great. I recommend using a multi-level directory hierarchy --- e.g., /home/t/y/tytso instead of having gazllions of directories in /home.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.