5

I'm working on an archival project and currently I'm storing 514600 images (ranging from 200kb to 2mb) on my 4TB NAS drive. The drive itself is formatted as ext4 and I'm writing the files over the network via cifs/smb.

Is there any implications of storing such a huge number of files? Will the filesystem be victim to fragmentation?

Improve this question
1
  • I've read that Btrfs does efficiently store small files and directories. I don't know if it performs better. Just want to give a hint on that so maybe more modern filesystem do better and could be considered.
    – user63835
    Commented Aug 4, 2020 at 15:43

3 Answers 3

Reset to default
5

Fragmentation ocures when a file already on disk is changed such that it becomes larger than it previously was, and no file smaller than your cluster size should end up fragmented. to correctly answer your question, we need to know your cluster size, the percentage of the drive in use, and whether the files get changed after the initial allocation of disk space.

IO operations on many small files vs a few large files are much much slower, due to all the files system overhead (and scan/seek time), so if you consider that degraded performance, then yes. it really depends more on whether you are accessing 5000 of them at once, or just a few at a time.

Ext4 in particular doesn't require defragmentation the same way windows file systems do because of the way it allocates space to files, so as long as your partition is not 90% full, I wouldn't worry about that.

Improve this answer
0
2

Is there any implications of storing such a huge number of files?

I'll add that you should watch for the maximum number of files your filesystem can handle. Files (on ext4 which you use) consume inodes. The inode table is statically allocated when the filesystem is created and the only way to increase it is to recreate the filesystem.

You can check your current inode used/free numbers with df -i or tune2fs -l /path/to/device

The default values should be an issue only when storing very small files.

Improve this answer
0

The access speed decreases significantly. It is a common pain in Java projects, where you have tons of relatively small files. It is more efficient to store them in one single archive, the access is faster in spite of the fact that they have to be unpacked.

In your case, your files are not small, so there will probably not be problems with fragmentation.

You work on an archival project, so you should probably use ZIP or TAR/GZIP. If your files are already packed, you may use just TAR to stick them together into one bigger file. Find some reasonable "unit of packing" - so you may easily locate all your files later.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .