I work (create, read, copy, delete, count the number of) with millions of files under Windows XP (and NTFS). Are there any other filesystems or OS better optimized for dealing with millions of files? Can you provide any suggestions on how I can significantly speed up things (say, copy/delete/count)? Any useful utilities?
-
3Millions of files in a directory will kill pretty much any OS and/or filesystem.– Ignacio Vazquez-AbramsJun 13, 2011 at 22:25
-
So maybe the approach should be to store data in fewer files?– LeoJun 13, 2011 at 22:46
-
Or to use a hierarchy of folders to partition the files.– Ignacio Vazquez-AbramsJun 13, 2011 at 22:47
3 Answers
I use FAR Manager when dealing with large numbers of files. It's ideally suited for this sort of task because you can set up filters to limit the number of files you see at one time:
FAR Manager (free and open source)
http://www.farmanager.com/
Both 32-bit and 64-bit editions are available, which is particularly important when dealing with really large numbers of files since 64-bit processing will yield a definite advantage for you.
Check DiskBoss.
DiskBoss is an automated, rule-based file and disk management solution allowing one to perform disk space utilization analysis, file classification and categorization, duplicate files detection and removal, high-speed file synchronization and data migration, disk change monitoring, rule-based file organizing, data wiping and much more.
- It sounds a design issue. You should revise your solution.
- Format your drive(s) with a lower cluster size to save some space.
Total commander
is a good utility for this purpose. Maybe you need to develop your ownbusiness-oriented
solution.
-
Regarding 1: I work with raw files produced in hundreds of thousands by mass spectrometers analyzing protein samples and with results of protein identification in these files. Maybe I should write to the major producers of mass spectrometers and to the developers of protein identification software that they should revise their solutions.– LeoJun 14, 2011 at 2:51
-
Odds are good that these software developers are interested in their customers feedback, and if you are paying for support, then this type of request should absolutely be submitted to them.– KirkJun 14, 2011 at 3:29
-
@Leo: So small data is not suitable to be written on file system. A database is more suitable here.– XaqronJun 14, 2011 at 3:43