up vote 1 down vote favorite
share [g+] share [fb]

I work (create, read, copy, delete, count the number of) with millions of files under Windows XP (and NTFS). Are there any other filesystems or OS better optimized for dealing with millions of files? Can you provide any suggestions on how I can significantly speed up things (say, copy/delete/count)? Any useful utilities?

link|improve this question
3  
Millions of files in a directory will kill pretty much any OS and/or filesystem. – Ignacio Vazquez-Abrams Jun 13 '11 at 22:25
So maybe the approach should be to store data in fewer files? – Leo Jun 13 '11 at 22:46
Or to use a hierarchy of folders to partition the files. – Ignacio Vazquez-Abrams Jun 13 '11 at 22:47
feedback

3 Answers

up vote 0 down vote

I use FAR Manager when dealing with large numbers of files. It's ideally suited for this sort of task because you can set up filters to limit the number of files you see at one time:

  FAR Manager (free and open source)
  http://www.farmanager.com/

Both 32-bit and 64-bit editions are available, which is particularly important when dealing with really large numbers of files since 64-bit processing will yield a definite advantage for you.

enter image description here

link|improve this answer
feedback
up vote 0 down vote
  1. It sounds a design issue. You should revise your solution.
  2. Format your drive(s) with a lower cluster size to save some space.
  3. Total commander is a good utility for this purpose. Maybe you need to develop your own business-oriented solution.
link|improve this answer
Regarding 1: I work with raw files produced in hundreds of thousands by mass spectrometers analyzing protein samples and with results of protein identification in these files. Maybe I should write to the major producers of mass spectrometers and to the developers of protein identification software that they should revise their solutions. – Leo Jun 14 '11 at 2:51
Odds are good that these software developers are interested in their customers feedback, and if you are paying for support, then this type of request should absolutely be submitted to them. – Kirk Jun 14 '11 at 3:29
@Leo: So small data is not suitable to be written on file system. A database is more suitable here. – Xaqron Jun 14 '11 at 3:43
feedback
up vote 0 down vote

Check DiskBoss.

DiskBoss is an automated, rule-based file and disk management solution allowing one to perform disk space utilization analysis, file classification and categorization, duplicate files detection and removal, high-speed file synchronization and data migration, disk change monitoring, rule-based file organizing, data wiping and much more.

link|improve this answer
feedback

Your Answer

 
or
required, but never shown
discard

Not the answer you're looking for? Browse other questions tagged or ask your own question.