1

I work (create, read, copy, delete, count the number of) with millions of files under Windows XP (and NTFS). Are there any other filesystems or OS better optimized for dealing with millions of files? Can you provide any suggestions on how I can significantly speed up things (say, copy/delete/count)? Any useful utilities?

3
  • 3
    Millions of files in a directory will kill pretty much any OS and/or filesystem. Jun 13, 2011 at 22:25
  • So maybe the approach should be to store data in fewer files?
    – Leo
    Jun 13, 2011 at 22:46
  • Or to use a hierarchy of folders to partition the files. Jun 13, 2011 at 22:47

3 Answers 3

1

I use FAR Manager when dealing with large numbers of files. It's ideally suited for this sort of task because you can set up filters to limit the number of files you see at one time:

  FAR Manager (free and open source)
  http://www.farmanager.com/

Both 32-bit and 64-bit editions are available, which is particularly important when dealing with really large numbers of files since 64-bit processing will yield a definite advantage for you.

enter image description here

1

Check DiskBoss.

DiskBoss is an automated, rule-based file and disk management solution allowing one to perform disk space utilization analysis, file classification and categorization, duplicate files detection and removal, high-speed file synchronization and data migration, disk change monitoring, rule-based file organizing, data wiping and much more.

0
  1. It sounds a design issue. You should revise your solution.
  2. Format your drive(s) with a lower cluster size to save some space.
  3. Total commander is a good utility for this purpose. Maybe you need to develop your own business-oriented solution.
3
  • Regarding 1: I work with raw files produced in hundreds of thousands by mass spectrometers analyzing protein samples and with results of protein identification in these files. Maybe I should write to the major producers of mass spectrometers and to the developers of protein identification software that they should revise their solutions.
    – Leo
    Jun 14, 2011 at 2:51
  • Odds are good that these software developers are interested in their customers feedback, and if you are paying for support, then this type of request should absolutely be submitted to them.
    – Kirk
    Jun 14, 2011 at 3:29
  • @Leo: So small data is not suitable to be written on file system. A database is more suitable here.
    – Xaqron
    Jun 14, 2011 at 3:43

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .