2

I'm using software that regularly scans thousands of files (probably the name and modification time). This takes quite a while since the files are accessed via a network share. Both machines run Windows XP SP3. Are there any simple, reliable tweaks to speed up the directory scan?

2
  • What application are we talking about? Feb 2, 2011 at 20:54
  • Bdb at work ...but somehow I don't think that makes a difference ;-)
    – VolkerK
    Feb 2, 2011 at 20:54

4 Answers 4

1

If you're working with directories that contain hundreds or thousands of files or subdirectories (or even a couple dozen files per directory, all with similar names), you're probably running into collisions with the 8.3 DOS-compatible filenames. There is a Microsoft KB article with instructions on how to improve directory enumeration performance by disabling 8.3 filename creation.

You can also disable the Last Accessed timestamp to improve performance. There's a small chance this may cause problems with other programs, but if you do notice problems you can easily re-enable the Last Accessed timestamp.

You may be able to get away with applying these changes to just the machine that's hosting the network share, but you could apply the changes on both machines just to be safe.

If you're running an antivirus program on both machines, you should also try disabling on-access scanning temporarily on both machines to see if that improves performance. In my experience, McAfee antivirus causes an insane performance hit every time you touch a file, but some other antivirus apps (such as CA and Avast) don't seem to have as severe a performance hit, even with on-access scanning enabled. Most antivirus programs allow you to specify directories to exclude from on-access scanning.

1
  • Both tips helped a lot, though I stick with only disabling the 8.3 filenames for now.
    – VolkerK
    Dec 12, 2009 at 9:04
1

A few ideas immediately come to mind:

  • See if it is possible to cache file data locally, within your application. That way it'd only have to do the scan once.

  • Upgrade the speed of your network. Gigabit ethernet PCI cards are $30-$40 CAD in my area. If you want a router, that's another $100. If it's only one other computer, however, then you can get away with running a piece of crossover (P2P) cat6 ethernet cable between the two.

  • The slowness may also be a function of the speed of the two computers in question. CPUs? Amount and speed of memory? What model of hard drives?

0

Accessing files over the wire is always going to be slower than if you access a local disk. You might be able to increase the performance of the scan if you set it up to run the scan locally. Whenever you start pulling files across the wire you will start noticing these types of slowdowns due to the number of files being accessed.

If you run the same scan remotely it will have to setup the connection to transfer/access each file which is why it is probably so slow.

It is possible to tweak the protocol you are using to access the files, but you will see much more dramatic results just moving the process to a local scan versus a network scan.

Something to note here, many enterprise class backup solutions also include a locally installed component that serves this very same purpose. The files can be transferred in a push versus a pull across the network which is a significant improvement.

1
  • It's not possible to change the location of the files or the way it is accessed, it's just the way this application works. You create a network drive, set the application's data dir to that network drive et volià you have a so-called multi-user installation. Works ok on a fresh install. But over time the number of files grows exceedingly and with it the time it takes to perform that (often used) operation.
    – VolkerK
    Aug 6, 2009 at 18:11
0

I have a similar issue. I was thinking about running an FTP server on the remote machine and mapping a FTP Drive mapping instead. I haven't tried it yet but I bet it would get better performance. If it doesn't then I might just use a FTP client to access it and just abandon microsoft drive mapping altogether. I was going to use NULL Ftp Server to try the scenerio.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .