3

I'm on a gigabit LAN with a 2-Core 3 Ghz server dedicated to serving up our Repository.

What I check out a project with about 30,000 mostly small files, it only streams out at about 1 MB/sec. There's about 140 MB so it kinda takes forever.

Surely the bottleneck isn't the hard-drive... it's on an SSD after all. Is it a processing issue?

Windows Server 2008

Improve this question
1
  • 1
    It's probably the network. I suppose you are not the only one using it. And 140 MB with 1 MB/s doesn't seem that much to me. That's little more than 2 minutes, and you only have to do it once, and later update only small parts.
    – petersohn
    May 18, 2010 at 19:06

2 Answers 2

Reset to default
3

Subversion has a really bad track record with performance, especially on Windows. The problem is with the assumptions that all file system operations are fast. And so they go ahead touching a few thousand files prior to an update, touching them again after an update. And generelly every file downloaded will result in at least a write and a rename.

The problem is that NTFS isn't that fast in touching a few thousand files (not even on an SSD; I can see that here ;-)) at once so it kinda breaks down. There probably isn't much you can do, but for comparision you might look how fast it is on Linux or another Unix-like.

Improve this answer
1

there are 2 problems:

a) ntfs: if you have enabled support for 8.3 file names, then windows has to come up with unique 8.3 names. it "attaches" these names as extra names and if the file names in a directory are very similar, than the algorithm to come up with unique 8.3 names has a lot of work to do. the way the algorithm works is explained very well in "windows internals 5th edition, page 946-947".you can disable that feature, checkout http://support.microsoft.com/?scid=kb;en-us;121007&x=15&y=12.

and keep in mind, you are not only checking out 30k files into the working directory but svn creates a .svn subfolder with a local copy of each of these 30k files...

b) explorer: the explorer has quite a job to display folders with lots of items in it. it is time consuming to display all the stats in a list, create icons and what not.

i would suspect mainly problem a)

Improve this answer
2
  • As for (a), depending on how many files there are in each folder this may not pose a problem. For example, most Java projects tend to have both massive numbers of files and massive numbers of directories, which makes the impact of 8.3 file name generation negligible. For as little as 30k files I doubt it poses much of a problem even in a single directory. As for (b), Explorer is remarkably quick with that, even faster than a few OFMs I'm using.
    – Joey
    May 19, 2010 at 16:19
  • a) the problem with 8.3 names exists only in the same directory, obviously. and b) i had problems with explorer when it came to only 10k files, and i created 90k files due a file converting program. and then explorer has a lot to do.
    – akira
    May 20, 2010 at 8:29

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .