Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have roughly around 5 million small (5-30k) files in a single directory that I would like to copy to another machine on the same gigabit network. I tried using rsync, but it would slow down to a crawl after a few hours of running, I assume due to the fact that rsync has to check the source & destination file each time?

My second thought would be to use scp, but wanted to get outside opinion to see if there was a better way. Thanks!

share|improve this question

migrated from stackoverflow.com Jun 2 '11 at 0:33

This question came from our site for professional and enthusiast programmers.

17 Answers 17

Bypass the filesystem.

Are you able to unmount this partition that the files live on it, or mount it readonly? Do that, then something like:

dd if=/dev/PARTITION | ssh username@host "dd of=diskimage.bin"

You can then mount diskimage.bin as a loopback device on the destination side, and copy files out of it to your actual destination file system, or perhaps use the proper tools to stitch it back into an empty partition on the destination side (dangerous, but probably possible, though I've never done it.)

If you are really courageous you can dd it directly back into a partition on the destination side. I don't recommend that.

share|improve this answer

There is something else to consider. Try this:

  • Create a VHD, dynamically sized
  • Mount it, possibly as a directory
  • Set the 'compress entire disk' attribute

By doing this, there is NO overhead for the directory iteration or compression, because that was done at the time the files were written. There is only one file to move--the VHD.

On Windows, I set the default TCP packet size to be larger, like 16348. This means less IP header overhead.

One thing I've run into, though, is that it's best to keep file sizes to under 100 Mb for a network or USB transfer. I use Rar.exe for that - to split up the files.

Works like a champ. This is the equivalent of 'dd' in Linux.. The concept of mounting a compressed filesystem to a directory is normal for Linux as well, so the same logic applies. You ought to ensure all files are closed before the operation starts, as in the other methods.

This has the added benefit of making it possible to put a size quota on a folder. If the VHD is a fixed size, going over that limit will not bring down the server, it will just cause an error creating or writing the file.

A VHD formatted as NTFS can handle millions of files in a folder as well.

share|improve this answer

I prefer using lz4 as fastest compression tool at the moment. SSH option -c arcfour128 uses faster encryption algorithm than default. [1]

So directory transfer looks something like:

tar -c folder | lz4 -c | ssh -carcfour128 somehost 'lz4 -d | tar -x > folder'

Please note that on Debian lz4 command is lz4c and on CentOS it's lz4.

share|improve this answer

To copy millions of files over a gigabit switch (in a trusted environment) you may also use a combination of netcat (or nc) and tar, as already suggested by user55286. This will stream all the files as one large file (see Fast File Copy - Linux! (39 GBs)).

# requires netcat on both servers
nc -l -p 2342 | tar -C /target/dir -xzf -   # destination box
tar -cz /source/dir | nc Target_Box 2342    # source box
share|improve this answer

We had about 1 million files in a directory (about 4 year's worth of files).

And we used robocopy to move files to YYYY/MM directory (about 35-45,000 files per month).. we put robocopy script in a .bat file like this:

ROBOCOPY /NS /NC /NFL /NP /LOG+:H:\BCK_REPORT\ROBO.LOG /MAXAGE:20081101 /MINAGE:20081201 /MOV H:\Cs\out\fix H:\BCK_REPORT\2008\11
ROBOCOPY /NS /NC /NFL /NP /LOG+:H:\BCK_REPORT\ROBO.LOG /MAXAGE:20081201 /MINAGE:20090101 /MOV H:\Cs\out\fix H:\BCK_REPORT\2008\12
ROBOCOPY /NS /NC /NFL /NP /LOG+:H:\BCK_REPORT\ROBO.LOG /MAXAGE:20090101 /MINAGE:20090201 /MOV H:\Cs\out\fix H:\BCK_REPORT\2009\01
ROBOCOPY /NS /NC /NFL /NP /LOG+:H:\BCK_REPORT\ROBO.LOG /MAXAGE:20090201 /MINAGE:20090301 /MOV H:\Cs\out\fix H:\BCK_REPORT\2009\02

brief notes.. /ns /nc /nfl /np is to avoid bloating the log file with additional info /log+... is to write summary information to log file.

/minage and /maxage is to copy files modified with in that date range. 

so for example files modified >= 01/Nov/2008 (inclusive) to files modified < 01/Dec/2008 (not inclusive)

ROBOCOPY /NS /NC /NFL /NP /LOG+:H:\BCK_REPORT\ROBO.LOG /MAXAGE:20081101 /MINAGE:20081201 /MOV H:\Cs\out\fix H:\BCK_REPORT\2008\11

/mov to move the files

then comes source directory

then comes destination directory (directories will be created on the fly as and when required).

It took about 40 - 60 minutes for 1 month worth of transfer (about 35-45,000 files) We reckon it takes about 12 hours or less for 1 year worth of transfer.

Using Windows Server 2003.

All the stuff is logged in the log file... Start Time, End Time and Number of files Copied.

Robocopy saved the day.

share|improve this answer

We are investigating this issue currently. We need to transfer about 18 million small files - about 200GB total. We achieved the best performance using plain old XCopy, but it still took a LONG time. About 3 Days from 1 server to another, about 2 Weeks to an external drive!

Through another process, we needed to duplicate the server. This was done with Acronis. It took about 3 hours!!!

We will be investigating this some more. The dd suggestion above would probably provide similar results.

share|improve this answer

In a similar situation, I tried using tar to batch up the files. I wrote a tiny script to pipe the output of the tar command across to the target machine directly in to a receiving tar process which unbundled the files.

The tar approach almost doubled the rate of transfer compared to scp or rsync (YMMV).

Here are the tar commands. Note that you’ll need to enable r-commands by creating .rhosts files in the home directories of each machine (remove these after they copy is complete - they are notorious security problems). Note also that, as usual, HP-UX is awkward - whereas the rest of the world uses ‘rsh’ for the remote-shell command, HP-UX uses ‘remsh’. ‘rsh’ is some kind of restricted shell in HP parlance.

box1> cd source_directory; tar cf - . | remsh box2 "cd target_directory; tar xf - "

The first tar command creates a file called ‘-’, which is a special token meaning ’standard output’ in this case. The archive created contains all the files in the current directory (.) plus all subdirectories (tar is recursive by default). This archive file is piped into the remsh command which sends it to the box2 machine. On box 2 I first change to the proper receiving directory, then I extract from ‘-’, or ’standard input’ the incoming files.

I had 6 of these tar commands running simultaneously to ensure the network link was saturated with data, although I suspect that disk access may have been the limiting factor.

share|improve this answer

Already tons of good suggestions, but wanted to throw in Beyond Compare. I recently transferred about 750,000 files between 5KB and 20MB from one server to another over a gigabit switch. It didn't even hiccup at all. Granted it took a while, but I'd expect that with so much data.

share|improve this answer

I know this may be stupid - but have you thought of just copying them onto an external disk and carrying it over to the other server? It may actually be the most efficient and simple solution.

share|improve this answer

Robocopy is great for things like this. It will try again after network timeouts and it also allows you set an inter-packet gap delay to now swamp the pipe.

[Edit]

Note that this is a Windows only application.

share|improve this answer

You know, I plus-1'd the tar solution, but -- depending on the environment -- there's one other idea that occurs. You might think about using dd(1). The speed issue with something like this is that it takes many head motions to open and close a file, which you'll be doing five million times. In you could ensure that these are assigned contguously, you could dd them instead, which would cut the number of head motions by a factor of 5 or more.

share|improve this answer

As suggested by sth you could try tar over ssh.

If you do not require encryption (originally you used rsync, but didn't mention it was rsync+ssh) you could try tar over netcat to avoid the ssh overhead.

Of course you can also shorten the time it takes by using gzip or other compression method.

share|improve this answer

Something like this should work well:

tar c some/dir | gzip - |  ssh host2 tar xz

Maybe also omit gzip and the "z" flag for extraction, since you are on a gigabit network.

share|improve this answer
1  
ssh will compress the stream if you pass "-C". Over a lan I wouldn't bother with compressing the stream; over the Internet I probably would, unless it were already compressed. –  Commodore Jaeger Jan 22 '09 at 3:52
5  
Personally I would leave gzip on: even over gigabit ethernet the bottleneck is very unlikely to be the CPU. –  Benji XVI Nov 17 '09 at 20:45
2  
@BenjiXVI the bottleneck will surely be the CPU as gzip will only ever execute on a single core. You can reasonably expect around 30 MB/s with the default compression level of 6 - but this will not max out Gigabit Ethernet. –  syneticon-dj Jul 9 '13 at 14:38

you can try the following (may be in batches of files)

  • tar the batch of files
  • gzip them
  • copy using scp if possible
  • gunzip
  • untar the files
share|improve this answer

Pack them into a single file before you copy it, then unpack them again after it's copied.

share|improve this answer

I'd see how a zip->copy->unzip performs

or whatever your favorite compression/archive system is.

share|improve this answer

I'm sure the fact that you have all FIVE MILLION files in a single directory will throw many tools into a tizzy. I'm not surprised that rsync didn't handle this gracefully - it's quite a "unique" situation. If you could figure out a way to structure the files into some sort of directory structure, I'm sure the standard sync tools such as rsync would be much more responsive.

However, just to give some actual advice - perhaps one solution would be to move the drive physically into the destination machine temporarily so you can do a copy of the files in the actual server (not over the network). Then, move the drive back and use rsync to keep things up to date.

share|improve this answer
2  
+1 for moving drive physically, it's way faster this way –  Robert Gould Jan 22 '09 at 3:44
1  
It sure beats copying everything on a jump drive and going back and forth... –  VirtuosiMedia Jan 22 '09 at 4:04

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.