I have roughly around 5 million small (5-30k) files in a single directory that I would like to copy to another machine on the same gigabit network. I tried using rsync, but it would slow down to a crawl after a few hours of running, I assume due to the fact that rsync has to check the source & destination file each time?

My second thought would be to use scp, but wanted to get outside opinion to see if there was a better way. Thanks!

  • The bottleneck is probably the filesystem on the receiving side. Most filesystems will end up being exponentially slower the more files you put in a single directory (that is, every time the rsync adds a new file on the receiving side, the receiving side slows down for the remaining part of the transfer). Many older filesystems cannot even contain more than 32K files in a single directory. Aug 8, 2019 at 12:26

17 Answers 17


Something like this should work well:

tar c some/dir | gzip - |  ssh host2 tar xz

Maybe also omit gzip and the "z" flag for extraction, since you are on a gigabit network.

  • Is it necessary to gzip it, or does ssh compress the stream anyway? Or can be made to do it?
    – Thilo
    Jan 22, 2009 at 3:50
  • 1
    ssh will compress the stream if you pass "-C". Over a lan I wouldn't bother with compressing the stream; over the Internet I probably would, unless it were already compressed.
    – Commodore Jaeger
    Jan 22, 2009 at 3:52
  • 6
    Personally I would leave gzip on: even over gigabit ethernet the bottleneck is very unlikely to be the CPU.
    – Benji XVI
    Nov 17, 2009 at 20:45
  • 6
    @BenjiXVI the bottleneck will surely be the CPU as gzip will only ever execute on a single core. You can reasonably expect around 30 MB/s with the default compression level of 6 - but this will not max out Gigabit Ethernet. Jul 9, 2013 at 14:38
  • 2
    use pbzip2? ...
    – Apache
    Nov 19, 2013 at 10:57

I'm sure the fact that you have all FIVE MILLION files in a single directory will throw many tools into a tizzy. I'm not surprised that rsync didn't handle this gracefully - it's quite a "unique" situation. If you could figure out a way to structure the files into some sort of directory structure, I'm sure the standard sync tools such as rsync would be much more responsive.

However, just to give some actual advice - perhaps one solution would be to move the drive physically into the destination machine temporarily so you can do a copy of the files in the actual server (not over the network). Then, move the drive back and use rsync to keep things up to date.

  • 7
    +1 for moving drive physically, it's way faster this way Jan 22, 2009 at 3:44
  • 1
    It sure beats copying everything on a jump drive and going back and forth... Jan 22, 2009 at 4:04
  • @RobertGould Let's use IPoAC as our transmission protocol :"D
    – coolcat007
    Dec 5, 2015 at 14:32

To copy millions of files over a gigabit switch (in a trusted environment) you may also use a combination of netcat (or nc) and tar, as already suggested by user55286. This will stream all the files as one large file (see Fast File Copy - Linux! (39 GBs)).

# requires netcat on both servers
nc -l -p 2342 | tar -C /target/dir -xzf -   # destination box
tar -cz /source/dir | nc Target_Box 2342    # source box
  • These days with more and more things trying IPv6 first you may need to also use the -4 switch with your nc command on both ends to make it work on an "old" IPv4 LAN. Sep 21, 2016 at 3:00
  • Very nice way! For some files like jpg the compression doesn't have sense, so it can be disabled by omitting z from tar flags in both commands.
    – Stalinko
    Jul 12, 2021 at 12:11

We had about 1 million files in a directory (about 4 year's worth of files).

And we used robocopy to move files to YYYY/MM directory (about 35-45,000 files per month).. we put robocopy script in a .bat file like this:

ROBOCOPY /NS /NC /NFL /NP /LOG+:H:\BCK_REPORT\ROBO.LOG /MAXAGE:20081101 /MINAGE:20081201 /MOV H:\Cs\out\fix H:\BCK_REPORT\2008\11
ROBOCOPY /NS /NC /NFL /NP /LOG+:H:\BCK_REPORT\ROBO.LOG /MAXAGE:20081201 /MINAGE:20090101 /MOV H:\Cs\out\fix H:\BCK_REPORT\2008\12
ROBOCOPY /NS /NC /NFL /NP /LOG+:H:\BCK_REPORT\ROBO.LOG /MAXAGE:20090101 /MINAGE:20090201 /MOV H:\Cs\out\fix H:\BCK_REPORT\2009\01
ROBOCOPY /NS /NC /NFL /NP /LOG+:H:\BCK_REPORT\ROBO.LOG /MAXAGE:20090201 /MINAGE:20090301 /MOV H:\Cs\out\fix H:\BCK_REPORT\2009\02

brief notes.. /ns /nc /nfl /np is to avoid bloating the log file with additional info /log+... is to write summary information to log file.

/minage and /maxage is to copy files modified with in that date range. 

so for example files modified >= 01/Nov/2008 (inclusive) to files modified < 01/Dec/2008 (not inclusive)

ROBOCOPY /NS /NC /NFL /NP /LOG+:H:\BCK_REPORT\ROBO.LOG /MAXAGE:20081101 /MINAGE:20081201 /MOV H:\Cs\out\fix H:\BCK_REPORT\2008\11

/mov to move the files

then comes source directory

then comes destination directory (directories will be created on the fly as and when required).

It took about 40 - 60 minutes for 1 month worth of transfer (about 35-45,000 files) We reckon it takes about 12 hours or less for 1 year worth of transfer.

Using Windows Server 2003.

All the stuff is logged in the log file... Start Time, End Time and Number of files Copied.

Robocopy saved the day.

  • robocopy these days has the switch /MT[:n] for Do multi-threaded copies with n threads (default 8) to achieve the same effect only better and not reliant on date ranges, and allows for a single command line, instead of one per thread. Though the MT switch is not available on Windows 2003. Sep 21, 2016 at 3:04

I prefer using lz4 as fastest compression tool at the moment. SSH option -c arcfour128 uses faster encryption algorithm than default. [1]

So directory transfer looks something like:

tar -c folder | lz4 -c | ssh -carcfour128 somehost 'lz4 -d | tar -x > folder'

Please note that on Debian lz4 command is lz4c and on CentOS it's lz4.

  • ssh encryption/decryption can be a bottleneck due to the cpu usage on either source or destination cpu and the single threaded nature of nearly all ssh implementations. It's a private gigabit LAN, so no need to encrypt. Sep 21, 2016 at 3:13

You know, I plus-1'd the tar solution, but -- depending on the environment -- there's one other idea that occurs. You might think about using dd(1). The speed issue with something like this is that it takes many head motions to open and close a file, which you'll be doing five million times. In you could ensure that these are assigned contguously, you could dd them instead, which would cut the number of head motions by a factor of 5 or more.


Robocopy is great for things like this. It will try again after network timeouts and it also allows you set an inter-packet gap delay to now swamp the pipe.


Note that this is a Windows only application.

  • Assuming you are on windows of course. The nice thing about robocopy is that the app is responsible for iterating over the files, Problem with unix utils is that you might run out of shell space expanding the names. Jan 22, 2009 at 4:00

I know this may be stupid - but have you thought of just copying them onto an external disk and carrying it over to the other server? It may actually be the most efficient and simple solution.


Already tons of good suggestions, but wanted to throw in Beyond Compare. I recently transferred about 750,000 files between 5KB and 20MB from one server to another over a gigabit switch. It didn't even hiccup at all. Granted it took a while, but I'd expect that with so much data.


We are investigating this issue currently. We need to transfer about 18 million small files - about 200GB total. We achieved the best performance using plain old XCopy, but it still took a LONG time. About 3 Days from 1 server to another, about 2 Weeks to an external drive!

Through another process, we needed to duplicate the server. This was done with Acronis. It took about 3 hours!!!

We will be investigating this some more. The dd suggestion above would probably provide similar results.


I'd see how a zip->copy->unzip performs

or whatever your favorite compression/archive system is.

  • yeah compressing them into one file would be a good idea too Jan 22, 2009 at 3:47
  • even just a tarball Jan 22, 2009 at 3:57

Pack them into a single file before you copy it, then unpack them again after it's copied.


In a similar situation, I tried using tar to batch up the files. I wrote a tiny script to pipe the output of the tar command across to the target machine directly in to a receiving tar process which unbundled the files.

The tar approach almost doubled the rate of transfer compared to scp or rsync (YMMV).

Here are the tar commands. Note that you’ll need to enable r-commands by creating .rhosts files in the home directories of each machine (remove these after they copy is complete - they are notorious security problems). Note also that, as usual, HP-UX is awkward - whereas the rest of the world uses ‘rsh’ for the remote-shell command, HP-UX uses ‘remsh’. ‘rsh’ is some kind of restricted shell in HP parlance.

box1> cd source_directory; tar cf - . | remsh box2 "cd target_directory; tar xf - "

The first tar command creates a file called ‘-’, which is a special token meaning ’standard output’ in this case. The archive created contains all the files in the current directory (.) plus all subdirectories (tar is recursive by default). This archive file is piped into the remsh command which sends it to the box2 machine. On box 2 I first change to the proper receiving directory, then I extract from ‘-’, or ’standard input’ the incoming files.

I had 6 of these tar commands running simultaneously to ensure the network link was saturated with data, although I suspect that disk access may have been the limiting factor.


Bypass the filesystem.

Are you able to unmount this partition that the files live on it, or mount it readonly? Do that, then something like:

dd if=/dev/PARTITION | ssh username@host "dd of=diskimage.bin"

You can then mount diskimage.bin as a loopback device on the destination side, and copy files out of it to your actual destination file system, or perhaps use the proper tools to stitch it back into an empty partition on the destination side (dangerous, but probably possible, though I've never done it.)

If you are really courageous you can dd it directly back into a partition on the destination side. I don't recommend that.


you can try the following (may be in batches of files)

  • tar the batch of files
  • gzip them
  • copy using scp if possible
  • gunzip
  • untar the files

As suggested by sth you could try tar over ssh.

If you do not require encryption (originally you used rsync, but didn't mention it was rsync+ssh) you could try tar over netcat to avoid the ssh overhead.

Of course you can also shorten the time it takes by using gzip or other compression method.


There is something else to consider. Try this:

  • Create a VHD, dynamically sized
  • Mount it, possibly as a directory
  • Set the 'compress entire disk' attribute

By doing this, there is NO overhead for the directory iteration or compression, because that was done at the time the files were written. There is only one file to move--the VHD.

On Windows, I set the default TCP packet size to be larger, like 16348. This means less IP header overhead.

One thing I've run into, though, is that it's best to keep file sizes to under 100 Mb for a network or USB transfer. I use Rar.exe for that - to split up the files.

Works like a champ. This is the equivalent of 'dd' in Linux.. The concept of mounting a compressed filesystem to a directory is normal for Linux as well, so the same logic applies. You ought to ensure all files are closed before the operation starts, as in the other methods.

This has the added benefit of making it possible to put a size quota on a folder. If the VHD is a fixed size, going over that limit will not bring down the server, it will just cause an error creating or writing the file.

A VHD formatted as NTFS can handle millions of files in a folder as well.

You must log in to answer this question.