Super User is a question and answer site for computer enthusiasts and power users. It only takes a minute to sign up.
Sign up to join this community
I have roughly around 5 million small (5-30k) files in a single directory that I would like to copy to another machine on the same gigabit network. I tried using rsync, but it would slow down to a crawl after a few hours of running, I assume due to the fact that rsync has to check the source & destination file each time?
My second thought would be to use scp, but wanted to get outside opinion to see if there was a better way. Thanks!
Something like this should work well:
tar c some/dir | gzip - | ssh host2 tar xz
Maybe also omit gzip and the "z" flag for extraction, since you are on a gigabit network.
gzip will only ever execute on a single core. You can reasonably expect around 30 MB/s with the default compression level of 6 - but this will not max out Gigabit Ethernet.
Jul 9, 2013 at 14:38
I'm sure the fact that you have all FIVE MILLION files in a single directory will throw many tools into a tizzy. I'm not surprised that rsync didn't handle this gracefully - it's quite a "unique" situation. If you could figure out a way to structure the files into some sort of directory structure, I'm sure the standard sync tools such as rsync would be much more responsive.
However, just to give some actual advice - perhaps one solution would be to move the drive physically into the destination machine temporarily so you can do a copy of the files in the actual server (not over the network). Then, move the drive back and use rsync to keep things up to date.
To copy millions of files over a gigabit switch (in a trusted environment) you may also use a combination of netcat (or nc) and tar, as already suggested by user55286. This will stream all the files as one large file (see Fast File Copy - Linux! (39 GBs)).
# requires netcat on both servers
nc -l -p 2342 | tar -C /target/dir -xzf - # destination box
tar -cz /source/dir | nc Target_Box 2342 # source box
z from tar flags in both commands.
We had about 1 million files in a directory (about 4 year's worth of files).
And we used robocopy to move files to YYYY/MM directory (about 35-45,000 files per month).. we put robocopy script in a .bat file like this:
ROBOCOPY /NS /NC /NFL /NP /LOG+:H:\BCK_REPORT\ROBO.LOG /MAXAGE:20081101 /MINAGE:20081201 /MOV H:\Cs\out\fix H:\BCK_REPORT\2008\11
ROBOCOPY /NS /NC /NFL /NP /LOG+:H:\BCK_REPORT\ROBO.LOG /MAXAGE:20081201 /MINAGE:20090101 /MOV H:\Cs\out\fix H:\BCK_REPORT\2008\12
ROBOCOPY /NS /NC /NFL /NP /LOG+:H:\BCK_REPORT\ROBO.LOG /MAXAGE:20090101 /MINAGE:20090201 /MOV H:\Cs\out\fix H:\BCK_REPORT\2009\01
ROBOCOPY /NS /NC /NFL /NP /LOG+:H:\BCK_REPORT\ROBO.LOG /MAXAGE:20090201 /MINAGE:20090301 /MOV H:\Cs\out\fix H:\BCK_REPORT\2009\02
brief notes.. /ns /nc /nfl /np is to avoid bloating the log file with additional info
/log+... is to write summary information to log file.
/minage and /maxage is to copy files modified with in that date range.
so for example files modified >= 01/Nov/2008 (inclusive) to files modified < 01/Dec/2008 (not inclusive)
ROBOCOPY /NS /NC /NFL /NP /LOG+:H:\BCK_REPORT\ROBO.LOG /MAXAGE:20081101 /MINAGE:20081201 /MOV H:\Cs\out\fix H:\BCK_REPORT\2008\11
/mov to move the files
then comes source directory
then comes destination directory (directories will be created on the fly as and when required).
It took about 40 - 60 minutes for 1 month worth of transfer (about 35-45,000 files) We reckon it takes about 12 hours or less for 1 year worth of transfer.
Using Windows Server 2003.
All the stuff is logged in the log file... Start Time, End Time and Number of files Copied.
Robocopy saved the day.
You know, I plus-1'd the tar solution, but -- depending on the environment -- there's one other idea that occurs. You might think about using dd(1). The speed issue with something like this is that it takes many head motions to open and close a file, which you'll be doing five million times. In you could ensure that these are assigned contguously, you could dd them instead, which would cut the number of head motions by a factor of 5 or more.
I prefer using lz4 as fastest compression tool at the moment. SSH option -c arcfour128 uses faster encryption algorithm than default. [1]
So directory transfer looks something like:
tar -c folder | lz4 -c | ssh -carcfour128 somehost 'lz4 -d | tar -x > folder'
Please note that on Debian lz4 command is lz4c and on CentOS it's lz4.
Robocopy is great for things like this. It will try again after network timeouts and it also allows you set an inter-packet gap delay to now swamp the pipe.
[Edit]
Note that this is a Windows only application.
I know this may be stupid - but have you thought of just copying them onto an external disk and carrying it over to the other server? It may actually be the most efficient and simple solution.
Already tons of good suggestions, but wanted to throw in Beyond Compare. I recently transferred about 750,000 files between 5KB and 20MB from one server to another over a gigabit switch. It didn't even hiccup at all. Granted it took a while, but I'd expect that with so much data.
We are investigating this issue currently. We need to transfer about 18 million small files - about 200GB total. We achieved the best performance using plain old XCopy, but it still took a LONG time. About 3 Days from 1 server to another, about 2 Weeks to an external drive!
Through another process, we needed to duplicate the server. This was done with Acronis. It took about 3 hours!!!
We will be investigating this some more. The dd suggestion above would probably provide similar results.
I'd see how a zip->copy->unzip performs
or whatever your favorite compression/archive system is.
Pack them into a single file before you copy it, then unpack them again after it's copied.
In a similar situation, I tried using tar to batch up the files. I wrote a tiny script to pipe the output of the tar command across to the target machine directly in to a receiving tar process which unbundled the files.
The tar approach almost doubled the rate of transfer compared to scp or rsync (YMMV).
Here are the tar commands. Note that you’ll need to enable r-commands by creating .rhosts files in the home directories of each machine (remove these after they copy is complete - they are notorious security problems). Note also that, as usual, HP-UX is awkward - whereas the rest of the world uses ‘rsh’ for the remote-shell command, HP-UX uses ‘remsh’. ‘rsh’ is some kind of restricted shell in HP parlance.
box1> cd source_directory; tar cf - . | remsh box2 "cd target_directory; tar xf - "
The first tar command creates a file called ‘-’, which is a special token meaning ’standard output’ in this case. The archive created contains all the files in the current directory (.) plus all subdirectories (tar is recursive by default). This archive file is piped into the remsh command which sends it to the box2 machine. On box 2 I first change to the proper receiving directory, then I extract from ‘-’, or ’standard input’ the incoming files.
I had 6 of these tar commands running simultaneously to ensure the network link was saturated with data, although I suspect that disk access may have been the limiting factor.
Bypass the filesystem.
Are you able to unmount this partition that the files live on it, or mount it readonly? Do that, then something like:
dd if=/dev/PARTITION | ssh username@host "dd of=diskimage.bin"
You can then mount diskimage.bin as a loopback device on the destination side, and copy files out of it to your actual destination file system, or perhaps use the proper tools to stitch it back into an empty partition on the destination side (dangerous, but probably possible, though I've never done it.)
If you are really courageous you can dd it directly back into a partition on the destination side. I don't recommend that.
you can try the following (may be in batches of files)
As suggested by sth you could try tar over ssh.
If you do not require encryption (originally you used rsync, but didn't mention it was rsync+ssh) you could try tar over netcat to avoid the ssh overhead.
Of course you can also shorten the time it takes by using gzip or other compression method.
There is something else to consider. Try this:
By doing this, there is NO overhead for the directory iteration or compression, because that was done at the time the files were written. There is only one file to move--the VHD.
On Windows, I set the default TCP packet size to be larger, like 16348. This means less IP header overhead.
One thing I've run into, though, is that it's best to keep file sizes to under 100 Mb for a network or USB transfer. I use Rar.exe for that - to split up the files.
Works like a champ. This is the equivalent of 'dd' in Linux.. The concept of mounting a compressed filesystem to a directory is normal for Linux as well, so the same logic applies. You ought to ensure all files are closed before the operation starts, as in the other methods.
This has the added benefit of making it possible to put a size quota on a folder. If the VHD is a fixed size, going over that limit will not bring down the server, it will just cause an error creating or writing the file.
A VHD formatted as NTFS can handle millions of files in a folder as well.