Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I'm running into a problem with scp under some conditions that are associated with transferring multiple GB files to a SUSE® LINUX Enterprise Server 9 (kernel 2.6.xx) system on an SGI Altix350 w/ 32GB of memory.

I used scp -r -p to copy a directory to another on on the same server for some reason starts using lots of memory, and the cached in top command growing very fast. Samba is running on the system.

If you have any comments either on scp or on a different way to copy a directory with a huge number of directories and files. I'd be interested to hear about it.

Thanks for your help, KhalilHL

share|improve this question

3 Answers 3

up vote 1 down vote accepted

Personally I prefer to use rsync as it lets me resume nicely from interrupted transfers.

To use it locally:

$ rsync -avP /path/to/source /path/to/destination

To transfer local to remote:

$ rsync -avP /path/to/source user@remote:/path/to/destination

To transfer remote to local:

$ rsync -avP user@remote:/path/to/source /path/to/destination

The switches I used are:

  • -a = Archive - create an exact copy of the tree as close as possible
  • -v = Verbose - speaks for itself really
  • -P = show progress of transfers and allow resume from partial files.
share|improve this answer
    
Thank you Matt, is there a limitation using rsync, since i have to copy more then 10millions files? –  AlBouazizi Apr 5 '11 at 8:53
    
It may take a moment or two to sort out its file lists, but it should be ok. I have never tried it on that sort of number of files before. –  Majenko Apr 5 '11 at 9:04
    
21hrs building files list and still not finished. I started the Rsync command: April05, start time: 10:35 today:April06 07:12:38 myaltix:~ # rsync -avP /mysource/my_imgs /mydest/ building file list ... 12236600 files... I hope it will not take days.. –  AlBouazizi Apr 6 '11 at 4:30
    
Ouch... thats alot of files... There must be a better way of handling that many files... –  Majenko Apr 6 '11 at 8:19
    
After building files rsync start copying files (3537073, 17.8% of 20415424). What im doing is copying from a backup dir to a new location im not copying from the original dir which is on production now. when rsync completed copying what will happened if i use rsync again but from the production dir to the destination will it be building files again? i don't if my question is clear. –  AlBouazizi Apr 10 '11 at 6:53

The same server??

SCP has a ton of encryption overhead, which is where your memory problems are coming from. It's also tunnelled through SSH, which means, if you're SCPing stuff between directories on the SAME machine, it's creating a SSH tunnel to itself, and then encrypting traffic that it has to then decrypt.

Why don't you just use CP?

share|improve this answer
    
Actually that's not true - scp has some intelligence. If you copy locally it just works like cp - no asking for passwords, no key exchanges, nothing. –  Majenko Apr 4 '11 at 19:28
    
Thank you for your prompt reply..still i need to use sp bcoz i will transferring the same directory to a different server.. so how to prevent scp to use lots of memory? –  AlBouazizi Apr 4 '11 at 19:29
    
i ran and SGI command ( bcfree ) this command cleared the cached but after few seconds it will be filled again.. –  AlBouazizi Apr 4 '11 at 19:34
    
@matt: I've never tried it, so I'll bow to your experience. Still, it seems like using any logic to determine that it was a local transfer would add some overhead. –  Satanicpuppy Apr 4 '11 at 19:44

I think you misinterpret the output of "top".

the cached in top command growing very fast

The term "cached" means "hard disk cache". You are messing with multi-GB files, so linux will use (almost) every bit of free RAM to help the HDD. Once a process needs more memory, the linux kernel will automagically free some of the cache and use it for the process. This is normal and expected behaviour.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.