2

I'm running into a problem with scp under some conditions that are associated with transferring multiple GB files to a SUSE® LINUX Enterprise Server 9 (kernel 2.6.xx) system on an SGI Altix350 w/ 32GB of memory.

I used scp -r -p to copy a directory to another on on the same server for some reason starts using lots of memory, and the cached in top command growing very fast. Samba is running on the system.

If you have any comments either on scp or on a different way to copy a directory with a huge number of directories and files. I'd be interested to hear about it.

Thanks for your help, KhalilHL

3 Answers 3

2

Personally I prefer to use rsync as it lets me resume nicely from interrupted transfers.

To use it locally:

$ rsync -avP /path/to/source /path/to/destination

To transfer local to remote:

$ rsync -avP /path/to/source user@remote:/path/to/destination

To transfer remote to local:

$ rsync -avP user@remote:/path/to/source /path/to/destination

The switches I used are:

  • -a = Archive - create an exact copy of the tree as close as possible
  • -v = Verbose - speaks for itself really
  • -P = show progress of transfers and allow resume from partial files.
11
  • Thank you Matt, is there a limitation using rsync, since i have to copy more then 10millions files?
    – AlBouazizi
    Apr 5, 2011 at 8:53
  • It may take a moment or two to sort out its file lists, but it should be ok. I have never tried it on that sort of number of files before.
    – Majenko
    Apr 5, 2011 at 9:04
  • 21hrs building files list and still not finished. I started the Rsync command: April05, start time: 10:35 today:April06 07:12:38 myaltix:~ # rsync -avP /mysource/my_imgs /mydest/ building file list ... 12236600 files... I hope it will not take days..
    – AlBouazizi
    Apr 6, 2011 at 4:30
  • Ouch... thats alot of files... There must be a better way of handling that many files...
    – Majenko
    Apr 6, 2011 at 8:19
  • After building files rsync start copying files (3537073, 17.8% of 20415424). What im doing is copying from a backup dir to a new location im not copying from the original dir which is on production now. when rsync completed copying what will happened if i use rsync again but from the production dir to the destination will it be building files again? i don't if my question is clear.
    – AlBouazizi
    Apr 10, 2011 at 6:53
0

The same server??

SCP has a ton of encryption overhead, which is where your memory problems are coming from. It's also tunnelled through SSH, which means, if you're SCPing stuff between directories on the SAME machine, it's creating a SSH tunnel to itself, and then encrypting traffic that it has to then decrypt.

Why don't you just use CP?

4
  • Actually that's not true - scp has some intelligence. If you copy locally it just works like cp - no asking for passwords, no key exchanges, nothing.
    – Majenko
    Apr 4, 2011 at 19:28
  • Thank you for your prompt reply..still i need to use sp bcoz i will transferring the same directory to a different server.. so how to prevent scp to use lots of memory?
    – AlBouazizi
    Apr 4, 2011 at 19:29
  • i ran and SGI command ( bcfree ) this command cleared the cached but after few seconds it will be filled again..
    – AlBouazizi
    Apr 4, 2011 at 19:34
  • @matt: I've never tried it, so I'll bow to your experience. Still, it seems like using any logic to determine that it was a local transfer would add some overhead. Apr 4, 2011 at 19:44
0

I think you misinterpret the output of "top".

the cached in top command growing very fast

The term "cached" means "hard disk cache". You are messing with multi-GB files, so linux will use (almost) every bit of free RAM to help the HDD. Once a process needs more memory, the linux kernel will automagically free some of the cache and use it for the process. This is normal and expected behaviour.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .