Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I'm using a VPS (running Debian) with 512mb RAM. I'm trying to copy some file locally on the server with rsync, but it fails fast with this message:

rsync: writefd_unbuffered failed to write 5 bytes to socket [sender]: Broken pipe (32)
rsync: connection unexpectedly closed (13505 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(601) [sender=3.0.7]

It's only 113MB of files (10.000 of them). But it fails long before reaching all of those files.

I believe this is because of the limited memory availible but it seems like i should be enough? Is there any way around this? Could i make rsync use a file as a memory stack?

Update on request

rsync command

rsync -av /srv .

ulimit -a

nine@www:~$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 16382
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

free

nine@www:~$ free
             total       used       free     shared    buffers     cached
Mem:        524288     231104     293184          0          0          0
-/+ buffers/cache:     231104     293184
Swap:            0          0          0

Swap would probably help. The problem is that i don't think my VPS supports adding that.

Try to make a swap

nine@www:~$ sudo swapon /home/nine/swap.file 
swapon: /home/nine/swap.file: swapon failed: Operation not allowed

Why I think it's a memory problem

I got the question why I think it's a memory problem. It's because of what the control panel for my VPS tells me. There is a table which shows errors for the machine in it looks like this:

resourcetype    held    maxheld     limit           failcnt
numiptent       24  24      800             0
numfile         1612    2324        4200            0
dcachesize      440391  547046      20480000        0
numothersock    104     139     6000            0
dgramrcvbuf     0   17224       262144000       0
othersockbuf    172272  442048      262144000       0
tcprcvbuf       114688  4962176     262144000       0
tcpsndbuf       122528  2177672     262144000       0
numsiginfo      0   3       256             0
numpty      1   3       32          0
numflock        6   13      4120            0
numtcpsock      7   34      6000            0
oomguarpages    14029   23497       131072          0
vmguarpages     0   0       131072          0
physpages       14029   23497       9223372036854776000     0
numproc         102     142         4000            0
shmpages        697     1993        512000          0
privvmpages     57818   131075      131072          51
lockedpages     0   0       8192            0
kmemsize        5566036 7796832     117760000       0

Watch the fail count for privvmages, which I believe shows when the memory burst above it's burst limits. This counts up whenever I'm running rsync.

share|improve this question
1  
Could you add the rsync command you are using, the output from ulimit -a and do you have any swap setup ( free ) –  Paul Mar 14 '13 at 21:20
    
@Paul: Yeah, done :) –  Niclas Nilsson Mar 14 '13 at 21:26
    
Why would you need support to add swap (which will probably help) ? Although I would not do it long term, you could create and enable a swap file with the line "dd if=/dev/zero of=/path/to/swap.file bs=1M count=512; mkswap /path/to/swap.file; swapon /path/to/swap.file " from a shellprompt. (this will make a 512 meg swap file, the count parameter will increase or decrease this figure) –  davidgo Mar 14 '13 at 23:58
    
Hmm. Looks fairly standard with limits, and 10,000 files should only be around 1MB, so there is memory. Are both directories on the local machine? As @davidgo says, you can set up swap pretty easily with a swap file, so it is worth trying. –  Paul Mar 14 '13 at 23:59
    
Actually, why do you think the problem is memory ? (Although its best practice to have some swap, nothing in your post implies a memory problem) –  davidgo Mar 14 '13 at 23:59

1 Answer 1

rsync takes more memory when there are more files.

Either split the files in separate folders and rsync them, or TAR all the 10,000 files at source location and rsync the 10000.tar to your vps and then untar it.

I don't know whether this problem is addressed in the latest version (3.1), but it was an issue with earlier versions.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.