Rsync is an open source command line tool for Unix and Windows systems that provides fast incremental file transfer to copy files and synchronize directories locally or between remote hosts. It is widely used for mirroring and backup.

learn more… | top users | synonyms

37
votes
16answers
47k views

How to synchronize the home folder between multiple computers?

I have three computers at home, and would like to have the /home/ folder tree synchronized between the three. Any files/folders (except those hidden) that are modified/added/removed in one of the ...
37
votes
3answers
27k views

rsync delete options

I see on the rsync man page that there are a number of delete options. I don't really understand the differences and implications of each of these - could someone fill me in please? --del ...
28
votes
2answers
22k views

Make rsync use less bandwidth?

I use rsync to backup and synchronize network shares and my computer. I have this figured out. The problem is, when using rsync to transfer files, it uses all the bandwidth it can. I want to cap it's ...
24
votes
15answers
31k views

Best way to copy millions of files between 2 servers

I have roughly around 5 million small (5-30k) files in a single directory that I would like to copy to another machine on the same gigabit network. I tried using rsync, but it would slow down to a ...
20
votes
7answers
17k views

Replacing Dropbox with: Amazon S3 + SSL + GPG/TrueCrypt + Mounting on OSX ?? [closed]

So, right now we're using Dropbox to share various data files around between approximately 10 Mac OS X systems. However, we already have an S3 account and everyone on the lowest Dropbox plan of ...
19
votes
6answers
18k views

Why is rsync -avz faster than scp -r?

I'm a bit puzzled by this? Why is rsync faster than scp? Doesn't rsync use scp beneath or does it do something more efficient? Is there some way to speed up scp?
17
votes
3answers
7k views

Resume rsync over SSH after broken connection?

I have to transfer large amounts of data (>80 GB) over ssh using rsync. Everything is working fine, but the DSL connection where the backup data is sent from will drop once every 24h for up to 3 ...
16
votes
4answers
13k views

Run rsync with root permission on remote machine

I want to sync a folder from my machine with a folder on a remote machine. The remote folder can only be manipulated by root. I have an account on the remote machine which can use sudo. How can I run ...
15
votes
4answers
22k views

How to speed up rsync?

I'm running rsync to sync a directory onto my external USB HDD. It's about 150 gigs of data. 50000+ files I would guess. It's running it's first sync at the moment, but its copying files at a rate of ...
15
votes
6answers
13k views

Parallelizing rsync

I just moved and discovered after some trial and error that somewhere between my home and my remote server, there is some throttling going on...but the throttling is not very intelligent. It only ...
15
votes
3answers
737 views

Moving 2TB (10 mil files + dirs), what's my bottleneck?

Background I ran out of space on /home/data and need to transfer /home/data/repo to /home/data2. /home/data/repo contains 1M dirs, each of which contain 11 dirs and 10 files. It totals 2TB. ...
14
votes
5answers
2k views

Transfering about 300gb in files from one server to another

I have about 200,000 files that I am transferring to a new server today. I haven't done anything on such a large scale before, and wanted to get some advice on how I should go about this. I am ...
13
votes
4answers
5k views

Rsync friendly gzip

I must not be the only one - I'm rsyncing .tar.gz files and notice that every time the full file gets rsynced rather than the differences. Reading into it it seems back in 1999 someone created an ...
12
votes
3answers
10k views

linux merge folders: rsync?

I have two copies of a folder src/ dest/ I want to merge them, doing the following: If a file is only in src, I want it to be moved to dest If a file is only in dest, I want it ignored IE left ...
12
votes
4answers
3k views

Resume recursive scp transfer (with rsync?)

I was transferring several thousand files each ~1MB via scp and my connection was broken after the first 2k files or so. I wanted to know if there was a way to resume the recursive transfer w/o ...
12
votes
2answers
6k views

Why does't rsync use delta-transfer for local files?

I have a big iso image which is currently being downloaded by a torrent client with space-reservation turned on: that means, file size is not changing while some chunks in in (4 Mib) are constantly ...
12
votes
1answer
8k views

Resume transfer of a single file by rsync

In Ubuntu, I want to copy a big file from my harddrive to a removable drive by rsync. For some reason or other, the operation cannot complete in a single run. So I am trying to figure out how to use ...
11
votes
3answers
14k views

rsync file permissions on windows

I have an rsync service that syncs files from remote machine to a machine that drops them on a network drive. I need the copied files to take on the native permissions of the destination folder. ...
11
votes
3answers
3k views

Do I need to have a passphrase for my SSH RSA key?

Before I started at my current job (at a small business), my office had no firewall on the network and literally nothing was ever being backed up. Now that I've signed on as a dedicated sysadmin / ...
11
votes
3answers
8k views

How to do rsync-like encrypted backup?

I want to save a backup of my data on a remote server, but never want the backup server to see the data unencrypted. Editing a single file and backing up should not result in everything being ...
11
votes
1answer
10k views

How to rsync a large file, with as little CPU and bandwidth expense as possible?

I have a 500 GB file that I plan on backing up remotely. The file changes often. I'll be rsyncing it from a desktop to a server. Both can run rsync client or server. What is the proper command for ...
10
votes
1answer
6k views

which is the rsync command to «smartly» merge two folders?

I have some folders that got duplicated by mistake and I have to merge them together. Let's say folder A contain file X and folder B contains file Y, I would like to get file X copied to folder B. ...
10
votes
1answer
7k views

How can I set environment variables for a remote rsync process?

When I use rsync to copy files from one computer to another, an rsync process is started on both ends. However, on the remote end, shell initialization files are apparently not read, so I cannot set ...
9
votes
6answers
6k views

Backup services that allow remote rsync

I'm looking for an online backup service that allows me to rsync my backups (1-3GB of incrementals per week, 50GB in total). I currently use my hosting space on Dreamhost for this but as it's ...
9
votes
2answers
4k views

Does anyone know why rsync would keep sending the files over and over again?

I'm trying to using rsync to backup some files, about half a TB. It's now it a state where it keeps sending the same files everytime it runs. for example: rsync -av /data/source/* ...
9
votes
10answers
5k views

Something like Dropbox for local use with a NAS as storage server

I am looking for a solution to sync folder pairs between a NAS and multiple local macs. Each of the macs could edit files and the other macs should then get synced automatically. Basically my own ...
9
votes
3answers
1k views

How to make a redundant desktop system with daily snapshots? (Is btrfs ready for use?)

I want to configure a desktop system in which the home filesystem would be redundant (e.g. RAID-1), and would have weekly snapshots taken. I've already done this with ZFS, the snapshot system is ...
8
votes
3answers
4k views

rsync on Windows

I'm trying to run rsync on Windows with the cwrsync port. I'm issuing the following command (note that this is just a dry run): rsync -nPaAz foouser@webxx.example.com:/home/foobar/webapps/barbaz/ ...
8
votes
1answer
4k views

How to exclude rsync excludes from delete?

I'm using rsync to sync files across multiple machines, using the following: rsync -az -e "ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no" \ --delete --delete-excluded --force ...
8
votes
2answers
9k views

rsync --remove-source-files deletes source files one by one or after rsync completes?

I have an rsync command in an application that copies the contents of a folder to another machine. I have the option --remove-source-files because I'm only interested in having the files on the ...
8
votes
2answers
1k views

How to copy Time Machine volume to another disk using rsync while preserving ACLs?

Recently I had to change my Mac's filesystem from case-preserving to being fully case-sensitive. I did that by using CCC and copying data to another disk, then returning back to newly formatted main ...
7
votes
3answers
3k views

Minimal rsync installation without need for cygwin

Is there a way to install rsync on Windows XP without having to use cygwin? I don't want to install cygwin because it takes lot of space. Moreover, I need to make it communicate with a rsync ...
7
votes
2answers
19k views

rsync set group owner, group permission

I want to use rsync to transfer files from my computer to a remote Linux system. Regardless of the local file's group ownership, I want to set these values on the remote side. If I was on the remote ...
7
votes
2answers
396 views

How to backup a 500gb server to my home computer?

what is the best way to backup my server? it has 500gb of data - most of it never changes (photos, videos) so only needs to be downloaded once. At the moment i do big tar.gz's and download them ...
7
votes
2answers
2k views

How to synchronize two instances of a virtual machine to be used in different computers?

I've been using VirtualBox on my desktop for quite a while. Having recently bought a notebook, I want to be able to take my Ubuntu virtual machine with me, and have the changes I do on the notebook ...
6
votes
3answers
4k views

How to gzip a directory, transfer via scp, and decompress in one command?

I frequently transfer large directories over scp and it would be sweet if I could somehow compress the directory, send it, and decompress it all in one. Is something like this possible?
6
votes
3answers
8k views

Is it possible to rate-limit an scp/sftp/rsync/etc transfer from the command-line? ie, manual QoS on a single command?

Specifically, I am looking to rate-limit an scp or sftp session (or other arbitrary network call) in the call itself. For example, let's say I want to copy 100MB to one server, and 1GB to another. ...
6
votes
2answers
6k views

Rsync, two-way sync, and syncing file-deletion operations

I use rsync in the following manner: rsync -Pvaz me@192.168.*.***:Documents/ /Users/me/Documents/ rsync -Pvaz /Users/me/Documents/ me@192.168.*.***:Documents/ The problem: if I delete a file on my ...
6
votes
4answers
4k views

Rsync over ssh, or something else?

I've been trying to use rsync on OSX to Linux or Unix box over which I don't have much control. I've been doing something like this rsync -avz -e ssh remoteuser@remotehost:/remote/dir /this/dir/ ...
6
votes
2answers
2k views

what is the difference between rsnapshot and rdiffbackup?

What is the difference between rsnapshot and rdiffbackup? Which is better for backing up a new snapshot of my laptop (over the Internet from some remote location) to my server (which already has the ...
6
votes
2answers
11k views

rsync: failed to set times on “<dir path>”

I am trying to rsync files from one server to another but getting this error rsync: failed to set times on "/dept/intranet/dept/atest/.": Operation not permitted (1) Can someone help me why so ...
6
votes
1answer
5k views

rsync files newer than 1 week

I want to run rsync on server A to copy all files from Server B when they are newer than 7 days.(find . -mtime -7) I don't want to delete the files on Server B. Thanks for you help.
5
votes
5answers
3k views

Copy a large file over an unreliable link

This isn't really a question about programming, but it is a question about working as a programmer. I hope this is an appropriate forum for this question. I work from home. My Windows XP based ...
5
votes
2answers
8k views

Rsync only changed files

Is it possible to have rsync transfer all files in a directory that have changed or have been created? I don't need something as sophisticated as a CRC diff check; files with different timestamps ...
5
votes
5answers
18k views

What similar rsync tool for Windows?

What can you recommend of an application that looks the same on rsync for windows? I will be doing migration where I need to transfer data from Windows to Linux. I like the rsync on linux where you ...
5
votes
5answers
4k views

cp -r -l in MacOS (recursive copy preserving hard links)

I'm trying to copy a directory tree recursively preserving hardlinks to the file. Using gnu cp, this would work with the -l flag. cp would then recreate the directory structure, but wouldn't need to ...
5
votes
1answer
3k views

Rsync to-check grows during backup

I am currently transferring about 40 Gigs of files from an external disk to my hard drive, and the rsync progress output always shows something like ... to-check=1100/24094) ... to-check=1099/24094) ...
5
votes
2answers
8k views

Use rsync to copy all files except for certain filenames with a certain extension

I have two same-size flash cards, and I want to copy contents of one to the other based on the following rules: I want all directories and subdirectories in place I want to exclude files of type ...
5
votes
2answers
10k views

How to Setup Rsync without password with SSH on UNIX / Linux?

How can I set up a rsync between two hosts without providing any password?
5
votes
3answers
2k views

Linux: Alternative to rsync? (ie, scp with resume)

I've been using rsync to automatically send files from one box to another, which is great compared to scp, since it supports resuming. However, when resuming a very large file (>10gb) rsync has to ...