39

I'm running rsync to sync a directory onto my external USB HDD. It's about 150 gigs of data. 50000+ files I would guess.

It's running it's first sync at the moment, but its copying files at a rate of only 1-5 MB/s. That seems incredibly slow for a USB 2.0 enclosure. There are no other transfers happening on the drive either.

Here are the options I used:

rsync -avz --progress /mysourcefolder /mytargetfolder

I'm running Ubuntu Server 9.10.

  • 2
    are you sure you're getting a USB2 connection? does a (non-rsync) copy or other write operation run at normal speeds? if not, have you tried a copy/other write op with another USB port/cable? – quack quixote Feb 17 '10 at 0:14
  • See also serverfault.com/questions/43014/… - there people also propose using two piped tar commands or cpio. – Blaisorblade Feb 23 '13 at 17:32
35

For the first sync just use

cp -a  /mysourcefolder /mytargetfolder

rsync only adds overhead when the destination is empty.

also.. the -z option is probably killing your performance, you shouldn't be using it if you are not transfering data over a slow link.

  • 3
    rsync is so called because it's for remote synchronization and is not really appropriate for a locally-connected volume for this very reason. – msanford Jun 27 '10 at 22:30
  • 5
    It's supposed to be usable also for local transfers, and it's much more flexible. It's only possibly overkill for the first sync. – Blaisorblade Feb 23 '13 at 17:31
  • 1
    rsync is also a one way sync. Very good for backing up to a server or from a server. However, if you want local TWO-Way sync to a removable drive, you may want to check out csync csync.org/get-it not to be confused with csync2 which is a completely different project. – Jesse the Wind Wanderer Dec 31 '14 at 13:32
  • 3
    rsync -avz --progress /mysourcefolder/ /mytargetfolder or you'll get a copy of mysourcefolder inside of mytargetfolder rather than mirroring the contents – editor Dec 1 '17 at 8:16
  • 2
    This answer does not answer the question. The question was about how to optimize rsync - not replace it with the cp command. – oemb1905 Nov 25 '18 at 4:08
37

If you're using rsync with a fast network or disk to disk in the same machine,

not using compression -z

and using --inplace

speeds it up to the performance of the harddrives or network

compression uses lots of CPU

not using inplace makes the harddrive thrash alot (it uses a temp file before creating the final)

compression & not using inplace is better for doing it over the internet (slow network)

NEW: Be aware of the destination... if there is NTFS "compression" enabled... this severely slows down large files (I'd say 200MB+) rsync almost seems stalled, it's caused by this.

  • NTFS compression is slowwww on big files – Scott Kramer Jun 3 '16 at 3:51
  • I don't see anything about '--inline' on the man page – Antoine Nov 18 '17 at 21:52
  • 1
    It's '--inplace' – Scott Kramer Dec 1 '17 at 17:17
24

Use the -W option. This disables delta/diff comparisons. When the file time/sizes differ, rsync copies the whole file.

Also remove the -z option. This is only useful for compressing network traffic.

Now rsync should be as fast as cp.

  • 5
    Minor note: -z is only useful for low speed network traffic. If your network is fast enough, it'll slow things down, since you'll be limited by CPU. – WhyNotHugo Jul 8 '13 at 2:28
  • 3
    These tips vastly sped up the transfer of my files between two NAS devices, thanks! – djhworld Sep 22 '13 at 13:30
  • 2
    but note that according to the man page says for -W: "This is the default when both the source and destination are specified as local paths, but only if no batch-writing option is in effect." – GuoLiang Oon Jul 5 '17 at 8:15
9

First - the number of files in this case is going to be a major factor. It's an average size of 3MB each. There's probably an io bottleneck influencing the speed in the OP's case. More here - that's a pretty dry read, but the cover picture is worth it.

So, using rsync to copy to an empty directory? Here are some ways to speed it up:

  1. No -z - definitely don't use -z as in the OP.
  2. --no-compress might speed you up. This could have the biggest impact... my test was 13,000 files, total size 200MB, and using rsync 3.1.3. I synced to a different partition on the same internal SSD drive. With --no-compress, I get 18 MBps, and without it I get 15 MBps. cp, by the way, gets 16 MBps. That's a much smaller average file size though. Also - I can't find any documentation for --no-compress. I learned about it from this post on stackexchange.com.
  3. -W to copy files whole - always use this if you don't want it to compare differences; never mind that the point of rsync is to compare differences and only update the changes.
  4. -S to handle sparse files well - can't hurt if you don't have sparse files.
  5. --exclude-from or something similar to exclude files you might not need will cut down the time, but it won't increase your transfer speed.
  6. It's possible if you send the output to a file like this rsync -a /source /destination >/somewhere/rsync.out 2>/somewhere/rsync.err - the first > basically prints a file with all the stuff you would normally see, and the 2> refers to error messages.
  7. Finally, running multiple instances of rsync for different parts of your transfer could be a big help.

My command would be:

rsync -avAXEWSlHh /source /destination --no-compress --info=progress2 --dry-run

If all looked well, I'd delete "--dry-run" and let it go. A, X, and E cover extended attributes and permissions not covered by -a, l is for soft links, H is for hard links, and h is for human readable.

Updating an already synced directory on a USB drive, or the same drive, or over a network, will all require different rsync commands to maximize transfer speed.

Bonus - here's the rsync man page, and if you want to test your hard drive speed, bonnie++ is a good option, and for your network speed, try iperf.


*The post is almost ten years old, but search engines sure like it, and I keep seeing it. It's a good question, and I don't think the top answer to "how to speed up rsync" should be "use cp instead."

2

You don't say what size distribution your files have. If there are many small files then this will reduce overall transfer rate by increasing head movement latency in both the source and destination drives as the tool opens new files and the OS keeps directory entries and other metadata (such as the filesystem's journal if you are using meta-data journaling like ext3/ext4 and NTFS do by default) up to date during the transfer. A file copy proces will only "get into its stride" for larger objects, when a simple bulk transfer is happening.

0

You definitely want to give rclone a try. This thing is crazy fast :

$ tree /usr [...] 26105 directories, 293208 files

$ sudo rclone sync /usr /home/fred/temp -P -L --transfers 64

Transferred: 17.929G / 17.929 GBytes, 100%, 165.692 MBytes/s, ETA 0s Errors: 75 (retrying may help) Checks: 691078 / 691078, 100% Transferred: 345539 / 345539, 100% Elapsed time: 1m50.8s

This is a local copy from and to a LITEONIT LCS-256 (256GB) SSD.

You can add --ignore-checksum on the first run to make it even more faster.

New contributor
Frédéric N. is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.