I'm using rsync for my own personal backup, but currently, I have an ISP that throttles prolonged transfers. Reading the rsync man pages, I haven't found an option, but I'm wondering if rsync has a setting to start a new transfer after transferring X number of bytes or holding a transfer for Y seconds.

In effect, unplugging the network cable and then re-connecting it every few minutes would do the same thing, but I'm not sitting at my home all day.

Wondering about this, and as a secondary question, I'm also wondering if rsync initiates a new file transfer session for each file or sends its data as one continuous stream.

  • While this doesn't directly answer your question, if backup transfers take too much time and this is due to you always making full backups, it would make sense to consider differential/incremental backups instead. These approaches have their drawbacks, but they can drastically reduce the amount of transferred data, if this is your primary goal. – vtest Sep 21 '10 at 20:22
up vote 3 down vote accepted

The -e option can be used to specify a transport program that it uses instead of the default ssh to open a shell on the other side. I don't know of any replacement that will do what you want though.

rsync works by running rsyncd, rsync daemon, on the remote side and communicating with it in one long stream.

  • Rather than an ssh replacement, I'd look for a wrapper, maybe autossh. Anyway I agree on tackling the problem at the ssh level rather than the rsync level. – Gilles Aug 18 '10 at 23:34

What's the nature of the throttling?

If it only kicks in for prolonged transfers of a certain speed, maybe you can use rsyncs rate throttling, from the man page:

              This  option  allows  you to specify a maximum transfer rate in kilobytes
              per second. This option is most effective when  using  rsync  with  large
              files  (several  megabytes and up). Due to the nature of rsync transfers,
              blocks of data are sent, then if rsync determines the  transfer  was  too
              fast,  it  will wait before sending the next data block. The result is an
              average transfer rate equaling the specified limit. A value of zero spec‐
              ifies no limit.

Alternatively, you could split up the files into equal sized chunks, then run a script that transfers each in turn, sleeping in between transfers.

rsync will use the same data stream for all the files. After the initial sync up, only file changes will be sent. I wouldn't worry about bandwidth throttling unless you routinely make huge changes to the the directory tree you are backing up.

Rsync uses a combination of timestamps and checksums to identify which files have changed and where the files have changed. For log type files there is some syncronization overhead and then the tail of the file is transfered. There should be some slowdowns in bandwidth utilization when files have changed as files need to be read and checksummed at both ends. These may be sufficient to keep bandwidth throttling from kicking in.

rsync backups can be interupted, and resumed later. It will then proceed to backup the current changes including any missed changes from prior runs. If you have a large file to backup, frequent interruptions may not work. You need to let it run long enough to transfer the file. Any strategy based on frequent interupts of the backup are likey to fail.

ssh is encrypted and should be exempted from throttling, although they may throttle it for large packets. If they are throttling on packet size, you will get throttled no matter how frequently you interrupt the process. As you are likely transferring only small part of your directory structure, throttling should still leave you with reasonably fast backup.

  • Currently, I'm doing the initial backup to a mostly-fresh drive. Once this is done, I don't have to worry about throttling so much. – palswim Aug 19 '10 at 16:04
  • In this case I would just run rsync as long as it takes. This is easier if you can arange to do it on a LAN rather than over the Internet – BillThor Aug 20 '10 at 5:26

I wonder why nobody had mentioned writing a script that sends SIGSTOP and SIGCONT every now and then to the rsync process.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.