39

It’s kind of funny, there are a lot of ways in which Linux command line tools are vastly superior to Windows ones, but one thing I have not found an equivalent to is robocopy. Robocopy is way more versatile than cp, and I can’t figure out how to do what I want with Linux tools.

A specific use case is that I have two directories with mostly similar files, but one has newer source files and one has content files that the other directory doesn’t have, and possibly newer source files (for a website). I want to copy the files from the latter directory into the former, adding new content files and such, but not overwriting newer files in the destination directory.

I tried to figure out how to do this in Linux for maybe half an hour, decided that I’d probably have to learn bash scripting or something to do what I wanted, and then I realized that I could just use robocopy. I also had some other requirements. I was basically converting a Python (Django) website from an SVN repository to a Git repository, and I wanted to make sure I didn’t have uncommitted changes in SVN. However, I wanted to conditionally exclude the .svn directories and the .pyc compiled Python files. The following robocopy command does exactly what I want:

robocopy source destination /XO /E /XD .svn /XF *.pyc

Is there any equivalent to Robocopy in Linux? I looked into rsync briefly, but it seemed like I’d have to set up an rsync server before attempting to sync the folders.

  • 1
    You do not need to set up a rsync server, rsync over ssh automatically runs rsync on the remote machine in server mode (in a way which is used only for that connection, running under your own account, and closing automatically when it is done). – CesarB Aug 25 '09 at 15:45
  • “…but it seemed like I’d have to set up an rsync server before attempting to sync the folders.” I have used Rsync for years and have never had to setup a formal Rsync server. In fact many syncing/backup tools on macOS use Rsync as their core without needing to setup a server such as Carbon Copy Cloner and—I believe—SuperDuper! – Giacomo1968 Jul 20 '17 at 20:34
  • Another thing that robocopy can do and rsync / cp cannot, seems to be parallel file transfers. – hey Jan 22 '20 at 22:45
40

It sounds like rsync is definitely what you are after. You do not need to set up an 'rsync server' to copy files from one machine to another. Rsync supports copying files over SFTP (SSH File Transfer) which most linux boxes have enabled already (if not manually disabled).

See Lifehacker's Mirror files across systems with rsync for more details:

Whether you want to backup your data, distribute files securely or mirror your working documents over the internet from the office to home, between computers on your local network, or from your computer to your web server, rsync can get the job done. Today we'll use rsync to mirror folders between a Mac and PC over a secure connection at the command line.

Rsync is free (as in speech) and cross platform, meaning it syncs files between operating systems (Windows/Cygwin, Mac OS, Linux); it works over ssh so it's encrypted and secure; unlike FTP it's incremental, so only the parts of changed files are transferred, not whole files, which makes it go like Speedy Gonzalez; and the fact that it's command line makes it scriptable and easily automated.

  • Read the rsync man page -- to use ssh (w/o an explicit rsync server) you'll use "-e ssh" – Doug Harris Aug 25 '09 at 14:32
  • 1
    rsync is awesome! And you don't need to use -e since any path like user@host.com:/blah/ will automatically use it. – The How-To Geek Aug 25 '09 at 15:49
  • Mmm, that's cool. So it can use basically scp-like syntax for transferring over ssh? I was actually trying to transfer files on one machine, but that is good to know. – Ibrahim Aug 25 '09 at 18:00
  • 2
    Unfortunately, this answer isn’t entirely correct: While it does use SSH, it does not use SFTP. Instead, it communicates with the remote rsync instance directly using the rsync protocol. – Daniel B Oct 21 '16 at 6:52
16

rsync will copy files from one directory to another directory on the same machine like robocopy. Here is the rsync command that is equivalent to your robocopy command:

rsync -auv --exclude '.svn' --exclude '*.pyc' source destination

This will recursively copy the source to the destination and exclude older files as well as '.svn' and '*.pyc' directories/files.

  • 1
    Except it will do so in a single threaded manner, which can result in over a 5x performance hit depending on the amount of indirection in your storage solution. – Mikhail Apr 10 '17 at 2:50
  • When a file is deleted from the source, it doesn't sync over to the destination. – Jirong Hu Jun 19 '17 at 17:11
1

If you can live without excluding the .svn directories and .pyc files, and if none of your file and directory names contain spaces, then you can just use cp at the GNU/Linux command prompt:

cp -uaT source destination

where source and destination are directories.

  • The -u option ensures that only newer versions from source will replace existing files in destination. See https://superuser.com/a/341296.
  • The -a option, AKA "archive", recursively copies directories within source while preserving time stamps and permissions, etc.
  • The -T option ensures that we copy the contents of source into directory, not directory source itself.

It's important that the GNU version of cp is used, as some of these options may be different or not available in other UNIX versions, e.g., macOS.

If you have spaces in your file or directory names, then you can use rsync as others have mentioned, or pipe find with -print0 to xargs with -0, which you can look into, but is beyond the scope of this answer. Both of these possibilities allow excludes.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.