Questions tagged [rsync]
Rsync is an open source command line tool for Unix and Windows systems that provides fast incremental file transfer to copy files and synchronize directories locally or between remote hosts. It is widely used for mirroring and backup.
460
questions with no upvoted or accepted answers
9
votes
1
answer
4k
views
Make rsync show progress on checksum comparison, even if no difference is being found
I run this command to check the identicalness of two folders and put all differences into a log file:
rsync -av --delete --checksum --dry-run --stats --itemize-changes --out-format='%t %i %f ' SOURCE ...
7
votes
1
answer
7k
views
Create a differential backup with rsync of a directory on my local drive to another directory on the same drive
How can I use rsync (but neither rsnapshot nor rdiff-backup nor any other application) to create a differential backup of a directory located on my local drive to another directory located on that ...
7
votes
1
answer
3k
views
Rsync backups on btrfs very slow
My environment is Ubuntu 15.04 with kernel 3.19.0-28-generic and Btrfs v3.17.
I have two identical external USB hard disks that I use with my backup script. One of them is formatted with btrfs and ...
6
votes
0
answers
3k
views
ignoring No data available (61) errors with rsync
Is there a way to suppress a warning where the origin changes during the rsync?
$ rsync --archive server::somemodule/ /dest/dir
rsync: read errors mapping "/somefile" (in somemodule): No ...
5
votes
0
answers
663
views
How to get output to STDOUT in Rsync while only logging changes to “--log-file” the way Rsync 2.6.9 did with Rsync 3.x
As a longtime macOS user I have used the built-in/bundled rsync binary that has been stuck at version 2.6.9 for various licensing reasons.
I recently decided to start using modern rsync via Homebrew. ...
5
votes
0
answers
639
views
rsync over ssh tunnel: What does this repeated message mean? debug2: channel 2: window 1966080 sent adjust 131072
I'm rsyncing through an SSH tunnel (with ssh -vv). It's not very fast and I'm getting this message about every second:
debug2: channel 2: window 1966080 sent adjust 131072
Once in a while I also get ...
5
votes
2
answers
900
views
Issue with China's Great Firewall (or other odd networking issue)
We periodically deploy an application to a computer located in China (from Canada). We deploy the changes using Subversion - we're talking a ~100Mb diff. We've noticed that the speeds are good until ...
5
votes
1
answer
2k
views
Copy folders and keep their modified date
I want to copy a couple of folders and preserve their modified date for historic reasons and not to change the sorting I'm used to.
I have tried using rsync with flags to preserve times and the ...
4
votes
0
answers
795
views
Copying files of Windows partition using rsync gives readlink_stat(...) Input/output error (5)
I want to copy all files of a Windows 10 system partition (NTFS) to another partition (NTFS).
I am using rsync under Ubuntu like this:
sudo rsync -axHAWXS --numeric-ids /media/windows /mnt/windows-...
4
votes
1
answer
5k
views
rsync issue failed: No such file or directory (2)
I am trying to use rsync files from my home directory. The error I get is:
rsync: link_stat "/home/xxx/.mozilla/firefox/**/*.sqlite" failed: No such file or directory (2)
I have checked and the file ...
4
votes
1
answer
116
views
Does Postgres ever write to tables without file system timestamps getting updated for a long time?
I'm running Postgres 9.6 and backing it up once a while simply by stopping the cluster and using rsync on file level. One day I've recognized that some old, already backed up files for tables still ...
4
votes
0
answers
5k
views
How to always let rsync continue when (permission denied) error occurs
The following error happened with a rsync task.
rsync: send_files failed to open "/home/user/scripts/@eaDir/script.js@SynoEAStream": Permission denied (13)
rsync error: some files/attrs were not ...
4
votes
0
answers
619
views
Is there a way to verify the integrity of a rsnapshot backups?
I'm using rsnapshot to make daily backups to a second hdd.
But how can I make sure, that no file in my backup is corrupted?
If I use the checksum option from rsync, the whole backup will take much ...
4
votes
2
answers
2k
views
rsync --remove-source-files but only those that match a pattern
Is this possible with rsync?
Transfer everything from src:path/to/dir to dest:/path/to/other/dir and delete some of the source files in src:path/to/dir that match a pattern (or size limit) but keep ...
3
votes
1
answer
1k
views
Rsync from Mac OS to synology with btrfs having issues with file and directories that contain umlauts and similar characters
I want to copy a folder from my local Mac (Catalina with APFS) to my Synology (btrfs filesystem) on my local network.
And all files that have an umlaut in it (ö,ä,ü,....) gets in trouble somehow. ...
3
votes
0
answers
1k
views
Is rsync's --link-dest option "better" than --backup-dir?
In his popular article Easy Automated Snapshot-Style Backups with Linux and Rsync, Mike Rubel asserts that using rsync to create hard-linked snapshots with --link-dest is "a better way" to perform ...
3
votes
0
answers
125
views
How to copy Epichrome (SSB Chrome) apps to other workstations with active Extensions?
Epichrome is an absolutely great tool when it comes to site-specific-browsing.
I've found it's useful to be able to copy Epichrome apps to other systems, however it appears that Chrome itself must ...
3
votes
0
answers
910
views
How to stop rsync from canceling when encountering a symlink without a referent
I use rsync to backup home dir of my computer 1 to computer 2 like this:
rsync --recursive --archive --copy-unsafe-links --hard-links \
--delete-after --fuzzy --compress --human-readable \
...
3
votes
0
answers
94
views
When do I need `--exclude=*` in an `rsync` call?
I sometimes read --exclude=* in rsync calls. But I don't really understand why I need them there.
What is the effect of it?
In wich use-cases should it be used?
The question is not about --exclude ...
3
votes
0
answers
335
views
systemd managing cgroups: assign specific binary to a cgroup
SHORT VERSION: I would like systemd to assign any instance of the rsync binary to a specific cgroup, automatically. How do I set up systemd to do this?
LONG VERSION: When I run rsync for (large) ...
3
votes
0
answers
272
views
Copying of large files failed
I have been running simulations in supercomputer often writing large output files in h5 format (around 750MB). For backing up data after simulations to my local computer, I used rsync command.
At the ...
3
votes
2
answers
1k
views
Creating a hard linked differential backup (trying with rsync)
I am trying to create a script that creates both a snapshot backup and a differential backup, but with hard links instead of duplicated files. In other words:
All unchanged files between the current ...
3
votes
0
answers
539
views
Rsync strangely gets auto killed and then binary is broken
I got a strange situation with Rsync on my remote Ubuntu 14.04.3 LTS (Trusty Tahir) instance.
Whenever I execute it, it runs for a while (not completed) then seems get auto killed. After that it's ...
3
votes
0
answers
2k
views
rsync SSD migration: lsetxattr failed: Operation not supported (95) -- consequences?
I just got an SSD and have been trying to migrate to it, poring through tons of how-to's I finally decided on this command (I have multiple partitions that needed migrating):
time rsync -avHAXx /...
3
votes
0
answers
456
views
ssh proxycommand equivalent on server
Lets say we have three machines me, proxy, server
I would like to have an ssh connection to server, so normally I would do
me$ ssh proxy
proxy$ ssh server
server$ # and here we are
ssh proxy ...
3
votes
1
answer
962
views
rsync ... blocked for more than 120 seconds
I'm running an rsync between two external USB 2TB drives to make a copy on a CentOS 5.9 server. It appears to have stalled, and it hangs when I cd to the source drive in the copy, while the ...
3
votes
0
answers
495
views
Rsync, how can I delete remote duplicates?
I'm trying to delete remote duplicate files without transferring the non-existing files, and without updating the existing files.
If I specify both --existing and --ignore-existing (along with -av --...
3
votes
1
answer
3k
views
Rsync to a WebDAV filesystem on OS X copies all files regardless of being changed or not
I am trying to sync from my (Mac) desktop to an iPad and an iPhone. MacOS X mounts WebDAV as a native file system, but syncing results in all files in my directory structure being copied again.
This ...
3
votes
2
answers
5k
views
Are cp/rsync asynchronous?
We're running a backup script which first copies a file to a destination and then runs tar over it.
DIR2BCK='/foo/bar'
TMPDIR=$(mktemp -d)
rsync -a ${DIR2BCK} ${TMPDIR}/ > /dev/null 2>&1
...
3
votes
1
answer
1k
views
Rsync: Only preserve meta (time, group, etc) on files and sub-directories, not root directory
I am copying some files (all except hidden ones) using rsync from one place to another using this command:
rsync -Cav --delete --exclude=.* /Some/Directory/ other-host:/Other/Directory
It works nice ...
2
votes
0
answers
288
views
Does rsync behave differently if it detects that a mount point is from a network instead of a local/external drive?
Note: I don't think I did a good job of explaining what I was looking for. The problem I'm running into is NOT which files rsync is selecting to mirror. The problem is that rsync is taking WAY ...
2
votes
0
answers
92
views
use rsync with autossh
I'm trying to use rsync with autossh on my laptop, so it can also work when internet connection is less reliable, but it doesn't really do as I'd like it to do.
The starting point is that I have an ...
2
votes
0
answers
22
views
How to change directories recursively in huge files
I tried to copy data between two huge NFS folders. I copied all the data, but I noticed that by mistake I made duplicate directories only in the 6th depth.
For example, instead of copying:
/mnt/nfs1/a/...
2
votes
0
answers
2k
views
Rsync not deleting files in destination directory despite “--delete” being set
I have the following rsync command:
rsync -ahv --progress --delete \
/source_dir1/file1.txt \
/source_dir2 \
/dest_dir/
It syncs everything as expected, but does not delete anything from ...
2
votes
0
answers
589
views
rsync with -X fails due to dot underscore files
I need to maintain sync copy from one external drive to another, on my macOS (Monterey).
ext drive 1 (source) = ExFAT
ext drive 2 (target) = APFS
rsync v3.2.3 is used (not macOS default v2.6.9)
rsync -...
2
votes
1
answer
317
views
rsync adding executable flag when syncing over network
I’m running rsync to synchronize two Debian machines. Both the source and the destination file systems are ext4.
Naturally I want to preserve the permissions so I’m running the command with the ...
2
votes
0
answers
904
views
rsync --ignore-missing-args not working
I'm trying to use rsync actually inside loop to backup a list of directories from remote.
I just want to ignore if src directory do not exists.
I got from most of the forums that --ignore-missing-args ...
2
votes
0
answers
346
views
Can rsync adjust sym-links to reflect their new location?
I have a directory tree that has internal, absolute sym-links, and I want to rsync it with a different disk. My problem is sym-links. After I rsync, the sym-links on the new disk all point to the ...
2
votes
0
answers
695
views
How to efficiently distribute large, incrementally updated sqlite database?
What is an efficient way to transfer deltas from a large sqlite database?
Note: this question is NOT about how to copy/distribute a "live" database. I.e. I am not concerned about corrupted ...
2
votes
0
answers
49
views
Copy symbolic link and its target subdir to new dir
I have a dir with a symbolic-linked subdirectory:
greg:/testA> ls
foo -> alice
alice/
I need to copy the link "foo-> alice" (preserve it at destination) AND the subdirectory alice/ to a new ...
2
votes
0
answers
885
views
rsync: send_files failed to open "filename": Invalid argument (22)
I use rsync to copy files from a mounted davfs box.com folder to an ext4 partition. For files containing the symbols [, ", and {, I receive the following error:
rsync: send_files failed to open "...
2
votes
0
answers
28
views
rsync: can rsync adjust to the directory layout insteaed of transferring?
I have three systems, but only the target and a outdated mirror are on a fast link.
Master_SLOW:
On a super slow link. Contains a updated directory structure and files
OldMirror_FAST:
On a fast link ...
2
votes
0
answers
464
views
Using rsync through intermediate host with `sudo` on jump-host
I am user1@host1 and would like to transfer data via rsync from user3@host3. However, I can access host3 only via user3@host2. The catch is, that I can become user3 on host2 only via su or sudo, but ...
2
votes
0
answers
814
views
Writing files to Android Phone SD card over SSH?
I successfully set up SSHDroid on my phone so I could both login and use rsync to copy files FROM the phone to the computer. However, when I try to push files back the other way I get the error:
$...
2
votes
0
answers
298
views
Stop rsync from deleting large amounts of data when disk accidentally umounted
I use rsync to mirror my hard drive onto a couple of devices with cron.
The command is essentially this:
rsync -avr --delete /home/username [email protected]:/home/username
The problem I have is ...
2
votes
0
answers
678
views
rsync: listing only changed files/attributes
I am trying to achieve the following:
In order to check whether I correctly rsynced all data from my old NAS to my new, I am executing:
rsync --rltD -hvinu /path/to/old/NAS/ /path/to/new/NAS
Now: -...
2
votes
0
answers
987
views
Rsync and accents in filename on macOS
When I use rysnc to backup my media hard drive that's plugged into my Ubuntu server using the command rsync -av --delete source destination it deletes and recopies all files that have accents in them, ...
2
votes
0
answers
265
views
MacOS 10.13.3 rsync and terminal: filenames with scandic letters
I have a rsync backup script that copies my personal files to an external drive with rsync:
rsync -vr —-delete --progress /Users/omistaja/Documents/ /Volumes/MacExternal/Backups.Omistaja/
Edit: It ...
2
votes
0
answers
976
views
Remote sync without rsync
My setup is as follows:
Two Linux machines running Debian 9
A "storage box" on the internet rented from a cloud hosting company. This box can only be accessed via the SFTP, Samba and WebDAV protocols ...
2
votes
0
answers
406
views
Best Practices for Transferring Data Between Hadoop Clusters
Situation:
I need to transfer around 100TB from an old Hadoop Cluster to a new Hadoop cluster. The two clusters have a direct 1GbE connection from the namenode on each cluster. The datanodes are not ...